ThisFor a RGB format, this seems to work well enough and is trivial:
diversity_score = (abs(r1 - r2) + abs(g1 - g2) + abs(b1 - b2)) / 765.0
This will give you a score between 0.0 and 1.0. The lower, the harder it is to distinguish two colors.
It should be obvious, but for the sake of completeness: the r, g, b values must be cast to a floating point number first, and it is assumed that their maximum value is 255.