You could now go really deep into color theory to answer the question of what exactly makes two colors visually distinct. RGB is a color system for machines which doesn't map linearly to human perception. So if you want a really good formula, then it can get amazingly complex. And if you want your game to be accessible for people with various forms of color blindness, then you might have to offer some completely different alternative formulas.
But a quick-and-dirty solution could be to simply look at their respective difference of their R, G and B components and add them up. So if you have the colors
col2, then you could calculate their difference with
abs(col1.r - col2.r) +
abs(col1.g - col2.g) +
abs(col1.b - col2.b)
But now that we have this formula, what do we do if our algorithm placed two very similar colors next to each other? We could change one of them to a random other color, but now it might be too similar to one of its other neighbors. So changing one color could cause a ripple effect affecting all the other colors.
One solution I could think of would be to create four different color palettes 1 to 4. Make sure that none of the colors in one palette is visually close to a color in a different palette (you can have visually close colors in the same palette, though). Then pick colors from these palettes at random according to this schema:
Or in technical terms:
- If X is even and Y is even, use palette 1
- If X is odd and Y is even, use palette 2
- If X is even and Y is odd, use palette 3
- If X is odd and Y is odd, use palette 4
This means that all neighbors of a tile will be from a different palette. And because no color has a similar color in a different palette, you will never have two visually similar colors next to each other.