A quick and easy way - though not 100% precise one - is to consider just the five extreme points white, black, red, green and blue.
First, let's transform RGB into linear space. Officially this is usually done by this formula (assuming the source data is in sRGB, which is the default for most graphic card operations on 8-bit data and nearly every image you see displayed on the web):

... where a = 0.055 for C = {R, G, B}
Or, if you just want a quick approximation, use Clinear = Csrgb2.2.
Black or White?
You can then calculate two helpful values, luminance Y and saturation S. Useful formulas (for our linear-space RGB values) are:
Y = 0.2126 R + 0.7152 G + 0.0722 B
S = (max(R, G, B) - min(R, G, B)) / max(R, G, B)
The second one is the calculation from the HSV colour space.
If the saturation is low enough (pick any value you like; something between 0.3 and 0.5 would work fine), check the luminance; if that's > 0.5, your contrasting colour is black, else if it's < 0.5 the colour is white. For exactly Y = 0.5, both work.
Red, Green or Blue?
For colours with high saturation, you can calculate a hue H and decide according to it. Since I'm lazy, I'll just copy the calculation from Wikipedia (hue is named H2):


If your hue is between 60° and 180°, your colour is green, if it's between 180° and 300°, it's blue, else it's red. Extend it to yellow, turquoise and magenta for more variation, or just use the opposite of the just-calculated hue at maximum saturation and lightness.