To answer your question succinctly, RYB (red, yellow, blue) would be most intuitive and closest to mixing physical colors, aka pigments, which blend subtractively.
To understand the mixing behavior of RYB color-space, see both the color wheel diagram in the question, as well as the color tree below. For your use case it seems to me it would be simplest to implement hard coded blending, roughly following these RYB principles.
However, since you mentioned you are opposed to implementing hard coded blending, if you want this mixing behavior to be dynamic you can use CMYK color space, which is a subtractive color model used primarily for printing. Blending two CMYK colors will behave analogously to pigmentary blending
With this approach however you may be restricted by the CMYK color space, which is rather compressed and can be difficult to convert to and from while preserving color.
For a visualization of the problem of converting between color spaces you could check out this video overview of color spaces, as well as this color space comparison diagram.
For those who care to go a bit deeper, read on.
As the question hints at, there are a variety of color spaces and blending types. To design the gameplay mechanic you have described I believe you are looking for pigment color / subtractive color blending.
The wiki page on pigments has far more details about the physics and theory behind the perception and behavior of these but the birds-eye overview of pigment color can be summarized in a few points:
First, when visible light mixes, it becomes white. This is called additive blending. When pigment mixes, it becomes black-ish. This is called subtractive blending.
Secondly, by defining primary colors (ideally distributed relatively evenly along the emission spectrum, commonly RBY, RGB, CMY etc) these colors can be mixed to create a color space. (You can mix them additively or subtractively but in this case we are looking at subtractive blending) Mixing primaries you can make secondary, tertiary, quaternary, etc... Folks rarely bother beyond quaternary colors because they all devolve into browns and various earth-tones the deeper you go, asymptotically approaching black. This will happen regardless of the primary colors, but choosing more evenly distributed primary colors will reduce the severity of this effect.
Color science and color theory are massive fields and you can go far deeper into the specifics but I think for feeling out the basic mechanics of the game all you need is that primary color based mixing behavior, which is demonstrated both in the diagram you included in your question, as well as in the chart below.
I would say hard coding this blending would be the simplest solution to give you flexibility for making a variety of potion colors, regardless of their distribution along the color space, while avoiding degenerating into muddy tones, but if you really want dynamic blending you can get it through CMYK color representation.
CMYK is primarily used for printers, and is subtractive, as opposed to RGB and most other color repesentations typically used in digital displays, which are additive. The wiki page for CMYK goes into more detail but basically if you add 2 CMYK colors they will give you the pigmentary blending behavior you are looking for.
Note though, if you are currently coloring your potions with RGB, Hex, etc you will need to convert to CMYK, do the blend, then convert back to whatever your normal color representation is to apply it to the new object, but this can be trickier than it sounds.
Blending CMYK colors can be difficult due to CMYK's limited color space, so your colors may get squished when converting back and forth, eg:
RGB(A) != RGB(B) but CMYK(A) == CMYK(B)
You can experiment with the compression of converting to and from CMYK and see the specific formula for implementing the conversion in this RGB->CMYK converter.
To understand the weird behavior here it helps to see a visualization of the color space, check out this video.
You can also look at the difference between CMYK and other color spaces representable colors this color space comparison, as well as this video overlaying several color spaces in 3D, demonstrating the gamut issues with CMYK conversion.
That should be more than enough theory, so for a bit of practical exploration, check out some of the other answers with code samples, or try comparing various colors with the Google color picker.