You can reverse the formula, assuming you want the "result" after colourization to look like your original texture:
OriginalTexture = GreyishTexture * (2 * Colorize * Mask + 1 - Mask)
GreyishTexture = OriginalTexture / (2 * Colorize * Mask + 1 - Mask)
You can choose Colorize
as the average green colour of your leaf. Then, as promised, any pixels of exactly that average colour get remapped by the formula above to (0.5, 0.5, 0.5), a medium grey.
All the other pixels in your texture get remapped to their relative value in proportion to this average: so bluer areas turn to a more bluish grey, darker areas become a darker grey, etc.
When you re-colourize this by running the "forwards" version of the formula with the same average colour as the Colorize
parameter, you recover the original texture (with some loss from quantization/compression). But you can also use different colours for Colorize
to get a new output with a different average colour, that still varies in colour in the same proportions.
Here's a quick mock-up, using a leaf texture by phaelax, shared on OpenGameArt:
The left image is the "grey" version, obtained by dividing the original texture colour by double the average colour, which I eyeballed to be around #8DC45B.
I added a grey stripe in the background so you can visually verify that this puts most of the texture close to a medium grey.
The middle image is the original green leaf, reconstructed by multiplying the "grey" version by twice #8DC45B.
The right image is the "grey" version multiplied by twice a different base colour, #BCA94F. You can see we've shifted the average colour to look more autumn, but the proportional variation in the veins and folds is preserved.