Algorithm for texture blending modification

Question

Im currently writing a terrain editor for my game (for the record: no commercial intentions) and after having implemented the terrain morphing to my liking im now working on the texturing of said terrain. One chunk of terrain can consist of up to 4 different layers of textures which are blended together using a 4-byte rgba texture with alpha values. The alpha values are 0-255 and together add up to 255, so the blending on the gpu happens like this:

float4 finalColor = color0 * alpha.r + color1 * alpha.g + color2 * alpha.b + color3 * alpha.a;

In a first approach ive been doing very simple modification of that blending (pseudocode):

selectedLayer.alpha += amount;
allOtherLayers.alpha -= (amount / numOtherLayers);

This doesnt work all too bad but it creates a lot of artifacts or sharp lines between chunks. Here is an example image:

Now im looking for different, better algorithms. The brush currently consists of an inner and an outer radius (as seen in the image). Inside the inner radius no interpolation depending on the distance to the center happens, between inner and outer radius linear interpolation between the pressure on the inside and 0 is created.

I hope my question is clear enough, please leave a comment if there has to be more information. Thank you in advance.

Answer 1

I think I understand what you are trying to do. You want the sum of all layers' alpha to be 1 so that there the total color is overdrawn.

You are on the right track. But your error seems in the way you compute the alphas.

Take for example this starting point:

1: 0.7 
2: 0.3 
3: 0.0

Now you apply 0.5 on layer 3, the result should look something like:

1: 0.35
2: 0.15
3: 0.5

You wan the original ratio between layer 1 and layer two preserved. The sum of remainder should be 1/amount.

So the correct way should be:

allOtherLayers.alpha *= 1/amount;

or

allOtherLayers.alpha /= amount;