Paint with soft brush (logic)

Question

During the last few days I was coding a painting behavior for a game am working on, and am currently in a very advanced phase, I can say that I have 90% of the work done and working perfectly, now what I need to do is being able to draw with a "soft brush" cause for now it's like am painting with "pixel style" and that was totally expected because that's what I wrote.

My current goal consist of using this solution :

1. Import a brush texture
2. Create an array that contain all The alpha values of that texture
3. When drawing use the array elements in order to define the new pixels alpha

And this is my code to do that (it's not very long, there is too much comments)

//The main painting method
//theObject     = the object to be painted
//tmpTexture    = the object current texture
//targetTexture = the new texture 
void paint (GameObject theObject, Texture2D tmpTexture, Texture2D targetTexture)
{
    //x and y are 2 floats from another class
    //they store the coordinates of the pixel 
    //that get hit by the RayCast
    int x = (int)(coordinates.pixelPos.x);
    int y = (int)(coordinates.pixelPos.y);

    //iterate through a block of pixels that goes fro
    //Y and X and go #brushHeight Pixels up
    // and #brushWeight Pixels right
    for (int tmpY = y; tmpY<y+brushHeight; tmpY++) {
        for (int tmpX = x; tmpX<x+brushWidth; tmpX++) {
           //check if the current pixel is different from the target pixel
           if (tmpTexture.GetPixel (tmpX, tmpY) != targetTexture.GetPixel (tmpX, tmpY)) {
               //create a temporary color from the target pixel at the given coordinates
                Color tmpCol = targetTexture.GetPixel (tmpX, tmpY);
                //change the alpha of that pixel based on the brush alpha
                //myBrushAlpha is a 2 Dimensional array that contain
                //the different Alpha values of the brush
                //the substractions are to keep the index in range
                if (myBrushAlpha [tmpY - y, tmpX - x].a > 0) {
                    tmpCol.a = myBrushAlpha [tmpY - y, tmpX - x].a;
                }
                //set the new pixel to the current texture
                tmpTexture.SetPixel (tmpX, tmpY, tmpCol);
            } 
        }
    }
    //Apply 
    tmpTexture.Apply ();
    //change the object main texture 
    theObject.renderer.material.mainTexture = tmpTexture;
}

Now the fun (and bad) part is the code did exactly what I asked for, but by asking to draw anytime with the brush alpha I found myself create a very weird effect which is decreasing the alpha value of an "old" pixel, so I tried to fix that by adding an if statement that check if the current alpha of the pixel is less than the equivalent brush alpha pixel, if it is, then augment the alpha to be equal to the brush, and if the pixel alpha is bigger, then keep adding the brush alpha value to it in order to have that "soft brushing" effect, and in code it become this :

if (myBrushAlpha [tmpY - y, tmpX - x].a > tmpCol.a) {
  tmpCol.a = myBrushAlpha [tmpY - y, tmpX - x].a;
} else {
  tmpCol.a += myBrushAlpha [tmpY - y, tmpX - x].a;
}

But after I've done that, I got the "pixelized brush" effect back, am not sure but I think maybe it's because I'm making these conditions inside a for loop so everything is executed before the end of the current frame so I don't see the effect, could it be that?

Am really lost here and hope that you can put me in the right direction.

Answer 1

New answer:

After giving it much thought I think you may not be blending the new clor with the existing one. I took the liberty to modify the code to what might work here:

for (int tmpY = y; tmpY<y+brushHeight; tmpY++) {
    for (int tmpX = x; tmpX<x+brushWidth; tmpX++) {
           //No need to check if the current pixel is different from the target pixel
            Color oldCol = tmpTexture.GetPixel (tmpX, tmpY);
            Color tmpCol = targetTexture.GetPixel (tmpX, tmpY);
            //change the alpha of that pixel based on the brush alpha
            if (myBrushAlpha [tmpY - y, tmpX - x].a > 0) {
                tmpCol.a *= myBrushAlpha [tmpY - y, tmpX - x].a;
            }
            //set the new pixel to the current texture
            double total_alpha = tmpCol.a + oldCol.a;
            tmpCol.a = oldCol.a + (1 - oldCol.a) * tmpCol.a;
            tmpCol.r = (tmpCol.r * tmpCol.a + oldCol.r * oldCol.a) / total_alpha;
            tmpCol.g = (tmpCol.g * tmpCol.a + oldCol.g * oldCol.a) / total_alpha;
            tmpCol.b = (tmpCol.b * tmpCol.a + oldCol.b * oldCol.a) / total_alpha;
            tmpTexture.SetPixel (tmpX, tmpY, tmpCol);
    }
}

Old answer:

This piece of code appears to be incorrect.

if (myBrushAlpha [tmpY - y, tmpX - x].a > tmpCol.a) {
  tmpCol.a = myBrushAlpha [tmpY - y, tmpX - x].a;
} else {
  tmpCol.a += myBrushAlpha [tmpY - y, tmpX - x].a;
}

I believe what you wish to do is mix the color of the brush with the current color of that pixel. It should be something like this:

// mba == myBrushAlpha[tmpY - y, tmpX - x]; 

double total_alpha = tmpCol.a + mba.a;
double alpha_result = tmpCol.a + (1 - tmpCol.a) * mba.a;
double red_result = (tmpCol.r * tmpCol.a + mba.r * mba.a) / total_alpha;
double green_result = (tmpCol.g * tmpCol.a + mba.g * mba.a) / total_alpha;
double blue_result = (tmpCol.b * tmpCol.a + mba.b * mba.a) / total_alpha;

tmpCol.a = alpha_result; // Assuming you will set the color to tmpCol.
tmpCol.r = red_result; 
tmpCol.g = green_result; 
tmpCol.b = blue_result; 

I also notice this:

if (myBrushAlpha [tmpY - y, tmpX - x].a > 0) {
    tmpCol.a = myBrushAlpha [tmpY - y, tmpX - x].a;
}

It looks like it might have been supposed to be something along the lines:

if (myBrushAlpha [tmpY - y, tmpX - x].a > 0) {
    tmpCol.a *= myBrushAlpha [tmpY - y, tmpX - x].a;
}