# Mixing between two colours in fragment shader causes normal mapping to be ignored

I'm making a game in which the world is black and white except from the immediate area around the player. It all works, except the normal mapping breaks when I turn it on. If I set the output at just the black and white version, normal mapping appears, if I set the output as the colour version, also there, but as soon as I mix between them the shader ignores the normal mapping. I'll post the shader code here, it's very long but I thought I'd include it all just in case.

Quick explanation: splatCalcs() uses a splat map to mix between 4 different textures and their corresponding normal maps and spec maps. In this example, the splat map is completely black, in order to isolate that variable. The normal is stored as "calcedNormal" during that method, which is then used in the applyLight() method. blackAndWhiteCalcs() takes in the final colour for the fragment and then mixes between a black and white version and the original. I've used that weird calculation on the radius etc to avoid if statements. It also means once it's working I can do partial mixes.

Thanks for any help.

Edit: I have reduced the shaders to a smaller function to show the normal mapping in action and the lack thereof when I apply a mix on the two colours. In this code, I simply change the mix value in the return from hard coded 1 to hard coded 0. This produces the following results:

vec4 blackAndWhiteCalcs(vec4 colour)
{
//Black and white effects for Dreameater
vec3 lum = vec3(0.299, 0.587, 0.114);

//the black and white version of tempFragColor
vec3 blackAndWhite = mix(vec3(dot(colour.xyz, lum)), colour.xyz, 0.1);
blackAndWhite *= 0.8f;

return vec4(mix(blackAndWhite, colour.xyz, 0) , colour.a);
}


Result when 1 is hard coded and result when 0 is hard coded However using a calculation to define the 1 or 0 causes all the detail gained by normal mapping to disappear. To be clear, the base texture is all one colour, and all the detail given above is from the normal mapping. So the area that is black and white is correct, only that detail is lost. Finally, I just to prove I wasn't going mad I replaced my eqn with a simple if set as seen below. This didn't change results.

vec4 blackAndWhiteCalcs(vec4 colour)
{
//Black and white effects for Dreameater
vec3 lum = vec3(0.299, 0.587, 0.114);

//the black and white version of tempFragColor
vec3 blackAndWhite = mix(vec3(dot(colour.xyz, lum)), colour.xyz, 0.1);
blackAndWhite *= 0.8f;

float bw = 0.0f;

if (circleRadius > length(inputFragment.vert - circlePos) )
{
bw = 1;
}
else
{
bw = 0;
}

//bw = clamp(bw, 0, 1);

return vec4(mix(blackAndWhite, colour.xyz, bw) , colour.a);
}


I'm still including the full final code below. Below is what produced the result above. Thanks again.

    #version 400

#define MAX_TEXTURES 4
uniform sampler2D bases[MAX_TEXTURES];
uniform sampler2D normals[MAX_TEXTURES];
uniform sampler2D specs[MAX_TEXTURES];
uniform int inverts[MAX_TEXTURES];
uniform sampler2D splat;

//Uniforms that have to be unique to each surface

#define MAX_LIGHTS 16

uniform float materialShininess;
uniform vec3 cameraPos_World;
uniform int numLights;

uniform int invertY;

uniform struct Light {
vec4 position;
vec3 colors;
float attenuation;
float ambientCoefficient;
float coneAngle;
vec3 coneDirection;
mat4 finalLightMatrix;
} allLights[MAX_LIGHTS];

in packet{

vec2 textureCoord;
vec2 tiledCoord;

vec3 normal;
vec3 vert;
mat3 TBN;

} inputFragment;

vec3 specColour = vec3(1,0,0);
mat3 calcedTBN;
vec3 calcedNormal;
vec4 tempFragColor;

#define CIRCLESMAX 60
uniform vec3 circlePos[CIRCLESMAX];

float bwMix = 1.0f;

// output packet
layout (location = 0) out vec4 fragmentColour;

vec3 applyLight(Light thisLight, vec3 baseColor, vec3 surfacePos, vec3 surfaceToCamera)
{
float attenuation = 1.0f;
vec3 lightPos = (thisLight.finalLightMatrix*thisLight.position).xyz;
vec3 surfaceToLight;

vec3 coneDir = normalize(thisLight.coneDirection);

if (thisLight.position.w == 0.0f)
{
//Directional Light (all rays same angle, use position as direction)
surfaceToLight = normalize((thisLight.position).xyz);
attenuation = 1.0f;
}
else
{
//Point light
surfaceToLight =  normalize(lightPos - (surfacePos));

float distanceToLight = length(lightPos -( surfacePos));
attenuation = 1.0 / (1.0f + thisLight.attenuation * pow(distanceToLight, 2));

//Work out the Cone restrictions
float lightToSurfaceAngle = degrees(acos(dot(-surfaceToLight, normalize(coneDir))));

if (lightToSurfaceAngle > thisLight.coneAngle)
{
attenuation = 0.0;
}
}

//Ambient light calculation
vec3 ambient = thisLight.ambientCoefficient * baseColor.rgb * thisLight.colors;

//Diffuse Light Calculation
float diffuseCoefficient = max(0.0, dot(calcedNormal,normalize(surfaceToLight)));
vec3 diffuse = diffuseCoefficient * baseColor.rgb * thisLight.colors;

//Specular Light Calculation
float specularCoefficient = 0.0;
if (diffuseCoefficient > 0.0)
{
specularCoefficient = pow(max(0.0, dot(surfaceToCamera, reflect(-surfaceToLight,calcedNormal))), materialShininess);
}

vec3 specular = specularCoefficient * specColour * thisLight.colors;//

return ambient + attenuation * (diffuse+ specular);
}

void splatCalcs()
{
vec4 splat = (texture2D(splat, inputFragment.textureCoord));

//read in each base colour texture
vec4 base0 = texture2D(bases, inputFragment.tiledCoord);
vec4 base1 = texture2D(bases, inputFragment.tiledCoord);
vec4 base2 = texture2D(bases, inputFragment.tiledCoord);
vec4 base3 = texture2D(bases, inputFragment.tiledCoord);

//Use an appropriate blend from each texture. use normal and spec maps for the most prevalent map
//Black = base 0
//red = base 1
//green = base 2
//blue = base 3
//a = 0 then base 4
float black = 1 - (splat.r + splat.g + splat.b);

base0 = base0 * black;
base1 = base1 * splat.r;
base2 = base2 * splat.g;
base3 = base3 * splat.b;

//frag colour combined =
tempFragColor = base0 + base1 + base2 + base3;

//Do everything again for normal maps
vec3 normal0 = texture2D(normals, inputFragment.tiledCoord).xyz;
vec3 normal1 = texture2D(normals, inputFragment.tiledCoord).xyz;
vec3 normal2 = texture2D(normals, inputFragment.tiledCoord).xyz;
vec3 normal3 = texture2D(normals, inputFragment.tiledCoord).xyz;

normal0 = normal0 * black;
normal1 = normal1 * splat.r;
normal2 = normal2 * splat.g;
normal3 =  normal3 * splat.b;

vec3 inputN = normal0 + normal1 + normal2 + normal3;
//input = (texture2D(normals, inputFragment.tiledCoord)).xyz;

//Normal map specific calculations
readIn.x = (2.0* inputN.r) - 1.0;
readIn.y = invertY *((2.0* inputN.g) - 1.0);

//calcec normal combined =

vec3 spec0 =  texture2D(specs, inputFragment.tiledCoord).xyz;
vec3 spec1 =  texture2D(specs, inputFragment.tiledCoord).xyz;
vec3 spec2 =  texture2D(specs, inputFragment.tiledCoord).xyz;
vec3 spec3 =  texture2D(specs, inputFragment.tiledCoord).xyz;

spec0 = spec0 * black;
spec1 = spec1 * splat.r;
spec2 = spec2 * splat.g;
spec3 = spec3 * splat.b;

//Set spec colour
specColour = spec0 + spec1 + spec2 + spec3;

}

vec4 blackAndWhiteCalcs(vec4 colour)
{
//Black and white effects for Dreameater
vec3 lum = vec3(0.299, 0.587, 0.114);

//the black and white version of tempFragColor
vec3 blackAndWhite = mix(vec3(dot(colour.xyz, lum)), colour.xyz, 0.1);

float bw = 0.0f;

//if the radius is smaller than the distance between the vert and the centre, mul = 0.
//if the radius is bigger than the distance between the vert and the centre, mul > 0
//the result of the following eqn is 0 if the point is outside of the circle, and (0< result < radius) radius if it is.

for (int count2 = 0; count2 < CIRCLESMAX; count2++)
{

// make sure the mul value is definetly either 0, or bigger than 1

}

bw = clamp(bw, 0, 1);

return vec4(mix(blackAndWhite, colour.xyz, bw) , colour.a);
}

void main(void) {

vec3 surfaceToCameraWorld = normalize((cameraPos_World)-(inputFragment.vert));

//get the base colour from the texture
splatCalcs();

tempFragColor = blackAndWhiteCalcs(tempFragColor);

vec3 tempColour = vec3(0.0, 0.0, 0.0);

for (int count = 0; count < numLights; count++)
{
tempColour += applyLight(allLights[count], tempFragColor.xyz, inputFragment.vert, surfaceToCameraWorld);
}

vec3 gamma = vec3(1.0 / 2.2);

vec4 final = vec4(pow(tempColour, gamma), tempFragColor.a);

fragmentColour = final;
}

• "In this example, the splat map is completely black, in order to isolate that variable." A better solution: create a copy of this shader and just delete all the parts of it that aren't required to demonstrate the problem you're solving. This is called creating a Minimal, Complete, Verifiable Example. You should also include image examples so we can see the symptoms of the problem. Mar 6, 2019 at 13:01
• Will get on that when I get home mate. Will update question Mar 6, 2019 at 13:31
• Have you checked that your shaders compile and link OK? This is something that sounds like it could be caused by a compilation failure and consequent fallback to the fixed pipeline. Mar 6, 2019 at 19:22
• They compile just fine and given I don't use any fixed function calls I don't think this would be possible... Besides the rest of the very specific behaviours I call in the shaders would be ignored if my shaders weren't being used, for example the coloured circle around the player. Mar 6, 2019 at 21:43
• @DMGregory I have editted the question as so. Mar 6, 2019 at 21:56