How can I avoid applying textures to a fragment in a shader when the surface normal is (0, 1, 0)?

I have a small GLSL shader with a vertex shader and a fragment shader. I want to avoid applying textures to faces that have a normal equal to (0, 1, 0). Is this possible? These are my shaders:

Vertex:

#version 120

varying vec2 UV;
varying vec3 normal;

void main()
{
UV = gl_MultiTexCoord0;
gl_Position = gl_ModelViewProjectionMatrix * gl_Vertex;
normal = gl_Normal;
}


Fragment:

#version 120

varying vec2 UV;
uniform sampler2D diffuseMap;

void main(void)
{
gl_FragColor = texture(diffuseMap, UV);
}

• How do you send your vertex and normal data to your shader? – Basaa Dec 4 '13 at 23:47
• can you elaborate why do you want to do this? maybe you are asking the wrong question – concept3d Dec 4 '13 at 23:48
• And what do you want to replace those fragments that you don't texture with? A solid color? A fully-transparent color? – Josh Dec 4 '13 at 23:54
• @Basaa Using Ogre3D. No idea how. I have a material script with some textures – Pacha Dec 4 '13 at 23:55
• @Pacha eventhough I did't fully understand what you are trying to do, I really believe this isn't the way to do it. What I think you should do, is using seperate draw calls with different shaders and textures. You don't even have to check for normal(0,1,0) just apply it for whatever objects you want. – concept3d Dec 5 '13 at 0:00

Use an if-statement in the fragment shader and pass through the normal or a flag from the vertex shader.

Note that sampling of textures must be in uniform flow, i.e. outside of the if-statement. Something like the following probably-not-legal-GLSL:

// vertex
in vec4 aPosition;
in vec3 aNormal;

out vec3 vsNormal;

void main(void)
{
gl_FragCoord = project_transform * aPosition;
vsNormal = aNormal;
}

// pixel
in vec3 vsNormal;

void main(void)
{
vec4 tex = texture2D(blah);
vec4 color = vec4(1, 1, 1, 1);

// depending on the input normal,
// you should better normalize it in the fragment shader before you use it
vec3 normal = normalize(vsNormal);
float dotN = dot(vsNormal, vec3(0,1,0));

if (dotN < 0.99)
color *= tex;
else
really_simple_effect();

gl_FragColor = color;
}


Note that you want to deeply minimize the amount of code you use inside of any branch in a shader. You might be best off simply rendering a 5-sided "cube" for the sides/bottom and then a separate plane for the top with two totally different materials/shaders. Batch as much as you can (e.g. draw all the 5-sides parts in one go, then draw all the tops in a second go).

• @Pacha that is the way to do it as you asked. But I still believe this is not the "correct" way to do it, checking if normal equals a certain value is far from ideal and will limit the use of the shader. I still think multiple draw calls are the way to go. – concept3d Dec 5 '13 at 0:15
• @Pacha also expect some unexpected failures because checking float with == and != is likely to fail sometimes, especially if you are calculating the normals. – concept3d Dec 5 '13 at 0:20
• @Pacha: I agree with concept3d. You really want to do what I suggested in the last paragraph. The == and != will likely work in the use case I think you're in but he's again right that you need to be wary floating-point imprecision depending on what you're doing. – Sean Middleditch Dec 5 '13 at 0:25
• I am using ranges though for floating points. (betweeen .99 and 1.01) – Pacha Dec 5 '13 at 0:36
• You still need to be careful. Is decimal .01 an appropriate range for you operations? Calculating the safe epsilons for floating-point operations is a black art few know and you're generally better off finding approaches that don't need them at all. – Sean Middleditch Dec 5 '13 at 0:44