# Fragment shader seems to floor() imprecisely

I'm trying to interpolate coordinates in my fragment shader. Unfortunately if close to the upper edge the interpolated value of fVertexInteger seems to be rounded up instead of beeing floored. This happens above approximately fVertexInteger >= x.97.

Example:

• floor(64.7) returns 64.0 --> correct

• floor(64.98) returns 65.0 --> incorrect

The same happens on ceiling close above x.0, where ceil(65.02) returns 65.0 instead of 66.0.

Q: Any ideas how to solve this?

Note:

• GL ES 2.0 with GLSL 1.0
• highp floats are not supported in fragment shaders on my hardware
• flat varying hasn't been a solution, because I'm drawing TRIANGLE_STRIP and can't redeclare the provoking vertex (only OpenGL 3.2+)

varying float fVertexInteger;
varying float fVertexFraction;
void main() {

// Fix vertex integer
fixedVertexInteger = floor(fVertexInteger);

// Fragment color
gl_FragColor = vec4(
fixedVertexInteger / 65025.0,
fract(fixedVertexInteger / 255.0),
fVertexFraction,
1.0 );

}

• Out of curiousity, how did you extract those values? Are you stepping through the shader somehow? – kevintodisco Nov 13 '13 at 16:48
• I'm using glReadPixels() in my Android app to retrieve the color values at a specific pixel. – Peter K. Nov 13 '13 at 20:20

You said highp floats are not supported on your hardware, so I assume you're using mediump. The OpenGL ES shading language spec says that mediump has a minimum relative precision of 2^(-10), or 1/1024. So if a and b are two adjacent mediump values, the ratio (b-a)/a can be as large as 1/1024.