Low quality bilinear sampling in WebGL/OpenGL/DirectX

Question

I'm seeing low quality bilinear texture sampling in WebGL, OpenGL and Directx, and was wondering if anyone knew how to make it higher quality? The picture below should help show what I mean.

The left uses the built in bilinear sampling, while on the right, i read in the 4 pixels of the texture and do linear interpolation myself, which results in a higher quality interpolation.

I've tried lots of things, including setting the precision to highp for everything, including the sampler, and also have tried using floating point textures and looking for any relevant extensions, but no luck.

Does anyone know if there's a way to get the smoother interpolation in WebGL?I have tried it on a couple different machines (all nvidia though) as well as the major browsers.

I know that I can make it higher quality by taking 4 samples and doing the bilinear interpolation myself, or by taking 2 samples where i left the hardware do the X axis, then i do the Y axis in my shader. Are there any other ways? Hopefully some solution that doesn't require more texture reads?

Here's a link to the actual WebGL page, where you can see the source: http://demofox.org/BilinearTest/bilineartest.html

Thanks for any help anyone can provide!!

EDIT: code details

I create a 2x2 texture with linear min and mag filters - no mips.

function createTexture(byteArrayWithRGBAData, width, height) {
    var texture = gl.createTexture();
    gl.bindTexture(gl.TEXTURE_2D, texture);
    gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, width, height, 0, gl.RGBA, gl.UNSIGNED_BYTE, new Uint8Array(byteArrayWithRGBAData));
    gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.LINEAR);
    gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR);
    gl.bindTexture(gl.TEXTURE_2D, null);
    return texture;
}

....

var textureData = [
    128, 0, 0, 255,     50, 0, 0, 255,
     50, 0, 0, 255,    200, 0, 0, 255
];

// create the texture
theTexture = createTexture(
    textureData,
    2, 2
    );

I'm drawing a full screen quad, where the lower left has a texture coordinate of 0,0 and the upper right has one of (2,1). In my pixel shader, on the left side i sample along the diagonal of the image (again, using bilinear sampling), using the x texture coordinate to define where in the image 0.25 to 0.75 my u,v coordinates should be.

The left side uses bilinear sampling and you can see that it's very rough. This is the problem I'm trying to overcome, and have heard (but have not seen proof) that the texture is interpolated using 24.8 fixed point in the hardware, which makes those jaggies.

On the right side, i remap the 1.0-2.0 x axis texture coordinate to 0-1 so that it shows the same thing.

On the right side however, instead of doing hardware bilinear texture sampling, I'm getting the 4 pixels of the image and doing the bilinear interpolation in the fragment shader myself. The results are smooth like you'd expect, due to the calculations happening with floating point values.

To get the actual pixel color on both sides, i check the texture coordinate's y axis versus the pixel value interpolated. If the texture coordinate's y axis value is smaller, i make the pixel green, else make it black.

The vertex shader is uninteresting and just a pass through, but here's the fragment shader

<script id="shader-fs" type="x-shader/x-fragment">
precision mediump float;
uniform sampler2D uSampler;
varying vec2 vTextureCoord;

vec4 SamplePixel (vec2 pixel, bool bilinearSampling) {
    // Bilinear sampling:
    // Hardware based bilinear sampling
    if (bilinearSampling)
        return texture2D(uSampler, (pixel + vec2(0.5)) / 2.0);

    // Nearest sampling:
    // Software bilinear sampling (higher quality)
    float fracu = fract(pixel.x);
    float fracv = fract(pixel.y);

    vec2 floorPixel = floor(pixel) + vec2(0.5,0.5);

    vec4 A = texture2D(uSampler, floorPixel / 2.0);
    vec4 B = texture2D(uSampler, (floorPixel+vec2(1.0,0.0)) / 2.0);
    vec4 C = texture2D(uSampler, (floorPixel+vec2(0.0,1.0)) / 2.0);
    vec4 D = texture2D(uSampler, (floorPixel+vec2(1.0,1.0)) / 2.0);

    return mix(mix(A,B,fracu), mix(C,D,fracu), fracv);
}

void main(void) {
    vec4 colorValue;

    if (vTextureCoord.x < 0.995)
    {
        float pixel = vTextureCoord.x / 0.995;
        colorValue = SamplePixel(vec2(pixel), true);
    }
    else if (vTextureCoord.x > 1.005)
    {
        float pixel = fract(vTextureCoord.x - 0.005) / 0.995;
        colorValue = SamplePixel(vec2(pixel), false);
    }
    else
    {
        gl_FragColor = vec4(1.0);
        return;
    }

    float value = step(vTextureCoord.y, colorValue.x);
    gl_FragColor = vec4(0.0, value, 0.0, 1.0);
}
</script>

Answer 1

As luck would have it Iñigo Quilez wrote an article about this which popped up in my Facebook feed this morning.

Hardware texture intepolation is fast and convenient. It is bilinear (plus mipmapping), and despite it can be somehow improved, it works great for most cases. Most cases being texture mapping of surfaces with color/albedo, normal and specular textures. Generally these types of textures don't need lots of color fidelity (where "color" means pixel value/content), and therefore the hardware's texture interpolation units implement some simple interpolation based on 24.8 fixed point arithmetic. That's what all GPUs' hardware implments these days.

A 24.8 precision texture interpolator means that there's a maximum of 256 intemediate values possible between two adjacent pixels of a texture. 256 values are a lot for albedo textures for sure, but often in computer graphics textures encode not only surface properties, but they serve as LookUp Tables (LUT), heighfields (for terrain rendering), or who knows what. In those cases, you can find yourself easily lacking more resolution than 256 values between pixels.

Here's an example image from the article:

(Top) Hardware interpolation: visually smooth, but the barcode shows the derivative is spiky. (Bottom) Manual interpolation: the constant colour in the derivative bar shows we're not getting step artifacts anymore.

So there's the answer. Hardware filtering is mainly intended to bridge sub-texel gaps for surface textures. So assuming our textures have enough resolution & mip maps for how close they are to the camera, we usually only have to interpolate a few steps - to deal with a texel center that's just a bit off from our rendered pixel center, or to fill a gap between two adjacent texels that are slightly more than 1 rendered pixel apart. For most cases, if we need anywhere near all 256 interpolation steps shown at once, it's because the player has crammed their camera right up against a wall, or we're trying to massively stretch a texture that's too low-res for the use it's being put to.

But if we're doing procedural graphics, it's often tempting to use a low-resolution texture stretched wide to give us low-frequency gradients, using the texture interpolator to give us free bilinear interpolation between samples. For colours this is often okay, since the human eye is pretty bad at spotting banding artifacts unless they're extreme. But if we care about the precise value at each sample - like for heightfields or sharp thresholds as you're doing here, then these staircase & plateau artifacts creep in.

So what can we do about it?

The most general-purpose solution (and what Iñigo Quilez recommends in his article) is exactly what you're doing: sample the texture four times and interpolate the results yourself in the shader, where you have full floating point precision.

If the case you've shown is representative of your real applications though, there may be more efficient methods:

• Increase the resolution of your input texture, say to 16x16. Now the texture interpolation from a single fetch only needs to cover 1/15th as much ground with its 256 intermediate values, so you have an effective range of 3840 intermediates from one edge to the other. Much better! And still with only 1 texture sample. The memory taken up by this texture is still tiny, so you needn't feel bad about being wasteful.

• If you only ever need to do bilinear filtering between the same 4 corners, you might as well do away with the texture and do it all in shader variables. This way you don't pay for even a single texture fetch. You can profile to double-check, but I'd wager this might be even be faster than using the "free" interpolation from the texture pipeline, since there's no contention or caching overhead.

Answer 2

I found this paper:
"Efficient GPU-Based Texture Interpolation using Uniform B-Splines" http://www.mate.tue.nl/mate/pdfs/10318.pdf

Which states:

It should be noted that there are some precision issues associated with the hard-wired linear texture interpolation. When, e.g.,an eight-bit texture is filtered, most people would expect that first the neighboring texture knots are queried, casted to floating point, and then weighted and added. This is, however, not the case; the texture knots are first weighted and added, and then casted to floating point, which limits the precision to the least significant bit of the texture data format [Ruijters et al. 08], as is illustrated in Figure 4. As a consequence, higher accuracies can only be obtained by using larger texture words, and thus at the cost of texture memory consumption.

A further precision issue of the linear texture interpolation is caused by the fact that the accuracy of the texture coordinates is limited to a fixed-point format with eight bits of fractional value [NVIDIA 08]. This means that there are only 254 discrete coordinate positions between two texture knots, as shown in the zoomed graph in Figure 4, which is especially of interest when the knots are far apart...

The two references it has are:

[NVIDIA 08] NVIDIA Corporation. “Linear Filetering.” In NVIDIA CUDA Compute Unified Device Architecture: Programming Guide, Appendix D.2, 2008. http://www.uni-graz.at/~liebma/CUDA/NVIDIA_CUDA_Programming_Guide_2.0.pdf

and:

[Ruijters et al. 08] Daniel Ruijters, Bart M. ter Haar Romeny, and Paul Suetens. “Accuracy of GPU-based B-Spline Evaluation.” In Computer Graphics and Imaging (CGIM), pp. 117–122. Calgary, AB, Canada: ACTA Press, 2008. http://www.dannyruijters.nl/docs/GPU_AccuracyBSpline.pdf