How to best select a texture pair to use in height blending when using a splatmap shader

Question

In my project, I'm implementing a shader to combine up to 4 textures in a terrain using a splatmap texture (https://en.wikipedia.org/wiki/Texture_splatting).

I want to combine the splatmap blending with the height blending technique, described in: http://untitledgam.es/2017/01/height-blending-shader/. This technique basically defines a function to interpolate between two textures more realistically based on height information.

The problem is, in order to use height blending, I need to know which pair of textures I'll be interpolating, while in the splatmap I actually have four different texture sets (channels, R G B and A).

So what I want to achieve is a shader that will select the pair of textures for which the splatmap channel information is strongest, and interpolate between the two while discarding others. My "naïve" attempt was something along the lines of (simplified version, pseudo-code and only for illustrative purposes):

uniform sampler2D tex1;
uniform sampler2D tex2;
uniform sampler2D splat;

void fragment_shader() {
    sampler2D sampler;
    if (splat.r > splat.g) {
      sampler = tex1;
    } else {
      sampler = tex2;
    }
}

The problem, I realized, is that I can't store samplers inside a variable like this due to limitations in GLSL. So I'm looking for a soultion that would be similar to this. Particularly, I would want to avoid a solution where I'm sampling all four textures for every texture lookup. That is because I will be using some expensive techniques inside the shader which require several texture lookups, and I don't want to increase the computation cost 2x.

EDIT My current attempt to do this consists of computing two vectors to select the maximum and "second maximum" channels. A channel vector would be something like (1.0, 0.0, 0.0, 0.0) to indicate the red channel should be used. The implementation is currently looking something like this:

vec4 c1, c2;
  float m = max(max(max(splat_val.r, splat_val.g), splat_val.b), splat_val.a);
  if (m == 0.0) {
      c1 = vec4(0.0);
      c2 = vec4(0.0);
  } else {
      c1 = vec4(float(splat_val.x == m), float(splat_val.y == m), float(splat_val.z == m), float(splat_val.a == m));
      vec4 splat_val_2 = splat_val * (vec4(1.0) - c1);
      float m2 = max(max(max(splat_val_2.r, splat_val_2.g), splat_val_2.b), splat_val_2.a);
      c2 = vec4(float(splat_val_2.x == m2), float(splat_val_2.y == m2), float(splat_val_2.z == m2), float(splat_val_2.a == m2));  
  }

Where c1 and c2 end up being the channel vectors for the largest and second largest splatmap channels. I'm still facing some issues when there are multiple elements with the same value though, and I keep thinking there should be a better way to do this.

After I compute the channel vectors, I'd still need a nasty static if structure like the following, so all in all I'm not sure my current attempted solution is a very good one:

if (dot(c1, red_constant) == 1.0) {
  if (dot c2, green_constant) == 1.0) {
    do_shader(tex_sampler_R, tex_sampler_G);
  }
  else if (dot c2, blue_constant) == 1.0) {
    do_shader(tex_sampler_R, tex_sampler_B);
  }
} else if (dot(c2, green_constant) == 1.0) {
  // ...
}
// ...

Answer 1

We can actually extend the height blending function to compute weights for all four terrain types at once.

We'll take as input a packed 4-vector, whose RGBA channels are the 0-1 height values of each of our four maps, and use it to modify the 4-vector of terrain type weights we read from our splat map.

(I've written this in Unity hlsl syntax, since that's what I used to test the shader, but it should be straightforward to translate to glsl equivalents)

void applyHeightsToWeights(float4 heights, inout float4 weights) {

    // Scale the height influence of a type toward 0 as we move away from its zone.
    heights *= weights;

    // Compute the lowest height that could have a non-zero blending weight.
    float height_start = max(max(heights.r, heights.g), max(heights.b, heights.a)) 
                       - _Height_Blend_Range;

    // Measure only heights above this cut-off.
    heights = max(heights - height_start, 0.0f);

    // Turn these heights into weights, balanced so all four channels sum to 1.0
    weights.rgba = heights/dot(heights, 1.0f);
}

The way I used this was to make one texture with my four height maps packed into it as the four channels. That way I could read all four heights with one tap, read the splat map with a second tap, and run them through the weighting function above after only these two texture samples.

This is convenient because you can defer your samples of the four albedo/normal/etc. textures and potentially modify them based on the blended result. Like if you were applying some per-pixel displacement based on height, and needed to shift the colour sampling point accordingly.

Or, if some weights come out "close enough" to zero, you could even skip reading those terrain types' albedo/normal/etc. textures entirely. This can save quite a lot of texture bandwidth, so it may be worth the branches to take advantage of it.

The downside is that all 4 terrain types need to have the same tiling frequency if you want to use this trick.

If you really want to pick just two textures to blend though, you can process this weight 4-vector like so:

// Find the greatest weight.
float top = max(max(weights.x, weights.y), max(weights.z, weights.w));

// Zero-out the greatest element.
float4 thresh = (weights < top);
float4 bottom = weights * thresh;

// Find the second-highest weight.
float second = max(max(bottom.x, bottom.y), max(bottom.z, bottom.w));

// Zero-out any elements below second-place.
weights *= (weights >= second);

// Re-balance so the remaining elements sum to 1.0
weights /= dot(weights, 1);

Technically this would give incorrect results if two weights tied at exactly the same value, but I haven't observed that occurring in practice.

In the animation above, you can see that limiting the blending to just the top two textures causes a harsher and more speckly-looking cut-off where the sand feathers out into the blending zone between grass and dirt/rock.

The areas where 3+ textures are all blending together at once tend to be quite small though, so you could get away with 2-texture blending most of the time, with a branch for the rare 3-4-way case.

Visualization of the number of textures contributing more than 1% to the result:

• Dark green: 1
• Bright green: 2
• Yellow: 3
• Red: all 4