Blending between many materials

Question

I am storing one material ID per vertex in my meshes. In the shader for the mesh, I have a texture array full of textures that the material ID can refer to. The fragment color is chosen using triplanar texturing. Unlike vertex colors, these material IDs cannot be blended.

For example, if a dirt texture had a material ID of 5, and a grass texture had a material ID of 1, fragments in between one vertex with material ID 5 and one with material ID 1 would potentially have a material ID between 2 and 4, which is not ideal. Another flaw with this method is the fact that there would be a hard transition in between materials, as the mix or lerp function in the shader is not used.

Is there a way to blend the textures of two neighboring vertices in the fragment shader? I can add additional per-vertex data if needed.

Answer 1

One way that this can be done is to divide your materials into layer slots. As one example, REDengine 3, powering the Witcher 3, uses a background layer and an overlay layer for its terrain materials.

You can store your material data in control map textures as in the case above, or in your vertex data. Either way, you end up storing the identity of the material in each slot, and a blending weight between them. This keeps the total amount of material data per point and total number of texture sampling operations tightly controlled: even if your library of materials is huge, only a fixed number of them are ever in play for a single triangle/fragment.

If storing this data in the vertices, then this means that any three vertices sharing a triangle need to agree on the materials in each slot. If a vertex sits at the junction of 6 triangles, 3 of which want "snow over rock," and the other 3 want "snow over dirt," then you can duplicate this vertex to handle each case - similar to how we create hard edges where a normal changes abruptly, or UV seams. For a control map, you can use point filtering for the material IDs so they never get interpolated.

This does place a constraint on how your materials are painted. In a 2-layer system, if two adjacent locations disagree about what material should go on layer 1, layer 2 needs to hit full opacity along their shared edge to prevent a texture seam being visible. Similarly for two locations disagreeing about what belongs on layer 2 - it has to hit zero opacity in between. In practice you can usually spread out material changes over a few triangles/cells to account for this, or stick a prop/some grass in the way so the seam isn't visible. ;)

Answer 2

The vertex value will indeed be interpolated when arriving to the fragment shader, so for example having two connected vertices, one with texture slot id=2 and one with id=17 will give the fragment shader everything in between 2 and 17 (including decimal values in between) depending on pixel position, causing lots of texture slots to be walked through on the way.

A quick though might be to actually describe the amount of each texture slot in each vertex, using a 0...1 style, ie. doing it "in parallel", in an order that is always fixed. A vertex typically connects to 6 other vertices (but can connect to more), so at most 6 textures would in that case be involved in the blending at any time. You'd however describe the amount of all textures in each vertex; those that are 0 you'd simply skip in the fragment shader and maybe count on doing at most 6 lookups.

A question that comes next is how many "floating point" channels can you allocate in the vertex data? Not too many, i guess, but at least a few, depending on what else you need to bake in the vertex data.

Having said that, i could point out that you maybe do not need a fine resolution of 1/256 for the "amount of each texture"? If you are willing to have for example 1/16 as resolution, you can bake in the texture amounts for 16 different textures in one 8-bit "float-part", since it can hold 256 different integer values. In that case, you could describe the blending of max 16 texture materials in for example normal.w. Or have 8 channels, each with a resolution of 1/32. Ofc you need to do some bit maths in the shader (and when building the mesh), but that's the fun part right? :-)

Edit:

About the segment above, i must admit a duration-of-a-shower amount of brainwork did not reveal any good solutions as to how to control the splitting of a single float so that one is able to grab individual segments of it's bit-sequence while GPU is interpolating it. This would be a question for the bit-guys, ie. a separate issue to ask the encryption and compression experts. Can you interpolate a 32-bit floating point value AAAA (4 bytes), so that individual bytes of it's bit representation remain decodeable along the interpolation and all become BBBB separately at the other end?