In a graph-based material system, how can I support a variety of input and output types?

Question

I'm trying to wrap my head around how material systems like this, this are implemented. These powerful and user-friendly, graph-like systems seem to be relatively common as a method of allowing programmers and non-programmers alike to quickly create shaders. However, from my relatively limited experience with graphics programming, I'm not entirely sure how they work.

---

Background:

So, when I've programmed simple OpenGL rendering systems before, I typically create a Material class that loads, compiles, and links shaders from static GLSL files that I've manually created. I also usually create this class as a simple wrapper for accessing GLSL uniform variables. As a simple example, imagine that I have a basic vertex shader and fragment shader, with an extra uniform Texture2D for passing a texture. My Material class would simply load and compile those two shaders into a material, and from that point on it would expose a simple interface for reading/writing the Texture2D uniform of that shader.

To make this system a little bit more flexible, I usually write it in a way that allows me to attempt to pass uniforms of any name/type [i.e.: SetUniform_Vec4( "AmbientColor", colorVec4 ); which would set the AmbientColor uniform to a particular 4d vector called "colorVec4" if that uniform exists in the material.].

class Material
{
    private:
       int shaderID;
       string vertShaderPath;
       string fragSahderPath;

       void loadShaderFiles(); //load shaders from files at internal paths.
       void buildMaterial(); //link, compile, buffer with OpenGL, etc.      

    public:
        void SetGenericUniform( string uniformName, int param );
        void SetGenericUniform( string uniformName, float param );
        void SetGenericUniform( string uniformName, vec4 param );
        //overrides for various types, etc...

        int GetUniform( string uniformName );
        float GetUniform( string uniformName );
        vec4 GetUniform( string uniformName );
        //etc...

        //ctor, dtor, etc., omitted for clarity..
}

This works but it feels like a bad system due to the fact that the client of the Material class has to access uniforms on faith alone - the user has to be somewhat aware of the uniforms that are in each material object because they're forced to pass them by their GLSL name. It's not a huge deal when it's just 1-2 people working with the system, but I can't imagine this system would scale very well at all, and before I make my next attempt at programming an OpenGL rendering system, I want to level up a bit.

---

Question:

That's where I am so far, so I've been trying to study how other rendering engines handle their material systems.

This node-based approach is great and it seems to be an extremely common system for creating user friendly material systems in modern engines and tools. From what I can tell they're based on a graph data structure where each node represents some shader aspect of your material and each path represents some kind of relationship between them.

From what I can tell, implementing that kind of system would be as simple a MaterialNode class with a variety of subclasses (TextureNode, FloatNode, LerpNode, etc.). Where each MaterialNode subclass would have MaterialConnections.

class MaterialConnection
{
    MatNode_Out * fromNode;
    MatNode_In * toNode;
}

class LerpNode : MaterialNode
{
    MatNode_In x;
    MatNode_In y;
    MatNode_In alpha;

    MatNode_Out result;
}

That's the very basic idea, but I'm a little bit uncertain about how a few aspects of this system would work:

1.) If you look at the various 'Material Expressions' (nodes) that Unreal Engine 4 uses, you'll see that they each have input and output connections of a variety of types. Some nodes output floats, some output vector2, some output vector4, etc. How can I improve the nodes and connections above so that they can support a variety of input and output types? Would subclassing MatNode_Out with MatNode_Out_Float and MatNode_Out_Vec4 (and so on) be a wise choice?

2.) Finally, how does this kind of system relate to GLSL shaders? Looking again at UE4 (and similarly for the other systems linked above), the user is required to eventually plug a some material node into a large node with various parameters that represent shader parameters (base color, metalness, gloss, emissiveness, etc.). My original assumption was that UE4 had some kind of hard coded 'master shader' with a variety of uniforms, and everything that the user does in their 'material' is simply passed to the 'master shader' when they plug their nodes into the 'master node'.

However, the UE4 documentation states:

"Each node contains a snippet of HLSL code, designated to perform a specific task. This means that as you construct a Material, you are creating HLSL code through visual scripting."

If that's true, does this system generate a real shader script? How exactly does this work?

Answer 1

I'll try to answer to the best of my knowledge, with little knowledge about the specific case of UE4, but rather on the general technique.

Graph based materials are as much programming as writing the code yourself. It just doesn't feel like it for people with no background on code, making it seemingly easier. So, when a designer links a "Add" node, he is basically writing add(value1,value2) and linking the output to something else. This is what they mean that each node will generate HLSL code, either its a function call or just straightforward instructions.

In the end, using the material graph is like programming raw shaders with a library of predefined functions that do some common useful things, and that's also what UE4 does. It has a library of shader code that a shader compiler will take and inject into the final shader source when applicable.

In the case of UE4, if they claim its converted to HLSL, I assume they use a converter tool which is able to convert HLSL byte code into GLSL byte code, so its usable on GL platforms. But other libraries just have multiple shader compilers, which will read the graph and directly generate the needed shading language sources.

The material graph is also a nice way to abstract from platform specifics and focus on what matters from a art direction point of view. Since its not bound to a language and much higher level, its easier to optimize for the target platform and dynamically inject other code like light handling into the shader.

1) Now to answer your questions more directly, you should have a data-driven approach to designing such a system. Find a flat format that can be defined in very plain structures, and even defined in a text file. In essence, each graph should be an array of nodes, with a type, a set of inputs and outputs, and each of these fields should have a local link_id to make sure graph connections are unambiguous. Also, each of these fields could have additional configuration to what the field supports (what range of data types are supported, for example).

With this approach, you could easily define a node's field to be (float|double) and let it infer the type from the connections, or force a type into it, with no class hierarchies or overengineering. It's up to you to design this graph data structure as rigid or flexible as you want. All you need is that it has enough information so that the code generator will not have ambiguity and therefore potentially handling wrong what you want to do. The important is that at the basic data structure level, you keep it flexible and focused on solving the task of defining a material alone.

When I say, "define a material" I am very specifically referring to defining a mesh surface, beyond what the geometry itself provides. This includes using additional vertex attributes to configure the aspect of the surface, add displacement to it with a heightmap, disturb the normals with per-pixel normals, changing physically based parameters, to change the BRDF's and so on. You don't want to describe anything else like HDR, tonemapping, skinning animation, light handling or a lot other things done in shaders.

2) It is then up to the renderer's shader generator to traverse this data structure and by reading its information assemble a set of variables and link them together using premade functions and injecting the code that calculates lighting and other effects. Just remember that shaders vary not only from different graphics API's, but also between different renderers (a deferred rendering vs tile based vs forward renderer all require different shaders to work), and with a material system such as this, you can abstract from the nasty low level layer, and focus only on describing the surface.´

For UE4, they came up with a list of things for that final output node you mention, which they think describes 99% of the surfaces in old and modern games. They developed this set of parameters over decades and proved it with the insane amount of games the Unreal engine has produced so far. Therefore you will be just fine if you do things the same way as unreal.

To wrap it up, I suggest a .material file just to handle each graph. During development, it would contain perhaps a text based format to debug and then be packaged or compiled to binary for release. Each .material would be composed by N nodes and N connections, a lot like a SQL database. Each node would have N fields, with a name and some flags for types accepted, if its input or ouput, if the types are inferred, etc. The runtime data structure to hold the loaded material would be just as flat and simple, so the editor can easily adapt it and save back to file.

And then leave the actual heavy lifting for the final shader generation, which is really the hardest part to do. The beautiful part is that your material stays agnostic to the rendering platform, in theory it would work with any rendering technique and API as long as you represent the material in its appropriate shading language.

Let me know if you need additional details or any fixing in my answer, I lost the overview over all the text.