How to procedurally compose shaders? [closed]

Question

I read several times that shaders can be composed by the engine on the fly, depending on graphical settings. How is this generally done?

Answer 1

A common way that shaders are adapted depending on settings is through compiler directives like define macros and conditional compilation.

You can see a lot of examples of this if you look through the shader code of a free-to-use game engine like Unity (not specifically written in OpenGL-style glsl, though they use transpilers to output their shader code to OpenGL-compatible versions):

output.screenUV = output.positionCS.xyw;
#if UNITY_UV_STARTS_AT_TOP
output.screenUV.xy = output.screenUV.xy * float2(0.5, -0.5) + 0.5 * output.screenUV.z;
#else
output.screenUV.xy = output.screenUV.xy * 0.5 + 0.5 * output.screenUV.z;
#endif

This lets the shader adapt to different graphics APIs' conventions on where the (0, 0) point of a texture is: top-left for DirectX-like APIs, and bottom-left for OpenGL-like.

Here UNITY_UV_STARTS_AT_TOP is a macro that gets replaced by a constant during compilation. When invoking the shader compiler, the engine will include a shader source file with the needed constants for the platform/graphics API it's targeting. For instance, in D3D11.hlsl there's a line saying:

#define UNITY_UV_STARTS_AT_TOP 1

So when the compiler gets to #if UNITY_UV_STARTS_AT_TOP, that compiles to #if 1 (ie. "yes") and the first version of the code gets compiled into the resulting shader, and the second version gets skipped over by the compiler.

If we were compiling this for OpenGL, we'd instead have defined this macro to be zero, and the first version of the code would be skipped over, compiling the second version into the resulting shader.

These macros can also define whole blocks of code - functions that do different things depending on needs, variable declarations that change their format or syntax to match what the platform wants, etc.

A deep material and lighting system will use these types of compiler switches for all sorts of things, including:

• Whether a feature is supported on a given graphics card. If not, replace the code that uses the feature with code that emulates the feature, or uses a graceful fallback instead.

• Whether this material is being lit by a spot- or point light (rays diverge) or a directional light (rays parallel) - this would swap out the math that calculates the lighting vector and attenuation factors, while leaving the code that uses these values to actually compute the colour common to both paths.

• Whether the material uses a particular feature, like detail maps. If not, the code that applies them can be compiled out of its shader entirely, rather than leaving a runtime branch that will never be taken.

• Whether the player has enabled an optional feature, like parallax occlusion mapping. If so, the code to do it is included. If not, it's stripped out or replaced with a cheaper fallback.

...etc.

This can result in a lot of code handling all these different cases, so in a large engine this will usually be parcelled off into several separate shader files that individual material shaders simply "include", rather than copy-pasting all this logic into every shader and creating a maintenance nightmare.

On modern graphics APIs, to save load times, a lot of this compilation is done in advance at edit/build time, stamping out the permutations of the shaders the engine will need for the given game content in a more efficient "intermediate representation" that can be digested by the graphics driver faster than parsing files upon files of human-readable code.

It's also possible to compile shaders from raw text, allowing you to just concatenate together a string with whatever code you want it to run. This is commonly how it's done in JavaScript for WebGL apps since we don't have a binary build in the same way as a native app. But for non-web applications, this is something you'd usually do only if you need extremely open-ended shader editing at runtime, like if you were making a shader graph material editor for users to create their own bespoke material effects. If you just want to apply some small, fixed set of settings tweaks, conditional compilation handles that in a more structured way, with fewer complications like needing to come up with non-colliding variable names in each block of code that's concatenated-on.

Answer 2

Different shaders are compiled/loaded depending on the settings selected. This is generally why games ask you to restart the game, so the correct shader can be loaded when the game boots and why you can't change some settings while the game is running, i.e. only from the main menu.