# How to perform some custom math between the model matrix and the projection matrix?

I'm trying to write a shader in Unity that will perform some custom math after the model*view matrix is applied but before the projection matrix is applied.

First off, I'm not sure whether I should be using a surface shader or an unlit shader. My impression is that a surface shader can do everything I need it to and the outcome will look better with less effort, is that right?

I'm thinking that what I can do is manually apply the model and view matrices in the vertex shader, and then I have to tell it somehow to only apply the projection matrix in the rasterization step. Any idea how to go about this, or a different way to achieve the same result?

• "I'm not sure whether I should be using a surface shader or an unlit shader." - do you want lighting? If not, an unlit shader is the way to go. Aug 1, 2020 at 17:05
• Note too that the projection matrix is not applied in the rasterization step. It's the vertex shader's job to do this. The rasterizer just takes the clip space positions output by the vertex shader joins them up to find the pixels covered by each projected triangle. Aug 1, 2020 at 18:02
• @DMGregory is it okay if I just ask you questions directly? I DMed you on Twitter Aug 1, 2020 at 18:29
• I won't always be at my desk, but you may also want to try Game Development Chat for quick informal help if you don't want to make a whole new Question post just yet. Aug 1, 2020 at 18:31

## 1 Answer

Here's a modified version of the default unlit vertex shader that shows how to break UnityObjectToClipPos into two steps, with a layover in camera view space in between.

v2f vert (appdata v)
{
v2f o;

// Ordinarily, we'd go all the way from object space to clip space in one operation:
// o.vertex = UnityObjectToClipPos(v.vertex);

// But instead, we'll first go from object space to world space to view space:
float3 viewSpace = UnityObjectToViewPos(v.vertex);

// Then do some math in view space... Here I exchange x & y for a weird effect. :)
viewSpace.xy = viewSpace.yx * float2(-1, 1);

// Then finally project the view space position into clip space:
o.vertex = mul(UNITY_MATRIX_P, float4(viewSpace, 1));

// And the rest of the shader can stay unchanged.

o.uv = TRANSFORM_TEX(v.uv, _MainTex);
UNITY_TRANSFER_FOG(o,o.vertex);
return o;
}


If you want to do this with a surface shader, it's a bit more tricky, since their vertex modification happens in local space. We need to transform to view space, apply our adjustments, then transform back to local space for the vertex to be in the right space for the rest of the pipeline to work with:

// Modify your surface shader declaration to point at the vertex-modifying function
#pragma surface surf Standard fullforwardshadows vertex:vert

// ...then provide the function lower down.

void vert(inout appdata_full v) {
// As before, we transform into view space first.
float3 viewSpace = UnityObjectToViewPos(v.vertex);

// Do some math on our vertex in that space...
viewSpace.xy = viewSpace.yx * float2(-1, 1);

// Now the tricky part: getting back. We don't have an inverse view
// matrix by default (though a script could provide this for us).

// Instead, we'll take advantage of the fact that the camera view matrix
// is just a rotation and translation, and the inverse of a rotation
// is its transpose (much cheaper to calculate in a shader):
float4x4 viewMatrix = UNITY_MATRIX_V;
float3 translation = float3(viewMatrix[0][3], viewMatrix[1][3], viewMatrix[2][3]);

viewMatrix[0][3] = 0;
viewMatrix[1][3] = 0;
viewMatrix[2][3] = 0;

// Pseudo-inverse: undo the translation, then the rotation.
viewSpace -= translation;
float4 worldSpace = mul(transpose(viewMatrix), float4(viewSpace, 0));
worldSpace.w = 1;

// Now for the final hop from world space back to object space.
v.vertex = mul(unity_WorldToObject, worldSpace);
}


Note that neither version I've shown touches your normal vectors. Be aware that these are not automatically recalculated when you change your vertices. So, if you alter your shape in such a way that its surface normals or lighting should be affected, be sure to apply a corresponding transformation to the normal vectors while you're at it.

• Cool, but how is the data passed to the surface shader if I don't write a fragment shader? Aug 1, 2020 at 17:38
• The first block of code is for an unlit shader, since your question didn't say whether you needed a surface shader or vertex-fragment style. If you're using a surface shader, you'd use the second block of code I've just added. Aug 1, 2020 at 17:46
• Sounds like using a vertex-fragment shader is the right move. Thanks! Aug 1, 2020 at 17:50