Camera Calibration custom Model-View-Projection matrix in Unity?

Question

Coming from this topic

I have a Model-View-Projection Matrix obtained from a Camera Calibration method, from where I want to render the scene:

-4.7415   -2.3964   -2.4075  253.0431
-2.7923    6.1067    6.1569  143.6573
 0.0000    0.0000    1.0000    1.0000
 0.0061   -0.0094    0.0145    1.0000

As far as I saw, all documentation shows methods to load ModelView and load Projection independently. I was hoping to find something like:

GL.PushMatrix();
GL.MultMatrix(theMVPMatrix);

-> draw/load all the scene 

GL.PopMatrix();

All examples I found show this would run OnPostRender(), but assuming there is a camera, at this point the camera has already rasterized the image. I guess?

edited to make my steps clearer:

Starting a new unity scene. A camera calibration algorithm returns me a matrix \$C\$ such:

 0.00166351   -0.00917993    0.0216692   0.885074
 -0.0120409     0.0087226    0.0119709   0.464458
-1.5839E-05  -1.43059E-05  1.55468E-05  0.0013842

I add a new camera object to the unity scene and name it "projector".

After decomposing \$C\$ into \$R\$, \$K\$ and |T| I can apply rotation and transformation on newly added unity camera, and also extract the projection to apply on the same camera, following this and this steps.

\$R\$

 0.703018  -0.00551996  0.711151
-0.38204    0.8405      0.384195
-0.599843  -0.541784    0.588778

\$K\$

 629.808  -0.986886   633.74
  0        626.035    361.485
  0          0        1

applying position and rotation from R and |T|

projector.transform.position = new Vector3(-30.2851,-30.2294, 30.3633);
projector.transform.rotation = Quaternion.Euler(new Vector3(33.126,-45.329,-0.45));
//Ground truth camera position: 30,30,-30 ; rotation: 33,-45,0

ProjMat:

0.98408 0.00154 0.00978 0.00000
0.0     1.73899 0.00413 0.00000
0.0     0.0     1.001   0.10005
0.0     0.0     -1.0    0.00000

I apply this projection matrix to the unity as suggest:

Projector.projectionMatrix = ProjMat;

Although not constant for each calibration matrix, so far I obtain projector(pink) and ground_truth(green) cameras almost matching.

// Undo view matrix 'V'
var antiView = Projector.transform.localToWorldMatrix;

// Undo model matrix 'M' (assuming this object is not batched).
var antiModel = transform.localToWorldMatrix;
    

//the moment I enable the line below, my "projector" camera frustum collapses into a line and points somewhere away from the projection center (orange) and values go strange.

Projector.projectionMatrix = ProjMat * antiModel * antiView;

edit2: Added the comparison between Unity and Processing screens while adjusting the calibration points. I am still aiming to obtain a stable matrix transformation, as seen in the Processing window.

Answer 1

The simplest way to do this is probably to override the camera's projection matrix (see another recent answer), since that will automatically work with any shaders you're already using. You can have a function like this...

public static OverrideProjection(Camera camera, Matrix4x4 newProjection) {
    camera.projectionMatrix = newProjection;
}

This replaces the default projection matrix for your camera, P, with your new matrix, N. So the MVP matrix you render with changes from P * V * M to N * V * M.

But even though this is stacked on the end, you can override the M and V matrices too if you need, something like so:

// Invoked on a MonoBehaviour just before the object gets rendered.
void OnWillRenderObject() {
    // Capture the camera currently being used for rendering.
    var camera = Camera.current;

    // Undo view matrix 'V'
    var antiView = camera.transform.localToWorldMatrix;

    // Undo model matrix 'M' (assuming this object is not batched).
    var antiModel = transform.localToWorldMatrix;

    camera.projectionMatrix = newMatrix * antiModel * antiView;
}

When this is combined with the other matrices in the rendering pipeline, you get...

$$N M^{-1} V^{-1} V M = N M^{-1} M = N$$

So we cancel out the built-in matrices in the pipeline and substitute our own.

If you do this, I'd recommend still trying to position the camera and objects used for rendering reasonably close to their effective arrangement after your matrix substitution. The reason for this is that culling and level of detail are performed based on how objects sit inside the unmodified camera frustum - that system doesn't see your custom matrix, so it might conclude an object is out of view and skip rendering it, even if your custom matrix would have brought it into view.

---

Another way you can approach this is to write your own shaders for the objects you're rendering, and just directly replace the vertex transformation.

If you make a new unlit shader, you'll see some code like this:

sampler2D _MainTex;
float4 _MainTex_ST;

v2f vert (appdata v)
{
    v2f o;
    o.vertex = UnityObjectToClipPos(v.vertex);
    o.uv = TRANSFORM_TEX(v.uv, _MainTex);
    UNITY_TRANSFER_FOG(o,o.vertex);
    return o;
}

That UnityObjectToClipPos() is basically a macro that maps to mul(UNITY_MATRIX_MVP, v.vertex) (plus some adjustment for stereoscopic rendering when you're using it). We can replace it with our own custom matrix:

sampler2D _MainTex;
float4 _MainTex_ST;
float4x4 _CustomMatrix;

v2f vert (appdata v)
{
    v2f o;
    o.vertex = mul(_CustomMatrix, v.vertex);
    o.uv = TRANSFORM_TEX(v.uv, _MainTex);
    UNITY_TRANSFER_FOG(o,o.vertex);
    return o;
}

You can assign a value to this _CustomMatrix uniform with either Material.SetMatrix(), if you want it to affect just one material/object, or Shader.SetGlobalMatrix(), if you want to set the matrix once and use the same version across all materials.

You can also chain it with any of Unity's built-in matrices, if you don't want to completely replace the model or view matrices.