How do I calculate a matrix to render a mirror about a plane using GLM?

Question

I've been trying to work out how to render a mirror for a while now, but a lot of resources I find either use very outdated tools I'm not familiar with, appear to be college slides referencing other material I cannot find, or simply don't seem to work correctly at all. (e.g. the answer to a previous OpenGL mirror question links to wikibooks' tutorial on portal rendering, which doesn't provide the appropriate math for reflections.)

I have all the stencil logic in place but am struggling to produce a reflection matrix which, when combined with my view matrix, creates a correct reflection. Notice that the in-game render below does not match up with a correct reflection of the same scene:

The setup for calculating the view prior to rendering the scene looks like this:

// Obtain the model to world-space matrix for the mirror plane
        auto N = node.getCachedWorldTransform();
        
        // Obtain the position and normal of the mirror plane in world-space
        auto pt = (N * glm::vec4(meshes[node.meshes[0]].firstPoint, 1.0)).xyz();
        auto norm = glm::normalize(glm::mat3(glm::transpose(glm::inverse(N))) * meshes[node.meshes[0]].firstNormal);

        // Calculate plane equation, and then reflection matrix
        auto plane = planeFromPointNormal(pt, norm);
        auto refl = glm::transpose(matrixReflect(plane));

        // Create a new view matrix combining our reflection matrix with the previously used camera matrix
        auto pCam = camView * refl;

The helper functions involved are:

glm::vec4 planeFromPointNormal(glm::vec3 pt, glm::vec3 normal) {
    auto norm = glm::normalize(normal);
    return { norm.x, norm.y, norm.z, -glm::dot(pt, norm) };
}

// The math here is ripped shamelessly from D3DX documentation, so it may have to be transposed
glm::mat4 matrixReflect(glm::vec4 plane) {
    auto P = glm::normalize(plane);

    return glm::mat4{
        -2 * P.x * P.x + 1,  -2 * P.y * P.x,      -2 * P.z * P.x,        0,
        -2 * P.x * P.y,      -2 * P.y * P.y + 1,  -2 * P.z * P.y,        0,
        -2 * P.x * P.z,      -2 * P.y * P.z,      -2 * P.z * P.z + 1,    0,
        -2 * P.x * P.w,      -2 * P.y * P.w,      -2 * P.z * P.w,        1
    };
}

Various attempts at fixing this (altering culling, transposing or inverting matrices, keeping regular back-face culling but applying negative scaling on some axis, swapping the multiplication order of components, etc.) do not seem to have any effect on the outcome other than making the reflection broken in dramatically different ways. My current attempt is to flip the culling face (glCullFace(GL_FRONT)) before drawing the reflected objects. (I am also concerned that this might cause the back-face of the floor beneath the mirror to be rendered!)

What might I be doing wrong? Is there some resource I am missing that lays out clearly how to do this?

Answer 1

I was working out the math for a reflection matrix from first principles, and found it matches your code exactly, except that you normalize the plane 4-vector first.

Don't do that.

We want the first 3 components to form a unit 3-vector, which they already do thanks to the normalization in the planeFromPointNormal function.

---

Here's the full derivation:

Take a basis direction vector \$\vec d = (1, 0, 0)\$. We'd expect this to be reflected by a unit normal \$\vec n = (n_x, n_y, n_z)\$ to:

$$\vec d - 2 \left( \vec d \cdot \vec n \right) \vec n\\ =(1, 0, 0) - 2n_x(nx, n_y, n_z)\\ = (1 - 2n_x^2, -2n_xn_y, -2 n_xn_z)$$

And so on for the other basis vectors. That gives us the first three columns of our matrix.

Next we'll take a vector on the reflection plane: \$\vec p = -w \vec n\$ makes a convenient choice, where \$w\$ is the fourth component of our plane vector. If we reflect that vector about the origin along our normal, we'd get \$\vec p^\prime = -w \left(- \vec n\right) = w \vec n\$, and the translation that brings it back to where it started is \$\vec t = \vec p - \vec p^\prime = -w \vec n - w\vec n = -2w\vec n\$, so that gives us our fourth column.

glm::mat4 matrixReflect(glm::vec4 plane) {

    return glm::mat4{
        1-2*plane.x*plane.x,  -2*plane.x*plane.y,  -2*plane.x*plane.z, -2*plane.x*plane.w,
         -2*plane.y*plane.x, 1-2*plane.y*plane.y,  -2*plane.y*plane.z, -2*plane.y*plane.w,
         -2*plane.z*plane.x,  -2*plane.z*plane.y, 1-2*plane.z*plane.z, -2*plane.z*plane.w,
                          0,                   0,                   0,                  1
    };
}

Note that I've baked-in the transpose here, so you don't need to transpose the matrix after computing it.