Should 3D transformations be represented by a 4x4 matrix or a 3x4 matrix?

Question

Since 3D transformations are represented by 4x4 homogeneous matrices we know that their last row is always (0,0,0,1), and as such the behavior of this final row is implied so long as we know whether or not the transformation is operating on a vector (a 4x1 matrix with a w element of 0) or a point (a 4x1 matrix with a w element of 1).

If we have classes that represent points and vectors separately as 3x1 matrices with implied w coordinates then we can overload the * operator so that its behavior on both classes is separate and thus there is no need to store the final row of the transformation matrix in memory.

Now I can see how this would be a great optimization at first as it increases cache coherency instead of adding what is essentially padding to each transformation matrix. On the other hand, we will have to eventually use a full 4x4 matrix to handle the projection and perspective divide later on so is it worth it to store a transformation matrix as a 3x4 matrix only to later convert it into a full 4x4? Or would a good solution here be to make the projection matrix a 4x4 matrix and make the product of a transformation matrix and a 4x4 matrix another 4x4 matrix?

I'm sorry if this seems basic or like a question that answers itself but I was reading a book on game engine development and it used implications to handle interactions between transformation matrices with points and vectors, and other transformations separately, never multiplying by the final row and using implications instead. But on creation they would still stored the final row as (0,0,0,1) in memory to only be used in multiplication between a transformation matrix and a 4x4 matrix specifically and so I'm mostly just wondering if there may have been a good reason for that or not. Thanks in advance for the help.

Answer 1

Since 3D transformations are represented by 4x4 homogeneous matrices

Correction: Might be represented by 4x4 matrices ;)

Now I can see how this would be a great optimization at first as it increases cache coherency instead of adding what is essentially padding to each transformation matrix.

I don't think that cache coherence is a major issue here. More relevant are the computations that you can avoid. A 4x4 matrix multiplied with a 4 element vector results in 16 multiplications and 12 additions. If you use a 3x4 matrix it is only 12 multiplications and 9 additions. So you have 25% less of the computational costs. You also save some memory but I don't think it is as relevant as the computational savings.

I can remember having a similar discussion about this topic on www.gamedev.net. It was going even further by just using 3x3 matrices and handling translations separately by vector addition (computations: 9 multiplications, 9 additions) . If I remember correctly the conclusion was, that 4x4 matrices are easier to use, since they contain translations and also handle point vs vector issues for you. This "luxury" comes with a performance impact. However, I can also remember that people stated, that this performance impact is negligible most of the time since the matrix multiplications are done during vertex processing and this is usually not a bottleneck in a game engine. To be sure, you have to benchmark it.

On the other hand, we will have to eventually use a full 4x4 matrix to handle the projection and perspective divide later on so is it worth it to store a transformation matrix as a 3x4 matrix only to later convert it into a full 4x4? Or would a good solution here be to make the projection matrix a 4x4 matrix and make the product of a transformation matrix and a 4x4 matrix another 4x4 matrix?

I think you are approaching this a little bit too complicated. You have probably something in mind like this:

$$v_p = P \cdot T \cdot v$$

where P is the 4x4 projection matrix and T an arbitrary 3x4 transformation matrix. v is the Vertex that should be transformed and v_p the fully transformed vertex. If I understood you correctly, then your problem here is, that 4x4 matrices can't be multiplied with 3x4 matrices, so you think about introducing a special operation for that. However, since v times a matrix always yields a vector, you can simply do the following (GLSL like pseudo-code):

vec3 v_t = T * v; 
vec4 v_p = P * vec4(v_t, 1.0);

So just extend the vector by one element after the 3x4 multiplication and multiply the 4x4 projection matrix afterward. However, there is one other problem with 3x4 matrices. You can't multiply them with each other and you can't multiply them with the result of a preceding 3x4 matrix-vector multiplication since they yield a vec3 but can only be multiplied with a vec4. So you have a problem in case you have multiple 3x4 transformation matrices. You can solve this exactly the same way as in the previous example by doing one matrix-vector multiplication at a time and extending the result vector to 4 components, but I don't think this is a smart solution. This is why you should either use 4x4 matrices or 3x3 matrices and handle translations by vector addition. However, since the 4. column of a 3x4 matrix is identical to the translations in this special case, you can still use it as a storage format and just separate it into a 3x3 matrix and a translation vector during the calculations.

classes that represent points and vectors separately as 3x1 matrices

Note, that you have to use 3x3 matrices during multiplication if you want to use vectors of size 3 since you can't multiply a 3x4 matrix with a 3x1 matrix. So I assume that you or the book you are reading separates a 3x4 matrix into a 3x3 matrix and a translation before using it.

Conclusion

Using 3x3 matrices + translations or alternatively 3x4 matrices gives you performance and storage benefits by omitting redundant operations and values but 4x4 matrices are more comfortable to use. The performance benefits might also not be noticeable if your game is not utilizing the full corresponding hardware capacity, so benchmark it, if you are not sure if this is relevant for you. In the end, it is a design decision where you trade more comfortable/readable code against some (possible!) performance gain.