Difference between column- and row vector matrix multiplication (vector transform)

Question

Let's say we have a 4-by-4 matrix \$A\$ which represents some transformation.

We can use this matrix to transform a vector in two ways.

• \$Ax\$ by assuming \$x\$ is a 4-by-1 column vector.
• \$xA\$ by assuming \$x\$ is a 1-by-4 row vector.

Up until yesterday, I have not seen this \$xA\$ variant and I thought it was unnecessary. Then I came across this DirectX code from Microsoft.

This function takes just the projection matrix, computes the inverse projection matrix and then does this \$xA\$ variant (not \$Ax\$) to transform (or unproject) a bunch of coordinates.

At first, I wasn't looking and did \$Ax\$ which came out nonsensical and now I'm trying to understand why.

I've noticed that \$Ax = xA^T\$ but when I look at the way the matrix multiplication is carried out. The components that make up the result are very different depending on whether I use \$Ax\$ or \$xA\$ (as it should be, matrix multiplication is not commutative).

The analytic solution for \$Ax\$ (3-by-3 matrix and row vector)

$$ \begin{pmatrix} x \\ y \\ z \end{pmatrix} \cdot \begin{bmatrix} a & b & c \\ d & e & f \\ g & h & i \end{bmatrix} = \begin{pmatrix} ax + dy + gz \\ bx + ey + hz \\ cx + fy + iz \end{pmatrix} $$

The analytic solution for \$xA\$ (3-by-3 matrix and column vector)

$$ \begin{bmatrix} a & b & c \\ d & e & f \\ g & h & i \end{bmatrix} \cdot \begin{pmatrix} x \\ y \\ z \end{pmatrix} = \begin{pmatrix} ax+by+cz \\ dx+ey+fz \\ gx+hy+iz \end{pmatrix} $$

I use Mathematica to test these things. On a hunch I tried the analyitic solution for \$xA\$ with the elements of \$A\$ transposed manually and of course, I get:

$$ \begin{bmatrix} a & d & g \\ b & e & h \\ c & f & i \end{bmatrix} \cdot \begin{pmatrix} x \\ y \\ z \end{pmatrix} = \begin{pmatrix} ax+by+cz \\ dx+ey+fz \\ gx+hy+iz \end{pmatrix} $$

Which is equivalent to \$Ax\$ except we now have a column vector not a row vector. I'm assuming all of this is purely conventional (and depends on whether I'm assuming row- or column-major matrix layout). It looks like I might have answered the question my self but if I'm wrong about anything here please correct me.

Answer 1

I think the answer I was looking for was laid out in @DmitryTolmachov's comment.

Row-major vs column-major is the one of main differences between OpenGL and DirectX. It seems OpenGL uses column-major, while DirectX uses row-major. It's pure conventional, but it starts to matter in what order to perform matrix vs vector multiplication in shader. In GLSL shader if use OpenGL based math library you would use "Ax" convention, but if you use DirectX based math library for your OpenGL app, you would need to use "xA convention". Also this makes difference in what order you need to multiply your matrices.

We can turn row-major into column-major by transposing the matrix. In my particular case I have been using row-major for my 4-by-4 matrix. I then transpose these matrices when I send them to OpenGL (I use OpenGL).

Since I'm using a row-major convention for my matrix math library, transforming a vector must be implemented as xA. If I use a column-major convention I implement vector transform as Ax.

It doesn't matter which you use but you cannot mix them (that would be undefined behavior) and the end result must be compatible (with regards to their respective shader language) with the 3D API we're using, be it DirectX or OpenGL or something else.