I have a problem in terms of what frame of reference, I should construct, scaling (S), rotation (R) and translation (T) for a model matrix (in that order). Let us assume the use of glm (or glsl), i.e. to mean column major representation.
I believe, it should be a matrix multiplication of:
M = T * R * S // S * R * T is what we want to apply, but due to being column major, the order gets reversed, due to (AB)^T =B^T*A^T
Now, if I have a previous transformation already applied (say M_prev), and I want to apply a transformation M on top of that, then is the correct form:
M_new = M_prev * M // M_new and M_prev are both column major. I'm post multiplying due to being column major, but this is incorrect, I guess?
I believe this is incorrect (at least in some experiments I carried out, please correct me if I'm wrong), and it should be the reverse. What is the flaw in my calculation of M_new?
T*R*S
means T is being applied after R which is in turn applied after S, thenM_prev * M
means M_prev is being applied after M; or vice versa, assuming all of your matrices are constructed similarly. Are you expecting that somewhere in between the matrices are being transposed? You mention one set being column-major and another row-major, but it's not clear why your application would be mixing two different matrix conventions. \$\endgroup\$