Let's assume we have 4x4 3D transformation matrix, that is the result of scale, rotation, and translation transforms. How to set its scale to (1,1,1) in the fastest way? Assume also that the matrix is represented by float[16] array.
By fastest way I mean, the most performance accurate way, that will be low CPU cost.
Assuming your matrix multiplication follows the convention...
M * v = (T * R * S) * v
(where M is your composed matrix, T is a Translation matrix, R rotation, S scale, and v is a vector you want to transform using the matrix)
...then you can normalize the first three columns of the matrix to get just the T * R part.
If you use the opposite matrix multiplication convention (v * M) then you'd normalize the first three rows instead. Either way, you only want to modify the 3x3 block of entries in the top-left of the matrix, ignoring the last row & column (which contain translation information and the homogeneous unit)
If you want to eke out every last CPU cycle, you can play with SIMD instructions to do the three vector normalizations with one multiply & square root, but this is likely to only be noticeable if you're processing big batches of these matrices in a very friendly data layout.
There's a trick you can use to numerically remove (or rather reduce as much as you want) any scaling from the upper 3x3 sub-matrix, assuming it's not singular. Let's call that 3x3 sub-matrix M. You can take the transpose of the inverse of M, and average it with M. That will be the new M for the next loop.
while (...) {
N = (transpose(inv(M)) + M)/2
if M is almost like N then stop
M <-- N
}
You can set a max number of loops, and stop earlier if M is close enough to N
