Assuming your matrix multiplication follows the convention...
M * v = (T * R * S) * v
M is your composed matrix,
T is a Translation matrix,
S scale, and
v is a vector you want to transform using the matrix)
...then you can normalize the first three columns of the matrix to get just the
T * R part.
If you use the opposite matrix multiplication convention (
v * M) then you'd normalize the first three rows instead. Either way, you only want to modify the 3x3 block of entries in the top-left of the matrix, ignoring the last row & column (which contain translation information and the homogeneous unit)
If you want to eke out every last CPU cycle, you can play with SIMD instructions to do the three vector normalizations with one multiply & square root, but this is likely to only be noticeable if you're processing big batches of these matrices in a very friendly data layout.