The best paper out there for performance comparison is from geometrictools according to the paper you need 12 multiplication and 12 addition to convert a quaternion to a matrix, but this is hardly a deciding factor.. you need to look at the bigger picture
Quaternions are great for interpolation because they are numerically more stable than matrices when dealing with interpolations because less operations are involved. This is especially important for skeletal animation. Other than than you can use a matrix for everything including rotations. But if you insist on using quaternions (which is fine btw) the scenario I can imagine you are talking about is that you can separately keep track of scaling vec3, translation vec3 and a quaternion in that you need to build a matrix from and sent it to the GPU, in this case you need to do this for every object every frame it moves. I hardly think this would be a bottleneck unless you profile and it proved to be the problem.
A deciding factor for using matrix vs quaternion for rotation is what your engine is supposed to do, matrices can be more than enough if no precise skeletal animation is involved. Quaternions have the advantage of less memory but unless you are dealing with limited memory (eg. consoles) this is not a problem.
I would hardly optimize unless it is proven to be a bottleneck or I am struggling to get every bit of performance. So the answer it depends on the scale of the game and where the bottleneck is. Read the paper I linked to get a detailed comparison. After all you can hide your transformation in a class and change the implementation if it proves to be troublesome.