I stumbled on this question because I had the same problem with my camera, and changing the multiplication order worked.
I ran into a similar problem when scaling a model. This code produced the wrong result:
// world is an existing Matrix var scale = world * Matrix.CreateScale(3.2f);
whereas this worked perfectly (scaled a model to 3.2x it's size):
var scale = Matrix.CreateScale(3.2f) * world;
I don't understand why the order matters though - in "normal" math (that is, 5*3 vs 3*5), swapping the factors yields the same result. I understand differences if there are more than 2 factors (like in this question), but I don't see the issue here?
I don't know much about how Matrices really work and so I don't know if that's an XNA quirk or if it would happen in OpenGL as well?