For every matrix rotation, there is an axis and an angle. You are trying to marry your rotations to your 3d world's basis axes and applying Euler angles about them. That can invite issues when one of your cube's local axis is not aligned with the world axis and you want to rotate your cube around it or a 2nd world axis.
So the root of your problem is that you are storing your cube's rotation as Euler angles to access and modify the rotation next frame. There are various ways to store a rotation; Euler angles, matrix, quaternion, axis/angle... etc. For what you are doing, Euler angles does not lend itself as the best way. If your were to choose a Matrix to store your cube's orientation/position, you could simply store the rotation in the
worldMatrix and do not use
In your process, you are taking two identity matrices and applying an x rotation to one and a y rotation to the other then trying to combine them.
But what if you didn't start with two identify matrices? Why not simply apply the current frame's mouse x movement to last frame's
worldMatrix directly and right after that take the current frame's mouse y movement and apply that to the "just modified"
worldMatrix directly. All rotations will be as expected. The only thing is that you now have no idea what the overall
rotation.X, rotation.Y is because you aren't modifying
rotation.X & Y directly, your modifying
worldMatrix directly. Hopefully the rest of your code can be OK with that. Trying to hang onto those
rotation values is leading you down an unnecessarily complex path.
All you need to determine to do this is what axis to rotate the cube/
worldMatrix around for each rotation and how much.
Since your rotations are tied to your mouse movements and the 3d world involvement of the mouse has to do with camera orientation and position, the rotational axes that you want your cube/
worldMatrix to rotate about are actually stored in the view matrix.
//extract mouse movement for this frame
float mouseX = input.GetMouseXMovementSinceLastFrame();
float moustY = input.GetMouseYMovementSinceLastFrame();
//scale mouseX & Y if necessary for your application
//get rotation axis
xAxis.X = camWorld._11;
xAxis.Y = camWorld._12;
xAxis.Z = camWorld._13;
yAxis.X = camWorld._21;
yAxis.Y = camWorld._22;
yAxis.Z = camWorld._23;
//set worldMatrix to origin for rotation purposes but save position info
cubePosition.X = worldMatrix._41;
cubePosition.Y = worldMatrix._42;
cubePosition.Z = worldMatrix._43;
worldMatrix._41 = 0;
worldMatrix._42 = 0;
worldMatrix._43 = 0;
d3dxMatrixRotationAxis(&worldMatrix, yAxis, mouseX);
d3dxMatrixRotationAxis(&worldMatrix, xAxis, mouseY);
worldMatrix has had rotations applied to it that are relative to the camera's vertical and horizontal axes and no matter what position the camera is at or what rotational orientation it has while looking at your cube, the cube will always rotate as expected. You just lost the overall rotation of your cube is relative to Euler angles. But that does not matter if the only reason you kept Euler angles anyway was to get
worldMatrix orientated correctly, because this does that in a simpler fashion for this particular application.
If your camera's position and orientation were locked and never moved, you could calculate the rotation axes once and reuse them each frame simplifying this even more.