You've already basically got the answer, your problem is the positioning of all objects is based off of the player, and when the player moves so do the objects in the world.
Instead, you need to change everything to be based off of a camera position. And when rotating the world, figure out how much the character was moved, and move the camera to compensate; this way the character doesn't look like it's moving, and the world is rotating around it.
I've whipped up a sample for you, and here it is: sketchpad.cc. Just hit play. :) In one editor I was experiencing issues in Chrome (Firefox works fine), though in this editor it seems to work okay; was having issues properly tracking the mouse with Processing.js.
Just like your function, I have a GetProjPos method, or Get Camera Space Position.
class GameCamera
{
float Rot;// Camera rotation
PVector Pos;// Camera position
void GetProjPos(PVector objectPosition)
{
return new PVector(
objectPosition.x * cos(Rot) - objectPosition.y * sin(Rot) + Pos.x,
objectPosition.x * sin(Rot) + objectPosition.y * cos(Rot) + Pos.y);
}
All objects are drawn using this function. To handle rotation around the camera we need the player position before and after the rotation happens.
oldMouse = new PVector(pmouseX, pmouseY, 0);
newMouse = new PVector(mouseX, mouseY, 0);
oldMouseFromCenter = PVector.sub(oldMouse, cam.GetProjPos(P));
newMouseFromCenter = PVector.sub(newMouse, cam.GetProjPos(P));
if (newMouseFromCenter != oldMouseFromCenter)
{
Cross product helps us determine which side a vector is to another vector. This is used for determining which way to rotate the world, since angleBetween (which uses acos) returns a positive angle.
sign = oldMouseFromCenter.cross(newMouseFromCenter).z >= 0 ? 1 : -1;
oldPlayerPosition = cam.GetProjPos(P);
angleBetween uses the dot product to determine the amount of radians needed to rotate a vector to match another vector. The typical formula is like this: angle = acos(v1.dot(v2)) / (v1.length() * v2.length())
cam.Rot += sign * PVector.angleBetween(oldMouseFromCenter, newMouseFromCenter);
newPlayerPosition = cam.GetProjPos(P);
deltaPlayerPosition = PVector.sub(newPlayerPosition, oldPlayerPosition);
And the amount the player moved in world position, just move the camera back by that amount so it doesn't look like the player moved.
cam.Pos.sub(deltaPlayerPosition);
}
I'm sorry you didn't get this answer sooner, but I hope you find it useful. I know my example isn't in C++ and SDL, but you should be able to apply this easily enough.