# Picking problem with rotated view matrix

I have picking code that works, if my view matrix has no rotation (only translation). Now If I rotate my camera, picking goes to hell.

I cant find whats wrong

My code:

/*-----------------------------------------------------------
Function:   GetViewportSystem
Returns:
viewportCoordSystem

Get viewport coordinate system (only for reading)
Forward ray goes through origin
d = distance |camera, projection plane|
cameraPos = position of camera in World Space
lookAt = camera target in World Space
-------------------------------------------------------------*/
ViewportCoordSystem Camera::GetViewportSystem() const
{
ViewportCoordSystem viewportCoord;
viewportCoord.w = this->cameraPos;
viewportCoord.w = this->cameraPos - this->lookAt;
viewportCoord.w.Normalize();

viewportCoord.u = MyMath::Vector3::Cross(viewportCoord.w, MyMath::Vector3::UnitY());

viewportCoord.v = MyMath::Vector3::Cross(viewportCoord.u, viewportCoord.w);

float d = (this->viewport.Height * 0.5f) * (1.0f / tanf(this->viewport.fov * 0.5f));

viewportCoord.origin = this->cameraPos;
viewportCoord.origin -= d * viewportCoord.w;

return viewportCoord;
}

/*-----------------------------------------------------------
Function:   MapViewport2Dto3D
Parametrs:
[in] viewportSystem - cameras viewport coordinate system
[in] point - 2D point on image
Returns:
3D mapped point in space

Map 2D image point to 3D space
Info about mappind 2D to 3D: http://meatfighter.com/juggler/
-------------------------------------------------------------*/
MyMath::Vector3 Camera::MapViewport2Dto3D(const ViewportCoordSystem & viewportSystem, const MyMath::Vector2 & point) const
{
MyMath::Vector3 res = viewportSystem.origin;
res += (point.X - this->viewport.Width * 0.5f) * viewportSystem.u;
res += (this->viewport.Height * 0.5f - point.Y) * viewportSystem.v;

return res;
}


And calculating ray (x, y are mouse coordinates within screen, my world is scaled, so I amd trasnforming ray and origin to transformed world)

    ViewportCoordSystem vpSystem = this->camera->GetViewportSystem();
MyMath::Vector3 pos = this->camera->MapViewport2Dto3D(vpSystem, MyMath::Vector2(x, y));

float inverseScale = 1.0f / scale;
MyMath::Matrix4x4 worldInv = MyMath::Matrix4x4::Scaling(inverseScale, inverseScale, inverseScale);

this->ray.dir = pos - this->camera->GetPosition();
this->ray.dir = MyMath::Vector3::TransformCoordinate(this->ray.dir, worldInv); //transform ray to object space
this->ray.dir.Normalize();

this->ray.origin = this->camera->GetPosition();
this->ray.origin = MyMath::Vector3::TransformCoordinate(this->ray.origin, worldInv); //transform ray to object space

• You are likely applying the rotation backwards. Happened to me the first time I wrote picking code. Aug 16, 2013 at 13:12
• @PieterGeerkens What rotation ? Rotation part of view matrix is now used exactly. I use only lookAt vector, and that is correct. Aug 16, 2013 at 13:16
• Not sure if it leads to your particular issue but viewportCoord.u and viewportCoord.v will not come out unit length when the view is rotated. You might need (or it may help) to normalize .u Aug 16, 2013 at 13:49
• @SteveH Thanks.. that was it... post your comment as Answer, and I will accept it Aug 16, 2013 at 13:52

viewportCoord.u and viewportCoord.v will not come out unit length when the view is rotated. You might need (or it may help) to normalize .u