I have picking code that works, if my view matrix has no rotation (only translation). Now If I rotate my camera, picking goes to hell.

I cant find whats wrong

My code:

Function:   GetViewportSystem

Get viewport coordinate system (only for reading)
Forward ray goes through origin
d = distance |camera, projection plane|
cameraPos = position of camera in World Space
lookAt = camera target in World Space
ViewportCoordSystem Camera::GetViewportSystem() const
    ViewportCoordSystem viewportCoord;
    viewportCoord.w = this->cameraPos;
    viewportCoord.w = this->cameraPos - this->lookAt;

    viewportCoord.u = MyMath::Vector3::Cross(viewportCoord.w, MyMath::Vector3::UnitY());

    viewportCoord.v = MyMath::Vector3::Cross(viewportCoord.u, viewportCoord.w);

    float d = (this->viewport.Height * 0.5f) * (1.0f / tanf(this->viewport.fov * 0.5f));

    viewportCoord.origin = this->cameraPos;
    viewportCoord.origin -= d * viewportCoord.w;

    return viewportCoord;

Function:   MapViewport2Dto3D
    [in] viewportSystem - cameras viewport coordinate system
    [in] point - 2D point on image
    3D mapped point in space

Map 2D image point to 3D space
Info about mappind 2D to 3D: http://meatfighter.com/juggler/
MyMath::Vector3 Camera::MapViewport2Dto3D(const ViewportCoordSystem & viewportSystem, const MyMath::Vector2 & point) const
    MyMath::Vector3 res = viewportSystem.origin;
    res += (point.X - this->viewport.Width * 0.5f) * viewportSystem.u;
    res += (this->viewport.Height * 0.5f - point.Y) * viewportSystem.v;

    return res;

And calculating ray (x, y are mouse coordinates within screen, my world is scaled, so I amd trasnforming ray and origin to transformed world)

    ViewportCoordSystem vpSystem = this->camera->GetViewportSystem();
    MyMath::Vector3 pos = this->camera->MapViewport2Dto3D(vpSystem, MyMath::Vector2(x, y));

    float inverseScale = 1.0f / scale;
    MyMath::Matrix4x4 worldInv = MyMath::Matrix4x4::Scaling(inverseScale, inverseScale, inverseScale);

    this->ray.dir = pos - this->camera->GetPosition();    
    this->ray.dir = MyMath::Vector3::TransformCoordinate(this->ray.dir, worldInv); //transform ray to object space

    this->ray.origin = this->camera->GetPosition(); 
    this->ray.origin = MyMath::Vector3::TransformCoordinate(this->ray.origin, worldInv); //transform ray to object space
  • \$\begingroup\$ You are likely applying the rotation backwards. Happened to me the first time I wrote picking code. \$\endgroup\$ Aug 16, 2013 at 13:12
  • \$\begingroup\$ @PieterGeerkens What rotation ? Rotation part of view matrix is now used exactly. I use only lookAt vector, and that is correct. \$\endgroup\$ Aug 16, 2013 at 13:16
  • \$\begingroup\$ Not sure if it leads to your particular issue but viewportCoord.u and viewportCoord.v will not come out unit length when the view is rotated. You might need (or it may help) to normalize .u \$\endgroup\$
    – Steve H
    Aug 16, 2013 at 13:49
  • \$\begingroup\$ @SteveH Thanks.. that was it... post your comment as Answer, and I will accept it \$\endgroup\$ Aug 16, 2013 at 13:52

1 Answer 1


viewportCoord.u and viewportCoord.v will not come out unit length when the view is rotated. You might need (or it may help) to normalize .u


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .