Picking problem with rotated view matrix

Question

I have picking code that works, if my view matrix has no rotation (only translation). Now If I rotate my camera, picking goes to hell.

I cant find whats wrong

My code:

/*-----------------------------------------------------------
Function:   GetViewportSystem
Returns:
    viewportCoordSystem

Get viewport coordinate system (only for reading)
Forward ray goes through origin
d = distance |camera, projection plane|
cameraPos = position of camera in World Space
lookAt = camera target in World Space
-------------------------------------------------------------*/
ViewportCoordSystem Camera::GetViewportSystem() const
{
    ViewportCoordSystem viewportCoord;
    viewportCoord.w = this->cameraPos;
    viewportCoord.w = this->cameraPos - this->lookAt;
    viewportCoord.w.Normalize();



    viewportCoord.u = MyMath::Vector3::Cross(viewportCoord.w, MyMath::Vector3::UnitY());

    viewportCoord.v = MyMath::Vector3::Cross(viewportCoord.u, viewportCoord.w);

    float d = (this->viewport.Height * 0.5f) * (1.0f / tanf(this->viewport.fov * 0.5f));

    viewportCoord.origin = this->cameraPos;
    viewportCoord.origin -= d * viewportCoord.w;

    return viewportCoord;
}

/*-----------------------------------------------------------
Function:   MapViewport2Dto3D
Parametrs:
    [in] viewportSystem - cameras viewport coordinate system
    [in] point - 2D point on image
Returns:
    3D mapped point in space

Map 2D image point to 3D space
Info about mappind 2D to 3D: http://meatfighter.com/juggler/
-------------------------------------------------------------*/
MyMath::Vector3 Camera::MapViewport2Dto3D(const ViewportCoordSystem & viewportSystem, const MyMath::Vector2 & point) const
{
    MyMath::Vector3 res = viewportSystem.origin;
    res += (point.X - this->viewport.Width * 0.5f) * viewportSystem.u;
    res += (this->viewport.Height * 0.5f - point.Y) * viewportSystem.v;

    return res;
}

And calculating ray (x, y are mouse coordinates within screen, my world is scaled, so I amd trasnforming ray and origin to transformed world)

    ViewportCoordSystem vpSystem = this->camera->GetViewportSystem();
    MyMath::Vector3 pos = this->camera->MapViewport2Dto3D(vpSystem, MyMath::Vector2(x, y));

    float inverseScale = 1.0f / scale;
    MyMath::Matrix4x4 worldInv = MyMath::Matrix4x4::Scaling(inverseScale, inverseScale, inverseScale);


    this->ray.dir = pos - this->camera->GetPosition();    
    this->ray.dir = MyMath::Vector3::TransformCoordinate(this->ray.dir, worldInv); //transform ray to object space
    this->ray.dir.Normalize();

    this->ray.origin = this->camera->GetPosition(); 
    this->ray.origin = MyMath::Vector3::TransformCoordinate(this->ray.origin, worldInv); //transform ray to object space

Answer 1

viewportCoord.u and viewportCoord.v will not come out unit length when the view is rotated. You might need (or it may help) to normalize .u