Picking objects with mouse ray

Question

I simply want to pick a few spheres in my scene using the mouse-ray. I have implemented(copied most of it but with little understanding) a ray-sphere collision code. Also I have implemented the code for converting the mouse window coords to OpenGL coords. And constructed the mouse function. I have a ray class. In other words I have made everything, but sth isn't working...

So, there is the ray-sphere collision code:

bool Sphere::CheckRayCollision(Ray mouseRay)
{
     double discriminant, b;

     b = -DotProduct(mouseRay.GetOrigin(), mouseRay.GetDirection());
     discriminant = b * b - DotProduct(mouseRay.GetOrigin(), mouseRay.GetOrigin()) + this->radius * this->radius;

     if(discriminant < 0)
     {
         std::cout<<"disc"<<std::endl;
         return false;
     }
     discriminant = sqrt(discriminant);

     double x1 = b - discriminant;
     double x2 = b + discriminant;

     if(x2 < 0) 
     {
          std::cout<<"x2"<<std::endl;
          return false;
     }
     if(x1 < 0)
     {
           x1 = 0;
           return true;
     }

     return true;
}

It is within the sphere class. The convertion between window and OGL coords:

Vector3d MouseClass::ConvertMouseToOGLCoordinate(int mouseX, int mouseY, int mouseZ)
{
    GLint viewport[4];
    GLdouble modelMatrix[16];   
    GLdouble projectionMatrix[16];  

    glGetIntegerv(GL_VIEWPORT, viewport);
    glGetDoublev(GL_MODELVIEW_MATRIX, modelMatrix);
    glGetDoublev(GL_PROJECTION_MATRIX, projectionMatrix);

    //float winZ;
    float winY = float(viewport[3] - mouseY);
    //glReadPixels(mouseX, (int)winY, 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT, &winZ);

    double x, y, z;
    gluUnProject((double)mouseX, winY, mouseZ, 
        modelMatrix, projectionMatrix, viewport,
        &x, &y, &z);

    return Vector3d(x, y, z);
}

And the function which checks whether the left mouse btn is clicked and if it is checks for the collision:

void HandleMouse()
{
    if(userMouse.IsLeftButtonDown())
    {
        int cursorX = int(userMouse.GetCurrentPosition().GetX());
        int cursorY = int(userMouse.GetCurrentPosition().GetY());

        Vector3d nearPoint = MouseClass::ConvertMouseToOGLCoordinate(cursorX, cursorY, 0.0f);
        Vector3d farPoint = MouseClass::ConvertMouseToOGLCoordinate(cursorX, cursorY, 1.0f);

        Vector3d direction = farPoint - nearPoint;
        direction.Normalize();

        positionInWorld = farPoint;

        mouseRay = Ray(nearPoint, direction);
        drawCube = true;
        for(int i = 0; i < 3; i++)
        {
            bool isCollided = sphere[i].CheckRayCollision(mouseRay);
            if(isCollided)
            {
                sphere[i].SetColor(0.0f, 1.0f, 0.0f);
            }
            else
            {
                sphere[i].SetColor(1.0f, 0.0f, 0.0f);
            }
        }
     }
}

The 'drawCube' bool var is just for me to show if the convertion func works well and the 'positionInWorld' var is for the cubes coords. The drawing is well, so I think the convertion function works, but I might've done sth wrong.

If anyone can help me sort out why isn't this code working I would be very gratefull! I have been struggling with this for a week and searched everywhere on the Internet and haven't found a solution yet...

Thanks in advance!

EDIT:

The picking isn't working here. Pressing the left mouse button doesn't select anything. No matter where I click...

Answer 1

When I was debugging mouse picking in my game, I had it set up to draw a line from my camera to where ever the picking function ended. It lets you know if the ray is going the right direction, and if the ray is ending too soon, or too late.

In your case, I'd have it draw a line between nearPoint and farPoint. Make sure those are the rays you want.

I imagine you'll find that it's not working the way you want it to, and here's why.

Vector3d nearPoint = MouseClass::ConvertMouseToOGLCoordinate(cursorX, cursorY, 0.0f);
Vector3d farPoint = MouseClass::ConvertMouseToOGLCoordinate(cursorX, cursorY, 1.0f);

That's not the way you want to get the nearPoint. The near point has nothing to do with where the mouse cursor is, it's about where your camera is. I created an answer a while back about picking: https://gamedev.stackexchange.com/a/12370/7191

You may want to look at how I get the ray in world coordinates.

Good luck!