Game Development Stack Exchange is a question and answer site for professional and independent game developers. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I simply want to pick a few spheres in my scene using the mouse-ray. I have implemented(copied most of it but with little understanding) a ray-sphere collision code. Also I have implemented the code for converting the mouse window coords to OpenGL coords. And constructed the mouse function. I have a ray class. In other words I have made everything, but sth isn't working...

So, there is the ray-sphere collision code:

bool Sphere::CheckRayCollision(Ray mouseRay)
     double discriminant, b;

     b = -DotProduct(mouseRay.GetOrigin(), mouseRay.GetDirection());
     discriminant = b * b - DotProduct(mouseRay.GetOrigin(), mouseRay.GetOrigin()) + this->radius * this->radius;

     if(discriminant < 0)
         return false;
     discriminant = sqrt(discriminant);

     double x1 = b - discriminant;
     double x2 = b + discriminant;

     if(x2 < 0) 
          return false;
     if(x1 < 0)
           x1 = 0;
           return true;

     return true;

It is within the sphere class. The convertion between window and OGL coords:

Vector3d MouseClass::ConvertMouseToOGLCoordinate(int mouseX, int mouseY, int mouseZ)
    GLint viewport[4];
    GLdouble modelMatrix[16];   
    GLdouble projectionMatrix[16];  

    glGetIntegerv(GL_VIEWPORT, viewport);
    glGetDoublev(GL_MODELVIEW_MATRIX, modelMatrix);
    glGetDoublev(GL_PROJECTION_MATRIX, projectionMatrix);

    //float winZ;
    float winY = float(viewport[3] - mouseY);
    //glReadPixels(mouseX, (int)winY, 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT, &winZ);

    double x, y, z;
    gluUnProject((double)mouseX, winY, mouseZ, 
        modelMatrix, projectionMatrix, viewport,
        &x, &y, &z);

    return Vector3d(x, y, z);

And the function which checks whether the left mouse btn is clicked and if it is checks for the collision:

void HandleMouse()
        int cursorX = int(userMouse.GetCurrentPosition().GetX());
        int cursorY = int(userMouse.GetCurrentPosition().GetY());

        Vector3d nearPoint = MouseClass::ConvertMouseToOGLCoordinate(cursorX, cursorY, 0.0f);
        Vector3d farPoint = MouseClass::ConvertMouseToOGLCoordinate(cursorX, cursorY, 1.0f);

        Vector3d direction = farPoint - nearPoint;

        positionInWorld = farPoint;

        mouseRay = Ray(nearPoint, direction);
        drawCube = true;
        for(int i = 0; i < 3; i++)
            bool isCollided = sphere[i].CheckRayCollision(mouseRay);
                sphere[i].SetColor(0.0f, 1.0f, 0.0f);
                sphere[i].SetColor(1.0f, 0.0f, 0.0f);

The 'drawCube' bool var is just for me to show if the convertion func works well and the 'positionInWorld' var is for the cubes coords. The drawing is well, so I think the convertion function works, but I might've done sth wrong.

If anyone can help me sort out why isn't this code working I would be very gratefull! I have been struggling with this for a week and searched everywhere on the Internet and haven't found a solution yet...

Thanks in advance!


The picking isn't working here. Pressing the left mouse button doesn't select anything. No matter where I click...

share|improve this question
You forgot to tell what exactly isn't working. – o0'. Dec 26 '11 at 14:17
The picking. The mouse won't select anything, although the left button is pressed. – Tsvetan Dec 26 '11 at 15:44
up vote 3 down vote accepted

When I was debugging mouse picking in my game, I had it set up to draw a line from my camera to where ever the picking function ended. It lets you know if the ray is going the right direction, and if the ray is ending too soon, or too late.

In your case, I'd have it draw a line between nearPoint and farPoint. Make sure those are the rays you want.

I imagine you'll find that it's not working the way you want it to, and here's why.

Vector3d nearPoint = MouseClass::ConvertMouseToOGLCoordinate(cursorX, cursorY, 0.0f);
Vector3d farPoint = MouseClass::ConvertMouseToOGLCoordinate(cursorX, cursorY, 1.0f);

That's not the way you want to get the nearPoint. The near point has nothing to do with where the mouse cursor is, it's about where your camera is. I created an answer a while back about picking:

You may want to look at how I get the ray in world coordinates.

Good luck!

share|improve this answer
I am currently trying the code from the post. I hope it works. I have one question, though, what Vector4f is about? – Tsvetan Dec 26 '11 at 17:57
Ok, I found out that the ray is correct. But still no detection for collision. Maybe the collision code isn't right? Btw, I made the near vector point my camera (nearPoint = Vector3d(0.0f, 0.0f, 99.0f)) – Tsvetan Dec 26 '11 at 18:16
How do you know the ray is correct? You viewed the line for the ray? And the ray does pass through the sphere, but doesn't detect is? So your camera is fixed then? Otherwise setting it to a static point isn't going to work. – Byte56 Dec 26 '11 at 18:23
Yes, my camera is fixed. The ray does pass through the sphere and it doesn't detect it. – Tsvetan Dec 26 '11 at 18:39
Just curious, how can you see that the ray passed through the sphere if your camera is fixed? You would be looking at the ray from directly behind, it would just look like a point. Anyway, if it is passing through the sphere then you've probably narrowed your problem down to the sphere collision code. It looks fairly correct, the only thing I can't tell on is the DotProduct function. Have you double checked that? Put a break point at the top, calculate the variables by hand, then step through your code and ensure you get the same answer. – Byte56 Dec 26 '11 at 18:50

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.