How to determine what triangle in a mesh the cursor is pointing to?

Question

I have a game I'm working on in which items are "placed" and "selected".

In order to do this, I need to determine what triangle in a mesh the cursor is pointing to. I have no trouble with getting the cursor position or the orientation/position of the camera - the problem is figuring out how they relate.

I'm guessing that this involves ray-picking, but I have no idea on how to convert a cursor position to a ray that I can test for intersection with a triangle.

A little bit of background:
Game engine: Irrlicht
Language: C++

Answer 1

I found a solution, though not an in-depth explanation - seems like Irrlicht has a built in function for this:

irr::scene::ISceneCollisionManager::getRayFromScreenCoordinates

This will calculate a ray given a screen coordinate.

Answer 2

To convert a cursor position to a ray, you can back-project the 2D co-ordinates onto two planes parallel to the view plane. Typically you might use the near and far planes. Two points define a ray.

This is simply a matter of doing the maths used to project a 3D point onto the screen, in reverse.

Typically from world space you would multiply by a view and projection matrix to get into clip-space, divide by Z to do the actual projection, and then scale the resulting values into pixel-values.

So in reverse you would scale the screen-coordinates into clip space, multiply by Z (taken from the plane you're interested in), and transform by the inverse of the view-projection matrix to get back into world space.

However as you've noticed, most engines and libraries can do this for you...

Answer 3

Irrlicht Tutorials This link explains the triangle selection and ray casting, just need to convert screen to the world coords.

scene::ITriangleSelector* selector = 0;
scene::IAnimatedMeshSceneNode* node = 0;

    // Add an MD2 node, which uses vertex-based animation.
    node = smgr->addAnimatedMeshSceneNode(smgr->getMesh("../../media/faerie.md2"),
                                            0, IDFlag_IsPickable | IDFlag_IsHighlightable); 
    node->setPosition(core::vector3df(-70,-15,-120)); // Put its feet on the floor.
    node->setScale(core::vector3df(2, 2, 2)); // Make it appear realistically scaled
    node->setMD2Animation(scene::EMAT_POINT);
    node->setAnimationSpeed(20.f);
    video::SMaterial material;
    material.setTexture(0, driver->getTexture("../../media/faerie2.bmp"));
    material.Lighting = true;
    material.NormalizeNormals = true;
    node->getMaterial(0) = material;

    // Now create a triangle selector for it.  The selector will know that it
    // is associated with an animated node, and will update itself as necessary.
    selector = smgr->createTriangleSelector(node);
    node->setTriangleSelector(selector);
    selector->drop();
    scene::ISceneCollisionManager* collMan = smgr->getSceneCollisionManager();

In the main loop

while(device->run())
    if (device->isWindowActive())
    {
            driver->beginScene(true, true, 0);
            smgr->drawAll();
            scene::ISceneNode * selectedSceneNode =
                    collMan->getSceneNodeAndCollisionPointFromRay(
                                    ray,
                                    intersection, // This will be the position of the collision
                                    hitTriangle, // This will be the triangle hit in the collision
                                    IDFlag_IsPickable, // This ensures that only nodes that we have
                                                    // set up to be pickable are considered
                                    0); // Check the entire scene (this is actually the implicit default)

            // If the ray hit anything draw the triangle that was hit.
            if(selectedSceneNode)
            {
                    driver->setTransform(video::ETS_WORLD, core::matrix4());
                    driver->setMaterial(material);
                    driver->draw3DTriangle(hitTriangle, video::SColor(0,255,0,0));

            }

            // We're all done drawing, so end the scene.
            driver->endScene();
     }