# How can I remove the view and projection matrix from ray position to get only the ray model?

I've got a little problem with my picking module and I need your help.

I have a method to project a ray from the camera to a fixed point when I click somewhere. The computed ray looks good, it go to the asked direction and passes through the pointed box, so I'm thinking everything is OK here.

When I'm trying to pick an object a little further from the origin, there is no intersection found.

I'm sure that the ray is correctly projected and computed to the screen (I draw it) but coordinates are bad during intersection test.

Currently, I compute the ray with this code :

const glm::vec3 position    = glm::vec3( mouse.x, mouse.y, 0.f );
const glm::mat4& view       = camera.getViewMatrix();
const glm::mat4& projection = camera.getProjectionMatrix();
const glm::vec4& viewPort   = camera.getViewport();

ray.origin =  glm::unProject( glm::vec3(mouse.x, mouse.y, 0.f ),
modelView,
projection,
viewPort );

ray.direction =  glm::unProject( glm::vec3( mouse.x, mouse.y, 0.f ),
modelView,
projection,
viewPort );


So, I think the ray origin and direction are computed to world position (with camera matrix: projection and view).

When I test intersection, I use object model (without camera transformation), so I'm thinking error is here.

The question: How can I remove view and projection matrix from ray position to get the ray model only ?

Thanks a lot!

PS: To test intersection, I create a bounding box from my object's model matrix and I test intersection with the ray computed previously.

EDIT:

Code to compute ray positions in world space :

////////////////////////////////////////////////////////////
bool computeRayPositions( Ray* ray )
{
glm::vec3 start = ray->getOrigin();
glm::vec3 end   = ray->getDirection();

glm::mat4 startMatrix   = glm::translate( glm::mat4(1.f), start);
glm::mat4 endMatrix = glm::translate( glm::mat4(1.f), end);

const glm::mat4& viewMatrix       = camera->getViewMatrix();
const glm::mat4& projectionMatrix = camera->getProjectionMatrix();

glm::mat4 startWorldMatrix  = startMatrix * glm::inverse(viewMatrix * projectionMatrix) ;
glm::mat4 endWorldMatrix    = endMatrix * glm::inverse(viewMatrix * projectionMatrix);

ray->setOrigin( glm::vec3 (startMatrixFinal[3][0], startMatrixFinal[3][1], startMatrixFinal[3][2] ) );
ray->setDirection( glm::vec3 (endMatrixFinal[3][0], endMatrixFinal[3][1], endMatrixFinal[3][2] ) );

}


Your ray is possibly in view (camera) space. I'm unsure exactly how glm::unproject works. If I'm right, pass its end points through the inverse camera matrix to put then in world space.

Remember that a coordinate is expressed in some basis, which for our purposes can be considered a vector space (mathematically there is some difference) or commonly just a space in graphics terms. The vertices on a model are in model-space. You run those through the model-specific model-world matrix (sometimes called a model-view matrix, though that term is highly misleading IMO). Then you run the vertex through the world-view matrix (the camera matrix) to put the vertex in view space (camera space). Then finally you run through the view-projection matrix and divide by the W component to end up in NDC space. A final step performed automatically by the graphics API is to multiply another matrix moving from NDC space to screen space.

To move backwards in spaces, you simply use the inverse of the corresponding matrix, paying special attention to NDC space and the projection matrix (and that divide by W). We normally call matrices from-to to indicate which spaces/basis they transform from and to. The inverse of any such matrix should be named accordingly, e.g. the inverse of the world-view matrix is the view-world matrix.

glm::unproject is possibly doing the equivalent of multiplying by the screen-ndc matrix and then the projection-view matrix, with the W divide as necessary (often you can ignore it if you construct your points appropriately). This means the the output of glm::unproject would be in view space. You need to multiply by the view-world matrix (inverse camera matrix) to then take the point from view space to world space, which is generally where you'll be doing collision and intersection tests. (Although for individual model test, you'll often take the incoming ray and multiply it by the world-model matrix, e.g. the inverse model-world matrix, to make the math way easier and possibly more efficient.)

glm::unproject also takes the modelView matrix in your example which indicates it is maybe trying to unproject through the whole pipeline, so it's possible that it's just expecting that either the projection matrix or the modelView matrix passed in is already concatenated with the camera matrix and that its results are intended to be in model-space. You might try concatenating that matrix, or if you're looking for world-space coordinates, pass in the camera matrix as the model-view matrix.

• Thanks for your answer! I understand better the problem, but I can't resolve it. I have edited my post to show how I compute ray's points position. W is the glm::unproject "z" parameter, right ? (So 0.f and 1.f) – user30088 Jun 17 '13 at 8:22
• The GLM documentation are an exercise in frustration for me, I can tell you the math to do by hand yourself, not how to use that API. Edited answer to indicate my lack of surety with that API. – Sean Middleditch Jun 17 '13 at 17:10
• @SeanMiddleditch I understand your frustration with glm, but somehow they are trying to mimic what glu used to do so people feel familiar, I don't think it's the best thing to do, but it's fine for most people. – concept3d Jan 14 '14 at 7:53