Your use-case is two-fold:
- Projecting your 2d mouse coordinates to 3d world coordinates, to create a ray.
- Performing collision detection on the new coordinates, compared to your game objects.
NDC to 3D world coordinates
To do this, you need your view and projection transform matrices:
mat4 view = lookat(camPos, camPos+camDir, vec3(0.0f,1.0f,0.0f);
// For 3D
mat4 projection = perspective(fovAngle, aspectRatio, znear, zfar);
mat4 projection = ortho(0, width, height, 0);
combine these matrices using affine transformations:
mat4 worldToScreen = projection * view;
then invert them:
mat4 screenToWorld = inverse(worldToScreen);
Now, we take our mouse coordinates, and transform into world coordinates:
vec3 worldMouse = screenToWorld * vec4(mouseX,mouseY,-1.0f,1.0f);
Then finally we create our ray:
vec3 ray = normalize(worldMouse - camPos);
Now, we can iterate through a vector of game objects, to determine which one has been clicked on:
gameObject *selectedObject; = nullptr;
for (auto &gameobject: gameObjects)
selectedObject = gameobject;
Then you can have some rendering code that draws a halo on the ground at the targeted object position:
if (selectedObject != nullptr)
// then draw your game objects as normal.
This technique is known as "3D picking". There are plenty of more detailed tutorials available, but I covered the basic gist of it.