I'm trying to create a 3D cube from rectangle drawn by ther user's mouse (think click-and-drag selection). My end goal is to use this to implement entity click-and-drag box selection. If you've played a traditional strategy game, basically that but everything is in 3D.

The interesting bit is that each entity in the scene has a potential 3D transform (translation/rotation/scale).

My approach is to create a cube in world space, based on where the user clicks (in screenspace).

  1. Set min at the left-top of the mouserect (with a Z value matching the Camera's Z value).
  2. Set the cube's maximum value to the right bottom (with a Z value of FAR, as defined by the projection matrix).
  3. Use standard AABB vs OBB collision algorithm to determine if they overlap.

I'm having trouble with implementing this correctly. I have a working AABB vs OBB collision algorithm. I'm struggling with converting the user screen coordinates to a 3D cube.

Question 1: How would I go about this? Is there a better approach for this? (I'd like to ignore methods that involve comparing colors in the framebuffer).

Question 2: I have some code I've written that shows my understanding of the problem, maybe it's obvious what I'm doing wrong?

Question 3: Do you know of any example code doing this?

// NEAR, FAR values the proj matrix was constructed with
#define NEAR -500
#define FAR  +500

// Here invert the viewport Y component (glUnproject expects (x,y) to be in the lower-left of the
// screen)
auto vp        = viewport.size_vec4();
vp.y = vp.w - vp.y;

// The next bit is what I'm having trouble with.
// GOAL1: Convert the LEFT_TOP of the mouse-drawn rect from screen-space to world-space.
auto const lt  = mouse_rect.left_top();
auto const lt3 = glm::vec3{lt, 0};
glm::vec3 const mmin = glm::unProject(lt3, view, proj, vp);

// GOAL2: Convert the RIGHT_BOTTOM of the mouse-drawn rect from screen-space to world-space, but
// with a Z value equal to FAR.
auto const rb             = mouse_rect.right_bottom();
auto const rb3            = glm::vec3{rb, FAR};
glm::vec3 const mmax      = glm::unProject(rb3, view, proj, vp);

// Combine min/max to create a cube with
// min => User drawn rect left-top with Z at NEAR.
// max => right-bottom with Z at FAR.
Cube const mouse_cube{mmin, mmax};

// Determine whether the mouse is overlapping the cube.
auto const cube_mouserect_overlap = [&](auto const& cube_entity) {
  auto const& cube        = cube_entity.cube();
  auto tr                 = cube_entity.transform();
  auto const obb_onscreen = OBB::from_cube_transform(cube, tr);

  Transform mouse_tr;
  auto const obb_mouse = OBB::from_cube_transform(mouse_cube, mouse_tr);
  return collision::overlap(obb_onscreen, obb_mouse);

for (auto &ce : cube_ents) {
  ce.selected = cube_mouserect_overlap(ce);



Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.