I have a sketch that produces the following:

enter image description here

public void settings() {
  size(600, 600, P3D);

float rot_y;

public void draw() {

  float z = (float) ((height/2.0) / Math.tan(PI*30.0 / 180.0));
  translate(width/2, height/2, z);

  if (mouseX > width * 0.9) {
    rot_y += radians(0.5f);
  } else if (mouseX < width * 0.1) {
    rot_y -= radians(0.5f);


  for (int i = 0; i < 8; i++) {
    rotateY(radians(45) * i);
    translate(0, 0, z);
    rect(0, 0, 400, 400);
    // how can I know it the mouseX, mouseY hits the rect?
    // and if so, also where it hit's the rect?

Now I would like to know if the mouse is over the rect, and if so where the hit is on the rect is in x and y.

I have seen some topics about rays and quad detection and I found them hard to understand. Since I always have a quad that is basically a AABB but rotated in 3d and the mouse if always a ray coming straight out of the screen; I am hoping this would make a possible solution more easy.

Any help is welcome.

  • \$\begingroup\$ for clarification... do you mean a plane or a quad? a plane is a flat 2d surface that extends infinitely. \$\endgroup\$
    – Kyy13
    Commented Jan 8, 2019 at 18:01
  • \$\begingroup\$ I mean a quad then. \$\endgroup\$
    – clankill3r
    Commented Jan 8, 2019 at 20:33
  • \$\begingroup\$ Do you want a general solution, or can we make assumptions based on your code here like "the camera position will always be fixed" and "two rects will never overlap"? \$\endgroup\$ Commented Jan 8, 2019 at 21:25
  • \$\begingroup\$ A general solution, the camera position etc can change. \$\endgroup\$
    – clankill3r
    Commented Jan 8, 2019 at 23:20

2 Answers 2


Since the comments made clear you're looking for a general solution, I'll describe that process here. Note the code is to show the principle, you will need to translate it to the framework you're using. I also assume you are familiar with projection, view and world matrices- and their role in 3D graphics.

3D Mouse picking

The idea is to create a ray from the mouse (screen) towards the distance.

A Ray is basically a line with one startpoint and a direction:

Ray r = new Ray(startpoint, directionVector);

Now let's create a ray from the mouse pointer. To do this, we consider a 'near' and a 'far' plane. The line is from the screen towards the horizon- however we only need the direction, so it does not really matter if the far point is actually near the horizon. So:

Vector3 nearPlaneMousePointer = new Vector3(mouseX, mouseY, 0);
Vector3 farPlaneMousePointer = new Vector3(mouseX, mouseY, 1);

Now we have two coordinates in the 3D space- in relation of the screen- the viewport.

To convert the coordinates in the worldspace, we actually need to "unproject" (inverse of the projection) to get the correct coordinates. Most frameworks have a 'Viewport.Unproject()' method- using the projection and view matrices, to calculate a given point to it's original unprojected object space.

Vector3 nearPlanePoint = Viewport.Unproject(nearPlaneMousePointer, _projection, _view, Matrix.Identity);
Vector3 nearPlanePoint = Viewport.Unproject(farPlaneMousePointer, _projection, _view, Matrix.Identity);

With these two points you can now get the direction and construct the ray:

Vector3 rayDirection = farPoint - nearPoint;
Ray r = new Ray(nearPoint, rayDirection);

Now we have a ray that is in the object space.

We can still not use it to do the ray picking. Why? Because the object space does not take the world matrices into account. If you only have AABBs or the worldspace is static, this step may not be needed. I'll provide the theory anyway in the event you plan to move things around a bit more and use world matrix.

So translate the ray into world space. To do this, use the inversion of the world matrix.

Matrix inverted = _world.Invert();
Ray worldRay = new Ray(Vector3.Transform(r.Origin, inverted),
                       Vector3.Transform(r.Direction, inverted));

If the target quads are also transformed to world space, a ray-triangle intersection test will provide your answer. Loop through the quads and check is the quad is hit. Some frameworks have a hit test function for rays and quads or triangles.

The result of the hit test can also be checked for distance from the ray's origin: this way you can select the first quad that is hit- if the viewpoint has overlapping quads. You can optimise this part als by checking only quads that were in the viewport (use the same culling mechanism as you would in drawing the world).


This is not a full answer, but these are the general steps to go through to get what you want

Create a mouse-viewport ray, this is a line segment that starts from the camera eye position and shoots out into the scene through the mouse position.

Ray-plane intersection, use some maths to check if and where the ray intersects the infinite plane defined by a position and normal of your square.

Check if the point lies in the square, because the square is AABB aligned you can check the intersection point's x, y and z co-ordinates to see if it's in the square.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .