How to implement translation, scale, rotation gizmos for manipulating 3D object's transforms?

Question

I am in the process of developing a basic 3D editor. It uses OpenGL for rendering a 3D world. Right now my scene is just a few boxes of different sizes and I am at the stage where I want to be able to select each box and then move/scale/rotate it to achieve any transform I want.

How can I solve the problem of implementing both the rendering of these tool's gizmos(or handles, or how people usually call them), and also picking them on each axis to perform the change in the transform with my mouse? For clarity:

My research so far suggested the cleanest approach is to have an axis aligned bounding box per arrow in the gizmo and another one per square (the ones that move the object in a plane rather than a single axis) and then cast a ray from the mouse position and see what it collides with. But this is still a bit too abstract for me, I would appreciate further guidance in how this algorithm would go (pseudocode is more than enough)

Answer 1

At some point in my time at e-on I have maintained the gizmos of Vue product line.
I can tell you, it will take you multiple days, full time.
Unless you find some library or super clever way, the classic way is to get the coordinate of the mouse in the window when you click, if its a relative coordinate to the viewport, you can simply divide x and y by width and height, you get a vector (float 2d) in [0,1] range. subtract (0.5,0.5) to it to get into [-0.5, 0.5] range for both x and y.
Then, you make a ray from this coordinate by using the x and y simply as the ray x and y, and you set z to the focal distance. sometimes aspect ratio is a pain in the ass in this operation. A bit of fiddling and trial error will get you fixed.
Then, you need to check intersection with your gizmos elements, either you have a mesh that you generated, or modeled in blender or other DCC, or mesh parts that can articulate between each other... Just use those mesh part as a ray/triangle intersection query.
Or if you have it, ray/cylinder, ray/sphere according to your gizmo looks and parts.
You need to have intersection routines that are capable to apply a transformation matrix on the primitive they collide. Extremely important because your gizmo will translate with the object it serves to move, it will rotate, and it will scale with the inverse of the distance to the camera, so that it keeps a fixed projected size on screen.
Then you have the interaction part, the easiest is the take the delta of the point when the mouse was first "mouse down" event, and the current "Mouse move" position, in pure 2D, and use this delta as the current axis movement in world space, multiplied by some k that you decide empirically. According to your internal units versus pixel versus current scale of zoom etc..
The final step is simply to apply the gizmo's matrix to the manipulated object, so that it follows it.

I tell you it is quite a journey in hell to implement, and if you're doing it in your free time excpect more than one week. Multiple weeks if you're completely discovering the field. More than one month if your week-ends are busy with other activities :)

I suggest you download Embree 2.0 from intel to do the ray/triangle intersection query for you, so you don't have to worry about coding that. Or you could mercilessely copy/paste and adapt code from blender... I think they moved to Apache license ? Should be possible legally.

Answer 2

For manipulator-translator I use follow algorithm:

1) When mouse down, we need check if ray intersects arrow. For example we consider X arrow. We construct Ray in world space (based on camera frustum and mouse position). We build a plane in which the x axis lies: its normal is equal V cross X cross V, where V - vector from center to camera, X - represents x axis. Then we intersect the ray with the plane and therefore we find the intersection point in world coordinates. Then we project the x axis segment and the resulting point back onto the screen, find the distance between the projected segment and the projected point on the screen. if it is less than a few pixels, the mouse intresects the axis. Also we calculate world space delta vector between selection’s center and our intersection.

This procedure we do for 3 axis, so we find distances to all axis. Find the minimum distance. so we found with which axis the mouse is intersects.

2) When mouse move. we know by what axis the object moves (from 1). we find the world space intersection of the ray with the plane (as in 1). additionally we project the intersection point on the line the object moves along. final manipulator position = intersection + delta.