How to get the center position for AABB of rotated mesh right?

Question

I want to calculate the AABB for my rotated & translated mesh, but whatever I tried, I can't calculate the center position correctly if the mesh is rotated.

My mesh is defined by an array of vertices, located around the coordinate origin (0,0,0) and a TRS matrix (containing translation & rotation only). To calculate the AABB I multiply each vertex of my mesh with the TRS matrix and check for min/max values on all three axes. My AABB is simply defined as glm::vec3 center, extend; (extend = length from center to outer border of the AABB; extend * 2 = length of the AABB). My TRS matrix is defined as:

 glm::translate(glm::mat4(), meshPosition) * glm::eulerAngleY(glm::radians(angle));

For each vertex I calculate:

glm::vec4 vert4 = glm::vec4(vertex.x, vertex.y, vertex.z, 1.0) * trsMatrix;
vert4 = vert4 / vert4 .w;

And search for the minimum/maximum x, y and z components in all vert4. Once the min/max values of all these vertices is found, the AABB is calculated as following:

extend = (max - min) * 0.5f;
center = min + extend;

First strange thing I noticed, is: even though my TRS matrix contains the translation for my model, the resulting AABB is still centered around the coordinate origin rather than the translation contained in my transformation matrix. I couldn't fully understand the reason yet, but it's not a big deal because I have the mesh position available anyway: so I thought I could just set the center position to the sum of the mesh position and the AABB center position.

center = center + meshPosition;

Unfortunately wrong: this works only as long as the angle of my rotation is around 0 or 180 degrees: In case of 90 or 270 degrees the position of the AABB relative to the mesh position is wrong (inverted on one axis).

I figured out the reason must be because the center of my mesh and the center of the AABB are different; the AABB center is "rotated" around the mesh center, when the mesh is rotated. If the mesh is facing in x+ or x- position, the center is correct calculated, if the mesh is facing in z+ or z- position the center of the AABB is in front of the mesh, instead behind it (and because of that not fully covering the mesh). How can I fix this/where is my error?

Note: This question is loosely related to my last question: How to recalculate AABB after rotation?

Answer 1

According to this tutorial, the vector you are transforming should be the right-hand operand in the matrix multiplication expression, when using glm. Having it on the left will cause your vector to be multiplied against the matrix's translation instead of being summed with it. This will also cause your vector to be multiplied against the transpose of your transform's rotation component. You don't see an issue when rotating 180 degrees about a single axis because the transpose of this transformation is equivalent to the same axis-rotation at 0 degrees.