This is a bit easier if you have a quaternion "look rotation" method handy, but I don't see one in the library you're using, so I'll describe the algorithm in high level pseudocode and you can translate it as needed. That can include implementing your own lookRotation(forward, up)
, which returns a quaternion that rotates (0, 0, 1) to point along forward
and (0, 1, 0) to point as close as possible to up
using the remaining degree of freedom.
So, we have object 1 with a triangular face ABC and object 2 with a triangular face PQR, each defined in the objects' own local coordinates. We want to move object 2 so that P meets A, Q meets C, and R meets B.
First we'll compute the normal to each face, the centroid of each face, and a vector from the centroid to the vertex we want to match up on each face:
n1 = cross(B - A, C - A)
n2 = cross(Q - P, R - P)
c1 = (A + B + C)/3
c2 = (P + Q + R)/3
v1 = A - c1
v2 = P - c2
We'll use that to compute an orientation of each face within its object's local coordinate space:
q1 = lookRotation(-n1, v1)
q2 = lookRotation(n2, v2)
We negate n1
to flip the face, so we dock object 2 to the outside of the face, not the inside.
Now ABC's orientation in world space is:
worldSpaceOrientation = object1.rotation * q1
And we can undo the effects of PQR's local orientation by inverting it, to get...
object2.rotation = worldSpaceOrientation * q2.inverse
Now we've rotated the second object so PRQ is in the right orientation to match ABC, but it's still probably in the wrong position. All we have to do now is get the vector between the two centroids, and translate object 2 by that vector:
c1World = object1.matrix * c1
c2World = object2.matrix * c2
object2.position += c2World - c1World
Make sure you do this after rotating object 2, otherwise the rotation can change the translation you need, twisting them out of alignment again.