I'm trying to find the correct way to build a rotation matrix from a unit vector. I have two arbitrary points in space (p1 and p2), and I'm trying to create the vertices for a rectangle to span between them.
From the points, I can get a unit vector easily enough. From there I've been trying to create a rotation matrix that I can apply to a set of hard-coded vertices to get them oriented correctly. The rectangle extends along the X axis, so I'm trying to rotate it's X axis to match the unit vector.
So far I've tried GLM::orientation:
glm::vec3 direction = glm::normalize( glm::vec3( p2 - p1 ) );
glm::mat4 rotation4 = glm::transpose( glm::orientation( direction, glm::vec3( 0, 1, 0 ) ) );
glm::mat3 rotation = glm::mat3( rotate4 );
As well as the manual matrix creation method mentioned in this similar question:
glm::vec3 direction = glm::normalize( glm::vec3( p2 - p1 ) );
glm::vec3 rotationX = direction;
glm::vec3 rotationZ = glm::normalize( glm::cross( direction, glm::vec3(0,1,0) ) );
glm::vec3 rotationY = glm::normalize( glm::cross( rotationZ, direction ) );
glm::mat3 rotation( rotationX, rotationY, rotationZ );
However, in both cases I end up with rectangle rotations that seem fine for axis-aligned rotations, but which veer off in weird directions in all other cases. In both cases, I'm attempting to specify the positive Y axis as up, to keep the rectangle level, but the rotated rectangle always ends up twisted around despite that.
I'm obviously doing this wrong, but I may just not be understanding the problem well enough. If someone could point me in the right direction, I'd appreciate it.