As mentioned in the comments, rotation in 3 dimensions has 3 degrees of freedom (you can think of them like yaw, pitch, and roll), but a unit vector in 3 dimensions has only 2 degrees of freedom (latitude and longitude). So we need to store more than just a single unit vector to describe our orientation.
The smallest change from your current code would be to store two perpendicular unit vectors: one representing your forward/facing direction, and one representing your "up" direction. That way the second vector can track any "twist" around the first vector's axis.
When you rotate your forward vector using one of the three methods you've shown, also rotate your up vector by the same method.
To keep rounding errors from building up and making your vectors no longer perpendicular, you'll want to periodically orthonormalize this two-vector basis, something like this:
forward = normalize(forward);
up = normalize(up - dot(up, forward) * forward);
You can expand this to a full rotation matrix just by computing your third basis vector as the cross product of the other two:
// In a left-handed coordinate system. Flip the order for a right-handed system.
right = cross(up, forward);
Then [right | up | forward] is an orthonormal rotation matrix ready to use in transforming your points.
A more conventional solution to this problem is to store your object's orientation as a quaternion. That's four floats instead of six for two basis vectors, and quaternion rotations compose quite nicely:
Quaternion Compose(Quaternion after, Quaternion before) {
Quaternion q;
q.x = after.w * before.x + after.x * before.w + after.y * before.z - after.z * before.y;
q.y = after.w * before.y - after.x * before.z + after.y * before.w + after.z * before.x;
q.z = after.w * before.z + after.x * before.y - after.y * before.x + after.z * before.w;
q.w = after.w * before.w - after.x * before.x - after.y * before.y - after.z * before.z;
return q;
}
You can construct an axis-aligned rotation like so:
Quaternion ZRotation(float angle) {
Quaternion q;
q.x = 0;
q.y = 0;
q.z = sin(angle/2f);
q.w = cos(angle/2f);
return q;
}
Or one around an arbitrary unit vector like so:
Quaternion AngleAxis(float angle, t_vec unitAxis) {
Quaternion q;
float s = sin(angle/2f);
q.x = unitAxis.x * s;
q.y = unitAxis.y * s;
q.z = unitAxis.z * s;
q.w = cos(angle/2f);
return q;
}
Then you can compose your rotations to get your new orientation:
orientation = Compose(rotationChange, orientation);
You can transform a vector by the quaternion like so:
t_vec Rotate(Quaternion q, t_vec v) {
float x = q.x * 2f;
float y = q.y * 2f;
float z = q.z * 2f;
float xx = q.x * x;
float yy = q.y * y;
float zz = q.z * z;
float xy = q.x * y;
float xz = q.x * z;
float yz = q.y * z;
float wx = q.w * x;
float wy = q.w * y;
float wz = q.w * z;
t_vec rotated;
rotated.x = (1f - (yy + zz)) * v.x + (xy - wz) * v.y + (xz + wy) * v.z;
rotated.y = (xy + wz) * v.x + (1f - (xx + zz)) * v.y + (yz - wx) * v.z;
rotated.z = (xz - wy) * v.x + (yz + wx) * v.y + (1f - (xx + yy)) * v.z;
return rotated;
}
Though you can see here, most of these intermediate calculations don't depend on the vector being rotated. So if you're going to rotate a whole bunch of vectors, it can be more efficient to turn the quaternion into a transformation matrix first, then use the matrix to rotate all the vectors.