Cylindrical billboarding around an arbitrary axis in geometry shader

Question

I found an answer on this site relating to this question already, but it doesn't seem applicable in the context of my project.

Basically I'd like to create a method which fits this signature:

float3x3 AxisBillboard(float3 source, float3 target, float3 axis)

That is to say, when given a source point (i.e an object's position in world space), a target (camera position in world space), and an axis (the object's up vector, which is not necessarily the global y axis), it produces a 3x3 rotation matrix by which I can multiply the vertices of my point so that it's properly rotated.

I've found many solutions and tutorials online which work nicely assuming I only want to rotate around the y axis.

For example, here's a solution which billboards around the global y axis:

float3 dir = normalize(target - source);
    
float angleY = atan2(dir.x, dir.z);
c = cos(angleY);
s = sin(angleY);
    
float3x3 rotYMatrix;
rotYMatrix[0].xyz = float3(c, 0, s);
rotYMatrix[1].xyz = float3(0, 1, 0);
rotYMatrix[2].xyz = float3(-s, 0, c);

For context, I'm working on a grass shader, and each individual blade of grass should be billboarded to face the camera while remaining aligned with the normal of the terrain.

edit:

Thanks to @DMGregory I'm using his matrix method in the following way:

float3 viewDir = normalize(vertexPosition); // camera is at (0, 0, 0)
float3x3 billboardMatrix = transpose(AxisBillboard(worldUp, viewDir));
float3 worldPosition = vertexPosition + mul(billboardMatrix, localPosition);

This works correctly when grass is rotated to align with world space. In other rotations, grass is still billboarding 'correctly' around its local y axis but also has another rotation applied. Any ideas?

Answer 1

This isn't as hard as it might seem.

We can simplify two of your arguments to one:

float3 viewDirection = target - source;

Then our strategy will be to cross the vertical axis and view direction to get a vector perpendicular to both (the right/sideways direction).

Next, we cross this with our vertical axis to complete the basis.

A rotation matrix is just a unit vector in each basis direction, so we can assign our vectors directly to the rows/columns of our matrix.

float3x3 AxisBillboard(float3 upAxis, float3 viewDirection) {
    float3 rightAxis = normalize(cross(upAxis, viewDirection));
    float3 forwardAxis = cross(rightAxis, upAxis);

    float3x3 result;
    result[0].xyz = rightAxis;
    result[1].xyz = upAxis;
    result[2].xyz = forwardAxis;

    return transpose(result);
}