Clamp quaternion relative to plane normal

Question

How would one be able to clamp a quaternion based on the angle relative to a plane surface (defined by a normal)?

My diagram below provides more of a visual explanation of exactly what I am talking about.

The quaternions are clamped based on the angle between the planes surface, and the quaternion's forward vector (i.e. q * (0, 0, 1)). The quaternions are global quaternions. The plane normal (vector) N defines the upwards direction of the plane surface. Angles that go opposite of the normal vector is negative, whereas in the direction is positive. With that in mind, the quaternion's angle cannot be greater than MAX degrees and less than MIN degrees. These bounds are visualized by the cones (MIN is -90˚ in the diagram, so the cone is infinitely thin).

Thoughts:

My thought is that this algorithm would have to work in regards to the quaternions delta (unless the quaternion is in the MIN/MAX zones, which I will talk about later). If the change (delta) of the quaternion results in the quaternion entering, or passing, the bounds, the change is capped (clamped) so the final quaternion never has a resulting angle that dissatisfy the bounds.

Clamping the delta could be achieved by determining the beginning and end angles, and obtaining a ratio to be used as the t value (the interpolator) of a Slerp function, so the final quaternion's angle is that of the bound. The issue I see with this is how the roll would be manipulated due to the Slerp. Would the intended, final roll of the quaternion be lost due to this Slerping?

This idea of change seems more important in the example of if the quaternion was to go from -89˚ to -91˚. The final quaternion should be snapped back to -90˚ (if the bound was -90), instead of considering the new quaternion as -89˚ but from the other side. Without this proper interpretation of the quaternion's angle, the quaternion could rotate around constantly and consistently due to the angle never being considered less than -90˚ and greater than 90˚.

Edge Case:

If the quaternion was to start inside the MIN/MAX zones (the cones), the quaternion would not have a change to define as the direction that the quaternion needs to move to get back to legal territory/angles (see angle a3 in diagram). My thoughts is that a fallback plane normal that is orthogonal to plane normal N would have to be used, so the quaternion has a plane to rotate around.

Use Case:

Orbiting camera with user or author definable pitch angle clamping, and a varying change in "up". Example of varying change of "up": Mario Kart 8 Deluxe.

Answer 1

One idea is that you transform to a coordinate system where up is actually Vector3.up then do whatever clamping you need via standard Euler angles, and then transform it back.

So I think something like this (this is the version in Unity C#, but take it as pseudocode) would work:

    public Quaternion GeneralEulerClamp(Vector3 targetNormal, Quaternion inputrotation) {
        Vector3 xyz = Vector3.Cross(Vector3.up, targetNormal).normalized * Mathf.Sqrt((1 - Vector3.Dot(Vector3.up, targetNormal)) / 2);
        Quaternion coordchange = new Quaternion(xyz.x, xyz.y, xyz.z, Mathf.Sqrt(1 - xyz.sqrMagnitude));
        Vector3 EA = (coordchange * inputrotation).eulerAngles;//Note that quaternion multiplcation is not commutative!
        EA = new Vector3(Mathf.Clamp(EA.x,-90,70), EA.y, EA.z);//set EA.z to zero if you don't want roll
        Quaternion result = new Quaternion();
        result.eulerAngles = EA;
        result = Quaternion.Inverse(coordchange) * result;//It's possible that the first application of coordchange is the inverse and this one isn't
        return result;
    }

targetNormal here needs to be normalized.

So here, coordchange is a rotation rotates targetNormal to Vector3.up. We apply that rotation to the input (by multiplying quaternions), then find the Euler angles. Then we do some clamp operations on the Euler angles and turn in back into a quaternion. Then we do the inverse of the coordchange operation to move Vector3.up back to targetNormal.

Not 100% sure if I worked out my trig correctly since I don't have a way to test this (most likely error is that I got the sign of xyz wrong), but the general concept can work like this.