As I mention in the linked answer, for full 6 DoF controls like spaceflight or swimming, it often works well to think of a corrective twist that you gradually apply to reassert your desired horizontal plane.
Here's one way we can do this:
First, we'll track a normal for the plane we want the ship to orient to when the player is not deliberately rolling.
Vector3 _planeNormal;
void Start() {
_planeNormal = transform.up;
// ...
}
Next, we'll think of our yaw and pitch controls as rotating only our forward vector. We'll play with the up vector next so we can control the twist:
float rotLeftRight = Input.GetAxis("Mouse X") * mouseSensitivity * Time.deltaTime;
float rotUpDown = -Input.GetAxis("Mouse Y") * mouseSensitivity * Time.deltaTime;
Quaternion yawPitch = transform.rotation * Quaternion.Euler(rotUpDown, rotLeftRight, 0f);
Vector3 newForward = yawPitch * Vector3.forward;
Next we're going to look at how we're currently oriented toward our reference plane.
float alignment = Vector3.Dot(transform.up, _planeNormal);
Vector3 referenceUp = Mathf.Sign(alignment) * _planeNormal;
This gives us the perpendicular to our reference plane that's closest to our current vertical direction (to avoid a nasty flip/lurch if we fly in a loop-de-loop - we'd rather tolerate flying upside-down than doing a sudden snap upright)
If we snap straight to this up vector, we can still get a nasty lurch when we're flying nearly perpendicular to our plane - its projection on our view can swing from pointing up to sideways to down and back from just small wiggles in our heading. So instead, we'll blend toward it gradually, with a strength that falls off to zero when we're perpendicular to the reference plane:
Vector3 targetUp = Vector3.Lerp(transform.up, referenceUp, alignment * alignment);
(Note that you'll want to do this at a consistent framerate, or adjust the above for deltaTime if your framerate is variable)
Now we're ready to set our new rotation based on our yawed/pitched forward vector and our calculated up vector:
Quaternion newOrientation = Quaternion.LookRotation(newForward, targetUp);
Before you apply this, you can add in your roll:
float roll = Input.GetAxis("Roll");
if(Mathf.Approximately(roll, 0) == false) {
newOrientation = newOrientation * Quaternion.Euler(0f, 0f, roll * rollSpeed * Time.deltaTime);
// Save our new reference plane, according to the player's input.
_planeNormal = newOrientation * Vector3.up;
}
transform.rotation = newOrientation;
At this moment, the player has told you "I want my horizon to be rotated like this," so we save that information into our reference plane normal. Wherever possible, we'll try to level-out the ship's orientation to align with this newly-twisted plane, until the player manually twists it back.
(Of course, you can also add a "twist decay" over time, if you want the roll to unroll itself if the player doesn't touch the roll controls for a while).
All together, this lets the player's local yaw/pitch/roll win in the short term, so the player immediately sees consistent response to their controls, and it's always correct relative to their current orientation - even if it introduces an unwanted twist globally. Then as we can, we gradually hint the ship back toward the last horizontal plane the player told us was good for them by deliberately rolling toward it.
