Steering constantly turning in circles

Question

I've been working on a fish system in a compute shader but bugs have been drowning me. The latest bug had me strip almost all the functionality to track down:

Fish keep turning in circles. It starts slowly, the fish gains velocity in the proper direction, then it reverses. The fish slowly loses velocity until it does a 180 and then the forward wander then applies acceleration behind it repeating the process. However the reverse isn't exactly 180 degrees behind it. It goes from doing roughly 180 degree instant turns to tracing smooth circles over a period of roughly 10 seconds.

float desiredSpeed = 0.2f;
// Wander Behavior substitute
float3 acceleration = (fish.mat._21_22_23 * desiredSpeed  - fish.velocity) * velocityMultiplier;

// Acceleration
float accelMag = length(acceleration);
float3 steerDirection = fish.mat._21_22_23;
if (accelMag > 0)
    steerDirection = acceleration / accelMag;
if (accelMag > desiredAccel)
{
    acceleration = steerDirection * desiredAccel;
    accelMag = desiredAccel;
}

// Direction
if (accelMag > 0)
{
    // Removed rotation smoothing
    fish.mat._21_22_23= normalize(acceleration); // forward
    fish.mat._11_12_13 = normalize(cross(fish.mat._21_22_23, float3(0, 0, 1))); // right
    fish.mat._31_32_33 = normalize(cross(fish.mat._11_12_13, forward)); // up
}

// Velocity
fish.velocity += fish.mat._21_22_23 * accelMag * fishDeltaTime;

speed = length(fish.velocity);
if (speed > desiredSpeed)
{
    fish.velocity = fish.velocity / speed * desiredSpeed;
    speed = desiredSpeed;
}
fish.mat._41_42_43 += fish.velocity * fishDeltaTime;

Once the fish is going in circles I understand what is happening. The fish has a velocity off to one side and it calculates the desired velocity as (forward * speed - velocity). That translates to an acceleration added at an angle pointing inwards away from the velocity so the fish continues to turn. You can sort of see it in this picture:

The light gray line is the current velocity, dark blue the desired wander velocity (forward), light blue the wander acceleration that is causes the fish to turn to the left.

My questions are:

Why does the fish slow and start to turn if acceleration is only added in the forward direction?

How do I rewrite the code so that the this sort of situation doesn't happen? Ideally the current velocity would become more forward and the corrective acceleration would lower until it went back to a normal straight line.

Answer 1

So it turns out the turn starts because of what I assume is accumulated floating point error from increasing velocity in this line:

float3 acceleration = (fish.mat._21_22_23 * desiredSpeed  - fish.velocity) * velocityMultiplier;

A small difference in direction causes the fish to turn slightly which is enough to start a positive feedback loop that ends with the fish going in circles.

My current thought is to based the acceleration on the current velocity rather than the current direction:

float3 wanderDirection = forward;
if (any(fish.velocity))
    wanderDirection = normalize(fish.velocity);
float3 acceleration = (wanderDirection * desiredSpeed  - fish.velocity) * velocityMultiplier;

This makes sense once it is expanded to a wander behavior and I believe it solves most of my problems.