Of course there is no definite answer on this one. Only different possibilities, endless I would say. Still the question isn't easy to answer. Here are a few ideas that come to my mind, if they play out according to your wishes will only be decidable by you - probably in game.
First possibility: Loosen up on your "mitigation" model. Right now you strictly separate the incoming damage to either go to the armor or the hp value. This gives you a pretty clean mathematical model.
But I can find two reasons why a player may expect another behavior from his armor:
It takes less damage when in lesser shape. If my armor is only at half its original strength it somehow only takes less damage from incoming blow, while the health of the hull takes more and more. As a player at this point I may feel betrayed by my protection or feel that this is an inefficient design.
With the damage to the shields declining, so will the mitigation percentage difference. At first the mitigation potential of the armor will drop by 3 for the first three hits, then by 2 for nearly 10 hits and then by 1 for the last 15.
(These numbers and observations are based on the following calculation of your model: Original Mitigation Model)
Still from a game design perspective, I would value the first point a lot more. The second though fits your desire to change the curve.
One radical way to achieve that is to fight the first irritation: Do not reduce the damage to the shields, always subtract the full incoming damage from your armor. That would lead to linear mitigation decline and nearly linear damage increase. The ships then would loose their complete armor quite some time before destruction and would since then receive the full incoming "pure" damage.
To rebalance your ships you might then have to create them with higher initial armor which will make the difference between initial damage protection and final damage protection (0) even greater. (For the calculation of the hits until armor destruction view this image: Linear Mitigation Decrease)
Second possibility: Keep the mitigation concept like you have it (alas: keep dividing the damage between reducing hull or armor), but choose a different formula to calculate mitigation from armor. Right now, the mitigation value is proportional to the armor value, determined by the proportion factor 60/800.
Instead there are quite a few possibilities to link these to values.
You want a model that starts high and falls steeply. A square function, like you proposed would do that for you.
To test that out I choose one extra point of the curve, say at armor value 600 you only want 30 percent mitigation left, and you got the two points to determine the factors in the quadratic function
Mitigation(Armor) = a * Armor² + b * Armor
For the example given these are the resulting coefficients:
a = 1/8000
b = -1/40
This will give you a model which at the first hit declines the mitigation by 6 percent and then by 4 and then by 3 and then by 2, quickly dropping from 60 percent to 30 in 10 hits. (Details)
This is only one example of an quadratic connection of the two values, you could choose different, even steeper points to determine the coefficients, since I did the calculation by hand I tried to keep it manageable.
If you would like to look at another example altogether for different mathematical solutions I would suggest exponential regress.
I will however not cover this here, hoping with the idea given you could apply the transfer to exponential curves. Be advised that these models would require more mathematical handiwork every time you would want to change the balancing of your game. (And of course the first time you would want to implement it. But for these topics there are a lot of resources out there, like the wikipedia math section or the stackexchange math forum.)
[Side Note: In my (humble) experience simple equations make your balancing job a lot easier and may allow the player a greater understanding of your games mechanics, thus allowing him (and you as tester) a more strategic experience.]
For my third (and final) suggestion I would return to try to find a different algorithm completely rather than a different mathematical solution. By that this suggestion is pretty similar to the first one in concept and in implementation.
Additionally it is really simple, we keep the Mitigation/Armor-Proportionality and the damage divide, but we artificially increase the weakness of the armor. After the damage was divided into hull/armor damage we take the armor damage and multiply it by two. So the first hit takes 20 of our hp and a high 60 of our armor.
By this we keep the original concept, with all mathematical effects: As the armor drops down, the damage the armor draws drops down too and the mitigation decline stagnates. But till then the armor drops down a lot lot faster and the mitigation drops even faster than with the second solution from 60 to 30 in 10 hits until finally calming down at hit 15. (Details)
Like the first concept this third one would call for an increase of your initial armor values (while not necessarily increasing initial mitigation). And of course the factor "2" is one to be tested, higher values increase the dropping rate of mitigation, and therefore steepen the curve.
You will have noticed, that I did not answer the question what you should do, but the question, what you could do. And I hope I did so in your interest. As game designer only you have all the information to decide between the presented options. All of them have different side effects which I tried to outline here and all will have to be paired with different rebalancings of your other values. But I consider all of them a possible solution to your problem and hope you find the same.