Does Unity do work for every pair of objects on compatible layers?
Will Unity do some amount of work on each pair of compatible colliders even if they aren't near each other?
No. Both the PhysX engine for 3D physics and the Box2D engine for 2D physics include spatial partitions and broadphase passes designed to avoid doing detailed collision checks for every pair of colliders.
Both of these are extremely widely-used pieces of technology, in and outside of Unity, and have each been through many years of development and refinement. It's probably safe to assume they're not making a rookie mistake like checking all pairs in a \$O(n^2)\$ fashion wherever it can be easily avoided. 😉
How much less efficient is it to, say, have everything on the default layer and quickly break from OnTriggerEnter? As opposed to separating colliders into as many layers as possible to indicate colliders which should never be able to interact?
This will vary depending on the kinds of physics load you're throwing at it. The only way to get an exact number (like "30% more expensive") is to create a test and profile it in a scenario approximating your use case.
You know you'll be paying at least the cost to invoke a managed C# script function from unmanaged code, perform the check, and (possibly mispredict and backtrack to) the early-out. So it's reasonable to expect this will be more expensive. Whether it's a meaningful contributor to your game's total frame workload is something only your testing can determine conclusively.
Are there negative consequences to having too many layers?
If you manage to measure a negative performance impact from "too many" layers, I would find it very surprising. The things that tend to choke games are when we're manipulating hundreds, thousands, or millions of objects. Given that there are only a maximum of 32 layers, even a substantial per-layer overhead just doesn't have that much room to stack up.
As above, measure to be certain, but I have never seen the addition of a layer cause a performance problem.
Is it best practice to separate my colliders into layers with as many specific purposes/interaction masks as applicable?
I would recommend separating out layers to the degree that it's convenient. You usually hit diminishing returns trying to apply any optimization exhaustively, so start with the ones that cause the greatest inconvenience. Is it annoying to have to check whether a projectile hit a target of the same faction to avoid friendly fire? Split that into layers so the collision doesn't happen in the first place. Is this one off script use case easy enough to solve with a tag check without futzing with the layers? Probably safe to leave it as is until your profiling tells you otherwise.