I haven't actually done any profiling, but the off-hand guestimate based on the way the physics engine should be optimized and some knowledge on the approximate complexity of various operations, my suggestion is:
Because you have one player, it will be very cheap for each of your enemies to check the square magnitude during their update tick. Very very cheap. Although it does go against the idea that the enemies shouldn't always know where the player is. But for a quick-and-dirty check prior to more intensive checking, it will work.
However, this won't necessarily be good for objects that have to check against multiple actors. E.g. an automatic door that opens whenever a guard patrols by it, or the player gets near. Or in a game I'm working on, guards that attack any intruder. And the player isn't always the only intruder (nor are the other intruders necessarily friendly to the player!). So what I do is:
- first do a overlapSphere. overlapSphere is cheap. The physics engine is likely performing a squareMagnitude call on every collider in the scene (there are other optimizations, such as maintaining a quadtree of the scene, so it doesn't have to check every collider, just some, as well as having a good layer mask; a bitwise AND will be faster than squareMagnitude) and returns a list of the ones that passed.
- For every collider returned, do a frustum check. This will vary by enemy, some can only see in front of them, some can see in all directions, but not very far up and down (e.g. can't see above themselves), another can only see above and below, but not horizontally. This is pretty cheap.
- Finally, on the remaining colliders, perform a raycast. Raycasts are expensive (relative to any of the above methods), but super accurate. sphereCast (and capsuleCast) also falls into this category, but is more useful for knowing if an object of a given size will reach. Good for knowing if an AI can walk down a narrow passage or fit under a low hang. But a small sized sphere in a sphereCast might be "friendlier" to the player than a raycast, as it means they can peek out around corners a little more (the larger the sphere the more they can peek out).
You can get away with a lot of raycasts in a frame without bogging down the FPS, but you definitely don't want every enemy performing a raycast on every ally every frame. Its just not needed. As the game I'm working on is a stealth game, I actually perform 5 raycasts: one against the center of the player (as you're going to be doing) and then four more towards approximated corners of the player's bounding box (relative to the enemy's view direction), which does the opposite of the spherecast I mentioned earlier: it makes it tougher on the player to stay out of sight. I have other mechanics in place to avoid this being harshly punishing (not only does the player have to be visible, the AI needs to perform a successful ID verification to recognize the player as hostile and the results are cached for a couple of seconds, avoiding even more raycasts: the raycast is irrelevant if the AI already "knows" that the player isn't hostile!).
All of this said, however:
Unless you're seeing a problem with frame rate, you're likely performing a premature optimization. For your usage, overlapSphere (followed by a raycast so things don't try to shoot through walls, assuming that's relevant) will be the best. Cheap enough and accurate enough to be good enough.
Watch your profiler and make optimizations based on where the lag is coming from. Often it will be from something you don't expect. (Hint: manage your garbage well).