Your tried solutions can work together. You don't have to choose one over the other since you want to simulate both gaze direction and line of sight.
Let's start with your base assumption: enemies must detect the player if the latter is within their detection range. An overlap circle check (or, similarly, a distance-to-point within range) will detect the player within a radius, but nothing more. The pseudocode would look like this:
// Detect the player
if ( player_in_circle() )
{
detect_player()
}
On top of distance, we check for direction. Instead of considering the Y-coordinate difference, I suggest checking whether the angle between each enemy and the player lies within a given range. We can do this with trigonometry: we first define a "limit angle" relative to the ground beyond which enemies cannot "see"; then, we check the enemy-to-player direction doesn't exceed such limit value.
Since we are ignoring all directions beyond the limit angle (let's say 75°), the enemies' detection area will be altered as follows:
Our pseudocode will now look like this:
// Detect the player
if ( player_in_circle() )
{
if ( player_direction_in_range() )
{
detect_player()
}
}
Finally, we want to check for line of sight. Platforms and solid objects should obstruct the enemies' vision, therefore a raycast check will be enough for this purpose. This enables ignoring all player positions where a raycast would return a hit ("I can't see the player"), and will alter the enemies' detection area like this:
Our ugly pseudocode will finally look like this:
// Detect the player
if ( player_in_circle() )
{
if ( player_direction_in_range() and not platforms_raycast_hit() )
{
detect_player()
}
}
The above code is supposed to run at every frame. If you are using a trigger collider, you may use the OnTriggerStay()
callback to check only as long as the player overlaps the circle. A quick (untested) sample code here:
using UnityEngine;
using System.Collections;
public class PlayerDetector : MonoBehaviour
{
[Range(0f,90f)]
public float maxAngle = 75f;
void OnTriggerStay(Collider other)
{
if (other.CompareTag("Player"))
{
// Is the player within the enemy's vision range?
Vector2 _en2pl = other.transform.position - transform.position;
if (Mathf.Abs(_en2pl.normalized.x) <= Mathf.Cos(maxAngle * Mathf.Deg2Rad)
{
// Check for platforms and solid objects
RaycastHit2D hit = Physics2D.Raycast(transform.position, _en2pl, ...);
if (hit.collider == null)
{
// PLAYER DETECTED
}
}
}
}
}