Unity Raycast not detecting layers

Question

I am having problems with layers detection using raycasts.

Context: I created a grid to use in a pathfinding algorithm on a 2D plane. I want to implement now different weights according to the layer each node of the grid is in e.g. a node in a normal terrain will have a "terrainPenalty" of 1 and more difficult terrains 2 or more. In order to detect the terrain type on the 2D plane, I am shooting a raycast on the node position for each node previously considered "walkable". The problem comes up now, because I observed that the paths created are neglecting such values, this is the slice of code which is giving me problems:

            if (walkable)
            {
                Ray ray = new Ray(worldPoint + Vector3.back * 10, Vector3.forward);//PROBLEM HERE (NOT DETECTING PENALTIES)
                RaycastHit hit;
                walkableMask = ~walkableMask;
                if (Physics.Raycast(ray, out hit, 100, walkableMask))
                {
                    walkableRegionsDictionary.TryGetValue(hit.collider.gameObject.layer, out terrainPenalty );
                }
            }

At first I thought that the problem was how the layers were called, because I created a LayerMask combining 2 layers using the bitwise OR operator, but then I tried to play around with all the layers (hence why there is walkableMask = ~walkableMask; now). The result was always a false output for Physics.Raycast(ray, out hit, 100, walkableMask).

I thought that maybe I was getting the wrong coordinates for the ray, but I created some gizmos to actually check that the ray was piercing perpendicularly my plane and it did, as well as trying to change layers to see if anything worked, but the output is always false. What am I getting wrong? Here is the rest of the function, if needed I can upload the whole script as well:

    void CreateGrid()
{
    grid = new Node1[gridSizeX, gridSizeY];
    Vector3 worldBottomLeft = transform.position - Vector3.right * gridWorldSize.x / 2 - Vector3.up * gridWorldSize.y / 2;

    for (int x = 0; x < gridSizeX; x++)
    {
        for (int y = 0; y < gridSizeY; y++)
        {
            Vector3 worldPoint = worldBottomLeft + Vector3.right * (x * nodeDiameter + nodeRadius) + Vector3.up * (y * nodeDiameter + nodeRadius);
            bool walkable = !(Physics2D.OverlapCircle(new Vector2(worldPoint.x, worldPoint.y), nodeRadius, unwalkableMask));
            //bool walkable = !(Physics.CheckSphere(worldPoint, nodeRadius));

            int terrainPenalty = 0;

            if (walkable)
            {
                Ray ray = new Ray(worldPoint + Vector3.back * 10, Vector3.forward);//PROBLEM HERE (NOT DETECTING PENALTIES)
                RaycastHit hit;
                walkableMask = ~walkableMask;
                if (Physics.Raycast(ray, out hit, 100, walkableMask))
                {
                    walkableRegionsDictionary.TryGetValue(hit.collider.gameObject.layer, out terrainPenalty );
                }
            }

            grid[x, y] = new Node1(walkable, worldPoint, x, y, terrainPenalty);
        }            
    }
}

EDIT : Here is how I create the layerMask:

        foreach (TerrainType region in walkableRegions)
    {
        walkableMask.value |= region.terrainMask.value; // | is the bitwise OR operation, I could have simplified by writing " --- |= ---; "
        walkableRegionsDictionary.Add((int)Mathf.Log(region.terrainMask.value, 2), region.terrainPenalty);
    }

Where TerrainType is the following class:

    [System.Serializable]
public class TerrainType
{
    public LayerMask terrainMask;
    public int terrainPenalty;
}

I basically create an array of TerrainType objects that will store their layerMask and penalty and then upon positive output from the raycast I access the dictionary to retrieve the terrain penalty. For the terrain and obstacles I am using a 2D BoxCollider and 2D PolygonCollider respectively, they do not interact with any other layer.

Thanks for any possible hints, I don't know what else to try :\ .

Answer 1

It looks as though you're trying to fire a ray with the 3D physics engine PhysX, represented by the Physics class, and expecting it to interact with a collider managed by the 2D physics engine Box2D, represented by the Physics2D class.

These are two completely separate universes that know nothing about and cannot interact with one another.

(As a hint, look at the fact that a 3D RaycastHit can only tell you about a 3D Collider it hit. It couldn't tell you about a Collider2D hit if it tried!)

To fire rays against 2D colliders, you need them to be 2D rays, ie. Physics2D.Raycast. But as noted in this answer and comment thread, these are rays in the plane, that check where they cross a collider's edge. They can't fire through the plane along the third dimension because there is no third dimension in 2D.

So really it looks like what you want is Physics2D.OverlapPoint, to get the 2D collider with the least Z value that overlaps this point in 2D.

It looks like your problem never had anything at all to do with layers or layer masks, so including that in your question was a red herring. Please try to do tests when preparing your question — like "what if I don't use a layermask at all, does the problem persist? What if I have only one object in my scene?" etc. — so you can eliminate distracting elements and focus on solely the details that are relevant to the root cause.