3
\$\begingroup\$

Many questions have been posted on this topic but I'm facing a weird behavior which I could not find an explaination for.

I perform a raycast giving it a LayerMask parameter which is defined as public variable and set in the inspector with thick on layer "Wall" and "Enemy".

At first glance it appeared to work fine because raycast rightly stops on GameObjects with that layer.

But then I saw that it stopped also on GameObjects with default layer, like if the LayerMask was ignored. Is there something I'm missing?

According to this answer I set up everything correctly and I expected it to work fine.

Looking at the inspector at Runtime I can clearly see that my mask is set only to "Wall" and "Enemy", and that the GameObject the collider is attached to has default layer set. I'm then stating that the mask the way I'm using it does not exclude other layers. The only way I found to have this working is by setting that GameObject layer to Ignore Raycast but in this case the LayerMask is not necessary.

I'm calling the function this way:

RaycastHit2D inSight = Physics2D.Raycast(start, end, lm); 

Maybe is there something different using Physics2D?
I would really enjoy touse LayerMasks instead of giving specifically Ignore Raycast to whatever I want to ignore. Any clue?

\$\endgroup\$
5
\$\begingroup\$

I've had this happen with the 3D raycast in the past, and it seems to be because the raycast functions have so darned many optional parameters, and LayerMasks aren't picky about what type they're treated as.

From the docs:

public static RaycastHit2D Raycast(
       Vector2 origin, 
       Vector2 direction, 
       float distance = Mathf.Infinity, 
       int layerMask = DefaultRaycastLayers, 
       float minDepth = -Mathf.Infinity, 
       float maxDepth = Mathf.Infinity
);

layerMask is the fourth parameter. So if you provide only three parameters, Unity wants to interpret them as origin, direction, and distance.

It might seem like providing an input of type LayerMask should make your intent unambiguous, or at least generate a compile-time error, but LayerMask can implicitly convert to int, and int can implicitly convert to float.

So when you put a LayerMask in the third argument, the compiler tries to convert it to a float, succeeds, and so silently re-interprets your mask as a distance. The fourth parameter layerMask is then unspecified, so it defaults to DefaultRaycastLayers, basically ignoring the layers you chose in the Inspector.

You can fix this by providing a dummy distance value (say, infinity, if you don't have any more conservative upper bound) so your layer mask selections end up in the fourth argument where Unity expects it:

RaycastHit2D inSight = Physics2D.Raycast(start, end, float.PositiveInfinity, lm);

\$\endgroup\$
  • \$\begingroup\$ Thank you gregory but I think this is wrong. For a simple reason which is that layerMask in my raycast function is a LayerMask and cannot be mismatched for a float. Simply unity has overload for Raycasy(vector, vector, LayerMask) \$\endgroup\$ – Leggy7 Oct 15 '15 at 8:10
  • 1
    \$\begingroup\$ I've explained how the LayerMask type can in fact be mismatched for a float. Try it — the compiler might not be using the overload that you think it should. I've had this happen in my own code too. \$\endgroup\$ – DMGregory Oct 15 '15 at 12:09
  • \$\begingroup\$ Also, @Leggy7, I can't find the overload that you describe. In both Unity 4 & 5, Physics2D.Raycast reports only the one definition, with its various optional parameters, and distance always comes before layerMask. Physics.Raycast (the 3D version) does have an overload that takes (ray, distance, layerMask) which is the only 3-argument version I can find containing layerMask, but even there distance comes first. \$\endgroup\$ – DMGregory Oct 15 '15 at 13:51
  • 1
    \$\begingroup\$ Now, at home, I tried it. Shame on me, it works perfectly. I got fooled by the fact that no compilers errors occurred giving overload with 3rd parameter as LAyerMask insted of float: I assumed such overload existed. Instead, apparently, unity can convert a LayerMask to a float implicitly. Thank you Gregory,your answer is correct! \$\endgroup\$ – Leggy7 Oct 15 '15 at 20:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.