Raycast Masks not working Unity3D

Question

I'm having a problem with filters/masks when raycasting.

I've created a LayerMask and then in the inspector selected the layer I want to ignore when Raycasting. however then I print out the name of the object hit by the raycast I am still getting hits on objects on the layer masked out, as well as on other layers.

public LayerMask rayMask;

RaycastHit hit;
Ray ray = Camera.main.ScreenPointToRay(Input.mousePosition);
if (Physics.Raycast (ray, out hit,rayMask)){
    Debug.Log (hit.collider.name);
}

I have paused the Game and checked that the object I'm getting a hit on is indeed on the layer in the mask.

Answer 1

Sorry folks. First Post here on Stack Exchange. Thought maybe after a while I could just mark one of your comments as an answer guess not.

So my solution was:

Physics.Raycast () needs 4 veriables in order to use a Layermask. So i had to add a distance to the ray. i.e.

Physics.Raycast (ray, out hit,mathf.infinity,rayMask)

that got the layermask working Although it was the Inverse of what i thought, the ray hit only what was on the mask. So i added

rayMask = ~rayMask;

to invert the Layermask.