# How can I offset a direction when raycasting?

I'm trying to figure out how I can best offset a direction when raycasting.

So let's say I have an object and I want to cast a ray from my mouse position to said object, that's easily done with (for example):

// "obj" being the transform of the object I'm raycasting to.
Vector2 mPosition = Camera.main.ScreenToWorldPoint(Input.mousePosition);
Vector2 dir = (Vector2)obj.position - mPosition;
Debug.DrawRay(mPosition, dir, Color.green);


But what if I want the direction to always be slightly offset to the right of the object from the mouse position? And I mean not just to the right of the object itself, but to the right from where the mouse is.

I hope I'm making any sense. Here are some pictures to try and make it a little more clear: (Pretend the line is the ray and the circle is the object)

I want to offset the ray so that when my mouse is under the circle, it goes to the right of it, like this:

However, when I move my mouse to the left of the object, the ray should instead go like this:

Meaning the ray will always cast to the right of the object from where the mouse is.

The reason I want to do this is because I'm trying to raycast to the corners of objects, but also see what's past them:

I need to use offset rays to just barely miss the corners and allow them to continue on and see if there's a wall behind them like on the picture above. I'm offsetting my rays in the picture above, but the way I'm doing it isn't the best and I'm trying to find the proper way of doing it.

• I'm not sure I understand you. Are you saying that you want to find a ray that will pass to the right (right being defined in screen space?) of the object directly under the mouse? – Jack Aidley Sep 10 at 14:40
• I edited the post a bit with some pictures to try and visualize it a bit more. – olawrdhalpme Sep 10 at 14:53
• Can you tell us a bit about what you're using this offset ray to accomplish? Having this context can help answer a lot of questions we might have about the behaviour you need, because it gives us a specific application to serve and test potential solutions against. – DMGregory Sep 10 at 14:58
• I'm trying to do something similar to this: ncase.me/sight-and-light He is using raycasts on the corners of objects, but in order to find out if the rays can hit walls behind the corners he needs to create additional rays with a slight offset so that they just barely miss the corners and continue on. I don't really know how to explain it. – olawrdhalpme Sep 10 at 15:03
• Why are you excluding the method given in that link: "For each (unique) line segment end point, I cast a ray directly towards it, plus two more rays offset by +/- 0.00001 radians." – Jack Aidley Sep 10 at 15:52

The X/Y Problem strikes again!

I'm trying to figure out how I can best offset a direction..I want the direction to always be slightly offset to the right of the object from the mouse position? And I mean not just to the right of the object itself, but to the right from where the mouse is.

This is complicated, messy, and error prone. How much offset is juuuuuust enough to miss while not unduly changing the resulting hit on the wall behind? What if we have two corners at similar angles, with just a narrow wedge in between, so our offset from one corner hits the other corner and vice versa, and we miss the gap entirely? :o

I need to determine where my ray hits a wall on the far side of a corner, for the purpose of determining the polygonal area visible / illuminated from my ray origin.

This is your real problem. Not offsetting rays. Offsetting rays was just a guess at the solution, and I'd argue it's not the best solution we have within our reach.

To fire a ray that continues past the corner of the object and out the other side to detect the next hit, just use a RaycastAll query (specifically the non-allocating version for efficiency). Something like this:

// Make this conservatively large so you don't overflow it.
RaycastHit2D[] _hits = new RaycastHit2D[10];

RaycastHit2D RaycastPastCorner(Vector2 cornerPosition, Vector2 rayOrigin) {
int hitCount = Physics2D.RaycastNonAlloc(
rayOrigin,
cornerPosition - rayOrigin,
_hits,
maxRayRange,
raycastLayers);

// If we hit nothing past the corner, return a miss.
if(hitCount < 2)
return default(RaycastHit2D);

// Otherwise, return the first hit past the corner.
// (If needed, you can inspect these hits and skip forward by some tolerance)
return _hits[1];
}


There we go, no fudge factor offsets required. Much cleaner and less error-prone!

• RaycastAll will return all objects along the line, though, so it will return objects that should be shadowed by the first object too, no? – Jack Aidley Sep 10 at 16:42
• It will return hits along the terminator that separates light from shadow, which is exactly what you need to compute the visibility polygon. By selecting the second hit, you skip over the hit at the corner and get the hit on the next object behind it, at the exact point where light should give way to shadow, rather than a point slightly offset from this terminator line. – DMGregory Sep 10 at 16:45
• Note that this code assumes you've already verified that the corner itself is visible. If you don't know that yet at this stage in your code, you can modify this to compare the hits in the array against the distance to the corner, and return an appropriate output if the corner itself is shadowed. – DMGregory Sep 10 at 16:50
• This assumes that you've identified the extreme corners (which is not what the linked code does), otherwise if have a corner in the middle of the object (seen from the source), then the ray cast at this corner will misidentify what you want. – Jack Aidley Sep 10 at 17:35
• You can modify this to check whether the first hit is glancing or penetrating, using information about the primitive struck, and return the next result only in the case of a glancing hit. – DMGregory Sep 10 at 17:37

You can rotate it around the Y axis a bit. E.g. 10 degrees:

direction = Quaternion.Euler(0, 10, 0) * direction;


To rotate a 2d vector (x, y) by a angle A to form a new vector (x2, y2):

x2 = x * cos(A) - y * sin(A);
y2 = x * sin(A) + y * cos(A);


(A mathematical discussion of why is here if you want it)

Because you want to rotate in both directions, you can simply form vectors with the angles +0.0001 and -0.0001 to get the rays either side of your central ray. Almost every implementation of cos and sin works with Radians so you won't need to perform a conversion.