How can I offset a direction when raycasting?

Question

I'm trying to figure out how I can best offset a direction when raycasting.

So let's say I have an object and I want to cast a ray from my mouse position to said object, that's easily done with (for example):

// "obj" being the transform of the object I'm raycasting to.
Vector2 mPosition = Camera.main.ScreenToWorldPoint(Input.mousePosition);
Vector2 dir = (Vector2)obj.position - mPosition;
Debug.DrawRay(mPosition, dir, Color.green);

But what if I want the direction to always be slightly offset to the right of the object from the mouse position? And I mean not just to the right of the object itself, but to the right from where the mouse is.

I hope I'm making any sense. Here are some pictures to try and make it a little more clear: (Pretend the line is the ray and the circle is the object)

I want to offset the ray so that when my mouse is under the circle, it goes to the right of it, like this:

However, when I move my mouse to the left of the object, the ray should instead go like this:

Meaning the ray will always cast to the right of the object from where the mouse is.

The reason I want to do this is because I'm trying to raycast to the corners of objects, but also see what's past them:

I need to use offset rays to just barely miss the corners and allow them to continue on and see if there's a wall behind them like on the picture above. I'm offsetting my rays in the picture above, but the way I'm doing it isn't the best and I'm trying to find the proper way of doing it.

Answer 1

The X/Y Problem strikes again!

I'm trying to figure out how I can best offset a direction..I want the direction to always be slightly offset to the right of the object from the mouse position? And I mean not just to the right of the object itself, but to the right from where the mouse is.

This is complicated, messy, and error prone. How much offset is juuuuuust enough to miss while not unduly changing the resulting hit on the wall behind? What if we have two corners at similar angles, with just a narrow wedge in between, so our offset from one corner hits the other corner and vice versa, and we miss the gap entirely? :o

I need to determine where my ray hits a wall on the far side of a corner, for the purpose of determining the polygonal area visible / illuminated from my ray origin.

This is your real problem. Not offsetting rays. Offsetting rays was just a guess at the solution, and I'd argue it's not the best solution we have within our reach.

So, moral of the story: always ask about the problem you're trying to solve, not about the kludge that might help you solve it.

---

To fire a ray that continues past the corner of the object and out the other side to detect the next hit, just use a RaycastAll query (specifically the non-allocating version for efficiency). Something like this:

// Make this conservatively large so you don't overflow it.
RaycastHit2D[] _hits = new RaycastHit2D[10];

RaycastHit2D RaycastPastCorner(Vector2 cornerPosition, Vector2 rayOrigin) {
     int hitCount = Physics2D.RaycastNonAlloc(
                      rayOrigin,
                      cornerPosition - rayOrigin,
                      _hits,
                      maxRayRange,
                      raycastLayers);

     // If we hit nothing past the corner, return a miss.
     if(hitCount < 2)
         return default(RaycastHit2D);

     // Otherwise, return the first hit past the corner.
     // (If needed, you can inspect these hits and skip forward by some tolerance)
     return _hits[1];
}

There we go, no fudge factor offsets required. Much cleaner and less error-prone!

Answer 2

You can rotate it around the Y axis a bit. E.g. 10 degrees:

direction = Quaternion.Euler(0, 10, 0) * direction;

Answer 3

To rotate a 2d vector (x, y) by a angle A to form a new vector (x2, y2):

x2 = x * cos(A) - y * sin(A);
y2 = x * sin(A) + y * cos(A);

(A mathematical discussion of why is here if you want it)

Because you want to rotate in both directions, you can simply form vectors with the angles +0.0001 and -0.0001 to get the rays either side of your central ray. Almost every implementation of cos and sin works with Radians so you won't need to perform a conversion.