The math behind
Physics.LineCast(A, B) is what we call a ray intersection test.
A to be the origin of the ray,
B-A to be the direction of the ray, and look for potential intersections along the line
P(t) = A + (B-A)*t for
0 <=t <= 1 (or other equivalent formations that differ only in details)
For many shapes, we have a closed-form formula we can apply to find the specific
t value where the ray hits it, if it does at all. If there's no such value of
t, or the intersection happens at
t < 0 or
t > 1, then there's no intersection with our line segment.
Different shapes will use different formulas:
So this gives us a simple algorithm: loop over all the collision shapes in your scene, apply the corresponding ray intersection formula for each one, and keep the lowest value of
t you find in the range 0...1.
This will give you correct results, but it's not fast. We waste a lot of time evaluating the formula on objects nowhere near the ray, or on distant objects likely to be occluded by something closer.
So we apply an acceleration structure to help us zero in on just the shapes worth checking, ideally in roughly front-to-back order. That lets us keep the correctness, while making the test much cheaper/faster.
There are a lot of forms this could take, depending on your needs:
We could divide your world into a grid of cells, and store collision shapes in each cell they overlap.
Then we can march the ray through this grid (kind of like Bresenham's line algorithm), so that we visit each cell in front-to back order.
For each cell we visit, we run the intersection formulas on just the shapes in that cell, skipping any cells our ray doesn't touch.
We could group objects into a bounding volume hierarchy, and test our ray only against the bounding volume of the root node of that tree. If we find a hit, we test against the bounding volume of its children, and recurse this way until we narrow in on the intersecting objects to run the fine-grained formula on.
We usually use a very simple shape for the bounding volumes, so that test is fast, letting us quickly discard large swaths of the scene.
For all these approaches, it's important to know when to stop. The first hit you found isn't necessarily the nearest hit - it could be that the next object you check has a closer hit point. We can again use a bounding volume to estimate the closest possible hit on an object, and stop our search once the closest possible estimate is further away than the closest hit we've found, guaranteeing that we do indeed have the closest result now.