The easiest way to do this would be:
Vector3 localPoint = bigCircle.transform.InverseTransformPoint(
smallCircle.transform.position
);
This local
variable now holds the position of the small circle within the big circle's local coordinate system. That is, a coordinate system where (0, 0, 0) is the origin of the big circle, +z points to the big circle's "forward" direction (blue arrow of the local transformation gizmo), +x points to its right (red arrow), and +y points to the circle's up (green arrow).
If you then want to get a signed angle in the big circle's horizontal plane, that could look like:
float angleDegrees = Mathf.Atan2(localPoint.x, localPoint.z) * Mathf.Rad2Deg;
That will give you a bearing angle where 0 = directly ahead / "N", 90 = to the right / "E", 180 = exactly behind / "S".
Since you say you want to calculate this with math, not use a pre-built function, what InverseTransformPoint
is doing is taking bigCircle.transform.localToWorldMatrix
and inverting it to get worldToLocalMatrix
, then multiplying the provided point by this matrix:
Vector3 localPoint = bigCircle.trandform.worldToLocalMatrix
* smallCircle.transform.position;
You can get a similar result without matrix math like so:
// Remove bigCircle's translation.
Vector3 worldOffset = smallCircle.transform.position - bigCircle.transform.position;
// Remove bigCircle's rotation.
Quaternion antiRotation = Quaternion.Inverse(bigCircle.transform.rotation);
Vector3 localOffsetScaled = antiRotation * worldOffset;
// Remove bigCircle's scale (if applicable).
Vector3 antiScale = bigCircle.transform.localScale;
antiScale = new Vector3(1f/antiScale.x, 1f/antiScale.y, 1f/antiScale.z);
Vector3 localOffsetUnscaled = Vector3.Scale(localOffsetScaled, antiScale);