It looks like your trouble isn't in computing the direction - it's that your arrow asset was imported with an unusual orientation, so when you use methods like LookAt or LookRotation they give you the "right" result for a standard orientation, leaving your arrow visually pointing somewhere else.
You tried to correct for this by adding angular offsets to the rotation, but the transform.rotation property is not measured in angles - it's a Quaternion, and so we need to use Quaternion methods if we want to modify it.
Usually the cleanest fix here is to re-export your asset in the orientation you really want, or parent it to an empty game object and then rotate the child to align properly with the parent - using the new parent as the root of your prefab.
I'll show you how we can correct the orientation in code though:
void TransformArrow(Vector3 start, Vector3 end, float height)
{
// This is similar to what you already have.
Vector3 arrowPosition = (start + end)/2f;
arrowPosition.y = height;
arrow.transform.position = arrowPosition;
Vector3 direction = (end - start);
// If you want to prevent the arrow from tilting up or down on slopes,
// uncomment this line.
// direction.y = 0f;
Quaternion arrowOrientation = Quaternion.LookRotation(direction);
// Your prefab arrow points along -x ("left"), with +z ("forward") pointing up.
// We want to rotate this to Unity's standard: +z forward, y+ up.
// We'll construct the orientation facing "left" with "forward" up, then invert it.
Quaternion correction = Quaternion.Inverse(
Quaternion.LookRotation(Vector3.left, Vector3.forward)
);
// Now we apply our orientation and the correction for the prefab's orientation.
arrow.transform.rotation = arrowOrientation * correction;
}