Compute the vector representing the displacement between the two points
v = p1 - p0
and then compute the length of that vector:
distance = sqrt(dot(v, v))
A vector in this case is an element of the real 3D coordinate space, so it has three components (X, Y and Z). A point also has the same three coordinates, and we can subtract two points from eachother to get the vector representing the displacement between those points. Both vector and point subtraction is done component-wise, which means you do the operation on each component separately. For the first line above (
v = p1 - p0), you're basically doing
v.x = p1.x - p0.x
v.y = p1.y - p0.y
v.z = p1.z - p0.z
v.x et cetera are the individual scalar components of the vector, and so on).
In the second equation we're taking the vector dot product of
v with itself. The dot product has lots of useful applications but in this case I'm just using it for brevity. The dot product of a 3D vector has a scalar result (a single number), and that number is calculated for two vectors
(a.x * b.x) + (a.y * b.y) + (a.z * b.z)
which in our specific case translates to
(v.x * v.x) + (v.y * v.y) + (v.z * v.z)
This is the same as "
v.x squared plus
v.y squared..." When you take the square root of that resulting scalar, you get what is known as the magnitude or length of the vector. The length of the vector is the distance you travel when, starting from some origin (
p0 in this case) you displace yourself by the vector arriving at some destination (
It is thus the quantity you want to sort your entities by. Note however that if you only need to compare distances, and not otherwise use the actual accurate value of the distance, you can omit the slightly-costly square root from the computation since it will not change the relative ordering of the values.