Selecting the closest point in perspective view mode

I'm trying to improve my algorithm for selecting points in perspective view mode (OpenGL/Qt/C++). The current implementation works as follows.

1. The user clicks on a certain (x,y) position (in Window space)
2. Traversing the pipeline from Window space back to World space (that is, Window → NDC → Clip → Eye → World), this defines a point a on the near clipping plane and a point b on the far clipping plane
3. The distances from the line between a and b and the vertices (available in World space) are determined (see e.g. this article on Wikipedia)
4. The point with the minimum distance to the line is selected

Now, this works fine when using orthogonal view mode. However, when visualizing something in perspective view mode, there are some cases where the result might be a bit off.

As example, let's consider the wireframe of a cube. In the illustration below, the point (x,y) is the point where all white lines meet. These lines are orthogonal to the line from a to b which is not visible in this orientation — the second illustration shows a rotated view.

The point that eventually gets selected is highlighted in green. However, as the blue point seems to be closer to (x,y) in perspective mode, the user probably expects this point to get selected instead. Somehow the computed distances have to be scaled in order to take this effect into account. How is this usually done?

• Sounds like rather than computing the 3D distance of each point from the unprojected line, you want to compute the 2D distance of each point's pixel position from the clicked point on the screen. Jan 10 '16 at 15:51