How do you triangulate a 3D point from a pair of stereo images?

Question

I'm working with a stereo pair of photos, and trying to obtain a set of 3D points from matching points on these 2 images. How exactly is the point triangulated? Do you throw a ray from the camera pinhole centerpoint? Or do you throw a ray perpendicular to the photos?

Edit: I'm also looking for code examples, that document the simple linear/polynomial math involved. Currently I'm simply using 3D line/line intersections to find the point. (actually line/plane intersections) I'm sure this is not accurate and the calibration data of the camera should also be used. Although I don't know how. For instance, with the points on the photos known, should I be drawing the 3D line from the camera pinhole 'origin', or is there a better way? Should I be taking the lens distort into account, or something fancier? The calibration data I have is as follows:

• ix, iy -- Image sensor width and height, i.e. the dimensions of the photograph.
• fx, fy -- Focal length in x- and y-dimensions measured in pixels.
• cx, cy -- Principal point coordinates, i.e. coordinates of lens optical axis interception with sensor plane.
• skew -- Skew transformation coefficient.
• k1, k2, k3 -- Radial distortion coefficients.
• p1, p2 -- Tangential distortion coefficients.

Answer 1

The first image is more correct, because image on the photo is perspective rather than orthographic. It can be orthographic on macro photos. Possibly it depends on what type of lens do you use.