If I have a mesh of triangles and going to make a selection on it using a region (and not rectangle using
Glu.gluPickMatrix(...);), how can I implement it?
Here's how I would do it: Using the dimensions of the box you've selected on the screen, define a new 'camera' (view and projection matrices) which defines that region of the screen and the frustum that extrudes out from that region into world space.
From there, you can go through your primitives and emulate the rasterization pipeline by converting vertex positions into Normalized Device Coordinates. Anything that falls in the range [-1,1] in each of the axes is a primitive which is being selected.
Note that my explanation is really suited for finding selected vertices, and some additional logic would have to be added to determine which triangles were selected.