I remember there is a specific function in Direct3D which optimizes a mesh or a 3D object in some way. What is the purpose of this function? How does it reorganize vertices to make it better?
The ID3DXMesh interface had Optimize and OptimizeInPlace methods, which is what I presume you are thinking of. The purpose of these methods was to reorganize the mesh data (vertex buffers, index buffers, et cetera) to improve drawing performance.
The details of the optimization performed depended on various flags you'd specify when you call the function. You can find a list of flags here, which will reveal some of the specifics about the kinds of operations performed.
I'd venture to guess that the specific optimizations weren't documented in detail to allow for flexibility in adjusting them as hardware evolved. However, most of them centered around removing unused or duplicated data (to achieve better bandwidth utilization) and to reorganize the buffers to allow for better cache coherency on the card.