I am working on the generation of Hexahedral mesh generation using octree based discretization. I have a volume mesh (Hexahedral), using isomorphism I need to project the outermost surface vertices of inner mesh onto original mesh. Could anyone help me on how to extract the surface of inner mesh?
I think the term you are looking for "Surface nets".
In Surface nets, instead of doing an exact projection onto the mesh surface, an filtering-based approximation is used. For example, in this paper, they use an iterative smoothing process, that keeps all vertices in their original volume boundary.
I think you need to create a signed distance field based on the original mesh. Then you can use gradient descent to iteratively move the outer vertices along the gradient of the distance field towards the orginal mesh surface.
The gradient of a scalar field can be constructed like this:
Hope that helps.