2
\$\begingroup\$

I'm playing around with 3D graphics programming and it's only a few weeks I'm into OpenGL and able to render meshes with textures with few basic transformations.

I've a hobby project where I want to apply a picture of human face on a 3D mesh head. I'm not very clear about the best programmatic approach to do that.

I read about how textures are created using a UV map which is created using some sort of projection of vertices in 3D onto a 2D image plane. This UV map is taken as reference to create the texture so that the texture appears correctly when it folds around the 3d mesh object.

So for my application let's say I've 3D mesh head with a UV map. I'd like my application to map the picture of my face onto this UV-map and create a texture which folds as seamlessly as possible on the 3D mesh head.

Please suggest some approaches I can explore. Also what kind of image processing algorithm I should refer and what libraries would best suit my needs?

Edit 1: Please check a similar app which is in market. https://play.google.com/store/apps/details?id=itseez3d.game.avatar&hl=en They are taking only one selfie image and superimposing it with a UV-map template to create it. I'd like to do something on similar lines.

\$\endgroup\$
  • \$\begingroup\$ Use a format that support uv coordinates (like wavefront) and use a modelling software. You can easily texture the face there. \$\endgroup\$ – Bálint Jun 20 '17 at 13:47
  • \$\begingroup\$ How to do it programmatically? Like I feed the face image and it outputs the 3D model with my face applied to it. \$\endgroup\$ – anurag-jain Jun 20 '17 at 13:50
  • \$\begingroup\$ You should probably go with something that takes less than a year to do. \$\endgroup\$ – Bálint Jun 20 '17 at 13:53
  • \$\begingroup\$ @Bálint Thanks for the motivation. Let's talk about your year long roadmap to tackle this problem? Shall we ? \$\endgroup\$ – anurag-jain Jun 20 '17 at 14:00
  • 2
    \$\begingroup\$ Relevant: xkcd.com/1425 \$\endgroup\$ – Elva Jun 20 '17 at 14:08
7
\$\begingroup\$

I am one of the developers of the itSeez3D application you linked in your post. Accidentally stumbled upon your question.

@Kevin van der Velden provided an entirely relevant reference here, I would not say it took us five years, but definitely not less than a year :)

There are several problems with the simplistic approach you're describing. First of all, if you just get the static mesh and try to use human photo as a texture you won't get anywhere near as convincing results. Human faces are intrinsically different in shape and to infer the 3D shape we use deep learning algorithms. With a bunch of trained neural networks we can predict the 3D shape of the face from the selfie (not precisely, but rather accurately). Thus we don't use static models, but rather each generated model is unique.

The other problem is an alignment of the texture. Of course, eyes on the texture should correspond to the eyes on 3D mesh, same with nostrils, etc. This is actually easier to achieve, you can use face landmarks to roughly align the texture.

Actually, the app mentioned in the post uses publicly available Web API that we created, it is called Avatar SDK. You can find the documentation here.

With this API you can send any human photo to our server, the neural networks will do their job and will generate a textured model of the human head for you with a bunch of available haircuts (currently in .ply format). There's also a Unity plugin for even easier integration. We're actively developing this SDK constantly adding features and we're strongly encouraging you to try it. It is free for trial and demo use, so you're not losing anything I guess. Please ask me here or write to support@itseez3d.com if you have any questions. Good luck!

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.