I found this algorithm in some old codebase, it takes two triangles from a mesh ABC and PMN, where ABC is the triangle that will be rendered and PMN is an extra triangle that is only used to generate the UV coordinate for ABC and is not rendered.
I am trying to create a reverse algorithm to calculate the PMN triangle positions based on UV coordinates of a triangle and ABC positions of the same triangle.
Vector3f A = /* stored in mesh data */;
Vector3f B = /* stored in mesh data */;
Vector3f C = /* stored in mesh data */;
Vector3f P = /* stored in mesh data */;
Vector3f M = /* stored in mesh data */;
Vector3f N = /* stored in mesh data */;
Vector3f PM = M.sub(P, new Vector3f());
Vector3f PN = N.sub(P, new Vector3f());
Vector3f PA = A.sub(P, new Vector3f());
Vector3f PB = B.sub(P, new Vector3f());
Vector3f PC = C.sub(P, new Vector3f());
Vector3f PMxPN = PM.cross(PN, new Vector3f());
// Calculate the U coordinates
Vector3f U = PN.cross(PMxPN, new Vector3f());
float Mu = 1.0F / U.dot(PM);
float Ua = U.dot(PA) * Mu; /* 1st U coordinate (for vertex A) */
float Ub = U.dot(PB) * Mu; /* 2nd U coordinate (for vertex B) */
float Uc = U.dot(PC) * Mu; /* 3rd U coordinate (for vertex C) */
// Calculate the V coordinates
Vector3f V = PM.cross(PMxPN, new Vector3f());
float Mv = 1 / V.dot(PN);
float Va = V.dot(PA) * Mv; /* 1st V coordinate (for vertex A) */
float Vb = V.dot(PB) * Mv; /* 2nd V coordinate (for vertex B) */
float Vc = V.dot(PC) * Mv; /* 3rd V coordinate (for vertex C) */
Formatted LaTex represention of the same algorithm: $$ \vec{PM} = \vec{M} - \vec{P} \\ \vec{PN} = \vec{N} - \vec{P} \\ \vec{PA} = \vec{A} - \vec{P} \\ \vec{PB} = \vec{B} - \vec{P} \\ \vec{PC} = \vec{C} - \vec{P} \\ \vec{PMPN} = \vec{PM} \times \vec{PN} \\ $$
$$ \vec{U} = \vec{PN} \times\vec{PMPN} \\ Mu = \frac{1} {\vec{U} \cdot \vec{PM}} \\ Va = \vec{U} \cdot \vec{PA} \times Mu \\ Vb = \vec{U} \cdot \vec{PB} \times Mu \\ Vc = \vec{U} \cdot \vec{PC} \times Mu \\ $$$$ \vec{U} = \vec{PN} \times\vec{PMPN} \\ Mu = \frac{1} {\vec{U} \cdot \vec{PM}} \\ Ua = \vec{U} \cdot \vec{PA} \times Mu \\ Ub = \vec{U} \cdot \vec{PB} \times Mu \\ Uc = \vec{U} \cdot \vec{PC} \times Mu \\ $$
$$ \vec{V} = \vec{PM} \times\vec{PMPN} \\ Mv = \frac{1} {\vec{V} \cdot \vec{PN}} \\ Va = \vec{V} \cdot \vec{PA} \times Mv \\ Vb = \vec{V} \cdot \vec{PB} \times Mv \\ Vc = \vec{V} \cdot \vec{PC} \times Mv \\ $$