Mapping a Touchinput-device in a 3D World for VR

Question

I have a Touch-Input device which is linked to my Unity Program and a VR Device. The Touch-Input device which is bolted somewhere BUT the position can be changed.

I can get the "2D Coordinates" X / Y from the touch input device (from 0 to 1)

For each of those 2D coordinates I also have a 3D-VR Coordinate , grabbed by a LEAP or gloves. (Those coordinates are not always the corner ones.)

Here the problem on a wonderful paint art:

So what I have is the 2d and 3d positions of my fingertip (on the touch and inside the 3d world

And what I need is the a 3d Object (for example Cube) with correct rotation and position.

Bonus points for Scale!

Answer 1

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.UI;

public class Calibrate : MonoBehaviour
{
    public Vector2 Current_2D_TouchscreenPosition;
    public GameObject PointingFingerTip_DummyPosition;

    private Vector3 CalibrationPos_3D_1;
    private Vector3 CalibrationPos_3D_2;
    private Vector3 CalibrationPos_3D_3;

    private Vector2 CalibrationPos_2D_1;
    private Vector2 CalibrationPos_2D_2;
    private Vector2 CalibrationPos_2D_3;

    private int CalibrationIndex;
    private bool CalibrationCompleted;

    private Matrix TouchToWorldMatrix;

    public void SetCalibration(Vector3 World3DPosition, Vector2 Touch2DPosition )
    {
        //TODO: Check if values are valid!
        if (CalibrationIndex == 0)
        {
            CalibrationPos_3D_1 = World3DPosition;
            CalibrationPos_2D_1 = Current_2D_TouchscreenPosition;
            CalibrationIndex++;
        }
        else if (CalibrationIndex == 1)
        {
            CalibrationPos_3D_2 = World3DPosition;
            CalibrationPos_2D_2 = Current_2D_TouchscreenPosition;
            CalibrationIndex++;
        }
        else if (CalibrationIndex == 2)
        {
            CalibrationPos_3D_3 = World3DPosition;
            CalibrationPos_2D_3 = Current_2D_TouchscreenPosition;

            //Creating The Matrix-
            TouchToWorldMatrix = CreateTouchToWorldMatrix();
            CalibrationIndex = 0;
            CalibrationCompleted = true;
        }

    }

    Vector3 GetWorldPosition(Vector2 TouchPosition)
    {
        if (!CalibrationCompleted)
        {
            return new Vector3(0,0,0);
        }
        return MultiplyVectorWithMatrix(TouchPosition, TouchToWorldMatrix);
    }

    private Matrix CreateTouchToWorldMatrix()
    {
        //Create an empty Matrix with 4 Rows and 3 Columns
        Matrix M_LHS = new Matrix(4, 3);
        //Create another empty Matrix with 3 Rows and 3 Columns
        Matrix M_RHS = new Matrix(3, 3);

        //Fill Left Hand-Side Matrix (
        //  Row, Column
        M_LHS[0, 0] = CalibrationPos_3D_1.x;
        M_LHS[1, 0] = CalibrationPos_3D_1.y;
        M_LHS[2, 0] = CalibrationPos_3D_1.z;
        M_LHS[3, 0] = 1;

        M_LHS[0, 1] = CalibrationPos_3D_2.x;
        M_LHS[1, 1] = CalibrationPos_3D_2.y;
        M_LHS[2, 1] = CalibrationPos_3D_2.z;
        M_LHS[3, 1] = 1;

        M_LHS[0, 2] = CalibrationPos_3D_3.x;
        M_LHS[1, 2] = CalibrationPos_3D_3.y;
        M_LHS[2, 2] = CalibrationPos_3D_3.z;
        M_LHS[3, 2] = 1;


        //Fill Right Hand Side Matrix
        //  Row, Column
        M_RHS[0, 0] = CalibrationPos_2D_1.x;
        M_RHS[1, 0] = CalibrationPos_2D_1.y;
        M_RHS[2, 0] = 1;

        M_RHS[0, 1] = CalibrationPos_2D_2.x;
        M_RHS[1, 1] = CalibrationPos_2D_2.y;
        M_RHS[2, 1] = 1;

        M_RHS[0, 2] = CalibrationPos_2D_3.x;
        M_RHS[1, 2] = CalibrationPos_2D_3.y;
        M_RHS[2, 2] = 1;

        //Create a Touchscreen to Worldspace Matrix
        return M_LHS * M_RHS.Invert();
    }




    public Vector3 MultiplyVectorWithMatrix(Vector2 vector, Matrix m)
    {
        float X_Multed = (vector.x * m[0, 0]) + (m[0, 1] * vector.y) + (m[0, 2]) ;
        float Y_Multed = (vector.x * m[1, 0]) + (m[1, 1] * vector.y) + (m[1, 2]) ;
        float Z_Multed = (vector.x * m[2, 0]) + (m[2, 1] * vector.y) + (m[2, 2]) ;

        return new Vector3(X_Multed, Y_Multed, Z_Multed);

    }

}

Allright I managed to solve this issue.

I used this answer: https://math.stackexchange.com/questions/3108619/how-to-find-a-2d-coordinate-fields-corners-in-a-3d-coordinate-field-if-i-have-3/3109504?noredirect=1#comment6409762_3109504

And it works.

You can basically copy-paste this script and use it. It works with creating a matrix ...