# Mapping a Touchinput-device in a 3D World for VR

I have a Touch-Input device which is linked to my Unity Program and a VR Device. The Touch-Input device which is bolted somewhere BUT the position can be changed.

I can get the "2D Coordinates" X / Y from the touch input device (from 0 to 1)

For each of those 2D coordinates I also have a 3D-VR Coordinate , grabbed by a LEAP or gloves. (Those coordinates are not always the corner ones.)

Here the problem on a wonderful paint art:

So what I have is the 2d and 3d positions of my fingertip (on the touch and inside the 3d world

And what I need is the a 3d Object (for example Cube) with correct rotation and position.

Bonus points for Scale!

• I'm not sure I understand this currently, do you mean the user has a glove in one hand, and touching the Touchinput device with their other hand? And you want the cube to be created on the 3D position of the glove? If you could describe this into more details I might be able to provide an answer, but right now I'm having trouble picturing what you are trying to achieve. – TomTsagk Feb 8 '19 at 16:15
• I updated my question! – OC_RaizW Feb 8 '19 at 21:29
• The user is using a glove- or a similar device to track his hand. and with the SAME hand he touches a touchscreen. And now I want to know where the touchscreen is . – OC_RaizW Feb 8 '19 at 21:30

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.UI;

public class Calibrate : MonoBehaviour
{
public Vector2 Current_2D_TouchscreenPosition;
public GameObject PointingFingerTip_DummyPosition;

private Vector3 CalibrationPos_3D_1;
private Vector3 CalibrationPos_3D_2;
private Vector3 CalibrationPos_3D_3;

private Vector2 CalibrationPos_2D_1;
private Vector2 CalibrationPos_2D_2;
private Vector2 CalibrationPos_2D_3;

private int CalibrationIndex;
private bool CalibrationCompleted;

private Matrix TouchToWorldMatrix;

public void SetCalibration(Vector3 World3DPosition, Vector2 Touch2DPosition )
{
//TODO: Check if values are valid!
if (CalibrationIndex == 0)
{
CalibrationPos_3D_1 = World3DPosition;
CalibrationPos_2D_1 = Current_2D_TouchscreenPosition;
CalibrationIndex++;
}
else if (CalibrationIndex == 1)
{
CalibrationPos_3D_2 = World3DPosition;
CalibrationPos_2D_2 = Current_2D_TouchscreenPosition;
CalibrationIndex++;
}
else if (CalibrationIndex == 2)
{
CalibrationPos_3D_3 = World3DPosition;
CalibrationPos_2D_3 = Current_2D_TouchscreenPosition;

//Creating The Matrix-
TouchToWorldMatrix = CreateTouchToWorldMatrix();
CalibrationIndex = 0;
CalibrationCompleted = true;
}

}

Vector3 GetWorldPosition(Vector2 TouchPosition)
{
if (!CalibrationCompleted)
{
return new Vector3(0,0,0);
}
return MultiplyVectorWithMatrix(TouchPosition, TouchToWorldMatrix);
}

private Matrix CreateTouchToWorldMatrix()
{
//Create an empty Matrix with 4 Rows and 3 Columns
Matrix M_LHS = new Matrix(4, 3);
//Create another empty Matrix with 3 Rows and 3 Columns
Matrix M_RHS = new Matrix(3, 3);

//Fill Left Hand-Side Matrix (
//  Row, Column
M_LHS[0, 0] = CalibrationPos_3D_1.x;
M_LHS[1, 0] = CalibrationPos_3D_1.y;
M_LHS[2, 0] = CalibrationPos_3D_1.z;
M_LHS[3, 0] = 1;

M_LHS[0, 1] = CalibrationPos_3D_2.x;
M_LHS[1, 1] = CalibrationPos_3D_2.y;
M_LHS[2, 1] = CalibrationPos_3D_2.z;
M_LHS[3, 1] = 1;

M_LHS[0, 2] = CalibrationPos_3D_3.x;
M_LHS[1, 2] = CalibrationPos_3D_3.y;
M_LHS[2, 2] = CalibrationPos_3D_3.z;
M_LHS[3, 2] = 1;

//Fill Right Hand Side Matrix
//  Row, Column
M_RHS[0, 0] = CalibrationPos_2D_1.x;
M_RHS[1, 0] = CalibrationPos_2D_1.y;
M_RHS[2, 0] = 1;

M_RHS[0, 1] = CalibrationPos_2D_2.x;
M_RHS[1, 1] = CalibrationPos_2D_2.y;
M_RHS[2, 1] = 1;

M_RHS[0, 2] = CalibrationPos_2D_3.x;
M_RHS[1, 2] = CalibrationPos_2D_3.y;
M_RHS[2, 2] = 1;

//Create a Touchscreen to Worldspace Matrix
return M_LHS * M_RHS.Invert();
}

public Vector3 MultiplyVectorWithMatrix(Vector2 vector, Matrix m)
{
float X_Multed = (vector.x * m[0, 0]) + (m[0, 1] * vector.y) + (m[0, 2]) ;
float Y_Multed = (vector.x * m[1, 0]) + (m[1, 1] * vector.y) + (m[1, 2]) ;
float Z_Multed = (vector.x * m[2, 0]) + (m[2, 1] * vector.y) + (m[2, 2]) ;

return new Vector3(X_Multed, Y_Multed, Z_Multed);

}

}


Allright I managed to solve this issue.

And it works.

You can basically copy-paste this script and use it. It works with creating a matrix ...