# Find appropriate camera-projector coordinates to project contours of real objects

I am developping a game (unity3d + opencv for image processing) where i detect real-life objects from the webcam on every frame, find their edges and then draw them on the real objects. I need to find the real-world coordinates in order to place my camera and projector in a way to get edges that align with their respective objects. I looked into camera calibration which returns the camera's intrinsics, extrinsics, and lens distortion parameters but i don't know how to use them in my case. Is there a possible way to calculate these positions and angles?

Basically my real-world view looks something like this:

And the image that I should project is this:

I think i should also mention that it's a top-down projection and i'm not using any depth sensing camera.

EDIT: When i put the camera on the same axis as the projector it keeps displaying the projected image infinitely (still didn't figure out the right distance for display)... So where should i place the camera (is there an angle to be mesured?) to be able to detect the objects but not the projection? I appreciate your help.

• Can you show us an example of what this should look like, just to ensure we're on the same page? Are you calculating your edges in image space (eg. using a Sobel operator) or as a collection of 3D points? Do you have a source of depth information? Jun 1, 2018 at 11:50
• It's a top-down projection and i'm using the canny operator for edge detection (i'll add images to explain)
– Safa
Jun 2, 2018 at 0:33
• I suspect that doing this is going to be very very hard as if the camera and projector are independent, even if stationary, the distortion can vary wildly. You'd need some kind of calibration setup where you use a fixed object target and "shine" a dot on it and adjust offsets until the dot sits on the center of the target. Repeat in n² different locations, where n > 1 (larger values increase accuracy). Jun 2, 2018 at 1:39
• I have another problem, when i put the camera on the same axis as the projector it keeps displaying the projected image infinitely... So how can i place the camera to be able to detect the objects but not the projection?
– Safa
Jun 2, 2018 at 10:43
• This is sounding less like a game development problem and more like computer vision & media installation art — something most of our game experts here might not know much about. You might have better luck elsewhere. One trick you can use though to reduce feedback is to use an infrared camera, to try to keep the spectrum of the projection and the spectrum of the recording from overlapping. Jun 2, 2018 at 11:43

This is going to be tough.

When you have a camera and a projector with an offset between them, there will be parallax between them, shifting the positions and projections of objects in the image plane.

You can compensate for shallow parallax to some degree by re-projecting your scene from a different point of view, but to do this you need to know the depth of each part of the surface away from the camera - something you don't normally get from a webcam.

If your objects are fairly flat and stay close to a fixed plane, you might be able to get away with pretending they're all in a flat plane, and re-project based on that.

For aligning the two, you can use the fact that one shines light and one sees light. Run a calibration step at the start of your session, where you use the projector to shine a calibration pattern onto the object plane.

(If you're using an IR camera to avoid feedback effects, take out the filter that blocks visible light for this step, or use the projected grid as a guide to make marks/place tokens that the camera can see)

For instance, you could project an image that's a dense, uniform grid of dots. By varying their intensity or flashing them one at a time, you can register which dot in the projected image corresponds to which dot in the captured image.

Now you have a series of point-to-point correspondences between coordinates in the camera's image plane and coordinates in the projection image plane.

You can run some fancy pose inference algorithms and lens distortion compensation algorithms on this to find an exact mapping between the two, but I'd suggest a more quick & dirty approach:

Construct a mesh, where each vertex's position corresponds to the position of one of your calibration points in projection space, and its texture coordinate corresponds to the position of that point in camera space. By displaying your edge-processed image feed on this mesh, using normal texture mapping, and capturing that appearance to shoot through the projector, you precisely match the distortion at every calibration point that was visible to the camera. The denser the mesh, the less interpolation error in-between.

Phones are generally terrible at knowing exactly where they are. GPSes have a couple of meters of error and the only other way you have is using the accelerometer, which gives you the second derivative of position, which makes it very vulnerable against small precision issues (of which there is a lot). You could technically do it the hard way by following points on the video feed and trying to guess the position based on the relative position to each other, but even video editing software has to stop for a second to calculate that.