How to determine if a 2D image represents what player sees in 3D game world?

I'm working on a game idea where the player is shown a 2D image (say, a "photo") and the player must find a spot in a 3D world where the view is about equivalent to that photo - not necessarily the exact same spot, but something that at least somewhat represents what is seen in the photo. How could that be approached?

To elaborate a bit, the player's objective is not to find the specific position and viewing direction in the game world. Rather, it is to find any spot in the world that matches the photo. For instance, if the photo is of a blue cube, then any view that represents a blue cube would be acceptable. Similarly, if the photo was of a triangle, any view that ends up representing a triangle (say, a pyramid viewed from a distance) would be OK.

As already discussed in the answers so far, a single correct point would be rather easy to do, or even if there were multiple possible solutions. However, if anything goes as long as it represents the photo, things seem to get a lot more difficult. I'd guess it boils down to processing the 3D view and the photo somehow and do a percentage matching between these two images to determine if they are alike..?

• This isn't enough to be an answer but you should look at The Witness as an example of the single correct point that you mention. From what I remember, there's something where when the user is close, there's an outline that the user can see that gets bolder the more correct the user is. Jul 13, 2021 at 19:07
• It feels like this needs to be fleshed out a bit more... in the case of the blue cube, does a blue square satisfy the request? How about a blue hexagon? That's just a cube viewed from a certain angle. How blue does it need to be? Might lighting matter? Games are all about clever fakery; actual computer vision seems unlikely to be the right solution, unless you intend for that to be "the thing" that sets your game apart. And even then, you'll need to define what qualities are important to "a match". The OpenCV docs might help you pin things down (also check out SIFT algorithm)
– A C
Jul 14, 2021 at 6:02
• First you build a neural-network, then add a few thousand layers to its hidden-layer, then train it with a few million sample-sets, then... Jul 14, 2021 at 19:51
• Just for clarification: The photo is basically the rendering of your scene from a specific viewpoint, so I'd assume that you have the generation process under full control. If you are interested in some specific objects in the image, why not store which objects are of interest alongside the photo and check for their visibility in the current user view rendering? This avoids the problem altogether. Jul 15, 2021 at 7:14

To handle a symmetrical object like a cube that could be photographed from many different directions, you could try comparing silhouettes: Render the quest object's silhouette and match that against the current camera view.

Create the quest

Use a render texture, set your camera to clear with transparent and to only renders that one object, render. Convert the render texture data to the quest silhouette by storing all transparent pixels as 0 and nontransparent as 1. (To ignore lighting and shading on the quest object.) Now set your camera to normal and render. Now you have the quest texture.

Check if quest is satisfied

Query for the quest object in the camera frustum. If so, repeat the above steps to get the silhouette. Compare the resulting silhouette against the quest silhouette with some amount of tolerance (are 90% of pixels the same). You might need to tune a different tolerance per quest object.

If you're picking a random object in the scene, you could do both the create and check steps with rendering all quest objects. That would allow background duplicates of the quest object to be in the photo and you wouldn't have to query for the quest object. However, it might make the gameplay feel uneven -- background objects might make the positioning more finicky.

• I think the silhouette method is the way to go, it should match my use case quite nicely. Jul 13, 2021 at 18:29
• I would not recommend doing any image space tests or pixel manipulation if your images to match consist of something as simple as a cube in a specific location/angle relative to the camera. This is something you can solve in a resolution-independent way with a little geometry math, no rasterization or large array traversal required. Jul 15, 2021 at 1:47
• The problem with this (it is in other answers too) is that it would force the player to specify the direction of view along with the position (maybe even in three dimensions, with height of the direction), if you don't want to brute force all possible views from one point. Note that a quite small move of the direction can make the image recognition harder Jul 15, 2021 at 7:35
• @DMGregory: Do you mean using math to find symmetrical camera positions and then checking for correct player position like TomTsagk's answer? I agree that solution seems more reliable method and likely easier to tune for fun gameplay at the cost of more up-front setup (defining symmetry planes). However, computing symmetry seems challenging if the quest objects are player-defined? Jul 15, 2021 at 21:40

I would say it depends a lot if you have one spot for every "photo" or if there are several possible place where the photo could be used.

If you got one spot for every photo, you can simply check for the coordinates and the players viewing direction. Than you can add some tolerance to both and that should be enough.

But if you got several places where this photo could fit or the player has to place objects first (lamp on the table in the photo, but on the floor right now), so the photo does fit, then i would go another way. Similar to what TomTsagk said, create a scene from that with a camera perspectiv. You could render it now and use that as the photo or you could implement it like a frame to this scene. Basically you hold the window to the scene in your hand. As long as you keep the scene, its all good.

Now you can check, if the scene of the picture and the scene infront of the player match. Are the objects at the same position, are the angles the same?.

Another way would be to implement some sort of raytracing. Do the objects in the photo match the objects infront of the player.

• I thought of the first one as well. If there was only a single point in the game world for the picture, it would be rather easy to match against the position and the view direction. Also, that could be done like you guys mentioned with another camera. However, what if the correct solution can be found from a number of places? Let's say the photo is of a blue cube, and any blue cube in the world could match it. I guess it would come down to some image manipulation and comparison of how similar the two images are. Jul 13, 2021 at 13:59
• Unlikely, @manabreak. You'd instead tag all the blue cube objects in your world, and use a volume query to check whether one of those tagged objects is inside a region of interest relative to the player's camera frustum, and optionally apply other tests like is its relative orientation within your tolerance range. That sidesteps the computer vision problem entirely. Jul 13, 2021 at 14:17

This might look different depending on the tools you are using, but here is how I would approach this.

Getting the Texture

In the scene where the place of the photo exists, create a camera object, that is only available to you (and other developers), but not the end user. When needed, take a screenshot of what that camera sees, and use that as a 2D texture. Optionally you can edit this texture with an image manipulation tool, depending on the needs of your project.

During development, if the scene changes (things move around, lighting, etc), you can easily re-take a screenshot using the same camera, and simply replace the previous texture with the new one, without extra work. You can even re-position the camera if needed.

This can work with multiple such places that need to exist in photos. Its only drawback being that it doesn't really scale, unless you can develop a way to take screenshots of hundreds of such cameras with the press of a button, but that might be out of scope of this question.

Finding out if the player is in the correct position

Having the texture ready, you would then need to find out if a user is looking at the spot in the photo. The implementation will depend on the tools you are using (Game engine / Libraries).

The general idea is you would need to compare the player's camera transformation, to that of the set camera's transformation for each place that needs to be looked at.

The easiest way would be to compare the distance between the two cameras, and the difference in their rotation, and if they are small enough that means the player is (roughly) looking at the same spot as the photo.

• To determine whether the player is standing in the right spot, you'd do some type of comparison between their current camera transformation and the placed camera object's transformation, correct? What would this comparison look like? Jul 13, 2021 at 10:31
• @DMGregory Thanks, that's a good question. I've updated the answer to include more details. I didn't add to many details on how to do the comparisons, as I feel it might be a bit off topic, and something easily searched online, depending on the tools used. Feel free to edit the question if you feel more details are needed. Jul 13, 2021 at 11:10
• This seems like a good start. However, if there's lots of places that could match the photo, it becomes rather impossible to use the "secondary scene / camera" approach... Jul 13, 2021 at 14:01
• If you have hundreds of blue cubes in the scene that are possible photo targets, then you could put the relative offset for the photo inside the cube (either make it a prefab or put data on the cube storing the offset and orientation). Then you can iterate over all cubes (near the player) and see if the player is in the right position+orientation for any of them. Jul 13, 2021 at 14:51
• @manabreak could you maybe give an example of how you picture this functionality is implemented? Is the user meant to take pictures of an object, or an entire scenery? Is there any existing video game that has a similar functionality? Jul 13, 2021 at 18:14

It depends on how much you want framing the shot to count

Basically, there are two ways you can look at this problem. You can either treat the player as someone trying to exactly recreate the scene from the photo, or you can treat it more like a scavenger hunt where the photo is just a "thing" you need to find. If you want the player to have to recreate the exact scene, then you want to use an Image Recognition API or SDK. However, if you just want to reward them based on how similar of a "thing" they can find, then you want to use meta-tag comparisons.

Option A: Image recognition API or SDK

Other programmers have already done the hard part for you. Depending on what API/SDK you go with, it could be as simple as taking a screenshot of your viewport to create a 2-image of what you are looking at, and comparing it to the photo the player is supposed to be looking for, and the function/API call should either return a true/false based on what certainty threshold you defined, or it will return the certainty threshold (usually a percentage).

If you go with one that returns the actual certainty threshold, then you could actually use that as a basis for scoring; so, if you get a really close matchup, you might get a 95% certainty, but if you just find something that is roughly the same shape and color, you still get credit, but less points if it is only a 60% certainty.

Personally I would recommend using some manner of exponential scoring if you do this to give more accurate shots significantly better scores... so a 60% certainty might yield 3600 points(60*60) whereas a 95% certainty would yield 9025 points (95*95)... just a suggestion

Option B: Meta-tag comparisons

If you don't care so much about matching the scenes as you do the properties of the object, a better approach would be to do meta-tag comparisons. Meta-tagging is any method of assigning hidden properties to an object that helps to define or label what it is. So if you have a photo of a teapot you could give it an associative array of properties that describe it, and a point value for each property that you match:

\$metaTags = {'teapot' => '50', 'cookware' => '15', 'black' => '15', 'spherical' => '10',  'metal' => '10'};


Then when you point your camera at an object you wish to match, you just do a simple hit-test to see what object is in your crosshairs and you then compare its tags to those on the photo. So, if you find a black sports car, you might get 25 points for matching 'metal' and 'black' or if you find a red teapot, you might get 85 points for matching everything but the color.

The reason this works better for scavenger hunts is because you just need to find the object. If you have a photo of a brown teddy bear facing the camera for example, using Image Recognition will give a very low score if you take a photo from its backside, but using tag comparison, the teddy bear could be upside-down and backwards and still give full points as long as all of its tags match the photo's tags.

Meta-tag comparisons can also be cheaper since they don't require paying for any third-party licenses.