11
\$\begingroup\$

The SmoothVideo Project uses frame interpolation to increase the fps of video from 24 to 60. The results are pretty impressive. I was wondering if this could be applied to, and whether it would look good in video games?

It uses much less resources than rendering all the frames so would allow lower end rigs to render at the quality of much better rigs at some level of compromise. I know it won't be as accurate, and would slightly increase input latency as it needs to hold on to the newest frame to be able to generate and insert the interpolated one. It's not as bad as a full frame though, by my reasoning only the lag would be the interpolation time plus half the original fps refresh time. So for 30 fps it would be 33ms/2 + interpolation time.

Maybe this lag would make it unsuitable for fast past first person games, but I doubt it would be a hindrance in slower paced games.

The lag becomes lower at higher start rates, so I would think it would be certainly worth it when going from 60fps to 100+fps which improves the experience though increasingly marginally, while being extreme taxing on the system.

\$\endgroup\$
1
  • \$\begingroup\$ In some regard this is already being done with motion blur. The lack of derived motion data (described in some answers) is solved by using already known objects' motions. The only difference is that instead of interpolating smoothly, renderers make several interpolations and create a mix of them. Interpolation as you described is probably not used because occlusion changes and it creates an undesirable result. \$\endgroup\$
    – transistor09
    Commented Jul 8, 2015 at 12:39

4 Answers 4

Reset to default
6
\$\begingroup\$

A system along these lines has been used in The Force Unleashed. I'm not aware of other titles that have used it though.

\$\endgroup\$
5
  • 1
    \$\begingroup\$ Thanks for the info. Found a publication on it: dl.acm.org/citation.cfm?id=1837047 It seems successful, so why hasn't it seen wider use? \$\endgroup\$
    – cybrbeast
    Commented Jul 8, 2015 at 4:10
  • \$\begingroup\$ Mostly for the reasons outlined in Byte56's answer: it's not without drawbacks and in many cases those drawbacks aren't worth it, given that there are other ways to achieve higher frame rates as well. \$\endgroup\$
    – user1430
    Commented Jul 8, 2015 at 4:17
  • 1
    \$\begingroup\$ According to this article by using predictive interpolation that method can actually reduce latency! eurogamer.net/articles/… So it seems better in all metrics. But something must be missing as you'd think it would be used everywhere if this was true. \$\endgroup\$
    – cybrbeast
    Commented Jul 8, 2015 at 4:18
  • \$\begingroup\$ @David: the article means that it reduces apparent latency compared to running at 30 fps, for certain types of input. Running at 60 fps natively is still preferable when feasible (better latency and no interpolation artifacts), so many devs consider that their first-choice target. When games fail to hit 60 fps, it's not always with enough time or budget left to author an interpolation system to smooth the gaps - this system is quite complicated and, in TFU's case, is aided by some facets of their rendering pipeline which are not universally shared by all games. \$\endgroup\$
    – DMGregory ♦
    Commented Jul 8, 2015 at 9:02
  • \$\begingroup\$ I was trying to say better in all metrics compared to 30 fps unenhanced, not rendered 60fps \$\endgroup\$
    – cybrbeast
    Commented Jul 8, 2015 at 16:01
9
\$\begingroup\$

Yes it's possible, but it's not without its complications.

While frame interpolation can work real-time on videos, that isn't necessarily the case with video games. Even though this is processing real-time on videos, the software is able to "look ahead" to the next frame. This is a pretty critical component of interpolation. This is where the issue comes into play with games. Most of the time, next next frame has not been rendered yet! So the software doesn't know the next point in the interpolation.

There's certainly the possibility of running the game a frame or two behind in order to give your interpolation software the frames it needs to work with. However, this doesn't work as well for media that is interactive. Now what's being shown on screen is delayed from the input being received. This does make things more tricky for processing input and providing a good responsiveness to the game. It's like building in an artificial performance lag. Additionally, it's unlikely to have native support in any of the major game engines, which means writing your own.

\$\endgroup\$
3
  • \$\begingroup\$ Any reaction to this smarter interpolation method mentioned below , which claims to prevent any increase in latency? eurogamer.net/articles/… \$\endgroup\$
    – cybrbeast
    Commented Jul 8, 2015 at 5:29
  • \$\begingroup\$ In that case it's primarily the final sentence that applies. I'm not sure how much their technology abstracted away that feature. It may have taken significantly more work to not break the interpolation, or it could have "just worked". I imagine if it was the latter, we might see it show up in some of the major engines (if it's not patented). I'm not sure this feature alone is worth creating and maintaining your own engine. \$\endgroup\$
    – House
    Commented Jul 8, 2015 at 5:57
  • \$\begingroup\$ Unsure of the claim that running a frame behind is a major obstacle for interactivity. Any multiplayer games using an authoritative server w/ (game state) interpolation between the last two updates will often show a greater lag, yet this includes some of the fastest-paced shooter games. We also don't necessarily need the next frame to construct intermediates in games, because unlike in video, games can generate ground-truth object masks & screenspace velocity information (similar to motion blur effects) which can be extrapolated. \$\endgroup\$
    – DMGregory ♦
    Commented Jul 8, 2015 at 8:52
2
\$\begingroup\$

Yes, this is not only possible, but available now: hook up your game PC/console to a TV that uses motion interpolation. Opinions vary, and this is less suitable for twitch games like FPS due to the interpolation lag, but for upscaling frame rates from 60 to 120Hz it works fine.

As for whether this can be done in the game itself, there's not enough impetus yet, in that most monitors can't output those high frame rates. 120+Hz monitors for computers are less common, although judging from how TVs are going, this may happen soon. Having a high refresh rate monitor has advantages even if the game can't reach those frame rates: besides the aforementioned TV-based motion interpolation, it can offer smoother frames if the game is allowed to output frames as they are ready, instead of synced up ala v-sync. Once 120+Hz monitors are commonplace, I expect game developers to catch up and start using more tricks, including motion interpolation, to reach those high frame rates.

\$\endgroup\$
1
  • 3
    \$\begingroup\$ My experience has been that these systems have tremendous input latency, like 200+ms (12+ frames at 60fps). This is the reason those TV's typically have a "gaming mode" that disables that feature. \$\endgroup\$
    – BlueRaja - Danny Pflughoeft
    Commented Jul 8, 2015 at 7:54
0
\$\begingroup\$

The lag between when the user does something and when the result appears on the screen should never exceed 100ms or the user might notice the delay.

The monitor of the user might take about 30 ms to display the picture it receives. Much faster monitors are available, but many average users don't have these. If the video card calculates 30 frames per second, it takes 33 ms to calculate a frame. I assume 30 fps because we need no frame interpolation if the game already runs at 60 fps. If we use triple buffering this gets doubled to 66 ms. That's 90 ms.

I think we need triple buffering for frame interpolation to make sense. So if we use frame interpolation on top of that to get 60 fps, we increase the delay by one 60Hz frame, which is another 17 ms + interpolation time X, bringing us to 107 ms + X. The problem isn't the interpolation as such, but the fact that we are already close to the point where latency is noticeable before we introduce the interpolation.

It would probably be fine for a game that's mostly a virtual movie, but in a FPS the users would notice that something is off with the aiming.

\$\endgroup\$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .