This is sort of a question following onto a previous question I wrote specifically targeting XNA. Understanding that it's an involved enough process that I'd want to manage memory myself, I'm now approaching from C++. But the language isn't directly relevant to the process...
For the game Ghost Trick: Phantom Detective, the developers found that the DS' 3D renderer wasn't fast enough to render the game's characters with the depth they wanted. So, since the whole game takes place from a side viewpoint, they pre-rendered the characters' many lengthy, acted-out animations, so the whole game is technically 2D. This wasn't the first time such a thing was done, but being a "drama", they tended to have longer and smoother animations than say, a top-down shooter.
What I'm targeting is something similar, but on the desktop; my game would basically consist of several static, basic sprites serving as backgrounds, and then characters that are rendered through FMV, with an Alpha color component (likely a 24fps video made through greenscreening, then converted to whatever suits the engine best). What may make my thinking different from Ghost Trick is that on a Desktop, the character sizes would be much bigger. Currently, I'm trying to think through what people may have tried for this approach previously.
- The thick-skulled straightforward approach would be to convert the video files into PNG spritesheets, in some kind of format that crops down to just a box that is non-transparent in all X and Y lines. If this is all in graphics memory at once, it would kind of add up though. If RGBA is 32 bits, an 1152x800 character's 4-second idle animation at 24fps would be 345 MB, if my math is right. One could of course save more memory by reducing character detail (ie, scaling a smaller video up) or by having a lower framerate.
- A hybrid approach of that might involve keeping the frames compressed in-memory, and only keeping about 5 to 10 uncompressed at a time. I feel kind of worried that this would keep the game going through a lot of CPU/GPU work while not very much is happening, though.
- Since only one of these characters is going to be onscreen at a time, my primary thought was to simply treat it as a video, and find a way to play it using the FFMPEG/LibAV libraries. It's obvious the FFMPEG developers don't have issues like keeping a 2-hour movie's frames in graphics memory, so it would certainly use better memory-management techniques. But, part of what makes this difficult for me is that I'm a simplistic Visual Studio developer who's had a lot of difficulty getting those libraries working; even being able to compile the example files after finding a pre-compiled Windows DLL. I'd also have to modify their standard operation so that videos can loop themselves, or switch video file on a dime.
Since this is an area I'm mostly figuring out for myself and I haven't found many resources on the topic, I was curious if anyone else had input in which I miss something obvious, or what they feel would be the most logical approach.
If it matters, I'm planning to try this project in either SDL or SFML. I'm a pretty experienced programmer who's just been avoiding C++ a bit longer than I should have, so I'm trying to dive back in for more practice with this project.