Why do console game developers use large files to store game data instead of small files like on PC? Less streams in memory? Need to access file tree many times? Other reason?
\$\begingroup\$
\$\endgroup\$
4
-
\$\begingroup\$ Perhaps an analog in the web world: CSS Sprites \$\endgroup\$– Nick TCommented Nov 26, 2011 at 23:53
-
\$\begingroup\$ CSS sprites is a nasty downside of still using that old rusty HTTP 1.1 which apparently makes servers unable to send many resources with a single connection. I wish for a new standard. \$\endgroup\$– KosCommented Nov 27, 2011 at 9:57
-
1\$\begingroup\$ @Kos Your wish has been granted, although unfortunately not yet implemented. \$\endgroup\$– phihagCommented Nov 27, 2011 at 15:26
-
\$\begingroup\$ Thanks for this! I want to bake them a cake so that they'll work faster. \$\endgroup\$– KosCommented Nov 27, 2011 at 15:43
Add a comment
|
2 Answers
\$\begingroup\$
\$\endgroup\$
4
Having a single big file has a few benefits for performance, regardless of platform. Many PC games use big files too.
You can manually cache the offset of each file within the big file, to avoid the need to read file system directory entries (which can involve multiple seeks per file opened if you miss the cache).
You get full control of the order of the data within the file, so you can arrange it to minimize the need for slow seeks.
Data compression is generally more effective on a single big file, than on individual files.
Note that avoiding seeks is very effective at speeding up load times. A typical hard drive might have a 10ms seek time, and a 100MB/s transfer rate. Let's say you need to load 100 shader files with a total size of 1MB. If you have to do a seek for each one it would take one second to load them all. With a single seek you're looking at 20ms to load all of them.
Of course not all consoles have hard drives, and optical drives have even worse seek performance, so those shaders might take 5 or more seconds to load off a DVD when seeking. Also consoles have much less memory for use as cache than a PC does, so they tend to load data more often.
-
9\$\begingroup\$ Very nice answer. To the point, with examples of seeking one file vs multiple files. \$\endgroup\$– WilliamCommented Nov 26, 2011 at 18:47
-
1\$\begingroup\$ Additionally, for competitive online games, it's much, much faster (and uses a lot less disk space) to verify that the player hasn't tampered with a single huge file, than it is to individually validate thousands of small files. \$\endgroup\$ Commented Nov 27, 2011 at 2:09
-
\$\begingroup\$ Google keyword: virtual file system. Also the well-stated need for virutal file systems seems to indicate that non-virtual file systems could use some more polishing :(. \$\endgroup\$– KosCommented Nov 27, 2011 at 9:58
-
\$\begingroup\$ However this is bad for those of us using SSDs - we can't move the rarely-loaded stuff onto HDDs, because all the game data is inside a big file. Say for WoW I never ever go to Outland; I'd just move that file off the SSD. Not so doable for say... CoD 4. \$\endgroup\$ Commented Nov 27, 2011 at 11:02
\$\begingroup\$
\$\endgroup\$
1
Apart from very good answer given by Adam I would also like to point out some other important factors:
The fewer files you have the easier is the deployment of your application. This is especially true if you want to deploy your application on mobile platforms, like iOS, Android or Windows Phone. Even if in case of Windows Phone you still can have as many files as you want, since XAP-files, used for deployment, are just a zip archive, it is
Having all your assets in one single files ensures that they are all downloaded/read "in piece". Many gamers will excuse you a noticeable delay for downloading assets or unpacking them before the game starts, but they will be non-indulgent as long they face even subtle pauses during the game. Of course you can pre-cache your resources in RAM, but loading them "in piece" is still easier for this purpose than loading them on per-file basis.
-
\$\begingroup\$ I have done Android and BlackBerry development and never ran into any problem with having too many small files. I was unaware of the benefit of larger files until now however... \$\endgroup\$– ADBCommented Nov 28, 2011 at 23:15