I am using .pngs for my textures and am using a virtual file system in a .zip file for my game project. This means my textures are compressed and decompressed twice. What are the solutions to this double compression problem? One solution I've heard about is to use .tgas for textures, but it seems ages ago, since I've heard that. Another solution is to implement decompression on the GPU and, since that is fast, forget about the overhead.
\$\begingroup\$
\$\endgroup\$
1
-
2\$\begingroup\$ Related: if you used jpegs, then a lot of zip programs with their own format are able to compress even those by 20% so its not necessarily that bad. What is the point in the double compression that you worry most about? Does it take too long? A lot of formats even store already compressed data just verbatim and don't do any decompression on them at all. \$\endgroup\$– PlasmaHHCommented May 27, 2014 at 10:42
Add a comment
|
4 Answers
\$\begingroup\$
\$\endgroup\$
6
The zip-format supports several different compression algorithms. You can use a different algorithm for each file in the archive. When you want to store already compressed files which do not benefit from additional compression (like PNG) in a zip-archive, you can encode these files with the "stored" algorithm which doesn't compress at all. The "Add to archive" dialog of 7-zip lets you chose this under "Compression strength".
But when you do not only have images but also other, more compressible resources in your archives, it might be quite tedious to choose the algorithm for every single file. In that case you might rather opt for an uncompressed image format in a compressing archive.
The TGA format knows a lot of different modes, of which some are compressed and some are not. When you do not want to use compression, make sure you pick the right one in the export options of the graphic editor you are using. Another non-compressing image format is BMP (Windows Bitmap).
Here is a test I made. I added the same image (an asset from my current project) in different formats multiple times to a zip-archive, some with "deflate"-algorithm on normal strength and one with "store". Sorry for the German GUI. 2nd column is uncompressed size, 3rd column is compression algorithm and 4th column is compressed size.
As you can see, deflate-encoding the PNG only saved a meager 0.3%, while the deflate-encoded BMP is reduced to one-tenth of the original file which is even smaller than the PNG version. This quite surprised me. I would have expected the PNG to be smaller because the compression method of PNG should be optimized for image-data while ZIP is not. A likely explanation is that my image editor (GIMP) added quite a lot of meta-information to PNG files which it doesn't do for BMP.
Uncompressed TGA behaved similar to BMP regarding filesize before and after zipping while the compression of the compressed TGA file was further improved by ZIP, although not as much as the uncompressed versions.
It might be worth to experiment with other algorithms than deflate and with other compression-strength setting. Which combination will have the best results will likely depend on the style of your textures. But you might also consider to benchmark the asset-loading of your game and have the decompression-performance influence your decision which setting you use.
Bottom-line: When you want to avoid double-compression while still having a low filesize, either use PNG with Store zip algorithm or BMP with a compressing zip algorithm.
-
5\$\begingroup\$ I did it on some pictures of mine and PNG (compression level 9) beat zip (compression level ultra) of BMP images all the time by about 20%. So it should be an effect like meta-information. \$\endgroup\$ Commented May 27, 2014 at 9:07
-
3\$\begingroup\$ png has an interlacing option with decreases compressibility but allows a low res image to be extracted from just the first part of the image file (like when downloading over a low bandwith connection), did your wall have that active? besides that png uses deflate for its compression already. \$\endgroup\$ Commented May 27, 2014 at 10:12
-
\$\begingroup\$ It seems like you have used a poor png compressor, give your files a run through PNGOUT and you should get a result that will not be beaten by a zipped bmp file. \$\endgroup\$ Commented May 27, 2014 at 16:02
-
\$\begingroup\$ Are your BMPs or TGAs 24-bit or 32-bit? With BMP in particular they're more likely to be 24-bit, and the uncompressed TGA size suggests that's also 24-bit. You're probably not comparing like-with-like here. \$\endgroup\$ Commented May 27, 2014 at 21:31
-
\$\begingroup\$ @JimmyShelter Both the TGA and the BMP are 32bit with alpha channel - I made sure. \$\endgroup\$– PhilippCommented May 27, 2014 at 22:13
\$\begingroup\$
\$\endgroup\$
3
Don't worry about it.
When the "deflate" algorithm used in .zip files encounters a block of data that's already well-compressed, like the pixel data of a .png image, it finds that it can't compress it effectively, and it will store it as literal uncompressed data. This takes very little overhead on the decompression end to copy out.
-
3\$\begingroup\$ Sometimes it is that simple, a perceived problem isn't necessarily a real problem. Even if there actually was the overhead of decompressing twice, uncompressing deflate is a really quick operation, I guess the overhead would only just be measurable. \$\endgroup\$ Commented May 27, 2014 at 18:27
-
\$\begingroup\$ Would the compression end take a long time to realise that the file isn't compressible? \$\endgroup\$ Commented May 28, 2014 at 12:07
-
\$\begingroup\$ Relatively, yes. I don't know what heuristics typical .zip compressors use to choose a block's encoding, but it might be as straightforward as trying all the encodings and keeping the best result. During development, if you must have your data in a .zip, you'll probably want to force everything to use
storeinstead ofdeflateto save time, thendeflateeverything for release builds. \$\endgroup\$ Commented May 28, 2014 at 13:46
\$\begingroup\$
\$\endgroup\$
5
Looks like some very good answers have been given already, but I thought I'd give one more:
What are the solutions to this double compression problem?
Solution: Do Nothing.
Rationale: There has been no problem stated - yes you are compressing the information twice, but why are you worried about this? Is the data size too big? Is decompression too slow? Is this more or less important than the dozens of features you could be adding, refining,testing and/or debugging at this moment in time?
-
\$\begingroup\$ Doing something twice unnecessarily is a problem in itself. At least I've perceived it as such. But, of course, you're right with your suggestion as well. Both memory and decompression speed can be a problem, albeit not in this case, as I am not writing a commercial game. \$\endgroup\$ Commented May 28, 2014 at 6:41
-
\$\begingroup\$ I'm a professional game dev, and I can assure you unless there were real, documented performance issues we'd not be wasting a second worrying about this - it takes a lot of $0.99 purchases or ad impressions to pay for the say, 8 hours to properly 'rectify' this issue. Doing the something twice is not a problem, though it could be inefficient. Though in this case you're not doing the same thing twice - you've got image data that is being stored as a PNG for compression and a whole bunch of files that are being archived together. \$\endgroup\$– NPSF3000Commented May 28, 2014 at 8:51
-
\$\begingroup\$ True, but once it's done it's done for multiple projects. If you reflect on the DEFLATE algorithm - it has stayed the same since the eighties. You'd be an old gramps, if you programmed then, but , if you knew the algorithm, you could, say, implement it on the GPU and enjoy super-fast decompression speeds across multiple projects. \$\endgroup\$ Commented May 28, 2014 at 10:13
-
\$\begingroup\$ So you spend, what 2 weeks, a month implementing Deflate on GPU? And then you realize that you're just about to deploy on platform X than does Y and breaks your algorithm via Z. If you want to make games, FOCUS ON MAKING GAMES, don't worry about unnecessary things like deflate performance. \$\endgroup\$– NPSF3000Commented May 28, 2014 at 11:19
-
\$\begingroup\$ Once you know an algorithm (re)implementing it should be easy. A snap, not 2 weeks or a month. That's what I wanted to say. It's been around since the eighties, investing some time into it might be worthwhile. \$\endgroup\$ Commented May 28, 2014 at 12:03
\$\begingroup\$
\$\endgroup\$
3
If you use specialized formats such as PNG and OGG you won't need the compression of ZIP.
PNG, OGG and other already compressed formats won't get much smaller by compressing them again as ZIP. 100MB of PNGs compressed are still ~100MB.
Scripts, Configuration files and other text based formats benefit greatly from compression, however, typically they are tiny in comparison, they don't store that much data. If your game is 100MB, then the text-files maybe make 1MB of the whole game, even if you can make them 100KB through compression then you only won 900KB, less than 1%, hardly worth the effort.
You might even want to use the file system directly instead of using a virtual, zip based file system. It would make patching very easy: you can just swap out any files you modified.
-
2\$\begingroup\$ Sometimes one does not have a choice, as regards whether to use a
.zipfile or not. For example, on the android platform, the.apkis a.zipfile, which may contain resources. \$\endgroup\$ Commented May 27, 2014 at 9:48 -
3\$\begingroup\$ Plus, there are definite advantages to using an archive, and ZIP files provide both compression and an archive structure. \$\endgroup\$– MrCrankyCommented May 27, 2014 at 10:28
-
\$\begingroup\$ Yes I know, I am just saying that using the native file system has advantages too. I am a bit of a "Keep it simple" advocate. \$\endgroup\$ Commented May 27, 2014 at 10:38