Take the 2-minute tour ×
Game Development Stack Exchange is a question and answer site for professional and independent game developers. It's 100% free, no registration required.

I've noticed extracting PNGs from some game files that the image becomes distorted part-way through. For example, here are a couple of PNGs extracted from the Textures file in Skyrim:

Illuminated J PNG from Skyrim Illuminated K PNG from Skyrim

Is this some unusual variation on a PNG format? What modifications would I need to make to view such PNGs properly?

share|improve this question
1  
Perhaps they've put in some special encoding into their files to prevent people from doing stuff like this. Or maybe whatever you're using to extract isn't work properly. –  Richard Marskell - Drackir Nov 26 '11 at 23:44
    
Maybe it's a kind of compression to make the images smaller in filesize. This is also done in iPhone apps. –  rightføld Nov 27 '11 at 0:05
1  
Little bit off topic, but is that a pony? –  jco Nov 28 '11 at 15:15

4 Answers 4

up vote 22 down vote accepted

Here are the “restored” images, thanks to tillberg's further research:

final1 final2

As expected, there is a 5-byte block marker every about 0x4020 bytes. The format appears to be the following:

struct marker {
    uint8_t tag;  /* 1 if this is the last marker in the file, 0 otherwise */
    uint16_t len; /* size of the following block (little-endian) */
    uint16_t notlen; /* 0xffff - len */
};

Once the marker has been read, the next marker.len bytes form a block that is part of the file. marker.notlen is a control variable such that marker.len + marker.notlen == 0xffff. The last block is such that marker.tag == 1.

The structure is probably as follows. There are still unknown values.

struct file {
    uint8_t name_len;    /* number of bytes in the filename */
                         /* (not sure whether it's uint8_t or uint16_t) */
    char name[name_len]; /* filename */
    uint32_t file_len;   /* size of the file (little endian) */
                         /* eg. "40 25 01 00" is 0x12540 bytes */
    uint16_t unknown;    /* maybe a checksum? */

    marker marker1;             /* first block marker (tag == 0) */
    uint8_t data1[marker1.len]; /* data of the first block */
    marker marker2;             /* second block marker (tag == 0) */
    uint8_t data2[marker2.len]; /* data of the second block */
    /* ... */
    marker lastmarker;                /* last block marker (tag == 1) */
    uint8_t lastdata[lastmarker.len]; /* data of the last block */

    uint32_t unknown2; /* end data? another checksum? */
};

I haven't figured out what is at the end, but since PNGs accept padding, it's not too dramatic. However, the encoded file size clearly indicates that the last 4 bytes should be ignored...

Since I did not have access to all block markers just before the beginning of the file, I wrote this decoder that starts at the end and attempts to find the block markers. It's not robust at all but well, it worked for your test images:

#include <stdio.h>
#include <string.h>

#define MAX_SIZE (1024 * 1024)
unsigned char buf[MAX_SIZE];

/* Usage: program infile.png outfile.png */
int main(int argc, char *argv[])
{
    size_t i, len, lastcheck;
    FILE *f = fopen(argv[1], "rb");
    len = fread(buf, 1, MAX_SIZE, f);
    fclose(f);

    /* Start from the end and check validity */
    lastcheck = len;
    for (i = len - 5; i-- > 0; )
    {
        size_t off = buf[i + 2] * 256 + buf[i + 1];
        size_t notoff = buf[i + 4] * 256 + buf[i + 3];
        if (buf[i] >= 2 || off + notoff != 0xffff)
            continue;
        else if (buf[i] == 1 && lastcheck != len)
            continue;
        else if (buf[i] == 0 && i + off + 5 != lastcheck)
            continue;
        lastcheck = i;
        memmove(buf + i, buf + i + 5, len - i - 5);
        len -= 5;
        i -= 5;
    }

    f = fopen(argv[2], "wb+");
    fwrite(buf, 1, len, f);
    fclose(f);

    return 0;
}

Older research

This is what you get when removing byte 0x4022 from the second image, then by removing byte 0x8092:

original first step second step

It doesn’t really “repair” the images; I did this by trial and error. However, what it tells is that there is unexpected data every 16384 bytes. My guess is that the images are packed in some kind of filesystem structure and the unexpected data is simply block markers that you should remove when reading the data.

I don’t know where exactly the block markers are and their size, but the block size itself is most certainly 2^14 bytes.

It would help if you could also provide a hex dump (a few dozen bytes) of what appears right before the image and right after. This would give hints about what kind of information is stored at the beginning or end of the blocks.

Of course there’s also the possibility that there is a bug in your extraction code. If you are using a buffer of 16384 bytes for your file operations, then I would first check there.

share|improve this answer
    
+1 very helpful; I'll continue to dig into this with the lead you've given me and post some additional info –  James Tauber Nov 27 '11 at 20:26
    
The embedded "file" begins with a length-prefixed string containing the file name; followed by 12 bytes before the 89 50 4e 47 magic for PNG files. The 12 bytes are: 40 25 01 00 78 9c 00 2a 40 d5 bf –  James Tauber Nov 28 '11 at 2:28
    
Nice work, Sam. I updated the python code that actually reads the BSA files directly to do the same. The results are visible at orbza.s3.amazonaws.com/tillberg/pics.html (I'm showing only 1/3 of the images there, just enough to demonstrate the results). This works for many of the images. There are some other things going on with some of the other images. I'm wondering if this has been solved elsewhere re Fallout 3 or Skyrim, though. –  tillberg Nov 28 '11 at 14:18
    
Excellent work, guys! I'll update my code too –  James Tauber Nov 28 '11 at 15:28

Based on Sam's suggestion, I forked James' code at https://github.com/tillberg/skyrim and was able to successfully extract n_letter.png from the Skyrim Textures BSA file.

The letter N

The "file_size" given by the BSA headers is not the actual final file size. It includes some header info as well as some random chunks of useless-seeming data scattered around.

The headers look something like this:

  • 1 byte (length of file path?)
  • the full path of the file, one byte per character
  • 12 bytes of unknown origin, as James posted (40 25 01 00 78 9c 00 2a 40 d5 bf).

To strip off the header bytes, I did this:

f.seek(file_offset)
data = f.read(file_size)
header_size = 1 + len(folder_path) + len(filename) + 12
d = data[header_size:]

From there, the actual PNG file starts out. It's easy to verify that from the PNG 8-byte start sequence.

I proceeded to try to figure out where the extra bytes were located by reading the PNG headers and comparing the length passed in the IDAT chunk to the implied data length inferred from measuring the number of bytes until the IEND chunk. (for details on that, check out the bsa.py file at github)

The sizes given by the chunks in n_letter.png are:

IHDR: 13 bytes
pHYs: 9 bytes
iCCP: 2639 bytes
cHRM: 32 bytes
IDAT: 60625 bytes
IEND: 0 bytes

When I measured the actual distance between the IDAT chunk and the IEND chunk after it (by counting bytes using string.find() in Python), I found that the actual IDAT length implied was 60640 bytes - there were an extra 15 bytes in there.

In general, most of the "letter" files had an extra 5 bytes present for every 16KB of total file size. For example, o_letter.png, at around 73KB, had an extra 20 bytes. Larger files, like the arcane scribblings, mostly followed the same pattern, though some had odd amounts added on (52 bytes, 12 bytes, or 32 bytes). No idea what's going on there.

For the n_letter.png file, I was able to find the correct offsets (mostly by trial and error) at which to remove the 5-byte segments.

index = 0x403b
index2 = 0x8070
index3 = 0xc0a0
pngdata = (
  d[0      : (index - 5)] + 
  d[index  : (index2 - 5)] + 
  d[index2 : (index3 - 5)] + 
  d[index3 : ] )
pngfile.write(pngdata)

The five byte segments removed are:

at 000000: 00 2A 40 D5 BF (<-- included at end of 12 bytes above)
at 00403B: 00 30 40 CF BF
at 008070: 00 2B 40 D4 BF
at 00C0A0: 01 15 37 EA C8

For what it's worth, I've included the last five bytes of the unknown 12-byte segment because of some similarity with the other sequences.

It turns out they're not quite every 16KB, but at ~0x4030 byte intervals.

To guard against getting close-but-not-perfect matches in the indices above, I also tested zlib decompression of the IDAT chunk from the resulting PNG, and it passes.

share|improve this answer
    
the "1 byte for a random @ sign" is the length of the filename string, I believe –  James Tauber Nov 28 '11 at 4:37
    
what is the value of the 5-byte segments in each case? –  James Tauber Nov 28 '11 at 5:35
    
I updated my answer with hex values of the removed 5-byte segments. Also, I had mixed myself up over the number of 5-byte segments (I was previously counting the mysterious 12-byte header as 7 bytes header and 5 bytes repeating divider). I fixed that, too. –  tillberg Nov 28 '11 at 6:24
    
note that (little-endian) 0x402A, 0x4030, 0x402B appear in those 5-byte segments; are they the actual intervals? –  James Tauber Nov 28 '11 at 8:08
    
I thought I'd already said this was excellent work, but apparently I didn't. Excellent work! :-) –  Sam Hocevar Nov 28 '11 at 15:21

Actually, the intermittant 5 bytes are part of the zlib compression.

As detailed on http://drj11.wordpress.com/2007/11/20/a-use-for-uncompressed-pngs/,

01 the little endian bit string 1 00 00000. 1 indicating the final block, 00 indicating a non-compressed block, and 00000 are 5 bits of padding to align the start of a block to on octet (which is required for non-compressed blocks, and very convenient for me). 05 00 fa ff The number of octets of data in the uncompressed block (5). Stored as a little-endian 16-bit integer followed by its 1′s complement (!).

.. so a 00 indicates a 'next' block (not an ending one), and the 4 next bytes are the block length and its inverse.

[Edit] A more reliable source is of course RFC 1951 (Deflate Compressed Data Format Specification), section 3.2.4.

share|improve this answer

Is it possible that you're reading the data from the file in a text mode (where line endings that happen to appear in the PNG data are possibly mangled) instead of in a binary mode?

share|improve this answer
1  
Aye. That sounds a lot like the issue. Considering this is the code that reads it: github.com/jtauber/skyrim/blob/master/bsa.py --- confirmed :-) –  Armin Ronacher Nov 27 '11 at 0:16
    
Nope, makes no difference. –  James Tauber Nov 27 '11 at 4:14
    
@JamesTauber, if you really are coding your own PNG loader as Armin's comment seems to imply, then (a) does it work on other PNGs you've tried, and (b) does a proven PNG loader such as libpng read the Skyrim PNGs? In other words, is it just a bug in your PNG loader? –  Nathan Reed Nov 27 '11 at 6:57
    
@NathanReed all I'm doing is extracting the byte stream and uploading it here; there is no "loader" involved –  James Tauber Nov 27 '11 at 7:40
3  
-1, this can not be the reason. If the PNG files were corrupted in this way, there would be CRC errors at the inflate stage far before the errors at the image decoding stage. Also, there are no occurrences of CRLF in the files apart from the expected one in the header. –  Sam Hocevar Nov 27 '11 at 11:01

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.