Why would some PNG files extracted from games display incorrectly?

Question

I've noticed extracting PNGs from some game files that the image becomes distorted part-way through. For example, here are a couple of PNGs extracted from the Textures file in Skyrim:

Is this some unusual variation on a PNG format? What modifications would I need to make to view such PNGs properly?

Answer 1

Here are the “restored” images, thanks to tillberg's further research:

As expected, there is a 5-byte block marker every about 0x4020 bytes. The format appears to be the following:

struct marker {
    uint8_t tag;  /* 1 if this is the last marker in the file, 0 otherwise */
    uint16_t len; /* size of the following block (little-endian) */
    uint16_t notlen; /* 0xffff - len */
};

Once the marker has been read, the next marker.len bytes form a block that is part of the file. marker.notlen is a control variable such that marker.len + marker.notlen == 0xffff. The last block is such that marker.tag == 1.

The structure is probably as follows. There are still unknown values.

struct file {
    uint8_t name_len;    /* number of bytes in the filename */
                         /* (not sure whether it's uint8_t or uint16_t) */
    char name[name_len]; /* filename */
    uint32_t file_len;   /* size of the file (little endian) */
                         /* eg. "40 25 01 00" is 0x12540 bytes */
    uint16_t unknown;    /* maybe a checksum? */

    marker marker1;             /* first block marker (tag == 0) */
    uint8_t data1[marker1.len]; /* data of the first block */
    marker marker2;             /* second block marker (tag == 0) */
    uint8_t data2[marker2.len]; /* data of the second block */
    /* ... */
    marker lastmarker;                /* last block marker (tag == 1) */
    uint8_t lastdata[lastmarker.len]; /* data of the last block */

    uint32_t unknown2; /* end data? another checksum? */
};

I haven't figured out what is at the end, but since PNGs accept padding, it's not too dramatic. However, the encoded file size clearly indicates that the last 4 bytes should be ignored...

Since I did not have access to all block markers just before the beginning of the file, I wrote this decoder that starts at the end and attempts to find the block markers. It's not robust at all but well, it worked for your test images:

#include <stdio.h>
#include <string.h>

#define MAX_SIZE (1024 * 1024)
unsigned char buf[MAX_SIZE];

/* Usage: program infile.png outfile.png */
int main(int argc, char *argv[])
{
    size_t i, len, lastcheck;
    FILE *f = fopen(argv[1], "rb");
    len = fread(buf, 1, MAX_SIZE, f);
    fclose(f);

    /* Start from the end and check validity */
    lastcheck = len;
    for (i = len - 5; i-- > 0; )
    {
        size_t off = buf[i + 2] * 256 + buf[i + 1];
        size_t notoff = buf[i + 4] * 256 + buf[i + 3];
        if (buf[i] >= 2 || off + notoff != 0xffff)
            continue;
        else if (buf[i] == 1 && lastcheck != len)
            continue;
        else if (buf[i] == 0 && i + off + 5 != lastcheck)
            continue;
        lastcheck = i;
        memmove(buf + i, buf + i + 5, len - i - 5);
        len -= 5;
        i -= 5;
    }

    f = fopen(argv[2], "wb+");
    fwrite(buf, 1, len, f);
    fclose(f);

    return 0;
}

Older research

This is what you get when removing byte 0x4022 from the second image, then by removing byte 0x8092:

It doesn’t really “repair” the images; I did this by trial and error. However, what it tells is that there is unexpected data every 16384 bytes. My guess is that the images are packed in some kind of filesystem structure and the unexpected data is simply block markers that you should remove when reading the data.

I don’t know where exactly the block markers are and their size, but the block size itself is most certainly 2^14 bytes.

It would help if you could also provide a hex dump (a few dozen bytes) of what appears right before the image and right after. This would give hints about what kind of information is stored at the beginning or end of the blocks.

Of course there’s also the possibility that there is a bug in your extraction code. If you are using a buffer of 16384 bytes for your file operations, then I would first check there.

Answer 2

Based on Sam's suggestion, I forked James' code at https://github.com/tillberg/skyrim and was able to successfully extract n_letter.png from the Skyrim Textures BSA file.

The "file_size" given by the BSA headers is not the actual final file size. It includes some header info as well as some random chunks of useless-seeming data scattered around.

The headers look something like this:

• 1 byte (length of file path?)
• the full path of the file, one byte per character
• 12 bytes of unknown origin, as James posted (40 25 01 00 78 9c 00 2a 40 d5 bf).

To strip off the header bytes, I did this:

f.seek(file_offset)
data = f.read(file_size)
header_size = 1 + len(folder_path) + len(filename) + 12
d = data[header_size:]

From there, the actual PNG file starts out. It's easy to verify that from the PNG 8-byte start sequence.

I proceeded to try to figure out where the extra bytes were located by reading the PNG headers and comparing the length passed in the IDAT chunk to the implied data length inferred from measuring the number of bytes until the IEND chunk. (for details on that, check out the bsa.py file at github)

The sizes given by the chunks in n_letter.png are:

IHDR: 13 bytes
pHYs: 9 bytes
iCCP: 2639 bytes
cHRM: 32 bytes
IDAT: 60625 bytes
IEND: 0 bytes

When I measured the actual distance between the IDAT chunk and the IEND chunk after it (by counting bytes using string.find() in Python), I found that the actual IDAT length implied was 60640 bytes - there were an extra 15 bytes in there.

In general, most of the "letter" files had an extra 5 bytes present for every 16KB of total file size. For example, o_letter.png, at around 73KB, had an extra 20 bytes. Larger files, like the arcane scribblings, mostly followed the same pattern, though some had odd amounts added on (52 bytes, 12 bytes, or 32 bytes). No idea what's going on there.

For the n_letter.png file, I was able to find the correct offsets (mostly by trial and error) at which to remove the 5-byte segments.

index = 0x403b
index2 = 0x8070
index3 = 0xc0a0
pngdata = (
  d[0      : (index - 5)] + 
  d[index  : (index2 - 5)] + 
  d[index2 : (index3 - 5)] + 
  d[index3 : ] )
pngfile.write(pngdata)

The five byte segments removed are:

at 000000: 00 2A 40 D5 BF (<-- included at end of 12 bytes above)
at 00403B: 00 30 40 CF BF
at 008070: 00 2B 40 D4 BF
at 00C0A0: 01 15 37 EA C8

For what it's worth, I've included the last five bytes of the unknown 12-byte segment because of some similarity with the other sequences.

It turns out they're not quite every 16KB, but at ~0x4030 byte intervals.

To guard against getting close-but-not-perfect matches in the indices above, I also tested zlib decompression of the IDAT chunk from the resulting PNG, and it passes.

Answer 3

Actually, the intermittant 5 bytes are part of the zlib compression.

As detailed on http://drj11.wordpress.com/2007/11/20/a-use-for-uncompressed-pngs/,

01 the little endian bit string 1 00 00000. 1 indicating the final block, 00 indicating a non-compressed block, and 00000 are 5 bits of padding to align the start of a block to on octet (which is required for non-compressed blocks, and very convenient for me). 05 00 fa ff The number of octets of data in the uncompressed block (5). Stored as a little-endian 16-bit integer followed by its 1′s complement (!).

.. so a 00 indicates a 'next' block (not an ending one), and the 4 next bytes are the block length and its inverse.

[Edit] A more reliable source is of course RFC 1951 (Deflate Compressed Data Format Specification), section 3.2.4.

Answer 4

Is it possible that you're reading the data from the file in a text mode (where line endings that happen to appear in the PNG data are possibly mangled) instead of in a binary mode?