One source for such bugs can be a buffer over-read. A bug in your program reads beyond the boundaries of an allocated array, so it reads whatever is stored beyond it.
Another possible source is use after free. The program tells the runtime environment to deallocate the memory a pointer points to, but still keeps a pointer to it around. The runtime then allocates something else to that supposedly no longer used memory. When the program later reads data from the pointer, it reads whatever is now stored there but interprets it in the context of the pointer.
The bug in Commander Keen might be uninitialized variables. When you create a new variable and don't assign any value to it, most programmers intuitively assume that that data will be empty (full of 0's). With many modern high-level programming languages, that's in fact the case. But with many older or more low-level languages, the data will be filled with whatever was there in memory before. So when you forget to write to allocated data before reading from it, you might find strange things in there. The programmer of Commander Keen likely allocated an array as large as the screen to store the tiles currently on-screen, but then stopped writing to it when the border of the map was reached. So the off-level portion of the screen stayed uninitialized.
You might wonder why you see such glitches so rarely in modern games. The reason is that while these glitches are quite interesting in the context of a game, they can have disastrous consequences in any software where security and/or correctness matters. So the software development community (beyond game development) has put considerable effort into making them a thing of the past.
- The memory managers of operating systems try to detect and prevent such unusual memory access patterns, usually by crashing the program ("segmentation fault").
- With multitasking operating systems there are multiple processes allocating and deallocating memory. The memory management of the operating system is a lot more sophisticated and hence less predictable than in a single-task operating system like DOS. That means if such bugs occur and do something interesting, you might not be able to reproduce it.
- High level programming languages like C# with bound checking, garbage collection and implicit variable initialization built in are raising in popularity among game developers. With such languages, the above bugs are no longer that easy to create accidentally.
- And even the low-level programming languages like C/C++ have evolved in the past 30 years. This applies to both the language features themselves and to the best practices used by developers.