"System programming" (or "systems programming") tends to mean programming done at a lower level of abstraction than (for example) gameplay programming. Gameplay programming is usually about building the actual game mechanics and front-facing features that a user might see, whereas systems programming is more about building the frameworks upon which gameplay programmers work.
This might mean graphics, resource loading and streaming, audio, memory management, file IO, platform abstraction APIs, et cetera. The details vary quite a bit, and because there are no standards for job titles in the games industry there are similarly no standards for the names of programming domains. At one studio, you may find that "systems programming" means everything I listed above. At another, you may find that they distinguish "graphics programming" as a separate domain and call every other non-gameplay-programming task "systems programming." In yet another, they might not use the term at all and just call it "engine programming."
Since it's a lower-level domain, and typically involves interfacing more directly with the platform-specific APIs for whatever platforms the game is being built for, having knowledge of those platforms will be helpful, as will having knowledge of the more general domain (e.g., of OS concepts without regard to how specific OS's work, such as what virtual memory is, or how threads work, how IO buffering works, et cetera).