This question/answer is designed to be a community effort. Feel free to improve some aspects of it.
How a game can be modded when the source code is not available is a really broad topic.
It will depend on a lot of factors, including the type of modification you want to make to the game, and how the game is built.
What type of mod are you looking for? Different kinds of mods have different implementations.
- A simple reskinning?
- New level, enemy behaviour?
- Quality of life improvements?
Some games are made in such a way that modding can be done by:
- adding/swapping data/asset files or plaintext scripts without affecting compiled code
- decompiling the game and reverse engineering pars of it, then recompile theirrecompiling your own modified version
- allowing separate programs thatto modify the runtime behaviour of the original unaltered code, eg. via DLL injection, network packet sniffing, or manipulating itsthe game's internal data states in RAM
Further resources:
- If you have a specific question about modding a specific game, don't hesitate to ask a new question.
- How to make a game people can mod?.
- You can browse the modding tag to see how others approach this.