Let's say I have both a game and a modding system for it. The modding system loads DLL files containing custom features and allows mod creators to override certain functions to change game mechanics. This is all well and good until some mod creator decides to put a keylogger or a virus in one of those DLL files. What can be done to prevent this problem? Is there any way to allow a high level of customisability in a modding system while still maintaining a strong level of security for the end user?
You cannot prevent it (in situations which are equivalent to running arbitrary native code supplied by others).
- curate the mods
- screen submissions with a scanner
- restrict your API as much as possible
- select a language that is safety oriented
- sandbox foreign code
All of these increase safety to varying degrees. But none of them are a guarantee.
As others have stated in the comments, the easiest way is to either embed an interpreter in your game (for example, Lua scripts are fast-ish, and there are a number of good implementations scattered across the web for various languages) or create a simple text parser and read files. For example, you could roll your own text format for simple scripts, embed lua for mire complex scripts and gui-related tasks, and load object definitions from JSON or XML.
For example, in my strategy game;
1.) Unit types are defined in xml - safe.
2.) GUI layout is defined in xml - safe.
3.) I use a custom parsed scripting language for simple events.
4.) Lua is used for gui interaction. I have removed the io, and os functions effectively rendering these scripts safe.
 - As others have stated, these approaches aren't necessarily "safe" just "safer" than including compiled dll's.