Im working on a "game" that uses lua for scripting and i plan on making it very mod-able. However, malicious code can be written without much difficulty in lua... So how can i go about stopping that? I could block certain functions, but that wouldn't be nice for the modders and those who really want to will find another way.
could block certain functions, but that wouldn't be nice for the modders
It's perfectly nice for modders and is basically what everyone does. The only ones you need to restrict are the IO functions, of course, since Lua doesn't really include anything else out of the box.
You can instead of "blocking" them replace them with your own versions that do extra checks such as ensuring that all paths are contained within your game's script data folder (be sure to protect against
.. and funky versions of
and those who really want to will find another way.
Yes. Yes they will. They'll find a way to break the sandbox, or just abuse a bug in your code; a bug that might even be exploitable with a properly crafted texture file, because software is hard.
Which is why you should never ever never never auto-download mod packs for the player. That is, don't ever have the idea that you should auto-install required mods for a server that the player doesn't have. Instead have a mod repository where users can search, rate, and comment upon mods and from which any mod found to be malicious can be removed and the author banned and reported to authorities.
Miscreants still intent on abusing your players via your game will do so in so very many more stupid ways than trying to hack your script integration, even if you do allow unsafe scripts. The most common is still just good old social engineering: convincing players to download random malware in the guise of mods ("see $character$ in teh nudez!! jus download dis exe!!!~").