You might want to ask yourself if you really need client-sided scripting at all. You might be able to make many interesting mods possible by simply making your network protocol powerful enough. Most client-sided functions which you could consider exposing to the scripting engine could just as well be exposed to the server by adding another type of network message.
But if you really want client-sided scripts downloaded from untrusted servers, then the risk depends on what your scripting engine can and can't do. What's always dangerous is:
- Any form of file I/O: It might exfoliate sensitive information, delete/overwrite important files or infect executable files with malware code.
- Any form of network I/O with anything but the currently connected gameserver: It might be abused to turn your game clients into botnet zombies which send spam, perform DDOS attacks, distribute illegal content or other evil stuff. Even peer-to-peer connections between players currently playing on the same server would make me feel uneasy. There might be a lot of potential for abuse here when players use some hacking.
- Running other programs: You don't want your game client to run any 3rd party programs on the user's PC.
- Sharing data between servers: You want to properly sandbox servers against each other. For example, if your scripts have some way to create persistent information on the user's computer, it should not be allowed to read or overwrite information which was written by a different server. You also don't want a script to run code from scripts which originate from other servers.
- Any access to personal information: Your scripts should not be able to find out anything about the user or their setup. Personal information can be used by games in ways which seems like a cool feature at first glance but can end up being very creepy and potentially dangerous even if meant well. Extra Credits recently made a video about this. This is really something I would not trust some mod-building teens to use in a responsible manner.
- Any social interaction which might be mistaken for player action: For example, when your game has a chat, a malicious script might generate insulting chat messages which appear as if they came from a player. This might end up ruining real-life relationships between your players.
When you develop your own scripting language from scratch, you can easily avoid these things by simply not implementing them. But when you are using an existing scripting engine, you need to inform yourself well what it can or can't do and how it can be limited and/or sandboxed. Some are quite a lot more powerful than they seem at first glance. Lua, for example, which this question was tagged with, has quite a lot of potentially evil stuff in its standard library.
And then you need to be aware of security problems with your script execution environment itself. This question was tagged with
C++, which is a language potentially vulnerable to buffer overflow bugs which allow arbitrary code execution. If scripts are able to send arbitrary data to the C++ layer, they can potentially be used to exploit any such bugs remotely and drop some shellcode. For example, it might not be safe to assume that every
char* you get from your scripting engine points to a properly nul-terminated string.
Another possible problem is abuse of player computing resources. A script could be used to make player PCs perform expensive tasks for the server host, like mining cryptocurrency or cracking passwords etc.. This isn't evil per-se. Having players mine cryptocurrency might be a good way to finance a free server, if players are made aware of this. But it would not be in the best interest of your players to let others exploit them in this way without their knowledge. A good way to limit this is by limiting the amount of CPU time scripts can use. Usually there are few legitimate reasons why game scripts would perform lots of expensive calculations.