I am using an entity system for my MMO server and I was thinking about defining the behavior of "actions" using Lua scripts. The server is written in C++. I am not very familiar with the speed/memory usage of Lua in C++ but I have used it for scripting the GUI of the client. Would using Lua to define the game logic on the server side cut the performance by a lot?
TL;DR: Lua does have overhead, but if used properly it is negligible and easily mitigated. Don't use it for heavy math operations or transforming geometry. You will probably not see any performance problems at all using it to script a GUI.
I've done some basic benchmarks regarding Lua's performance as a game scripting language, and it's pretty damn fast. Using tolua++ to bind LuaJIT to my game engine, I spawned 2,000 actors, each actor controlled by a Lua script called every game loop (with a time-delta argument). Half of the actors had a flocking script and the other half were doing a sort of random walk (and were avoided by the flock).
Turning off the rendering component gave me a bit over 400 ticks per second on my Opteron 170 (2x2.0GHz, though my engine was single threaded at the time). I imagine I could have squeezed out quite a bit more than that if I dug in and optimized, perhaps moving some of the heavy work back into C++. Updating 2000 actors 400 times per second was still pretty impressive, and far exceeded my expectations at the time.
I now use Lua in all of my projects, and it actually constitues quite a large portion of the actual game code (AI, GUI layout/logic, Events/Messages). Making games is MUCH more fun when you can quickly change something and test it without having to exit, recompile, and reinitialize. I've run into some performance issues from time to time, but those are easily solved by reimplementing the offending code in C++ (and then calling it from Lua).
While slightly off-topic, EVE Online's servers are written almost completely in Stackless Python (I believe they defer most of their math operations to a C++ lib), which is considerably heavier than Lua, and, based on my own personal research and several available benchmarks, far less performant than LuaJIT. They manage to handle 30k+ concurrent players without too many issues. Granted, they do have a ton of expensive hardware running all of that, but I believe the majority of the cost is in their database cluster...
Apologies for the wall of text.
Short answer: Yes, yes it would.
Long answer: It depends how much game logic, how much it is run, and how complex it is, and if you need to run it for every player. If it is very simple and is not repeated a lot, you might be fine with it. But for most cases using LUA instead of C++ will give you much lower performance and will scale badly, assuming of course that the code is well designed and optimized.