Use duplicated data synchronized with message passing via 1 directional ques. It lets you scrap any kind of locking. It should also be a very flexible system.
Here are a few pointers, it's basically a system I worked out to tie in threading, multiplayer network replication and saving/loading. It's only theoretical, i've never implemented it so I cant say much about the performance.
This is a lot of text but typing it out helps me flesh it out too ☺
Firstly understand that mutex locks are going to kill performance and should be avoided. If you have 2 threads and they both lock the same game world state for their entire operation then they will be slower then just going unthreaded. Atomic stuff is better performance wise but if we get creative we can actually avoid it.
Don't base your synchronization around 'Objects' or 'Entities'. Base it around individual properties.
Give each property a unique id. Depending on your requirements, a simple game might just use an unsigned int, a more complex game like a MMORPG might use a UUID (32bytes) of which part is a timestamp that should allow ids to be generated across multiple separate servers that don't even know about each other with a statistically close to impossible chance of a id collision. If you keep those id's in sync rather than generating them dynamically this will help with network replication too (the simplest way is to transmit the whole list of dynamic properties, their current values and ids).
Named strings ids can help with scripting so you can do things like "dungeon12.dragon3.health = 1000" and/or print out a human readable list of all ids and their values for an object (which can help with things like writing level editors), but names add parsing overhead, so if you do want that it's probably best to have regular numbered ids too for things like threads and network.
Work out what properties are 'static' (or const) and what are 'dynamic'. The static stuff can be ignored with regards to synchronization. The memory could also be shared between threads without any synchronization and doesn't need to be transmitted to clients over a network (provided it's 'shipped' with your game (A custom level might need to be transmitted it the client doesn't have it though, and maybe you want to make a really flexible engine that allows you to alter vertex information on meshes which would be useful for a networked 3d editor). Things like unchanging objects in the level, the level itself, mesh information. The dynamic data will probably only be a small fraction of all your game data.
A good idea is to give each property a flag/timestamp that indicates it was changed (or created) at runtime (one flag for properties changed since file was loaded from disk, another one for properties that have changed since the last network/thread replication transmission). This allows you to make fast and compact saves since you only need to store the changes from the initial state. You can also let players jump into the middle of a game on a network and just upload a list of all the properties. It saves manually separating the properties in dynamic and static.
As a side note if you want almost instant loading/saving find a way to keep all the dynamic values separate from the static ones then when you save you can just 'snapshot' the current game state, copy it off to another block of memory and spawn a background thread to write it to the disk while the player keeps playing. Loading can just scrub the current dynamic state and restore the saved properties without effecting the unchanged ones.
Your properties are no longer going to just be basic data types in a class, they will be small classes of their own.
Look into serialization. That's the art of taking data from program classes and turing it into something that can be transmitted or stored to disk. Stuff can be serialized to bother binary, plaintext and XML. It might be handy to be able to choose between them (network and thread communication would want binary, XML might be usable for loading levels from disk and gives you an editable format).
For increased performance try to keep all the property information in a flat list somewhere so you don't have to walk up and down a tree, octree, BSP or whatever and query all the information inside of each object. It also helps with the above save/load system since you just copy the list.
Give each thread its own copy of the dynamic data, we will synchronize them manually. Ensure that 'Client' thread only has dynamic information it needs to know about, that is stuff the player can actually see or stuff with in range. In addition to reducing what we have to deal with, this also helps with preventing network cheating (the player can't wall hack if there is no entity information of what is behind the wall, of course that would require your server thread to work out what every player actually can see which is possible if you render a really low resolution low graphics version for each player but a a simple ranged/clipping frustum based system is more likely but that still prevents players from knowing about stuff on the other side of the map). You might be able to tie this into your 3D drawing code here since it will be doing similar things when working out what is visible and needs to be drawn, it's also handy to separate your render scene graph from your game world information, there's no sense in keeping mesh/texture/VBOs and so on inside your game objects, that's all render information. It also helps keep your game API agnostic when it comes time to port from DirectX to OpenGL or some kind of ASCII renderer.
Work out a property ownership/security. Just about everything will be 'owned' by the server except a few special classes for clients to send their information too. Stop clients from having the power to edit values they shouldn't but lets you use 1 global system for managing properties (ie rather than treating client information as different from the gameworld properties).
Work out the 'flow' of information. For example the 'Server' thread will be updating the 'Client' thread with regards to the entities in the game world. It won't normally be happening in reverse, the client shouldn't be updating entity information directly even on their own character otherwise it would allow hacks to teleport around the map. Instead a special controller object that they 'own' should be used that the server watches and uses to sync the game world's representation of a player.
To keep things in sync between each thread/network client use uni-directional FIFO message passing pipe RPC ques. Basically a simple FIFO that only has a single producer and a single consumer can be implemented in such a way that no mutex locks or atomic information is needed. Technically you might want a single atomic on the que itself depending on what your programming language/compiler do, it might be that all simple variable changes are atomic without having to do anything. Also depends on how 'theoretically' safe you need to be vs what actually works in the real world. C++ would probably work without issues in most places but technically the standard doesn't say anything about it so it's theoretically possible for your computer to explode, C++11 introduces atomic<>. I don't know what C# does. These FIFO ques can be used between threads and between network clients.
You can do some plumbing on your FIFO's to allow them to be split to multiple consumers or combine messages from multiple producers. (Basically just a list of FIFO's that actss like its just one big message que).
Use an Observer Pattern for your threads. Give each observer a dedicated que. You can thread threads and remote clients as basically the same just one goes across the network. Put your security in the network receiving part.
Work out a RPC language for your messages. What need to be transmitted across the network. The same can be used for inter-thread communication.
- UINT - Message sequence number
- BYTE - Op code
- Set int/uint.
- Set string.
- Set float.
- Set array.
- Add object to list. (we tread a list as a property, it lets you add objects to a level). Do you create new objects here or is that separate.
- Remove object from list.
- Request Ping (So we can calculate lag)
- Send Ping response (Pong).
- UINT - Property ID
- DATA - Property value, length depends on type. Some types will need their length appended at the start.
You could have special opcodes for things like setting the player name or a bit of text in chat, but otherwise you could just give the player ownership of special string properties that the server watches. Player set the string, server send a chatmessage/namechange request. Actually using that method you could perhaps decompose the above into something even simpler there the network codest just involve setting a few special state objects on the server. You would have SetRawDara, LengthOfData, Id. And floats, ints, strings, arrays and so on would all be treated the same way. Just leave it up to the server to check its valid (does player own that object, is the data size correct) and work out what to do with it.
You probably want to use both TCP and UDP. For things like player location UDP is fine. Things like chat or transmitting the initial state would need reliability.
For thread->thread communication you might actually want a different RPC. You actually might be able to use a list of lambada functions and ditch the protocol/parsing totally (On C++11 this should be possible, don't know about C#). Or use function pointers. If you do that network stuff would need to be treated somewhat separably but that's probably for the best. You don't need the security ownership stuff in thread->thread either and if the source is internal you can trust that the size of data being set it sane.
Each thread/client checks the que(s) before (or after) it calls its update function. If there are new messages it looks at the operation
Server thread has a Game World State. It contains every single object. When a property it owns is updated it will transmit it to all listening (or ones within range) 'client' threads via the FIFO. It will listen to a list of ques, one from each client. Its loop will basically be:
- Check incoming thread.
- Process incoming messages (This should be a FIFO that has combined all incoming client messages), update values as required.
- Perform its own updates.
- For each value updated (both from the incoming threads that need to be broadcast on to everyone else (unless you are doing p2p) or the internal update are added to a list of things to be broadcast to all clients.
- Send a broadcast message to all listening clients.
Come to think of it, working out what player is listening to a particular 'property' might be a bit hard since a health property won't have any concept of the object itself or where it is in space so maybe ensure that each property has a parent reference.
Network thread(s). Either one global one, one per-connection, or a pool that processes network packets. This is the same for both clients and server.
- Listen for a packet.
- Check the validity on the packet
- Does the client 'own' the property id it is trying to change.
- Is the size correct.
- If valid, convert it to the internal message representation (function pointer, lambada function, opcodes whatever). Add it to the FIFO que for this client.
- Process messages from server (possibly from other clients if your doing p2p).
- Update properties
- If using a separate rendering thread send relevant messages to that (ie just visible information, position, any new objects in view)
Rendering thread (this can be sound too ☺):
- Receive and process messages.
Physics thread is a bit more complicated. My recommendation would be to ensure that your physics engine works predictability so all clients will get an identical result. Rather than transmit a continuous stream of position updates, instead transmit the basic physics information (overall force being applied, velocity, current position) and let the clients work it out.