In addition to the article linked in other answers, I can tell a bit about the experience of the Arianne Project.
How to keep things synchronized?
We have build framework „Marauroa“ around the concept of actions and perceptions: Actions are sent from the client to the server carrying user input like (walk left, attack monsters #47, say 'hello'). And perceptions are sent from the server to the clients telling them about the state of the world closely around them. Those perceptions are sent every turn. Depending on the game we use turn times of 30ms to 300ms.
We have two types of perceptions: A full perception is sent on login and when the player enters a new area (zone). After that differential perceptions are sent including only the modified attributes (e. g. position) of the modified objects, and of course new and removed objects.
How to work around latency problems?
We strongly belief in “the server is always right”. The client does some predictions like smooth walking, collisions checks and so on. But if a client and the server disagree about something, the server wins. The subproject Stendhal (a 2D RPG) uses a turn time of 300ms by default (with a lot of smoothing done client side). This makes Stendhal very resistant against lag.
Note: Some other games trust the client to some extend to minimize the impact of network lag. In WoW it was often exploited in one of the battle field called “Warsong Gulch”. There are two ways which a player with the flag can choose: On in the middle through a tunnel and one on the right sight going up the hill. So a cheating player runs towards the tunnel, and then causes lag for himself. The server and the other clients will continue to see him running towards it. But after a little time this client can tell the server that it went towards the hill. WoW will check that the distance between the last transmitted coordinates and the current ones fits within the time segment and accept that.
Use of UDP vs. TCP
In early versions we used UDP to reduce the overhead of TCP. We handled lost packets on our own. This worked perfectly at the early days of the project. But when the server was moved from some home DSL connection to a real data center several years ago, we got huge issues. UDP is (or at least was 5 years ago) extremely demanding on CPU power of firewall hardware: The ruleset has to be applied to every single UDP packet. For TCP, however, the ruleset is only applied for the first 3 packets. After that the connection is established. All following packets will bypass the normal ruleset because they are in the connecting tracking table or because they don't have a SYN flag.
How to protect the communication and client from reverse engineering?
Arianne is completely open source, this include the client, server, graphics, music. And of course that includes our protocol documentation and even an analyzer used for debugging.
It is easy to protect the communication against unauthorized sniffing by third parties using SSL.
It is, however, impossible to protect it against reverse engineering. Sure you can obfuscate it and use anti debugging techniques. But in the end you cannot prevent reverse engineering of software you give away to users. There is a very interesting presentation on how Skype was reversed engineered despite the developers putting a lot of effort into anti debugging techniques: http://recon.cx/en/f/vskype-part1.pdf
Note: There are some countries in which reverse engineering is illegal or which allow to put a paragraph into the license or ToS disallowing reverse engineering. But there are other countries (like the one I am living in) which explicitly allow reverse engineering in the context of developing compatible data storage formats or transmissions protocols, paragraphs in the license or ToS trying to disallow that are void. (Everything in this section is as far as I know, I am not a lawyer)
Which things should be computed local and which things on the server?
We compute everything related to the game logic on the server. The client will predict certain events in order to make the game play smoothly. But in the end the server is always right.
Predicted events are for example stopping of movement when a collision is hit. Stendhal uses a grid to position elements. And from the server point of view, the top left corner of every entity is on exactly one square. But the client will move them around smoothly between tiles. It will draw the pseudo 3d effect, too. So an entity that has a base of 1x1 may be higher in the client.
How to balance load problems?
Try to keep this as simple as possible, to ease maintenance.
Load balancing of static content is well known in the area of http server clusters and content distribution networks.
An rather simple concept for load balancing of game services is to split servers across regions/zones. So zone A-C are on one server and zones D-F are on another. This is especially easy if you cannot look from zones in one set to zones in another set. You need to put some checks in there so that a client can only connect to a zone server responsible for the zone the player is in.
The easiest way to transfer players from one server to the other is to write them to the database, tell the client to connect to the other zone server, and disconnect them from the current one. The client will then connect to the new zone server which will load it from the database. (Since you need the load from / store to database code anyway, direct communication between the servers for handover can be implemented later).
Some additional global services are needed through: On login the clients have to be told to connect to the correct zone server. And you might want a world wide chat system.
I went into detail on this topic at How is load balancing achieved in MMOs?