Pub/Sub or message passing in multiplayer server backend?

Question

For my masters thesis I'm going to develop a scalable multiplayer server for locationbased games. It's going to be a "service oriented architecture" e.g. one dedicated service for positions, one for the database, login etc. However 1..n frontend services for the clients to connect to. Scalability will be tested with simulated clients on some other machines. However, I'm still undecided on how my services will talk to each other.

On the one hand, there is publish/subscribe. And even there are different ways to implement it, using Multicast or let each subscriber subscribe directly with the publisher. The good thing with pub/sub is, that each message will be sent to every service that is interested, e.g. the Logger-Service receives a position update as well as the position service. I can also have some "shadow" services that keep in sync in case the major services crashes. Depending on the implementation pub/sub feels like flooding my internal network with messages. And flooding is never a good idea.

Then there is message passing. When the Frontend-Service receives a message, it finds out what type of message it is, finds the service that deals with this type and forwards the message to it. But there has to be a service discovery first, which requires some multicast or service directory.

There are pros and cons for each type. In an article on high scalability I read that Playfish uses the message passing method. But using pub/sub sound appealing too. Maybe someone can tell me which way is better.

Edit: I have to admit it isn't so clear what I'm asking. Imagine one frontend service receives a "position update" message from one of the clients. With a pub/sub system, I think about calling something like this: Network.publish(Topic,Message) or Network.publish(PositionUpdateType, Message). Using a message passing system I might call: Network.send(PositionManager, Message). I think pub/sub seems to be like "fire and forget", where the message passing is some kind of "direct" connection. My question is: Which one is better suited for the job or should I even use a combination of both?

Answer 1

Pub/sub and message-based communication are not mutually exclusive. However, I think you're realizing that pub/sub abstracts the receiver, whereas message passing requires a recipient. They're both fire and forget unless structured otherwise (continuation-passing style comes to mind).

All in all, the two are not so different, we might argue that publishing a message is nothing more than sending a "publish" message to the "publisher" actor, which in turns delivers the message to any recipient that sent a "register" message to the publisher in the first place.

Anyway, to make it short: message-passing enables you to delay execution and share the load across workers. In our multicore age, this is handy, especially with task-stealing schedulers.

Which language are you targeting? I might give more details.

Edit: I just noticed the "Java" tag. I know nothing of Java :)

Edit 2:

Sharing the workload across services is just the beginning. You have to find a way to max out these services, and when the services are maxed out, you also need strategies to spawn new services and share the workload again. If you're using one server to the max, you can't just spawn a new process, but you need another physical server, ready to crunch numbers until the next load peak. And even then, you have to be careful about the strategies you use to spawn new workers.

Intra-server:

1. A main server broadcasts a discovery message.
2. Server 1 replies, says it can deal with logging
3. Server 2 replies, says it can handle player movement etc.
4. Server 3 replies, says it can handle trades and chat.
5. The main server will acknowledge the servers and subscribe them to specific service-dependant topics.
6. However, also server 1, 2, 3 will have to keep a list of subscribers, unless it's fine to let the main server do all the publishing.
7. Every n milliseconds, servers will have to show they're still alive by sending a message to the main server.
8. Intra-server and foreign events (from clients) will be dispatched to the correct subscribers.
9. At some point, the volume of events will cap the processing power of one or more servers, requiring more workers.
10. This is where things become messy; existing services will have to yield responsibility of a part of their processing to services on a different machine. Also, workload sharing criteria may be dynamic (you thought having workers per world region was enough, but turns out 90% of players is in the same city, in the very same region, nullifying your strategy).

As for maxing out a service, you really want to divide your work in tasks and, while order matters, let a task-stealing scheduler process them as fast as they come.

It's a wall of text already. Questions? Comments?

Answer 2

These aren't opposing paradigms, just different aspects of the same thing. Publish/subscribe can send messages. And service directories may be filled only with services that have subscribed to that directory. And if your subscriptions are 'flooding' anything then presumably you aren't subscribing to something fine-grained enough. So it's not really clear what you're asking.

Answer 3

Consider using a job server such as Gearman. Your services are workers that pull jobs from the job server, do some work, and put a result back which is forwarded to the job submitter. If you find that some service has a high number of jobs per unit time, simply add more workers to handle the load.