Packet handling system architecture?

Question

I'm working on a multiplayer Flash game (ActionScript 3) with an accompanying server written in Python and I'd like some tips regarding networking, and specifically, the handling of packets and implementation of such a system on both the client and server.

At this moment, I am using bytes to identify packets (e.g, 0x00 is login request, 0x02 is login response), sending and reading data in the order specified by my protocol. This is fine, and I am okay with this, but I'd like to clean up the code that handles this system because it seems redundant and unintuitive to use. I'm asking for experienced network programmers, specifically game developers, to answer this question as I lack experience in this field.

Right now, the packet system is made of two main components: The Networker and the PacketParser. The Networker's job is to determine when a packet begins and then passes the packet to the PacketParser which will read the packet contents determined by the identification byte, and return an associative array of the data, which is then handled by the Networker, for example, an Entity's position is updated.

When sending a packet, the user would send an associative array of data to the Networker, which is then sent to the socket through the PacketParser, and this code is rather ugly and ridiculous. Here is a sample in Python, packetparser.py:

def sendPacket(data, socket):
    byters = "" # Buffer - will be sent at once


    if data["Type"] == "LoginReply":
        byters = unpacker.sendUnsignedByte(byters, 1)
        byters = unpacker.sendString(byters, data["Message"])

    if data["Type"] == "Message":
        byters = unpacker.sendUnsignedByte(byters, 2)
        byters = unpacker.sendString(byters, data["Message"])

    if data["Type"] == "Tick":
        byters = unpacker.sendUnsignedByte(byters, 3)

    if data["Type"] == "TankInfo":
        byters = unpacker.sendUnsignedByte(byters, 4)
        name = data["Tank"].getName()
        velocity = data["Tank"].velocity
        baserot = data["Tank"].baserot
        x = data["Tank"].x
        y = data["Tank"].y

        byters = unpacker.sendString(byters, name)
        byters = unpacker.sendInt(byters, velocity)
        byters = unpacker.sendInt(byters, baserot)
        byters = unpacker.sendInt(byters, x)
        byters = unpacker.sendInt(byters, y)

As you can see, this is rather ridiculous, and repetitive.

So, I now would like to smoothen this system out. My initial idea is to create a Packet interface and for every packet I'd like to handle, I would implement the interface with the hardcoded data, so I can do something like this:

networker.sendPacket(new LoginRequestPacket("username", "password"));

This is rather nice to the eye and nice to use, but is it the best? Also, this works fine for sending packets, but what about receiving packets? Perhaps the Networker will have to do this?

ident = socket.readByte();

if (ident == 0x01)
{
    pack = new LoginRequestPacket();
    pack.read(socket);
}

Also, once packets have been received, where in the codebase should game logic be handled (relating to the packet received)? Right now, it's handled directly in Networker but this seems self-defeating - the Networker for network-related stuff, not game logic! That's handled in the main Game class! So, should I set up/use some sort of event system, for example, onLoginResponse() or onPositionUpdate? Or maybe, to make the Packet classes more useful, logic should be handled directly in there? Or maybe even queuing the packets somewhere, for both input and output? Would that work?

These are just things I've thought of on the top of my head, I have no idea on what the pros and cons of these suggestions are and I'd like someone to outline them.

I've had such little experience in this that I really want to hear how you have implemented such a system, to give me a baseline to develop off. Please tell me what I'm doing wrong, what I'm doing right, and what should be done better.

Thanks!

UPDATE

I didn't say so before, but right now, the client is using non-blocking sockets and the server is using blocking sockets, spawning a new thread per client. After some research it looks like the common consensus is to avoid blocking sockets for a game that expects >10 clients (source), so I will be redesigning the serverside codebase to use non-blocking sockets, but this still leaves the issue of where to handle the logic and how Packets are architecturally designed. Please advise me!

Answer 1

I'm writing game in HTML5 using websockets and my design should apply well for you too.

I've separated few things.

TRANSPORT. This is a module that takes care of reading data from network. Transport is for example TCP or UDP, but it might be also serial or something else. You should have functions in this module to connect, read, write and disconnect.

PROTOCOL. This is a way in which you are parsing data (not your game logic data yet, just whole messages). In example you can have "sign delimeter" protocol, which reads data from transport and creates messages from it by splitting data where "delimeter sign" occurs.

123%456%789%

this data read from transport would be converted to 3 messages [123,456, 789] if your delimeter is %. Special case of this protocol is "line receiver" where delimeter sign is "\n".

For binary packets you should create protocol that creates messages based on size of packet.

(3 1 3 4)

Let's assume that this is 4 bytes long data. Your protocol should read first byte (3) so it knows that this packet has 3 bytes. Then you read next 3 bytes so this protocol gets message (1 3 4). You can also have sizes constant for types, but it's not a good idea if you want to send strings.

PARSER. Messages that you get from protocol should be parsed. Parsing is based on some type of identifier sent in message. I have separated ID into type and subtype. My ID is 2 bytes long, and I store type id in 4 first bits, and subtype id in rest 12 bits. How to do this you have to read elsewhere. This way your switch (ifs) is more optimized. Store these identifiers at the top of parser module. In example:

    TYPE_SYSTEM = 1
    SUBTYPE_LOGIN_REQUEST = 1

    if type == TYPE_SYSTEM:
        if subtype == SUBTYPE_LOGIN_REQUEST:
            do_something()

In the end you should remember about design patterns MODEL, CONTROLLER, VIEW. On server you just need MODELS and CONTROLLERS. Parsers are controllers, while transport and protocol are models, you just use them in controllers. They should have no dependancy. Every model should be encapsulated.

Once again I'll explain flow of work in shorter form.

You separated modules that should have no dependancies on each other: - transport, - protocol, - game logics,

Controllers: - main controller, - main_parser, - specific_parsers.

In main controller in a loop you read data using transport model, that data is saved to buffer. Buffer is then passed to protocol model. If buffer has enough data then protocol returns list of messages that it created from data. This list of messages is passed to main parser controller. Main parser based on type and subtype calls specific parser and passes it data from message. Specific parser use functions from game logic model. And that's all about it.

I've separated main parser and specific parsers because in one module it would take too much space and it would be troublesome to find function you want later when you'll write like 100 parser functions.

Good luck.

Answer 2

Here is some thoughts about game server architecture, which I've used for packet handling in several projects:

1) If you use python at server side, don't use sockets directly, use twisted library. This library provide set of tools for creating great high-load applications using asynchronous approach. At first sight twisted seems to be very difficult to understand, but after comprehension several core concepts you will be happy. Here is great twisted introduction: http://krondo.com/?page_id=1327

2) Python have some problems with using threads because of GIL(Global Interpreter Lock), searching through this site could explain why.

3) It's very good idea to move the code which serialize your data to separate module, but the best idea, I think, to use some already existed serialization format like XML, JSON, BSON or msgpack ( or even AMF). They all have pros and cons and should be chosen carefully. However, if you use standard Python types like dictionaries and lists for processing data and returning result to client, serialization/deserialization procedures are trivial(not for XML :) ) .

4) Another good pattern will be to use simple RPC. When the client want to ask server perform some useful work, it creates a packet which include the name of function which should be called at server + provide some arguments for this function. Our server will process request according to game logic and return response to the client. Using this pattern it's easy to move separate pieces of game logic into separate modules.

Here is code-skeleton which is used in several twisted-based projects for processing packets from client at server side. I haven't provide the actual code, because it has many many details which are deeply unnecessary for understanding the main idea, but I think this is enough for understanding.

class Protocol(LineReceiver):
    def __init__(self, config):
        #here we create an adapter object according to 
        #configuration format. It could be XML, JSON, BSON, msgpack or
        #some other custom format
        self.adapter = getSerializationAdapter(type=config["serialization_format"])
        #here we define the packet delimiter "\0" is great for
        #non-binary protocol and also for some binaries like msgpack which 
        #can't produce "\0" symbol after serialization
        self.delimiter = "\0"
    ...
    def lineReceived(self, line):
        #here we unpack seriali
        packet = self.adapter.deserialize(line)

        #process requested line asynchronously, processRequest returns
        #special Deferred object which represent the result of packet
        #handling and could be used for attaching
        #some callbacks to it
        d = processRequest(packet)

        #add callback which will write to the client generated response 
        d.addCallback(self.writeResponse, packet)

        #add callback which will process server exception
        d.addErrCallback(self.writeErrResponse, packet)

    def writeResponse(self, result, packet):
        #... take some information from packet if nessesary

        #here we serialize the object which we obtain from
        #game logic layer
        self.sendLine(self.adapter.serialize(result))

    def writeErrResponse(self, err, packet):
        #... take some information from packet if nessesary

        log.err(err)
        #here we create internal game error object from
        #Failure object, which contains all info about error
        #(code, error message, traceback, local context of error, etc.)
        game_error = _create_game_error(err)

        #here we serialize game error object to stream of bytes
        self.sendLine(self.adapter.serialize(game_error))

    #a special twisted magic decorator which is used for simplifying
    #asynchronous code with callback chains, see twisted documentation     
    #for additional information
    @defer.inlineCallbacks
    def processRequest(self, packet):
        ...
        method_name = packet["method_name"]
        method_map = {
            "login": some_module1.do_login,
            "takeMoney": some_module2.take_money,
        }
        if method_name not in method_map:
                #return an error from generator function using special magic 
                #returnValue function
                defer.returnValue(_create_game_error("Invalid method"))

        #here we call method which implements some game logic, process request
        #and return some object as result
        res = yield method_map[method_name](method_args)

        #return result from generator function using
        #special magic returnValue function
        defer.returnValue(res)

And you should implement several serialization adapters for supporting needed formats, here is example of JSON-based serialization adapter:

class SerializationAdapterJson(object):
    def pack(self, arg_dict):
        import json
        return json.dumps(arg_dict)

    def unpack(self, packet):
        import json
        return json.loads(packet)