How should I collect user behavior data in an MMO?

Question

In an MMO, and I'm trying to collect data about user behaviors for the purpose of tweaking the game rules to achieve maximum user satisfaction.

Clearly one way to do it is to hand-roll specific things into an application, much like one might use Console.WriteLine() to view the contents of a variable, or the StopWatch() class to see how long something takes to execute. But you can use a profiler in lieu of StopWatch, and you can use a debugger in lieu of WriteLine.

How should I collect my data? Are there generalized techniques for instrumenting an application to observe user behavior, or some form of code instrumentation technique?

Also, I'm only interested in collection techniques; you can assume that I already know how to transmit, store and analyze said data.

Answer 1

We are using Command Oriented Interface/Architecture to audit user activity.

It means that requests to the server, cause the server side to instantiate instances of classes that represent commands (atomic actions the user can preform).

For instance, lets say the user makes a purchase in an item store:

//We could have done something like
someShop.sell(item, player); // This function deducts money from the player
                             // and stores the new item in their inventory

//Instead we do this:
$buyItemCmd = new BuyItemCommand(item); // The player is already known from the sessionId
$cmdExecuter.run($buyItemCmd);

Now we add logging / auditing features in the cmdExecuter that apply to all commands. This way there is little to no code duplication and we can turn off auditing, either by re-configuring a specific command class constants and / or modifying the cmdExecuter.

It helps to keep things organize. We currently create a database record for each "important" command. This helps track behavior that is interesting to us.

Further Reading
Command-Oriented Inteface by Martin Fowler

Answer 2

The "technique" is here is so simple that it almost doesn't deserve to be called as such: you just record the data from game systems you think are relevant.

How you record that data (to a database, to files, whatever) is too broad to discuss here (and the techniques and patterns are not particularly game-development specific in any case, making them better suited for discussion on SO), as is the question of which systems are relevant. There are no clever automatic techniques to answer those questions; that would be quite the feat of artificial intelligence.

The thing you want to make sure of with a game where there is a server component is that the server should do all the logging it can. Nothing you get from the client can be trusted, and especially if you're going to be using this data to tailor changes to the game you don't want players to be able to tamper with it.

The client should only record and transmit statistics that the server wouldn't normally have access too. For example of you want to record UI click heat maps, that's probably only feasible on the client.

Answer 3

In an MMO you already have data about where the user is going, which quests he takes and what equipment he uses sent to the server.

Other data you may be interested in is how they use the various guis, how long they keep stat and inventory screens open. You can for example record the number of times a feature has been activated with the mouse vs. the keyboard.

One important factor is to allow the user to opt-out (or make the collection opt-in) of the (non-critical) data collection to comply with privacy regulations. And to allow the server to decide what data should be recorded during the game session/next hour to spare your players some bandwidth.