most effective storage of logged information [closed]

Question

I am currently implementing basic logging of information on my multiplayer game and I want to have the most effective method.

To log when a player connects and disconnects to a the server it currently saves the info in one of two global txt files, one for the entire servers connections one for disconnections (a new pair for each day).

Taking into consideration the file sizes, that they are to be written to and not read from, organisation and problems with a large amount of players, would it be more effective to store the data:

A. In daily files

B. In monthly files

C. In yearly files

And should the files be:

1.combined in a single txt file

2.seperate in two txt files

3.combined individually for each player

4.seperate but for each player

Or 5.Another alternative I haven't thought of.

Which way should I use and why?

(also I know the difference between one or another could be dependent on the exact use so the best overall performance/organisation)

P.s if you don't understand the way I've said or worded something ask and I'll try to explain it better.

Answer 1

The overall problem by using any text-based solution is that they are highly unreliable and naive implementations are often prone to data integrity issues. This is where david's suggestion to use a database becomes important.

Databases offer you the ability to write entries to a table without worries about concurrent operations from other connections. Additionally, databases offer recovery mechanics where if the database crashes, it can attempt to recover and play back operations against the database that were in memory but not yet written to disk at the time of failure.

All this becomes critical, particularly in more transaction oriented operations such as trading items between players, sending in-game mail, and so forth. It's important things aren't capable of being duplicated by fooling the system and that the operation occurs atomicly without any side affects due to concurrency.

Logging in and of itself may not be a critical operation, but databases can also provide you a way to store information in a partitioned way to where information can be accessed in a much faster way than sequential searching through a text file. A single table can be split across multiple partitions to speed up queries that are against more recent activity versus that of older activity. Databases offer you indexing mechanics to where queries against specific column criteria can quickly find rows that fit your criteria.

If you're running a game server, chances are at some point you'll need a database to back a bit of your game's information. Might as well use it for this as well :P.

Answer 2

You might want to consider logging to a database, such as MySQL or one of the various "nosql" plain old tables.

(If your user base is "massive", there's paths to scaling with more servers or outside services like Amazon or Azure...)

In one table, you can have a row for each log entry, and a column with things like the time, the player, the action (log on, log off), or other events of interest.

This is convenient for aggregation of various metrics, it's exactly what databases are good at.

As for how often you switch to a fresh table, or delete your data, really depends on how full it's getting and if you have any privacy or data retention agreements with the players.

-- I was thinking of this a little more. For an easy an maybe good-enough approach, by all means just append to one file with log entries.

If there's threads involved, use a thread-safe queue of things to append, and then poll and append from just one place.

One file per day, and a folder for each year, as a wild guess to start, would probably be fine. Easy to think about, easy to label the folder & filenames appropriately, easy to grep.