I'm trying to compile a list things I should test to measure whether the development of a multiplayer game is going well. I've put a few links referring to Unreal to illustrate because that's what came up during my research, but I'm not focused on any particular engine at this point, I just want to focus on the process of testing.
- Take and compare screenshots: screenshots are taken of specific scenes and reviewed by humans who decide if new screenshots are acceptable
- Self-running demo with profiling: simulate part of the game and collect profiling data such as memory used, FPS etc... and define some thresholds for alerts (eg demo should not go below 40 FPS)
- Test on medium-spec hardware: seems like a good idea to run tests on a machine with medium specs to ensure the game runs decently there (obviously depends on the graphical ambitions of the game)
- Functional testing: making sure the game actually works (eg if a player presses the interaction key to open the door, we can assess the position of the door before/after and we expect it to change).
- Automate as much as possible: the more we can automate, the more systematic our tests and the sooner we can detect issues (if tests have to run manually and are painful to perform, there will be a tendency to run them less often)
- Save/Compare results over time: for all performance-related tests, save results of each test and graph them over time. Visualizing test results over several days/weeks/months alongside a timeline of changes made to the game might help us understand how the various choices we've made regarding the game design are affecting performance.
Now when it comes to the multiplayer part we could:
- Measure bandwidth consumed: run a demo and measure network bandwidth consumed by the game, profile latency in game etc...
- Compare player positions: while taking into account latency, packet loss etc... which obviously affect multiplayer performance, we can compare true X/Y/Z coordinates of a player vs. what is perceived by others as an indication for how laggy the game is. For instance we might decide to reduce the frequency of communication with the server which will reduce bandwidth usage, however we might find out that player true vs. perceived positions are deviating too much and decide to revert our changes.
Anything I've missed that's important when it comes to testing?