How do I run functional tests against my Unity3D game?

Question

Context

I am continuing some legacy code for a game in Unity3d, and I want to write some functional tests meant for regression, to ensure I don't break things when implementing new things or when refactoring.

I already know there is the 'Unit Test Tools' suite for Unity3d available as an asset. I have used it as a unit-test suite, so I test my models (classes).

Example of what kind of test I am thinking about

For "functional test" I mean things like this:

• Run the program
• When in the menu scene, assert there is a button that says "start"
• Click it
• Then assert a new scene is loaded
• Assert pixel XXX is red
• Click in coordinate XXX
• Assert now that the pixel changed to green
• etc.

Questions

Q1: How do I write and run functional tests for my game? Is this also typically done in the UnityTestTools (UTT)?

Q2: If UTT is more for Unit-Testing, then is there a separate suite for functional testing? Which one?

Notes:

I'm targetting Android, and running Unity5.3.1f1

Answer 1

As I understand it, you're looking to understand automation testing in Unity.

Unity Test Tools

This should be your goto method, but before digging into how, I'll examine the why.

The preferred way to do any form of automation testing is to do this in an environment as close to production as possible. Using something like the Unity Test Tools (living inside the Unity Editor) sounds like a bad idea for anyone who'd written conventional integration tests.

Why? Because with any integration testing, you want the least amount of extra stuff bogging down your application that's not there when the user opens it. The Unity Editor might have different performance or behavior compared to the iOS version (for example).

So, why use Unity Test Tools (UTT) then?

The main reason is convenience. Both the Unity Editor and UTT are built around being easy and visual to use. You'll have an easy time writing tests, hooking them with your specific game, and understanding when they fail.

What about the difference in behavior between the Unity Editor and production?

After all, the Unity Editor is a wrapper around your application. That means, there will be some differences between the Editor and production. But, integration testing aside, the Unity Editor has been a wrapper for a very long time now. It is a mature platform with thousands of game built on it. The results you get from running integration tests inside the Editor will be very accurate.

In the example scenario you provided, everything but the first item can easily be asserted using Unity Test Tools. You can open scenes and validate behaviors just like any other integration tests.

I suggest writing most (if not all) of your tests using UTT. To add compatibility with continuous integration tools (e.g. Jenkins), you might want to run those from the console using command line arguments.

Starting the application and manual testing

Starting the application isn't something you have precise control while in the Unity Editor. To validate that aspect of your game and more, you can use some basic manual tests.

Here's how to approach that:

1. Make sure your game emits some kind of signal when it reaches the specific milestones you care (or might care) about. Logging to a file is the most common approach (using the automatic builtin logging or a 3rd party version).
2. Periodically read and analyze the logs. This can happen manually or be scripted using your second-favorite scripting language.

Alternatives

Not surprisingly, as everything Unity related, we have more than a few alternatives. Here are some you might want to consider:

• Using Selenium and the Selenium driver for Unity. Sounds promising.
• The UI Testing Package which allows running the tests inside the player, too.
• Using Unium to send the data to an external testing framework.

Depending on the scale and scope of your project you might want to employ one or many of the options I outlined above.

Answer 2

For anyone finding this question post 2019 this is now possible with the Unity Test Framework. https://docs.unity3d.com/Manual/testing-editortestsrunner.html