For me the easiest way to start testing a code base which doesn't have any tests, is to test bugs. When a you discover a bug, the first thing to do is write a test to reproduce it, then fix the bug to make the test pass.
Other low-hangers are classes which contain logic or data. If you're using the MVC pattern, test your domain models first, and then your controllers. Its often easiest to do this while re-factoring, because you will probably need to make changes to the classes to make them testable (using Dependency Injection and Single Responsibility principles).
For code which is not using patterns, see where re-factoring to a pattern might help, then approach it in a test-driven manner. Write tests for the structure you would like to have, then port your code into that structure to make the tests pass. At the very least, create a façade to act as an organised API over the spaghetti code. Write tests to develop the façade, then after it's working, write integration tests to get it working with your current code. You can then re-factor or re-write the spaghetti code without impacting the rest of the system.
Testing client-server-interaction is also a big pay-off, as it is usually the most fragile code because it interacts asynchronously with a foreign data set. Start at very coarse granularity. So first test that the client-server operations happen in the expected order. Then write tests to ensure that the data is sent and received in the correct format, then test how it breaks when used in unexpected ways. Write tests to call operations out of order, pass invalid data, etc. In FlexUnit you can use [Test(expects="Error")] when you are testing conditions that are expected to throw exceptions. You can also use [Test(async)] in conjunction with the Async methods to test asynchronous operations.
Testing third-party code can also pay off for two reasons: firstly it provides documentation by example and use cases for the library, and also indicates compatibility with other versions of the library.
The trick with writing ad-hoc tests (as with all re-factoring) is to do it in small steps, one line at a time until you have covered a method, then a method at a time until you have covered a class, then a class at a time until you have covered the entire library.
Another tip to keep in mind, is to test behaviour, or rather test what the code does, not how it does it. So if you have collision detection code, test that it returns a positive result and depth vector for overlapping objects, and negative results for non-overlapping objects. You don't need to test the actual algorithm which provides these results, just that the correct results are returned. You may test specific optimisations in the algorithm, but keep these separate from the main test code in case you change the algorithm later.
After you have written tests to cover the functionality in the code, then test edge cases (array lower and upper indexes, maximum and minimum numbers of sprites, level locations, inventory items, quest items). If you have combinations of items or quests, simulate testing them in all possible combinations using paramaterized tests in FlexUnit (use the Paramaterized test runner).
Performance is important with a game engine While FlexUnit does provide timings for each test, it comes with quite a large overhead. For this reason it is probably better to write your own framework for testing performance. It doesn't need to be complicated, it can be a simple class which calls a set of methods in order, measuring the time taken to run each function.
A useful tool to have is an automatic game-play script. Either hand-roll or record a game sequence, then play it back and verify the game state at each frame. This can be especially useful when optimising to measure speed increases.
The hardest code to test is visual and audio, as it requires some sort of signal processing to validate, and the resulting tests are usually fragile. Steer away from these initially, and do them last (if at all).
http://www.flexunit.org/?page_id=6
