I am working at a localization agency and we have been approached by a client about testing their games on iOS as well as Android. Testing on iOS seems fairly easy as we can just buy a couple of devices and we should be covered. For Android it seems to be completely different. From what i found, the emulator can cover all API levels, screen sizes and such, but i hear it's buggy and nothing could replace testing on real devices. With the vast amount of Android devices out there and the rate at which new devices are released it seems impossible to keep up. How can i test games (localization and functional) on Android covering all compatible devices?
The only real thing that is different is the amount of devices, Apple just sell 1-2-3 new product each year, Android offers 1 new product every day/week.
The emulator it's not buggy, it's just not intended for profiling, if you want to profile an Android application you have to do the same thing that you have done for iOS: consider the lowest profile device that match your requirements and buying it.
You are supposed to have at least a basic know-how about the ARM architecture, otherwise you can make a difference between all the devices on the market, begin to outline the hardware features that are important for your application and buy that device for real testing.
In our company we prefer not to use emulators as they don't show real picture especially in terms of performance, instead we select several real devices to cover most popular screen size / gpu vendor / form factor combinations, more about how we select test devices on our blog http://developers.nravo.com/best-test-devices-for-game-development/
The Android emulator provided by the SDK has serious performance issues on all systems. Proper "functional" testing as you describe is probably completely invalid using emulators only. Also as far as I know, the emulator doesn't cover manufacturer-specific implementations of the Android OS. So even if you spoof, say, all the hardware of a Samsung Galaxy SII with the emulator, you may still get bugs specific to the Samsung implementation of Android. Lastly, you simply can't multi-touch with a mouse...
The only valid solution I can imagine is buying as wide a range of the most popular Android devices you can afford (in dollars) for your functional testing, and then use the emulator to test as many other specifications of phones as you can afford (in man hours). Fortunately the Android platform is robust enough this usually covers the majority of issues, but weird edge bugs still occur on some specific devices.
The solution to this might be test partitioning.
Consider partitioning into a low-end, medium/normal, and high-end device, for example. You can reasonably expect (in terms of performance) that if the performance is X on a certain low-end device, it will be reasonably similar on a similar low-end device.
With this in mind, you can judiciously choose devices that represent a "class" of devices and test of those.
And ultimately, as anyone who shipped an Android game will tell you, every phone has its own unique quirks and bugs. Make sure you use some sort of analytics tool (I used Flurry Analytics) that allows you to capture errors; you can eventually diagnose and fix a number of these, increasing the range of devices on which your game works on.
In my opinion, the emulator is only good for functional tests; it neither has decent performance, nor does it represent device-specific bugs. I would avoid it. If you want a decent alternative with decent performance, go with BlueStacks, which is an implementation of Android on Windows.