With automated testing on Android, we iterate as much as we can in order to improve our process before we launch each version of our app to the Play Store. Our setup included android unit tests that ran on an emulator in Jenkins and Robotium integration tests that ran on AppTwack. Whenever we released, we had a build task on Jenkins that created the Android APK, ran unit and integration tests, and then pushed the APK out to Google’s Play Store.
About two weeks ago, we noticed unit tests that had been passing for months suddenly started to fail without any distinct pattern. We checked code changes and tried reproducing the errors from the Jenkins box. Narrowing it down to be an issue with the emulator, we tried replacing it but didn’t have any luck. We synced it down from the Jenkins machine and tried running it locally but we still couldn’t reproduce the errors. Our Jenkins box was a headless server with a sketchy xvnc to fake a video buffer (basically cobbled together test rig that was flaky and hard to debug). We also experienced some growing pains with using an emulator for unit testing. For starters, it’s extremely slow and hard to write tests for. We got errors on Jenkins that we didn’t understand so we ended up rerunning the tasks. To make matters worse, now it was starting to break the actual tests without relevant code changes. In the past, we’d discussed switching over to using junit tests that run directly on JVM. With the release of Android Studio 1.2, we decided to bite the bullet and make the transition.
With Android Studio 1.2 came better support for unit testing so we initially tried refactoring some of the tests so that they were independent of running on an emulator but found out that you had to import the libraries separately in the testing environment. It turns out that common libraries such as Json or Uri are not included through the org.json package in the JRE. Instead they are provided by Android.jar. As a result, we were getting null values returned for initializing json objects, parsing strings, or encoding url params. If we had to import each library into Gradle, it would become unwieldy for developers and make it extremely difficult to debug/write tests.
Uri parsedUri = Uri.parse("market://details?id=" + packageName) // A simple line like the one above would return you parsedUri = null :
That’s where Robolectric came in. It has a lot of helpful resources and allows us to run android libraries as if we were running the application normally. By annotating each class to run with RobolectricGradleTestRunner, we can stub out methods used by the Android SDK and even add shadows to android specific classes so that we can add logic or mock out specific behavior.
By switching to Robolectric, there’s a clear distinction between our unit tests (Robolectric without emulation) and our integration tests (run with Robotium w/ emulation on AppThwack). The benefit of having emulation-less unit testing is that we now control the behavior to test out specific pieces of our code. We ended up having to quickly refactor several tests because some things weren’t being mocked properly when you stripped out the emulator (free lunch?) but for the most part, we were able to keep our tests the same.
In the end, we removed the emulator from the Jenkins box that was running the unit tests and our tests went from taking 8 minutes to run to a little over 30s. A nice speed improvement along with more clean and sane Android testing. Gotta love seeing everything blue 🙂
If you want to learn more about how things work at Jana, get in touch. We’re hiring!