Running Android Instrumented Tests on CI - from Bitrise.io to GitHub Actions
Tags: android, ci, github, testing
Published: 2019-11-24 09:36:00

It's almost 2020 and it remains a challenge to run Android Instrumented tests on CI, especially for **opensource** projects and small teams:

- Setting up and managing an in-house device farm is expensive and requires long-term investment and on-going infrastructure support - not a viable option for most teams.
- Cloud-based device farms such as **Firebase Test Lab** are a much more cost-effective solution for most teams as they take care of managing the infrastructure while offering simple APIs for [integrating with existing CI pipelines](https://firebase.google.com/docs/test-lab/android/continuous). However, for opensource projects with a large amount of tests, free plans offered by these services likely won't cover the needs, while paying for such service is usually hard to justify for an opensource effort.
- For small teams who are not quite ready to invest in a fully-fledged cloud-based testing service, running tests on the **Android Emulator** within a docker container has been the most viable and common approach. But today most cloud-based CI services are still lacking **hardware acceleration** support from the host VM, which is the no.1 blocker for running tests on modern Android Emulators (especially on recent API levels) on CI.

I recently released [FlowBinding](https://github.com/ReactiveCircus/FlowBinding) which has over **160** instrumented tests across 10 library modules. I wanted to be able to run all these tests on each PR / merge into master without worrying about things like usage limits and quotas. So **Firebase Test Lab** is not an option as the free plan has a daily limit of **10** test runs on virtual devices and **5** test runs on physical devices. Instead I needed something that checks these boxes:

- Free for opensource projects.
- Supports configuring the Android Emulator system images used - API level, target: (default or google_apis), arch / ABI (x86 or x86_64).
- Supports running emulators in **headless** mode (as of [Emulator 29.2.7 Canary](https://androidstudio.googleblog.com/2019/11/emulator-2927-canary.html) this is equivalent to running `emulator -no-window`).
- Supports running modern x86 and x86_64 based emulators with **hardware acceleration (KVM) enabled** for better performance.
- Provides enough RAM for running both **Gradle** and an instance of **Emulator**.

In this post I'll share my journey on finding the best solution for running Android Emulators on CI for opensource projects.

-------------------------

## Hardware-accelerated Emulators

Out of those requirements mentioned earlier, enabling **hardware acceleration** support on the host VM has been the most difficult one to meet.

The old ARM-based emulators were slow. But more importantly ARM-based system images are no longer supported by Google, as only x86 and x86_64 images are provided for recent API levels.

The modern Intel Atom (x86 and x86_64) emulators intend to provide much better performance with [GPU hardware acceleration](https://developer.android.com/studio/run/emulator-acceleration). However, installation of special software (HAXM on Mac & Windows, QEMU on Linux) is required for enabling hardware acceleration support. This presents a challenge on CI as to be able to run hardware accelerated emulators within a docker container, [KVM](https://www.linux-kvm.org/page/Main_Page) must be enabled by the host VM which isn't the case for most cloud-based CI providers due to infrastructural limits.

This means currently we are not able to run hardware-accelerated emulators on some of the most popular cloud-based CI services such as **CircleCI** and **Travis**. 


## Google wants to help

It's worth to note that recently Google has been putting more efforts into offering better support for running Emulators on **CI**.

- Earlier this year the Android Emulator Team added support for running the emulator engine in [headless mode](https://androidstudio.googleblog.com/2019/02/emulator-2818-canary.html) to reduce some of the implicit expectations on the host machine running the emulator in a CI environment. This feature has been unified with the `-no-window` mode in a [recent release](https://androidstudio.googleblog.com/2019/11/emulator-2927-canary.html).
- Google recently published a [blog post](https://android-developers.googleblog.com/2019/10/continuous-testing-with-new-android.html) highlighting some of the new tools they've been working on to improve deployability and debuggability of the Emulators on CI. They also opensourced a bunch of [scripts](https://github.com/google/android-emulator-container-scripts) for running Emulators in (Docker) containers.
- There was a dedicated session titled [Emulator in a Continuous Integration (CI) Environment](https://www.youtube.com/watch?v=70-FCrCKdRM) at Android Developer Summit 2019.
- There is also [Project Nitrogen](https://www.youtube.com/watch?v=-_kZC29sWAo) which is a larger effort on testing introduced back at Google IO 2018, but there hasn't been any public announcement on its progress this year.

It's encouraging to see Google's interest and commitment to continuously improve the experience of running Emulators on CI, but as of today the **KVM** dependency is still the main hurdle that prevents most users from being able to leverage these new tools and improvements.

## Bitrise.io

[bitrise.io](https://www.bitrise.io/) is a CI/CD platform dedicated for Mobile. It has its own ecosystem including an online [Workflow Editor](https://www.bitrise.io/features/workflow-editor) and an extensive library of workflow [steps](https://devcenter.bitrise.io/steps-and-workflows/getting-started-steps/), which probably appeal to new teams / projects starting out with basic Mobile CI needs.

But what got me excited were the provided steps for running Android UI tests (both physical and virtual devices), and **KVM** support in the host environment.

### Using Bitrise Steps

Bitrise provides the [Android Build for UI Testing](https://blog.bitrise.io/new-step-android-build-for-ui-testing) and [Virtual Device Testing for Android](https://github.com/bitrise-steplib/steps-virtual-device-testing-for-android) steps which use **Firebase Test Lab** to run tests for the chosen module / build variant with no limit on device hours / no. of test executions. But there are a couple of limitations:

* Only 1 build variant from 1 module can be run for each step. This means if you have multiple library modules with instrumented tests, you'll have to manually configure a **Android Build for UI Testing** or **Virtual Device Testing for Android** step for each module (or setup parallel workflows for running these in parallel if you wish to pay for additional containers).
* Running tests in a library module doesn't work unless an app APK is also present. The [workaround](https://discuss.bitrise.io/t/vdt-not-able-to-run-instrumentation-tests-on-android-library-project/3197/7) is to also run `app:assembleDebug` for your library module tests.

You could also use the [AVD Manager](https://github.com/bitrise-steplib/steps-avd-manager) and [Wait for Android emulator](https://github.com/bitrise-steplib/steps-wait-for-android-emulator) steps to spin up an Emulator on Bitrise locally and then run all your tests with Gradle, but you don't have full control over the Emulator configs and the specific SDK components needed for your workflow.

### Custom workflow

Since Bitrise's host VMs have **KVM** enabled, we can easily setup a custom Workflow to control exactly what SDK components we want to download / update, how we want to configure and start an Emulator by providing with a custom shell script and running tests with regular Gradle commands.

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/08w18sretir44jzo0lm8.png)

Here's a [repo](https://github.com/ychescale9/bitrise-ui-tests-workflow) showing how to setup such custom workflow on Bitrise.

At this point I was finally able to run Android instrumented tests on CI for **FlowBinding** on every PR! 

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/uldnev0gkmgfv75sugct.png)

### Getting greedy

Being able to run **unlimited** instrumented tests **for free** on CI is not something to be taken for granted. But as more modules and tests were being added to **FlowBinding**, build time for the Bitrise workflow also significantly increased.

When I first integrated the Bitrise workflow, **FlowBinding** had **2** library modules with **9** instrumented tests; the entire Bitrise workflow took about **10 mins** (this includes running a custom script to configure and spin up an Emulator, and running the tests with `./gradlew connectedCheck`).

By the time I was ready to publish the first version of **FlowBinding**, there were **10** library modules with a total of **160** instrumented tests; the entire Bitrise workflow was taking over **30 mins**.

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/udfzer83da1vf5qxdwuc.png)

While the number of modules and tests are unlikely to significantly increase for this project, an **above 30 mins** build time on CI for each PR is still quite an inconvenience.

Besides the long build time, I also wasn't very comfortable with the recommended way of configuring workflows with the online editor and having to rely on those built-in steps provided by Bitrise. Relying on the platform and ecosystem makes it harder to port our pipelines to potentially better alternatives in the future.

Given these concerns and dissatisfactions, I was set out to continue the search for better experience for running instrumented tests on CI. 

## GitHub Actions

**GitHub Actions** recently added [support for CI/CD](https://github.blog/2019-08-08-github-actions-now-supports-ci-cd/), and is free for public repositories. As a result many opensource projects on GitHub are starting to migrate their CI/CD workflows to GitHub Actions which has first-class integration with GitHub itself and strong infrastructure support from **Azure**.

The first thing I tried was running a **hardware-accelerated** Emulator within a docker container on a **Linux / Ubuntu** VM. Unfortunately (and perhaps unsurprisingly) **KVM** is not enabled on the host machines.

But **GitHub Actions** also offer free **macOS** machines for public repositories, and these macOS VMs also have [**HAXM** (Hardware Accelerated Execution Manager) installed](https://help.github.com/en/actions/automating-your-workflow-with-github-actions/software-installed-on-github-hosted-runners#android). This is promising as we should be able to setup a GitHub workflow that installs the required SDK components and spins up a hardware-accelerated Emulator directly from the host machine, and finally run our tests on the Emulator!

### Custom action on macOS

Setting up a workflow that runs on **macOS** though is not as easy as doing the same with **Docker**, as our base environment is no longer a docker image with all the tools installed and configured and we'll need to port our scripts to work in the macOS environment. The workflow would need to do the following:

- Install / update the required **Android SDK** components including build-tools, platform-tools, platform (for the required API level), emulator and system-images (for the required API level).
- Create a new instance of **AVD** with various configuration options such as the API level, CPU architecture / ABI, and device profile used.
- Launch a new instance of Emulator from the AVD created.
- Wait until the Emulator is booted and ready for use.
- Finally execute a Gradle command to run the tests - e.g. `./gradlew connectedCheck`.

We can package this process into a [custom GitHub Action](https://github.com/marketplace/actions/android-emulator-runner/) and provide a bunch of [configuration options](https://github.com/ReactiveCircus/android-emulator-runner#configurations) for the consumer.

A workflow that uses the custom action ([Android Emulator Runner](https://github.com/ReactiveCircus/android-emulator-runner)) to run instrumented tests on **API 29** may look something like this:

```
jobs:
  test:
    runs-on: macOS-latest
    steps:
    - name: checkout
      uses: actions/checkout@v2

    - name: run tests
      uses: reactivecircus/android-emulator-runner@v2
      with:
        api-level: 29
        script: ./gradlew connectedCheck
```

### Outcome

It was time to run the custom GitHub Action with **FlowBinding**, and the [result](https://github.com/ReactiveCircus/FlowBinding/runs/296389790) was promising!

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/wqn9ttd2irbou5ugqtv7.png)

The entire GitHub workflow took around **20 mins**, while the test suites can now be run against multiple API-levels in parallel (maximum concurrent macOS jobs for public repo is **5 per user/org**).

This is more than **30%** improvements over the previous solution with **Bitrise**.

On paper the specs of the build machines provided by **GitHub Actions** and **Bitrise** are similar - 2-core CPU / 7GB RAM. So I'm not really sure what the cause of the improvement is. One possibility is being able to run everything directly on the host VM without the **virtualization overhead** in a docker-based environment, but that depends on [many factors](https://www.brightcomputing.com/blog/containerization-vs.-virtualization-more-on-overhead).

### Leveraging build matrix

One of my favourite features of **GitHub Actions** is [Build Matrix](https://help.github.com/en/actions/automating-your-workflow-with-github-actions/workflow-syntax-for-github-actions#jobsjob_idstrategymatrix), which provides an easy way to run the same job across multiple configurations.

For example, to run instrumented tests on API 21, 23, 29 for both x86 and x86_64  ABIs, we can define the values for each configuration option under `matrix` and reference these values in the inputs of the action:

```
jobs:
  test:
    runs-on: macOS-latest
    strategy:
      matrix:
        api-level: [21, 23, 29]
        arch: [x86, x86_64]
    steps:
    - name: checkout
      uses: actions/checkout@v2

    - name: run tests
      uses: reactivecircus/android-emulator-runner@v2
      with:
        api-level: ${{ matrix.api-level }}
        arch: ${{ matrix.arch }}
        script: ./gradlew connectedCheck
```

The workflow will run the job with all 6 configurations:

- API 21, x86
- API 21, x86_64
- API 23, x86
- API 23, x86_64
- API 29, x86
- API 29, x86_64

We can also filter out certain configurations generated by the build matrix. If we only want to run the job with (API 21, x86) and (API 29, x86_64), we can easily define the configurations we don't want with `exclude`:

```
jobs:
  test:
    runs-on: macOS-latest
    strategy:
      matrix:
        api-level: [21, 29]
        arch: [x86, x86_64]
        exclude:
          - api-level: 21
            arch: x86_64
          - api-level: 29
            arch: x86
    steps:
    - name: checkout
      uses: actions/checkout@v2

    - name: run tests
      uses: reactivecircus/android-emulator-runner@v2
      with:
        api-level: ${{ matrix.api-level }}
        arch: ${{ matrix.arch }}
        script: ./gradlew connectedCheck
```

This will only generate 2 jobs:

- API 21, x86
- API 29, x86_64

You can even [go nuts](https://twitter.com/ychescale9/status/1192051899060457474) (please don't) and run your tests against all combinations of the configurations available, which when executed will generate a total number of **9** API levels (API 21 to 29) x **2** targets (default, google_apis) x **2** CPU / ABI (x86, x86_64) = **36** jobs.

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/lh5ndw5z80ujcud6hufg.png)

:upside_down_face:

**Fun fact** - while playing with build matrix I discovered that there is no system image available for `android-27;google_apis;x86_64`.

**Update** - [a response from Google](https://www.reddit.com/r/androiddev/comments/e5nulf/emulator_29211_and_amd_hypervisor_12_to_stable/f9l00cz?utm_source=share&utm_medium=web2x) re. the missing system image:

> For ‘android-27;google_apis;x86_64’, there may not be a good reason for it missing. That was in a transitional period where we just moved to x86_64 kernels with x86 32-bit userspace, and we needed to manage the extra complexity from there and never got around to making the x86_64 userspace version. We should have a x86 64-bit userspace version of it at some point, though, but it's not currently on our list of stuff to do (Sorry).


### Build Cache?

One of the things I love the most about **CircirCI** is its native support for **caching**. Since we are able to cache the entire `~/.gradle` directly to get incredibly fast incremental builds with **CircleCI**, I wondered if I could do something similar to my **Github workflow** to make it even faster.

Turned out there is this [actions/cache](https://github.com/actions/cache) provided by GitHub for caching dependencies and build outputs.

We can cache the `~/.gradle/caches` directory by adding the following step to the job:

```
- uses: actions/cache@v1
  with:
    path: ~/.gradle/caches
    key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle*') }}-${{ hashFiles('**/gradle/wrapper/gradle-wrapper.properties') }}
    restore-keys: ${{ runner.os }}-gradle-
```

~~Unfortunately this won't work right now as there's a **200 MB** limit on the cache size which isn't enough for Gradle, although they are planning to [increase the limit to **2GB**](https://github.com/actions/cache/issues/6#issuecomment-557099690) in the future.~~

**Update**: per-cache size limit has been increased to **2GB** so the caching config above should now work.

## Journey never ends

While the **custom GitHub Action** has been working really well for **FlowBinding** since the migration, I'm still hoping to find a more generic and future-proof solution where everything runs within a docker container rather than directly on a macOS VM. It's important for me to be able to easily try out / migrate to other CI/CD providers in the future, and using a workflow that runs things directly on a macOS VM instead of a standard Docker environment would likely make it harder to do so.

I recently came across a less-known cloud-based CI provider with native KVM support, and is completely free for opensource projects. In a future article I'll explore and share my experience with this product and compare it with my current CI workflow.

Stay tuned!

-------------------------

Featured in [Android Weekly #390](https://androidweekly.net/issues/issue-390).