Android: Automate Instrumented and Unit Tests With CircleCI
Tags: circleci, cicd, android, testing
Published: 2023-05-10 09:17:05

As an Android developer, testing is an integral part of the app development process. By running tests you can verify your app’s functional behaviour, correctness, and usability before you release it to the public.  CircleCI is a CI platform that supports running Android emulators from within an Android machine image. We will use CircleCI to set up automated testing for our project. Automated testing involves using tools that perform tests on the project for you, which is faster, repeatable, and gives you more actionable feedback about your app earlier in the development process.

This article describes how to automate  instrumentation and unit tests for an open source Android project using CircleCI. I assume you already know how to create an Android app and write tests for your app.  The cloned project and working pipeline can be found [here](https://github.com/Roseline-Bassey/androidlibrary).

## Instrumentation Testing
Instrumented tests, also known as instrumentation tests, are tests that run on Android devices, whether physical or emulated. It involves testing the overall behaviour of an application by testing the interaction between different components. It checks how the application performs when different components work together, including user interfaces, APIs, and external dependencies. This type of testing helps identify issues that may arise due to interactions between components, such as compatibility issues or performance problems. 

## Unit Testing
Unit testing involves testing individual components or units of code in isolation to verify their functionality, usually a single class or function. In unit testing, each unit of code is tested separately from the rest of the codebase to ensure that it behaves as expected. 

## CircleCI
CircleCI is a hosted and automated solution for continuous integration (CI) builds. It uses an in app configuration file that uses YAML syntax.  CircleCI is easier to set up and is hosted in the cloud.  After learning about instrumentation tests, unit testing, and CircleCI, we can now learn how testing in CI/CD works.


## Unit Testing and Instrumentation Testing in CI/CD

As a  DevOps principle, it is a good practice to automate testing as part of the CI process to test your code against any bug. Automated tests can be triggered at various stages of your CircleCI pipeline, including development, staging, and/or production. An effective automated test runs immediately after changes are committed to the version control system's (for example, GitHub) staging or development repository. If the test meets the success criteria, then the committed change(s) may proceed to the next stage or environment. If the test failed, the committed changes are blocked from proceeding until those issues are resolved. By automating tests, teams can have confidence that code changes have been thoroughly tested and have met the success and quality standards, thereby, protecting the production environment against disruptions.

## Setting up Automated Android Testing on CircleCI

Firstly, we will set up our `config.yml`file to run unit tests. I'll demonstrate two approaches for running instrumentation tests on CircleCI: using the No-orb and Orb examples. The No-orb example is a more complex process because you would have to write the steps yourself, while the CircleCI Android orb example is an easy process with many steps already written for you. There's no need to worry, the No-orb example is not as difficult as it seems because CircleCi offers clear [documentation](https://circleci.com/docs/android-machine-image/) that will make setup simple. Please take note that you should only use one of the examples in your `config.yml` file. The two instances given here are merely for demonstration. Let's get started.

## Create a `circleci.yml` File
At the root of your project’s folder, create a `.circleci` folder. Inside the folder create a `config.yml` file. Our workflow will be in this order:

- Unit Tests
- Instrumentation Tests

### Running Unit Test
We will call our job `unit-tests`. The `working_directory` specifies the location of the project files. The docker image section specifies the Docker image to use. In this case, it uses the `cimg/android:2023.02` image.

Here’s a full look at the unit test job:

```yaml
jobs:
version: 2.
unit-tests:
    working_directory: ~/project
    docker:
      - image: cimg/android:2023.02
    steps:
      - checkout
      - run:
          name: Run Unit Tests
          command: |
            ./gradlew clean test
        - store_test_results:
            path: app/build/test-results

```
The steps section includes three steps: first CircleCI checks out the repository where the code is stored. Then, run unit tests using the `./gradlew clean test` command, and store the test results at the path specified. 

## Running Instrumentation Tests 
As I mentioned earlier, there are different approaches to doing this. We will cover two examples or approaches. The first example is the No-orb example. 

### Using the No-orb Example 

Our instrumented test requires an emulator to run on a machine image. Below is a breakdown of the instrumentation tests job. The full code is provided below. We will add each step to the instrumented test `job` in the `circleci.yml` file.

```yaml
jobs:
instrumented_test:
    machine:
      image: android:202102-01
    resource_class: large
``` 
So in the instrumented test job, we specify the `image: android:202102-01` as the machine image and set the `resource_class` of the machine as "large" to improve build time. Under the `step` section we have the following steps:

```yaml
steps:
      - checkout
      - run:
          name: Create avd
          command: |
            SYSTEM_IMAGES="system-images;android-29;default;x86"
            sdkmanager "$SYSTEM_IMAGES"
            echo "no" | avdmanager --verbose create avd -n test -k "$SYSTEM_IMAGES"
```
The `- checkout` checks out the code from the repository while the command under `- Create avd` installs the Android system image required for the emulator to run, and creates a new virtual device named "test".

```yaml
- run:
          name: Launch emulator
          command: |
            emulator -avd test -delay-adb -verbose -no-window -gpu swiftshader_indirect -no-snapshot -noaudio -no-boot-anim
          background: true
```
The `Launch emulator` command launches the emulator and waits for it to start up. It runs in the background (background: true) so that other steps can execute in parallel.

```yaml
 - run:
          name: Generate cache key
          command: |
            find . -name 'build.gradle' | sort | xargs cat |
            shasum | awk '{print $1}' > /tmp/gradle_cache_seed
```
The `Generate cache key` command generates a cache key based on the contents of the build.gradle files in the project. This key is used to restore the Gradle cache later, to speed up the build process.

```yaml
  - restore_cache:
          key: gradle-v1-{{ arch }}-{{ checksum "/tmp/gradle_cache_seed" }}
```
`restore_cache` step attempts to restore the Gradle cache from a previous build, using the cache key generated in the previous step.

```yaml
- run:
          # run in parallel with the emulator starting up, to optimize build time
          name: Run assembleDebugAndroidTest task
          command: |
            ./gradlew assembleDebugAndroidTest
```
`assembleDebugAndroidTest` command compiles and build the debug version of the Android test APK and runs in parallel with the emulator starting up, to optimize build time.

This step can be modified to suit your project needs. I modified the above☝️ step to this:👇

```yaml
 - run:
          # run in parallel with the emulator starting up, to optimize build time
          name: Run compileDemoBasicDebugJavaWithJavac task
          command: |
            ./gradlew compileDemoBasicDebugJavaWithJavac
```

This step compiles the Java code in the project using the javac compiler. This runs in parallel with the emulator starting up, to optimize build time.

```yaml
  - run:
          name: Wait for emulator to start
          command: |
            circle-android wait-for-boot
```
As the name implies, the `Wait for emulator to start`step waits for the emulator to finish booting up and become available for testing.

```yaml
   - run:
          name: Disable emulator animations
          command: |
            adb shell settings put global window_animation_scale 0.0
            adb shell settings put global transition_animation_scale 0.0
            adb shell settings put global animator_duration_scale 0.0
```
The `Disable emulator animations` step disables the emulator animations to speed up the testing process.

```yaml
 - run:
          name: Run UI tests (with retry)
          command: |
            MAX_TRIES=2
            run_with_retry() {
               n=1
               until [ $n -gt $MAX_TRIES ]
               do
                  echo "Starting test attempt $n"
                  ./gradlew connectedAndroidTest && break
                  n=$[$n+1]
                  sleep 5
               done
               if [ $n -gt $MAX_TRIES ]; then
                 echo "Max tries reached ($MAX_TRIES)"
                 exit 1
               fi
            }
            run_with_retry
```

`Run UI tests (with retry)` runs the instrumented tests on the emulator, with a retry mechanism in case of failures. The `connectedAndroidTest` command is the main command which runs the tests.

```yaml
 - save_cache:
          key: gradle-v1-{{ arch }}-{{ checksum "/tmp/gradle_cache_seed" }}
          paths:
            - ~/.gradle/caches
            - ~/.gradle/wrapper
```

`save_cache` saves the Gradle cache for future builds, using the cache key generated earlier.

Let's take a look at the whole `config.yml` file, which includes both the unit tests and the instrumentation tests jobs:

```yaml
jobs:
version: 2.
unit-tests:
    working_directory: ~/project
    docker:
      - image: cimg/android:2023.02
    steps:
      - checkout
      - run:
          name: Run Unit Tests
          command: |
            ./gradlew clean test

      - store_test_results:
          path: app\build\test-results

instrumented_test:
    machine:
      image: android:202102-01
    resource_class: large
    steps:
      - checkout
      - run:
          name: Create avd
          command: |
            SYSTEM_IMAGES="system-images;android-29;default;x86"
            sdkmanager "$SYSTEM_IMAGES"
            echo "no" | avdmanager --verbose create avd -n test -k "$SYSTEM_IMAGES"
      - run:
          name: Launch emulator
          command: |
            emulator -avd test -delay-adb -verbose -no-window -gpu swiftshader_indirect -no-snapshot -noaudio -no-boot-anim
          background: true
      - run:
          name: Generate cache key
          command: |
            find . -name 'build.gradle' | sort | xargs cat |
            shasum | awk '{print $1}' > /tmp/gradle_cache_seed
      - restore_cache:
          key: gradle-v1-{{ arch }}-{{ checksum "/tmp/gradle_cache_seed" }}
      - run:
          # run in parallel with the emulator starting up, to optimize build time
          name: Run assembleDebugAndroidTest task
          command: |
            ./gradlew assembleDebugAndroidTest
      - run:
          name: Wait for emulator to start
          command: |
            circle-android wait-for-boot
      - run:
          name: Disable emulator animations
          command: |
            adb shell settings put global window_animation_scale 0.0
            adb shell settings put global transition_animation_scale 0.0
            adb shell settings put global animator_duration_scale 0.0
      - run:
          name: Run UI tests (with retry)
          command: |
            MAX_TRIES=2
            run_with_retry() {
               n=1
               until [ $n -gt $MAX_TRIES ]
               do
                  echo "Starting test attempt $n"
                  ./gradlew connectedAndroidTest && break
                  n=$[$n+1]
                  sleep 5
               done
               if [ $n -gt $MAX_TRIES ]; then
                 echo "Max tries reached ($MAX_TRIES)"
                 exit 1
               fi
            }
            run_with_retry
      - save_cache:
          key: gradle-v1-{{ arch }}-{{ checksum "/tmp/gradle_cache_seed" }}
          paths:
            - ~/.gradle/caches
            - ~/.gradle/wrapper
      - store_test_results:
          path: app\build\outputs\androidTest-results
```

## Using Android Orb Example
Since CircleCi has already done some of the work, this step is a bit shorter than the first example. So, you can use this instead of the No-orb example and it'll still function the same.

```yaml

version: 2.1
orbs:
  android: circleci/android@2.1.2
jobs: 
  android-test:
    executor:
      name: android/android-machine
      tag: "202102-01"
      resource-class: large
    #run instrumentation tests

    steps:
      - checkout
      - run:
          name: installing emulator and Running Instrumentation tests
          command:  |
            sdkmanager "platform-tools" "platforms;android-29" "build-tools;30.0.0" "emulator"
            sdkmanager "system-images;android-29;google_apis;x86"
            echo no | avdmanager create avd -n test-emulator -k  "system-images;android-29;google_apis;x86"
            emulator -avd test-emulator -noaudio -no-boot-anim -gpu off -no-window &
            adb wait-for-device shell 'while [[ -z $(getprop sys.boot_completed) ]]; do sleep 1; done;'
            adb shell wm dismiss-keyguard
            sleep 1
            adb shell settings put global window_animation_scale 0
            adb shell settings put global transition_animation_scale 0
            adb shell settings put global animator_duration_scale 0
            ./gradlew connectedAndroidTest
       
```
It's time to see our CI test in action on CircleCI.

## How to Setup CircleCI
Make sure you have created at the root of your project’s folder, a `.circleci` folder, and inside of the folder you have a `config.yml` file. Before moving on to CircleCI, let's add a build job at the beginning of the `config.yml` file and define the workflow of our build pipeline at the end of the file. Using the no-orb example, the final `config.yml` file will look like this: 

```yaml
version: 2.1
jobs:
  build:
    working_directory: ~/project
    docker:
      - image: cimg/android:2023.02
    steps:
      - checkout
      - restore_cache:
          keys:
            - v1-dependencies-{{ checksum "build.gradle" }}
            - v1-dependencies-
      - run:
          name: Install dependencies
          command: ./gradlew androidDependencies

      - save_cache:
          paths:
            - ~/.gradle
            - ~/.android
          key: v1-dependencies-{{ checksum "build.gradle" }}

      - run:
          name: Build project
          command: ./gradlew clean assemble
 
unit tests:
    working_directory: ~/project
    docker:
      - image: cimg/android:2023.02
    steps:
      - checkout
      - run:
          name: Run Local UnitTests
          command: |
            ./gradlew clean test

      - store_test_results:
          path: app\build\test-results


  instrumented_test:
    machine:
      image: android:202102-01
    resource_class: large
    steps:
      - checkout
      - run:
          name: Create avd
          command: |
            SYSTEM_IMAGES="system-images;android-29;default;x86"
            sdkmanager "$SYSTEM_IMAGES"
            echo "no" | avdmanager --verbose create avd -n test -k "$SYSTEM_IMAGES"
      - run:
          name: Launch emulator
          command: |
            emulator -avd test -delay-adb -verbose -no-window -gpu swiftshader_indirect -no-snapshot -noaudio -no-boot-anim
          background: true
      - run:
          name: Generate cache key
          command: |
            find . -name 'build.gradle' | sort | xargs cat |
            shasum | awk '{print $1}' > /tmp/gradle_cache_seed
      - restore_cache:
          key: gradle-v1-{{ arch }}-{{ checksum "/tmp/gradle_cache_seed" }}
      - run:
          # run in parallel with the emulator starting up, to optimize build time
          name: Run compileDemoBasicDebugJavaWithJavac task
          command: |
            ./gradlew compileDemoBasicDebugJavaWithJavac
      - run:
          name: Wait for emulator to start
          command: |
            circle-android wait-for-boot
      - run:
          name: Disable emulator animations
          command: |
            adb shell settings put global window_animation_scale 0.0
            adb shell settings put global transition_animation_scale 0.0
            adb shell settings put global animator_duration_scale 0.0
      - run:
          name: Run UI tests (with retry)
          command: |
            MAX_TRIES=2
            run_with_retry() {
               n=1
               until [ $n -gt $MAX_TRIES ]
               do
                  echo "Starting test attempt $n"
                  ./gradlew connectedAndroidTest && break
                  n=$[$n+1]
                  sleep 5
               done
               if [ $n -gt $MAX_TRIES ]; then
                 echo "Max tries reached ($MAX_TRIES)"
                 exit 1
               fi
            }
            run_with_retry
      - save_cache:
          key: gradle-v1-{{ arch }}-{{ checksum "/tmp/gradle_cache_seed" }}
          paths:
            - ~/.gradle/caches
            - ~/.gradle/wrapper
      - store_test_results:
          path: app\build\outputs\androidTest-results

workflows:
  build_and_deploy:
    jobs:
      - build
      - instrumented_test:
          requires:
            - build
      - unit tests:
          requires:
            - build
```

Next, login to CircleCI using your GitHub, BitBucket, or GitLab account. In the CircleCI web, click on projects at the sidebar and then select which project on your version control you want CircleCI to build on. In my case, I selected the `androidlibrary`project.

![CircleCI build dashboard](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/998tfgnpebbxagxbtnnf.png)
In the “Select your config.yml file” modal, select **Fastest**, and a pop-up will appear like so:

![CircleCI](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8er29i8oqz0padvhuid2.png)
Under **fastest** select to choose your preferred project and the branch where the project is located, then click **Set Up Project** to start building your project on CircleCI. You can monitor the build process on your dashboard. On a successful build, you'll have a green build like below:

![CircleCI Dashboard](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fvuqxxcujur7pc5woqdj.png)

Note that the instrumentation tests job may take a few more minutes than other jobs to build. The time taken is solely dependent on your project. It took almost five minutes for the instrumentation test to finish building in my case.

![CircleCi dashboard showing build time](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pjewk5lykmt2ye4uiz4m.png)

## Recap
In this article, you learned how to automate testing for your Android project using CircleCI. How to set up unit tests and also learned two approaches to setting up instrumentation tests on CirCleCI. You've learned the benefits of automating testing in CI and you've seen the CI test in action on CircleCI.