Around the World in 8 remote cache nodes
Tags: gradle, android, cache
Published: 2023-08-04 00:00:15

Nowadays, distributed teams across different regions or countries are common in organizations. Depending on the bandwidth and latency, we could experience a degraded experience in our builds when consuming the cache entries. 

This article shares the results of measuring builds in the project [nowinandroid](https://github.com/android/nowinandroid) using different Remote Cache nodes. 

**The Local environment**
To simulate the "local" build, we are using a GitHub Action runner with:
 * 2-core CPU (x86_64)
 * 7 GB of RAM 

We are not able to choose the runner location in GitHub Action. Following the doc, the supported Azure regions are:
* East US (eastus)
* East US 2 (eastus2)
* West US 2 (westus2)
* Central US (centralus)
* South Central US (southcentralus)

So we assume our "local" builds will be executed somewhere in the US. 

**The Project** 
The project used for this experiment is nowinandroid([commit id](https://github.com/android/nowinandroid/commit/2ac13e821e08a96ef10098784dc00d670fb04110)), build stack components at that point: 
* Gradle 8.2
* AGP 8.0.2
* KGP 1.8.22

**Cache Nodes**
We created a Remote Cache node for each scenario defined in different [GCP](https://cloud.google.com/about/locations) zones. 

| Id                  | Region         |
|---------------------|----------------|
| us-east             | South Carolina |
| us-west             | Oregon         |
| europe-west         | Belgium        |
| me-west             | Tel Aviv       |
| southamerica-east   | São Paulo      |
| asia-south          | Mumbai         |
| australia-southeast | Sydney         |
| asia-northeast      | Tokyo          |
 
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/itw1b9fj45htt3z9r7oz.png)

**Methodology**
* Each scenario is configured pointing to one specific Remote cache node.
* Each scenario executes 11 builds, the first build populates the cache. Then, 10 builds are executed in parallel reusing the cache entries. 
* The task executed in this experiment is "test". After populating the cache, the build gets 1859 tasks from the remote cache.
* All builds are tagged in the Build Scan, including the scenario and execution number.
* Using [Gradle Enterprise API](https://docs.gradle.com/enterprise/api-manual), we fetch and aggregate the build information. 

**Results**
We start analyzing the results of measuring the median for the task ":core:analytics:compileDemoReleaseJavaWithJavac":

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4mkupphzr1fgyk47j3xe.png)

We don't observe a significant difference using different locations for the remote cache. 
However, the output of this task is small (8.4 KiB). Let's see what happens when we analyze the median of a task with a bigger output size like ":app:mergeProdDebugResources" (950.1 KiB):

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ue0svovigb6gyrfvr0k3.png)

We observed a considerable difference across remote cache node locations once the task increased the output size.


Finally, instead of analyzing single tasks, we aggregated the results by task type. AGP task "MergeResources":

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wcjkz585wardet0g19fj.png)

For KGP, these are the results of the median aggregated task "KotlinCompile":

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wix9twfwb0c6abghjpoo.png)

Complete results table grouped by task type (ms):

| Task Type                           | us-east | us-west | europe-west | me-west | southamerica-east | asia-south | australia-southeast | asia-northeast |
|-------------------------------------|---------|---------|-------------|---------|-------------------|------------|---------------------|----------------|
| KotlinCompile                       |      71 |      82 |          67 |     207 |               162 |        259 |                 198 |            161 |
| MergeResources                      |      56 |      71 |          47 |   182.5 |               146 |        241 |                 183 |            141 |
| KspTaskJvm                          |    81.5 |    85.5 |          79 |   216.5 |             166.5 |      261.5 |               203.5 |          169.5 |
| AggregateDepsTask                   |     211 |     227 |         195 |     728 |               592 |      951.5 |               732.5 |          558.5 |
| PackageForUnitTest                  |      56 |      74 |          49 |     189 |               148 |        248 |               188.5 |          142.5 |
| MergeSourceSetFolders               |      48 |      63 |          41 |     179 |               143 |        233 |                 179 |            138 |
| CompileLibraryResourcesTask         |      47 |    58.5 |          40 |     178 |               143 |        228 |                 178 |            137 |
| ExtractDeepLinksTask                |      47 |      58 |          40 |     177 |               143 |        227 |                 178 |            137 |
| ZipMergingTask                      |      49 |      64 |          41 |   179.5 |               134 |        228 |                 180 |            137 |
| JavaPreCompileTask                  |      47 |      57 |          39 |     178 |               143 |        227 |                 178 |            137 |
| ParseLibraryResourcesTask           |      47 |      58 |          40 |     178 |               143 |        227 |                 178 |            137 |
| GenerateBuildConfig                 |      48 |    61.5 |          41 |     178 |             137.5 |      225.5 |               178.5 |          129.5 |
| ProcessMultiApkApplicationManifest  |      48 |      64 |          39 |     179 |               135 |      224.5 |                 179 |          129.5 |
| GenerateLibraryRFileTask            |      50 |      65 |          43 |     181 |               145 |        234 |                 180 |            138 |
| ProcessLibraryManifest              |      49 |      65 |          42 |     180 |               143 |        229 |                 180 |            138 |
| TransformClassesWithAsmTask         |      59 |      73 |          53 |     188 |               148 |        245 |                 186 |            144 |
| ProcessPackagedManifestTask         |      48 |      64 |          39 |     179 |             138.5 |        225 |                 179 |            127 |
| GenerateProtoTask                   |      61 |      73 |          51 |     192 |               149 |        239 |                 187 |            144 |
| ProcessApplicationManifest          |      70 |      76 |        62.5 |   189.5 |             152.5 |      248.5 |                 188 |            149 |
| KaptGenerateStubsTask               |      91 |     101 |          88 |     305 |               247 |        455 |                 313 |            234 |
| JavaCompile                         |      68 |      78 |          62 |     196 |               154 |        250 |                 191 |            151 |
| LinkApplicationAndroidResourcesTask |      82 |      87 |        75.5 |     222 |               162 |      267.5 |                 207 |          168.5 |
| KaptWithoutKotlincTask              |      83 |   101.5 |          81 |     302 |               244 |        453 |                 307 |            231 |
| AndroidUnitTest                     |      61 |      76 |          53 |     192 |               155 |        242 |                 190 |            149 |



**Final Words**

Caching continues to be the most simple and effective mechanism to optimize the build execution of Android projects. 
Highly modularized projects enjoy the savings provided by reusing the outputs of the tasks, not only in terms of duration but reducing the pressure on the environment's resources executing the build.

Cache Node Replication improves the experience of teams in regions far from the cache origin by creating closer cache components for the user. There are different resources and articles explaining this strategy:

* https://docs.gradle.com/enterprise/helm-admin/current/#replication
* https://gradle.com/blog/managing-a-remote-build-cache-at-scale-with-local-build-observability-at-cash-app/ 
* https://dev.to/thedancercodes/gradle-build-cache-node-replication-in-android-14bc

Happy Building!