Benchmarking Gradle Builds Using Gradle-Profiler
Tags: softwaredevelopment, gradle, android
Published: 2021-04-13 17:31:36

---
title: Benchmarking Gradle Builds Using Gradle-Profiler
published: true
date: 2020-11-12 17:38:59 UTC
tags: softwaredevelopment,gradle,android
canonical_url: https://goobar.io/benchmarking-gradle-builds-using-gradle-profiler/
---

In this post, we’re going to take an introductory look at benchmarking Gradle build performance using the gradle-profiler tool.  

By the end, you should have a basic understanding of how to use gradle-profiler to gain better insights into the performance of your Gradle build and how you might use those insights to improve Gradle build performance for your project.

## What is the Gradle-Profiler Tool?

The [gradle-profiler](https://github.com/gradle/gradle-profiler) project describes itself in the following way:

> “A tool to automate the gathering of profiling and benchmarking information for Gradle builds.”

What does that mean to you and your project?

Imagine you want to get a sense of how fast your Gradle build is.

You might run your target Gradle task, wait for the build to complete, and take that build time as your result.

Now, because build times are often variable, you may want to run your Gradle task multiple times and average the execution times.  So, you kick off your build, wait for it to finish, write down the total execution time, and repeat the process.  

This process of manually benchmarking your Gradle build will likely take a while.  Depending on how long your build takes to complete, and how many interactions you want to run, you could find yourself repeatedly coming back to your computer to check whether it’s time to start the next build or not.  Even if you stay busy with other things, this is still a drawn out, tedious process that relies on you, as the developer, manually recording build statistics for each iteration.

It’s this process that gradle-profiler aims to automate for you.

Rather than you having to manually kick off each build and recording the resulting build statistics, gradle-profiler allows devs to start a single command which will repeatedly run a specified Gradle task and record all the build stats in an easy to examine output format.

This takes Gradle build benchmarking from a tedious, manual task to something that can be highly automated with little need for human interaction or monitoring.

I’ve been using gradle-profiler recently in an effort to keep my primary project’s build times low, and to examine the build impact of proposed changes.  In this post, we’re going to walk through how you can start using gradle-profiler to gather benchmark data for your Gradle build, and how you may start using that data to understand the impact of changes on your project. 

## Installing Gradle-Profiler

Before you can start benchmarking Gradle build performance, you’ll need to install gradle-profiler to your development machine.

You can do this in one of several ways

### Install From Source

You could clone the git repository to your local machine, and [build the project from source](https://github.com/gradle/gradle-profiler#build-from-source).

```
→ git clone git@github.com:gradle/gradle-profiler.git
→ cd gradle-profiler
→ ./gradlew installDist
```

You would likely then want to add _`gradle-profiler/build/install/gradle-profiler/bin`_ to your path or create some kind of alias that enables you to invoke the tool by executing the _gradle-profiler_ command.

### Install With Homebrew

If you are using [Homebrew](https://brew.sh/) on your machine, installation is quite simple.

```
→ brew install gradle-profiler
```

### Other Installation Options

If building from source, or installing using Homebrew, aren’t good options for you, you could try either of the following installation methods:

- [Installing using SDKMAN](https://github.com/gradle/gradle-profiler#sdkman)
- [Downloading binaries directly from GitHub](https://github.com/gradle/gradle-profiler#download-binaries)

<section><div></div>
<div>
<div><div><div>
<div>
<div><h3>Examples On GitHub</h3></div>
<div><p>You can find example commands and example benchmark scenarios in my sample repo on GitHub.</p></div>
</div>
<div><div><a href="https://github.com/n8ebel/Gradle-Profiler-Sandbox" target="_blank" rel="noopener noreferrer"><span><span>Find The Code</span></span></a></div></div>
</div></div></div>
</div></section>

## Benchmarking Your Gradle Build

Now that gradle-profiler is installed, and the _gradle-profiler_ command is available to us, let’s start benchmarking Gradle build performance for your project.

### Running the Benchmarking Tool

To generate benchmarking results for our project, we need two things:

1. The path to the project directory containing the Gradle project
2. A Gradle task to run

With these, we can start benchmarking our build like this:

```
→ cd <project directory>→ gradle-profiler --benchmark --project-dir . assemble
```

Let’s break down this command into its individual parts.

- _`gradle-profiler`_ – invokes the gradle-profiler tool
- _`--benchmark`_ – indicates that we want to benchmark our build
- _`--project-dir .`_ – indicates that the Gradle project is located within the current working directory
- _`assemble`_ – this is the Gradle task to benchmark 

When this command is run, the benchmarking tool will begin.  You should see output in your console indicating that _warm-up_ and _measured_ builds are running.

When all the builds are completed, two benchmarking artifacts should be created for you:

1. _`benchmark.csv`_ – provides benchmarking data in a simple .csv format
2. _`benchmark.html`_ – provides an interactive webpage report based on the .csv data

The output paths should look something like this:

```
Results written to /Users/n8ebel/Projects/GradleProfilerSandbox/profile-out
/Users/n8ebel/Projects/GradleProfilerSandbox/profile-out/benchmark.csv
/Users/n8ebel/Projects/GradleProfilerSandbox/profile-out/benchmark.html
```

### Understanding Benchmarking Results

Once your outputs are generated, you can use them to explore the benchmarking results.

#### Interpreting .csv results

Here’s a sample _`benchmark.csv`_ output generated by benchmarking the _assemble_ task for a new Android Studio project.

| **scenario** | **default** |
| **version** | Gradle 6.7 |
| **tasks** | assemble |
| **value** | execution |
| **warm-up build #1** | 45574 |
| **warm-up build #2** | 2149 |
| **warm-up build #3** | 1778 |
| **warm-up build #4** | 1772 |
| **warm-up build #5** | 1436 |
| **warm-up build #6** | 1474 |
| **measured build #1** | 1247 |
| **measured build #2** | 1370 |
| **measured build #3** | 1267 |
| **measured build #4** | 1217 |
| **measured build #5** | 1305 |
| **measured build #6** | 1103 |
| **measured build #7** | 973 |
| **measured build #8** | 1007 |
| **measured build #9** | 999 |
| **measured build #10** | 1151 |

<figcaption>CSV benchmark results for <em>assemble</em> task</figcaption>

This report is very streamlined and highlights only a few things.  The most interesting data points are likely:

- Which version of Gradle was used
- Which Gradle tasks were run
- The task execution time, in milliseconds, for the _warm-up_ and _measured_ builds

Notice that the first _warm-up_ build took significantly longer than every other build?  Is this a problem?  Is something wrong with your project’s configuration?

By default, _gradle-profiler_ uses a warm Gradle daemon when measuring build times.  If you’re not familiar, the [Gradle daemon](https://docs.gradle.org/current/userguide/gradle_daemon.html) runs in the background to avoid repeatedly paying JVM startup costs.  It can drastically improve the performance of your Gradle builds.

So, for this first _warm-up_ build, task execution time is much longer as the daemon is started.  After that, you can see that subsequent builds are much faster.

If we ignore the _warm-up_ builds, and look only at the set of _measured_ builds, we see that the build times are consistently fast as one might expect for a new project.  

#### Interpreting HTML results  

In addition to being an interactive webpage, the _`benchmarking.html`_ results provide more data than the .csv file.  By viewing the results this way, you’ll have access to:

- Mean, Median, StdDev, and other statistics from the _measured_ build results
- Gradle argument details
- JVM argument details
- And more…

![gradle-profiler HTML benchmarking results for a single assemble task](https://i2.wp.com/goobar.io/wp-content/uploads/2020/11/Screen-Shot-2020-11-12-at-7.28.28-AM-1329778080-1605209269815.png?resize=800%2C562&ssl=1)<figcaption>HTML benchmark results for <em>assemble</em> task</figcaption>

The HTML results provide a graph of each _warm-up_ and _measured_ build to help you visually understand performance over time.

If you aren’t interested in automating the analysis of your benchmarking results, then viewing the HTML results is often the most convenient way to quickly understand how long your build takes, and to quickly see if there are any outlier executions to examine.

This HTML view becomes even more useful when benchmarking multiple scenarios at the same time as we’ll see later on.

## Detecting Gradle Performance Issues

Now that we have some understanding of the output generated by gradle-profiler benchmarking, let’s explore how we might begin to use that benchmark data to compare the performance of our Gradle builds.

The simplest approach is generally as follows:

- Collect benchmarking data for your current/default Gradle configuration
- Change your Gradle configuration
- Collect benchmarking data for updated configuration
- Compare the results

We could use this approach to compare the impact of caching on a build.  We might compare the performance of clean builds versus incremental builds. 

Any tweak to our build settings or project structure could be examined through this type of comparison.

### Comparing _clean_ and _up-to-date_ build performance

Let’s walk through a quick example of this kind of analysis using _gradle-profiler._  We’re going to compare the impact of an _up-to-date_ build over a _clean_ build.

First we’ll benchmark _up-to-date_ builds of our _assemble_ task and collect the results.

```
→ gradle-profiler --benchmark --project-dir . assemble
```

We’re referring this as the _up-to-date_ scenario because after the first execution of the _assmble_ task, each task should be up-to-date and subsequent builds should be extremely fast as there is nothing to rebuild.

Next, we’ll benchmark our _clean_ build.

```
→ gradle-profiler --benchmark --project-dir . clean assemble
```

In the _clean_ build scenario, we are discarded previous outputs before executing our _assemble_ task, so we should expect the _clean_ build to take longer than the _up-to-date_ build.

Once we’ve generated both sets of output, we can compare the benchmarked performance. I’ve taken the .csv results from each of those benchmarking executions and combined them into the following table for comparison.

| **scenario** | **default** | **scenario** | **default** |
| **version** | Gradle 6.7 | version | Gradle 6.7 |
| **tasks** | assemble | tasks | clean assemble |
| **value** | execution | value | execution |
| **warm-up build #1** | 14330 | warm-up build #1 | 26492 |
| **warm-up build #2** | 1887 | warm-up build #2 | 10345 |
| **warm-up build #3** | 1546 | warm-up build #3 | 9853 |
| **warm-up build #4** | 1440 | warm-up build #4 | 8757 |
| **warm-up build #5** | 1383 | warm-up build #5 | 8705 |
| **warm-up build #6** | 1301 | warm-up build #6 | 7377 |
| **measured build #1** | 1187 | measured build #1 | 7268 |
| **measured build #2** | 1230 | measured build #2 | 7378 |
| **measured build #3** | 1118 | measured build #3 | 7750 |
| **measured build #4** | 1104 | measured build #4 | 6707 |
| **measured build #5** | 1105 | measured build #5 | 6635 |
| **measured build #6** | 1082 | measured build #6 | 7542 |
| **measured build #7** | 1044 | measured build #7 | 7066 |
| **measured build #8** | 990 | measured build #8 | 6398 |
| **measured build #9** | 993 | measured build #9 | 6341 |
| **measured build #10** | 1037 | measured build #10 | 7416 |

With this, we can see that our _up-to-date_ build takes ~1 second while our _clean_ build is taking ~7 seconds. This seems in line with expectations about the relative performance of these two build types.

### Comparing caching impact

We’ve seen the performance of an _up-to-date_ build.  We’ve seen the impact of doing a _clean_ build. 

Let’s expand on our example by now benchmarking the impact of enabling [Gradle’s local build cache](https://docs.gradle.org/current/userguide/build_cache.html#sec:build_cache_enable).

First, we’ll once again benchmark a _clean_ build without enabling the build cache.

```
→ gradle-profiler --benchmark --project-dir . clean assemble
```

Next, we’ll enable Gradle’s local build cache by adding the following to our _`gradle.properties`_ file:

```
_org.gradle.caching_=true
```

Now, we can re-run our _clean_ build benchmark scenario; this time with the cache enabled.

```
→ gradle-profiler --benchmark --project-dir . clean assemble
```

Once again, we can compare results to get a sense of how enabling the local Gradle build cache can benefit build performance.

| No Caching | | Caching | |
| --- | --- | --- | --- |
| **scenario** | **default** | **scenario** | **default** |
| version | Gradle 6.7 | version | Gradle 6.7 |
| tasks | clean assemble | tasks | clean assemble |
| value | execution | value | execution |
| warm-up build #1 | 26492 | warm-up build #1 | 24568 |
| warm-up build #2 | 10345 | warm-up build #2 | 6838 |
| warm-up build #3 | 9853 | warm-up build #3 | 4901 |
| warm-up build #4 | 8757 | warm-up build #4 | 5145 |
| warm-up build #5 | 8705 | warm-up build #5 | 4382 |
| warm-up build #6 | 7377 | warm-up build #6 | 5309 |
| measured build #1 | 7268 | measured build #1 | 4825 |
| measured build #2 | 7378 | measured build #2 | 4674 |
| measured build #3 | 7750 | measured build #3 | 4428 |
| measured build #4 | 6707 | measured build #4 | 4577 |
| measured build #5 | 6635 | measured build #5 | 4053 |
| measured build #6 | 7542 | measured build #6 | 4236 |
| measured build #7 | 7066 | measured build #7 | 4388 |
| measured build #8 | 6398 | measured build #8 | 4131 |
| measured build #9 | 6341 | measured build #9 | 5818 |
| measured build #10 | 7416 | measured build #10 | 4016 |

<figcaption>Merged CSV results comparing the build speed impact of enabling the Gradle build cache</figcaption>

From these results, we see that enabling local caching seems to improve the performance of _clean_ builds from ~7 seconds to ~4.5 seconds.

Now, these build times are artificially fast as it’s a simple project.

However, this approach is applicable to your real-world projects, and these general results are what one might expect from a well-configured project; `up-to-date` < `clean with caching` < `clean w/out caching`.

## Configuring Benchmarking Behavior

When running these benchmarking tasks, you may find yourself wanting greater control over how benchmarking is carried out.  A few of the common configuration properties you may find yourself needing to change include

- changing the project directory
- modifying the output directory
- updating the number of build iterations

We’re going to quickly examine how you can update these properties when executing your benchmarking task.

### Changing Project Directory

Throughout these examples, we’ve been operating as if we are running _gradle-profiler_ from within the project directory.  

Here, we see that after the _`--project-dir`_ flag, we pass `.` to signify the current directory.

```
→ gradle-profiler --benchmark --project-dir . assemble
```

If we want to run _gradle-profiler_ from any other directory, we are free to do that.  We just need to update the path to the project directory.

In this updated example, we change directories into our user directory, and pass `Projects/GradleProfilerSandbox` as the path to our project directory.

```
→ cd
→ gradle-profiler --benchmark --project-dir Projects/GradleProfilerSandbox/ clean assemble
```

### Changing Output Directory

When _gradle-profiler_ is run, by default, it will store output artifacts in the current directory.  If you would prefer to specify a different output directory, you can do so using _`--output-dir`_.

```
gradle-profiler --benchmark --project-dir Projects/GradleProfilerSandbox/ --output-dir Projects/benchmarking/ clean assemble
```

This may come in handy if you want to automate the running of benchmarking tasks and would like to keep all your outputs collected within a specific working directory.

### Controlling Build Warm-Ups and Iterations

Another pair of useful configuration options are _`--warmups`_ and _`--iterations`_.  These two flags allow you to control how many _warm-up_ builds and _measured_ builds to run.

This might be useful to you if you want to receive your results more quickly, or if you want more data points, and hopefully greater confidence, if your benchmarking results.

If we wanted to have 10 _warm-up_ builds and _15_ measured builds, we can start our benchmarking task like this.

```
→ gradle-profiler --benchmark --project-dir . --warmups 10 --iterations 15 assemble
```

## Benchmarking Complex Gradle Builds

We’ve really only scratched the surface of what gradle-profiler is capable of.  Real world build scenarios are varied, and often quite complex.  Ideally, we’d be able to capture these complexities, and benchmark Gradle build performance for these real-world conditions.

Let’s take a look at how we can define benchmark scenarios that give greater control and flexibility into what is benchmarked.

### Defining a Benchmark Scenario

As our command line inputs become more complex, it may become difficult to manage. 

Look at the following command.

```
gradle-profiler --benchmark --project-dir Projects/GradleProfilerSandbox/ --output-dir Projects/benchmarking/ --warmups10 --iterations 15 clean assemble
```

This command is still executing a fairly simple benchmarking task, but has already become quite long and unwieldily to execute from the command line every time.

This is especially true if we want to start benchmarking multiple build types, or want to simulate complex incremental build changes.

To help define complex build scenarios, the _gradle-profiler_ tool provides a mechanism for encapsulating all of the configuration for a build scenario into a _`.scenarios`_ file.  This helps us organize our scenarios in a single place and makes it easier to benchmarking multiple scenarios.

To define a simple _`.scenarios`_ file, we’ll do the following.

- First, we’ll create a new file named _`benchmarking.scenarios`_.
- Next, we’ll define a scenario for our _up-to-date_ _assemble_ task.

```
assemble_no_op {
  tasks = ["assemble"]
}
```

In this small configuration block, we’ve defined a scenario named `assemble_no_op` that will run the `assemble` task when executed. We’ve used the “no op” suffix on the scenario name because this will test our _up-to-date_ build in which tasks shouldn’t have to be re-run each time.

### Benchmarking With a Scenarios File

With our scenario file defined, we can benchmark this scenario by using the _`--scenario-file`_ flag.

```
gradle-profiler --benchmark --project-dir . --scenario-file benchmarking.scenarios
```

From this, we get an HTML output similar to this.

![HTML benchmark results](https://i1.wp.com/goobar.io/wp-content/uploads/2020/11/assemble_no_op.png?resize=800%2C336&ssl=1)<figcaption>HTML output from running the assemble_no_op benchmark scenario</figcaption>

We can see that the _assemble\_no\_op_ name used to define our scenario is automatically used as the scenario name in the output.

In the next sections, we’ll see how to changes this to something more human-readable and why this can be important.

### Configuring Build Scenarios

Within our _`.scenarios`_ file, there are quite a few configuration options we can use to control our benchmarked scenarios.

We can provide a human-readable title for our scenario:

```
assemble_no_op {
  title = "Up-To-Date Assemble"
  tasks = ["assemble"]
}
```

We can provide multiple Gradle tasks to run:

```
assemble_clean {
  title = "Clean Assemble"
  tasks = ["clean", "assemble"]
}
```

You could pass in different Gradle build flags such as for parallel execution or for the build cache:

```
assemble_clean {
  title = "Clean Assemble"
  tasks = ["clean", "assemble"]
  gradle-args = ["--parallel"]
}
```

We can also explicitly define the number of _warm-ups_ and _iterations_ so they don’t have to be passed from the command line:

```
assemble_clean {
  title = "Clean Assemble"
  tasks = ["clean", "assemble"]
  gradle-args = ["--parallel"]
  warm-ups = 3
  iterations = 5
}
```

This is not an exhaustive list of scenario configurations.  You can find more examples in the _gradle-profiler_ [documentation](https://github.com/gradle/gradle-profiler#advanced-profiling-scenarios).

### Benchmarking Multiple Build Scenarios

One of the primary benefits of using a `.scenarios` file is that we can define multiple scenarios within a single file, and benchmark multiple scenarios at the same time.

For example, we could compare our _up-to-date, clean, clean w/caching_ builds from a single benchmarking execution, and receive a single set of output reports comparing all of them.

To do this, we first define each of our unique build scenarios within our `benchmarking.scenarios` file.

```
assemble_clean {
  title = "Clean Assemble"
  tasks = ["clean", "assemble"]
}

assemble_clean_caching {
  title = "Clean Assemble w/ Caching"
  tasks = ["clean", "assemble"]
  gradle-args = ["--build-cache"]
}

assemble_no_op {
  title = "Up-To-Date Assemble"
  tasks = ["assemble"]
}
```

Then we continue to invoke _gradle-profiler_ in the same way; by specifying the single _`.scenarios`_ file.

```
gradle-profiler --benchmark --project-dir . --scenario-file benchmarking.scenarios
```

From this, we will receive a merge report that compares the performance of each scenario.  

![HTML gradle-profiler output for multiple build scenarios](https://i1.wp.com/goobar.io/wp-content/uploads/2020/11/multiple_scenarios_no_baseline.png?resize=800%2C395&ssl=1)<figcaption>HTML output for multiple build scenarios</figcaption>

Scenarios will be run in alphabetical order based on the name used in the scenario definition; not the human-readable title.  

When viewing a report with multiple scenarios, you can select a scenario as the _baseline_.  This will update the report to display a +/-% on each build metric where the +/-% is the statistical difference between the current scenario and the baseline scenario.

What does that look like in practice?

In this example, we’ve selected the _clean build w/out caching_ scenario as our baseline.

![HTML gradle-profiler output for multiple scenarios with a selected baseline](https://i2.wp.com/goobar.io/wp-content/uploads/2020/11/multiple_scenarios_with_baseline.png?resize=800%2C427&ssl=1)<figcaption>HTML output for multiple build scenarios with a selected baseline</figcaption>

With the baseline set, we see that the mean build time was reduced ~4% for the _clean build with caching_ scenario and ~80% for the _up-to-date_ build scenario.

By setting a baseline, you let the tool do all the statistical analysis for you leaving you free to interpret and share the results.

## Benchmarking Incremental Gradle Builds

The last concept we’re going to touch on is that of benchmarking incremental builds.  These are builds in which we need to re-execute a subset of tasks because of file changes.  These files could change due to source file changes, resource updates, etc.

Incremental scenarios can be very important for real-world benchmarking of Gradle build performance, because in our day-to-day work, we’re often performing incremental builds as we make a small code change, and redeploy for testing.

When using _gradle-profiler_ we have [quite a few options available](https://github.com/gradle/gradle-profiler#profiling-incremental-builds) to us for defining and benchmarking incremental build scenarios.

If building with Kotlin or Java, we might be interested in:

- apply-abi-change-to
- apply-non-abi-change-to

If building an Android application, we might use:

- apply-android-resource-change-to
- apply-android-resource-value-change-to
- apply-android-manifest-change-to
- apply-android-layout-change-to

With these options, and others, we can simulate different types of changes to our projects to benchmark real world scenarios.

Here, we’ve defined a new incremental scenario in our _benchmarking.scenarios_ file.

```
incremental_build {
  title = "Incremental Assemble w/ Caching"
  tasks = ["assemble"]

  apply-abi-change-to = "app/src/main/java/com/goobar/gradleprofilersandbox/MainActivity.kt"
  apply-android-resource-change-to = "app/src/main/res/values/strings.xml"
  apply-android-resource-value-change-to = "app/src/main/res/values/strings.xml"
}
```

This scenario will measure the impact of changing one Kotlin file, and of adding and modifying string resource values.

The resulting benchmark results look something like this.

![HTML benchmark results from gradler-profiler displaying build times for 4 different build scenarios. The chart shows that an up-to-date build is the fastest of the 4 build types.](https://i1.wp.com/goobar.io/wp-content/uploads/2020/11/incremental.png?resize=800%2C474&ssl=1)<figcaption>HTML benchmark results including incremental build scenario</figcaption>

As expected, the incremental build was measured to be faster than a _clean_ build, but slower than an _up-to-date_ build.

This is a very simple example.  For your production project, you might define multiple different incremental build scenarios. 

If you’re working in a multi-module project, you might want to measure the incremental build impact of changing a single module versus multiple modules.  You might want to measure the impact of changing a public api versus changing implementation details of a module.  There are many things you may want to measure in order to improve the real world performance of your build.  

## What’s Next?

At this point, hopefully you have a better understanding of benchmarking Gradle build performance using gradle-profiler, and how to start using gradle-profiler to improve the performance of your Gradle builds.

There’s still more that can be done with gradle-profiler to make it more effective for detecting build regressions and to make it easier to use for everyone on your team. 

I’ll be exploring those ideas in upcoming posts.

If you’d like to start learning more today, be sure to check out gradle-profiler [on GitHub](https://github.com/gradle/gradle-profiler) and [Tony Robalik’s post](https://dev.to/autonomousapps/benchmarking-builds-with-gradle-profiler-oa8) on Benchmarking builds with Gradle-Profiler.

You can find more [Gradle-related posts here](https://goobar.io/tag/gradle/) on my blog.

The post [Benchmarking Gradle Builds Using Gradle-Profiler](https://goobar.io/benchmarking-gradle-builds-using-gradle-profiler/) appeared first on [goobar](https://goobar.io).