Reducing my Gradle plugin's impact on configuration time: A journey
Tags: gradle, performance
Published: 2020-09-09 17:29:29

---
title: Reducing my Gradle plugin's impact on configuration time: A journey
published: true
description: A story about diagnosing and fixing performance problems in a Gradle plugin
tags: gradle, performance
cover_image: https://dev-to-uploads.s3.amazonaws.com/i/ijbihamms08gxg8dsu6k.jpg
---

A friend of mine, [Kyle Lehman](https://twitter.com/kyhule) at Comcast, recently sent me this message:

> Hey Tony, I just noticed the dependency-analysis plugin is almost doubling our config time (adding ~3sec). Didn't see any open issues that would potentially be related to this. Should I open one up or do you think we might be doing something funky in our config or is this just a fact of life?

:scream:

I asked him to create an issue,[<sup>1</sup>](#endnote-1) <a name="source-1" /> but undercut my own process by immediately following up and also asking if he could provide a **profile.** I've had [success in the past](https://dev.to/autonomousapps/dependency-analysis-gradle-plugin-using-bytecode-analysis-to-find-unused-dependencies-509n) optimizing my code with the use of profiles, and find them to be invaluable tools.

# Synopsis

In this post, we'll see that optimizing build performance is often not a straight path, but a winding journey. We'll observe some Gradle worst practices, see how we can replace them with best practices, and with CPU profiles and build scans as our guides. You will learn how to create a CPU profile of your build process (it's quite easy!) and learn a bit about interpreting them.

# Starting with a build scan

Kyle started by providing me a [build scan](https://scans.gradle.com/), from which I have excerpted this:

![174 tasks created at a cost of over three seconds](https://dev-to-uploads.s3.amazonaws.com/i/3z7yajoix7gp2aukptxl.png)

Yeah, that looks pretty bad. I would not be happy to be working on a project where configuration time had a floor of three seconds.

While the build scan was sufficient to show that _something_ was going on, it didn't provide enough detail to understand _what_ that something was. What was taking all that time? And why were those tasks being created? I thought I used [task configuration avoidance](https://docs.gradle.org/current/userguide/task_configuration_avoidance.html) exclusively....

> Narrator: he didn't.

You see, [my plugin](https://github.com/autonomousapps/dependency-analysis-android-gradle-plugin) works by registering tasks for _each variant_ of an Android project,[<sup>2</sup>](#endnote-2) <a name="source-2" /> and then also registering an _aggregating_ task that combines all the variant-specific data into a single coherent whole. That holistic information is then the basis of the holistic "advice" it provides to users. And here's where it gets interesting. When I originally wrote the aggregating code, I was focused on getting it to _work_, and also apparently suffered a mild stroke?, because I completely ignored best practices and had done this:

```kotlin
val adviceTasks = tasks.withType<AdviceTask>()
val dependencyOutputs = mutableListOf<RegularFileProperty>()
adviceTasks.all {
  dependencyOutputs.add(adviceReport)
}

val aggregateAdviceTask = 
  tasks.register<AdviceSubprojectAggregationTask>("aggregateAdvice") {
    // a ListProperty<RegularFileProperty>
    dependencyAdvice.set(dependencyOutputs) 
  }
```

:scream:

For anyone who may not yet have gone through their ritual Gradle hazing, the issue here is `tasks.withType<AdviceTask>().all { ... }`, which eagerly realizes all tasks of type `AdviceTask`, defeating configuration avoidance.

I actually didn't notice this immediately after speaking with Kyle. At that point, I was waiting on the CPU profile he had promised, and so was working on something totally unrelated. But I had his issue in the back of my mind, tickling my neurons. When I stumbled across that snippet above, and finally saw what I had done (and then wept), I fixed it like so:

```kotlin
val aggregateAdviceTask
  = tasks.register<AdviceSubprojectAggregationTask>("aggregateAdvice")

val adviceTask = ...
aggregateAdviceTask.configure {
  // a ListProperty<RegularFile>
  dependencyAdvice.add(adviceTask.map { it.adviceReport })
}
```

(The commit is available [here](https://github.com/autonomousapps/dependency-analysis-android-gradle-plugin/commit/2bde52ebecf8797cad9d7a959a097c28f78875f7#diff-0d30468a393146b4ace7af5b9e62a3da).)

The difference is that I register the aggregating task immediately, _without_ a configuration block. Then I configure it piecemeal, each time I add a "sub," or variant-specific, task. Because I have a thorough end-to-end test suite, I was able to verify I didn't break anything by just running all the tests ✅

I published that version of the plugin and told Kyle about it.

He responded that it didn't resolve his problem. 

![Creating 0 tasks at a cost of three seconds](https://dev-to-uploads.s3.amazonaws.com/i/pp9vazw44m6u9la2cg4p.png)

Interesting! My plugin is no longer eagerly realizing all of its tasks, but it still takes three seconds to configure. Kyle and I then briefly debated the value of task configuration avoidance,[<sup>3</sup>](#endnote-3) <a name="source-3" /> but that's neither here nor there. It was clear we had to go deeper. I asked again for a CPU profile.

![We need to go deeper](https://i.imgur.com/i3SFJPP.jpg)

# Continuing with a CPU profile

The profile had a story to tell. I'll focus on one section of it (keep reading for instructions on how to create your own profile):

![CPU profile showing that 50% of cost of configuration was registering one set of tasks](https://dev-to-uploads.s3.amazonaws.com/i/idcttot4mmrfq3zqjxyu.png)

`AndroidAppAnalyzer` is the code responsible for analyzing a `com.android.application` project. And it is responsible for 50% of CPU usage! From the profile, I could tell it was due to `registerClassAnalysisTask()`, which in turn required some &mdash; apparently &mdash; quite expensive computation. Here's the code in question:

```kotlin
val javaVariantFiles = androidSourceSets.flatMapToSet { sourceSet -> 
  project.files(sourceSet.javaDirectories)
    .asFileTree
    .files
    .toVariantFiles(sourceSet.name)
}
```

This code does the following:
1. Iterates over the Android source sets (a `Set<SourceProvider>`) for the given variant
2. Gets the Java directories for each source set
3. Converts each directory into a `FileTree` (which points to _files_ instead of _directories_)
4. Gets that file tree as a `Set<File>`
5. Converts those into a custom data class called `VariantFile`, defined below.

```kotlin
/**
 * Associates a [variant] ("main", "debug", "release", ...) with a 
 * [filePath] (to a file such as Java, Kotlin, or XML).
 */
data class VariantFile(
  val variant: String,
  val filePath: String
)
```
(This is part of what enables the plugin to provide variant-specific advice for Android projects.)

The final two steps (`.files` and `.toVariantFiles()`) were the expensive bits, at 35% and 10% of the total cost, respectively.

I was able to find a solution to this problem by adding a new task that did this work (pushing the work out of configuration and into a cacheable task), and making that new task's output an input to my now much-cheaper-to-configure `ClassListAnalysisTask`. Here's [the commit](https://github.com/autonomousapps/dependency-analysis-android-gradle-plugin/commit/1a9f2224bcf53a4b3d88a8033e458d1d738a4db9) that does it.

And here's the new CPU profile:

![CPU profile showing that we've reduced the cost of registering this task to under 7% of the total](https://dev-to-uploads.s3.amazonaws.com/i/2ah1r7k0di07i13tz751.png)

From 50% of total CPU time to under 7%! Not bad. Configuration time also went down from over six seconds to just over three seconds. (Well, not _all_ their build's problems were from my plugin!)

# Interlude

While I was working on adapting my configuration-time code to instead work during task execution, Kyle continued to investigate with the help of build scans. He noticed this:

![A build scan showing that an unrelated script is eagerly realizing a lot of tasks](https://dev-to-uploads.s3.amazonaws.com/i/ea6051xukfak7xj5dqbr.jpeg)

This is from a custom script, `appcenter.gradle`, which configures publishing for one of their apps. By itself, it takes over 1s to apply, but it's also triggering the eager realization of a _lot_ of tasks, including all those from my plugin &mdash; which explains why my earlier fix didn't help, and reinforcing the value of continuing to work on improving my plugin's configuration performance _per se_.

When Kyle simply commented out that script as an experiment, he next saw this:

![A build scan showing configuration time for my plugin at about 200ms](https://dev-to-uploads.s3.amazonaws.com/i/3m4iqz4va9n3xbbs3ysr.jpeg)

Proving that his script was the proximal cause. Care to guess what that script was doing? If you can't bare to wait, keep reading:

```groovy
project.afterEvaluate {
    project.tasks.all { task ->
```

There's that damn `all` again, ruining everything. So I guess task configuration avoidance isn't a waste after all.[<sup>4</sup>](#endnote-4) <a name="source-4" />

# Using Gradle-Profiler to profile a build

First, please refer to [Benchmarking builds with Gradle-Profiler](https://dev.to/autonomousapps/benchmarking-builds-with-gradle-profiler-oa8) for instructions on how to install gradle-profiler. If you already have this installed, please ensure you have it up to date.

Now without further ado, here is how you can profile your build's configuration phase:

```
$ gradle-profiler —profile async-profiler --project-dir <root-dir-of-build> help
```

Note the use of the task `help` here. This task is very fast and just prints some basic help text to console. `help` therefore often serves as a useful proxy for configuration time. Once it finishes, it will produce output in the working directory under `profile-out-N`, where _N_ is a 1-based index. Within that directory, you will find a flamegraph, an icicle graph, and related files.

Enjoy!

# Endnotes
<sup>1</sup><a name="endnote-1"/> Just realized he never did. Tsk tsk. [<small>up</small>](#source-1)
<sup>2</sup><a name="endnote-2"/> For JVM projects, it also looks at "all the variants," which in practice is just "main." [<small>up</small>](#source-2)
<sup>3</sup><a name="endnote-3"/> At, like, an ontological level. [<small>up</small>](#source-3)
<sup>4</sup><a name="endnote-4"/> But what does it _mean?_ [<small>up</small>](#source-4)