Tools of the build trade: The making of a tiny Kotlin app
Tags: kotlin, gradle, shadow, proguard
Published: 2021-07-13 19:42:00

---
title: Tools of the build trade: The making of a tiny Kotlin app
published: true
description: Using Gradle to build a fat, minified jar
tags: kotlin, gradle, shadow, proguard
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/prpuj9z0joideg94t7zj.jpg
---

<a name="source-1" />Sometimes you need to solve a problem, and you _really_ don't feel like doing it with bash. Recently I took it upon myself to replace an intelligent-but-expensive Gradle task with dumb-but-cheap JVM app.[<sup>1</sup>](#endnotes) The Gradle task took an average of 2min per build, and my team cumulatively spent 43 hours over the past month waiting for it to complete. The Kotlin app that replaces it takes about 300ms to complete and I project we'll spend the next month cumulatively waiting on it less than 8 minutes. Since time is money, I've estimated this might recoup $100,000 in lost developer productivity over the next 12 months.

<a name="source-2" />This is not that story.[<sup>2</sup>](#endnotes)

It is instead the story of how to build such a thing with Gradle.  We'll learn how to use the [`application`](https://docs.gradle.org/current/userguide/application_plugin.html) and [`distribution`](https://docs.gradle.org/current/userguide/distribution_plugin.html) plugins to build the app and bundle a distribution; how to use the [`shadow`](https://imperceptiblethoughts.com/shadow/) plugin to turn it into a fat jar; and how to use [Proguard](https://www.guardsquare.com/manual/setup/gradle) to minify the whole thing. The upshot is we turn a 1.5M jar into a 12K "fat" jar, shaving 99.2% off the final binary.

> As soon as I saw that number, 99.2, I remembered that Jake Wharton had written about a [very similar experience](https://jakewharton.com/shrinking-a-kotlin-binary/) nearly a year ago. His post is a good read. Mine differs in that I will be explaining, step by step, how to achieve similar results with Gradle.

All the code for this is on [Github](https://github.com/autonomousapps/blog-minified-kotlin-app). ([Github drop ICE](https://github.com/drop-ice/dear-github-2.0).)

> This project is built with Gradle 7.1.1. This is important to keep in mind, as you would need to make some code changes if you were using Gradle 6.x.

## The app

<a name="source-3" />This post isn't about the app itself, but we need _something_ to build, so &hellip;[<sup>3</sup>](#endnotes)

```kotlin
// echo/src/main/kotlin/mutual/aid/App.kt
package mutual.aid

fun main(args: Array<String>) {
  val echo = args.firstOrNull() ?: "Is there an echo in here?"
  println(echo)
}
```

## The _real_ code

One thing I like to stress to people is that build engineering is its own proper domain. I do this because build engineering is how I make money. Let's look at the build code!

### Building an app with Gradle

The `application` plugin makes this very easy.

```groovy
// echo/build.gradle
plugins {
  id 'org.jetbrains.kotlin.jvm' version '1.5.20'
  id 'application'
}

group = 'mutual.aid'
version = '1.0'

application {
  mainClass = 'mutual.aid.AppKt'
}
```

We can now build and run this little app with
```bash
$ ./gradlew echo:run
Is there an echo in here?
```

...or if we want to customize our message...
```bash
$ ./gradlew echo:run --args="'Nice weather today'"
Nice weather today
```

We can run our program and provide any argument we want. This is the most fundamental building block for any JVM app.

### Turning the app into a distribution

Assuming that you want other people to actually run your app, you should bundle it as a distribution. Let's use the `distribution` plugin for that.

```bash
$ ./gradlew echo:installDist
$ echo/build/install/echo/bin/echo "Unless you're out West, I hear it's pretty hot out there"
Unless you're out West, I hear it's pretty hot out there
```

(Happily, with this version, we can drop the very un-aesthetic `--args=...` syntax.)

I'm being a little tricksy. We don't have to apply any new plugins, because the `application` plugin already takes care of applying the `distribution` plugin. The latter adds a task, `installDist`, which installs the distribution into your project's `build` directory. Here's what the full distribution looks like:

```bash
$ tree echo/build/install/echo
echo/build/install/echo
├── bin
│   ├── echo
│   └── echo.bat
└── lib
    ├── annotations-13.0.jar
    ├── echo-1.0.jar
    ├── kotlin-stdlib-1.5.20.jar
    ├── kotlin-stdlib-common-1.5.20.jar
    ├── kotlin-stdlib-jdk7-1.5.20.jar
    └── kotlin-stdlib-jdk8-1.5.20.jar
```

We can see that it has gathered all of the jars that are on our [runtime classpath](https://dev.to/autonomousapps/a-crash-course-in-classpaths-run-21go), including the new jar we've just built, `echo-1.0.jar`. In addition to these jars, we have two shell scripts, one for *nix and one for Windows. These scripts use the same template that Gradle uses for `gradlew[.bat]`, so they should be pretty robust.

We're already starting to see the problem. We have a "tiny" app, but it still drags along the full Kotlin runtime, despite using very little of it. That `lib` directory clocks in at 1.7M. All to echo a string! Not only that, but it just seems a little annoying to have all these individual files when all we really want is our program (the jar), and a script to easily invoke it from the command line.

### Layering on the Shadow plugin

They say that when you have a problem and you decide to solve it with regex, you now have two problems. The [Shadow plugin](https://imperceptiblethoughts.com/shadow/) really ups the ante on that: judging from [this article](https://www.alecstrong.com/posts/shading/) by Alec Strong, when you try to solve a problem by shading, you now have at least five problems.

> To be clear, I am joking. John Engelman, the maintainer of the Shadow Gradle Plugin, has done the community a service by making this tool available, free and open source. If the Shadow plugin is hard to use, it's because it's solving a hard problem.

```groovy
// echo/build.gradle
import com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar

plugins {
  id 'org.jetbrains.kotlin.jvm' version '1.5.20'
  id 'application'
  id 'com.github.johnrengelman.shadow' version '7.0.0'
}

def shadowJar = tasks.named('shadowJar', ShadowJar) {
  // the jar remains up to date even when changing excludes
  // https://github.com/johnrengelman/shadow/issues/62
  outputs.upToDateWhen { false }

  group = 'Build'
  description = 'Creates a fat jar'
  archiveFileName = "$archivesBaseName-${version}-all.jar"
  reproducibleFileOrder = true

  from sourceSets.main.output
  from project.configurations.runtimeClasspath

  // Excluding these helps shrink our binary dramatically
  exclude '**/*.kotlin_metadata'
  exclude '**/*.kotlin_module'
  exclude 'META-INF/maven/**'

  // Doesn't work for Kotlin? 
  // https://github.com/johnrengelman/shadow/issues/688
  //minimize()
}
```

Speaking of hard problems, we can see that I ran into a couple of issues while iterating on this. Feel free to upvote if you want to make my life easier!

I think the most interesting bits are the `from` and `exclude` statements. `from` is telling `shadow` what to bundle: our actual compilation output, plus the runtime classpath. The `exclude` statements are important for shrinking our fat jar. 

We can already run this fat jar and verify that it still works (the `runShadow` task is added by the `shadow` plugin due to its [integration](https://imperceptiblethoughts.com/shadow/application-plugin/) with the `application` plugin):

```bash
$ ./gradlew echo:runShadow --args="'I don't mind billionaires going into space, but maybe they could just stay?'"
I don't mind billionaires going into space, but maybe they could just stay?
```

And finally we can inspect the fat jar itself (this task is also run implicitly when we run the `runShadow` task):

```bash
$ ./gradlew echo:shadowJar
# Produces output at echo/build/libs/echo-1.0-all.jar
```

If we check its size, we see that it's 1.5M: already down about 12% from the original 1.7M. We can do much better, though.

### Layering on Proguard

I know I know. Proguard is so, like, 2019 or whenever it was that [R8](https://developer.android.com/studio/build/shrink-code) came out. But it's free, it's open source, and it has a Gradle task I can configure pretty easily. (And `shadow`'s `minify()` function didn't work :sob:)

The setup:

```groovy
// echo/build.gradle
import com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar
import org.gradle.internal.jvm.Jvm
import proguard.gradle.ProGuardTask

buildscript {
  repositories {
    mavenCentral()
  }
  dependencies {
    // There is apparently no plugin
    classpath 'com.guardsquare:proguard-gradle:7.1.0'
  }
}

plugins {
  id 'org.jetbrains.kotlin.jvm' version '1.5.20'
  id 'application'
  id 'com.github.johnrengelman.shadow' version '7.0.0'
}
```

For reasons that have completely passed me by, Proguard is not bundled as a Gradle plugin, but as a Gradle task, and so to add it to the [build script's classpath](https://dev.to/autonomousapps/build-compile-run-a-crash-course-in-classpaths-f4g), I must resort to this ancient black magic.

> There is a plugin for Android. And I suspect that someone, somewhere, has created a plugin for JVM projects. But the idea of looking for one that definitely works with Gradle 7 fills me with despair, so I decided to write this blog post instead.

Now we have access to the `ProGuardTask` task. How do we use it?

```groovy
// echo/build.gradle
def minify = tasks.register('minify', ProGuardTask) {
  configuration rootProject.file('proguard.pro')

  injars(shadowJar.flatMap { it.archiveFile })
  outjars(layout.buildDirectory.file("libs/${project.name}-${version}-minified.jar"))

  libraryjars(javaRuntime())
  libraryjars(filter: '!**META-INF/versions/**.class', configurations.compileClasspath)
}

/**
 * @return The JDK runtime, for use by Proguard.
 */
List<File> javaRuntime() {
  Jvm jvm = Jvm.current()
  FilenameFilter filter = { _, fileName -> fileName.endsWith(".jar") || fileName.endsWith(".jmod") }

  return ['jmods' /* JDK 9+ */, 'bundle/Classes' /* mac */, 'jre/lib' /* linux */]
    .collect { new File(jvm.javaHome, it) }
    .findAll { it.exists() }
    .collectMany { it.listFiles(filter) as List }
    .toSorted()
    .tap {
      if (isEmpty()) {
        throw new IllegalStateException("Could not find JDK ${jvm.javaVersion.majorVersion} runtime")
      }
    }
}
```

```proguard
// proguard.pro
-dontobfuscate

-keep class mutual.aid.AppKt { *; }
```

That's a mouthful. Since `ProGuardTask` isn't registered and configured by a plugin, we have to do that ourselves. The first part is telling it our rules, which are very simple; we want to shrink only, and we want to keep our main class entry point. Next, we tell it our `injars`, which is just the output of the `shadowJar` task: this is what is getting minified. (Importantly, the syntax I've used means that task dependencies are determined by Gradle without resort to `dependsOn`.) The `outjars` function very simply tells the task where to spit out the minified jar. Finally, we have [`libraryjars`](https://www.guardsquare.com/manual/configuration/usage#libraryjars), which I think of as the classpath needed to compile my app. These do not get bundled into the output. The most complicated part of this is the `javaRuntime()` function. The Proguard Github project has a much [simpler example](https://github.com/Guardsquare/proguard/blob/master/examples/gradle/applications.gradle#L37-L44), which you can use if you prefer.

Let's run it.
```bash
$ ./gradlew echo:minify
```

If we now inspect `echo/build/libs/echo-1.0-minified.jar`, we'll see it's only 12K :tada:

Similarly to how we verified that the fat jar was runnable, we can create a `JavaExec` task and run our minified jar:

```groovy
tasks.register('runMin', JavaExec) {
  classpath = files(minify)
}
```
and then run it:

```bash
$ ./gradlew echo:runMin --args="'How would space guillotines even work?'"
How would space guillotines even work?
```

We're not done yet, though. We still need to bundle this minified app as a distribution and publish it.

### The making of a skinny-fat distribution

Both the `application` and `shadow` plugins register a "zip" task (`distZip` and `shadowDistZip`, respectively). Neither are what we want, since they don't package our minified jar. Fortunately, they are both tasks of the core Gradle type [`Zip`](https://docs.gradle.org/current/dsl/org.gradle.api.tasks.bundling.Zip.html), which is easy-ish to configure. 

```groovy
// echo/build.gradle
def startShadowScripts = tasks.named('startShadowScripts', CreateStartScripts) {
  classpath = files(minify)
}

def minifiedDistZip = tasks.register('minifiedDistZip', Zip) { 
  archiveClassifier = 'minified'
  
  def zipRoot = "/${project.name}-${version}"
  from(minify) {
    into("$zipRoot/lib")
  }
  from(startShadowScripts) {
    into("$zipRoot/bin")
  }
}
```

The first thing we do is co-opt the `startShadowScripts` task (what this task does will be explained shortly). Rather than have it use the default classpath (as produced by the `shadowJar` task), we want it to use our minified jar as the classpath. The syntax `classpath = files(minify)` is akin to the earlier `injars(shadowJar.flatMap { it.archiveFile })` in that it carries task dependency information with it. Since `minify` is a `TaskProvider`, `files(minify)` not only sets the classpath, but also sets the `minify` task as a dependency of the `startShadowScripts` task.

Next, we create our own `Zip` task, `minifiedDistZip`, and construct it in a way analogous to the base `distZip` task. Understanding it is easier if we inspect the final product:

```bash
$ ./gradlew echo:minifiedDistZip
$ unzip -l echo/build/distributions/echo-1.0-minified.zip 
Archive:  echo/build/distributions/echo-1.0-minified.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  07-11-2021 17:02   echo-1.0/
        0  07-11-2021 17:02   echo-1.0/lib/
    12513  07-11-2021 16:59   echo-1.0/lib/echo-1.0-minified.jar
        0  07-11-2021 17:02   echo-1.0/bin/
     5640  07-11-2021 17:02   echo-1.0/bin/echo
     2152  07-11-2021 17:02   echo-1.0/bin/echo.bat
---------                     -------
    20305                     6 files
```

Our archive contains our fat, minified jar, as well as two scripts, one for *nix and one for Windows. The specific paths are important because the `echo` and `echo.bat` scripts that are generated include a `CLASSPATH` property:
```bash
CLASSPATH=$APP_HOME/lib/echo-1.0-minified.jar
```

There is one little gotcha here that wasted approximately 20 minutes of my life. You may have noticed this
```groovy
def zipRoot = "/${project.name}-${version}"
```

That starting `/` is very important! Without it, the zip contents had very weird paths. Fortunately, I have enough experience with Gradle at this point in my life to know when to take a win and just move on.

### Publishing our minified distribution

We now have a zip file with our minified jar, and the zip itself is only 15K, a huge improvement over the original 1.5M zip produced by the `distZip` task. We have just one more thing to automate, which is publishing this archive. We'll add the `maven-publish` plugin, and then configure it:

```groovy
// echo/build.gradle
plugins {
  id 'org.jetbrains.kotlin.jvm' version '1.5.20'
  id 'application'
  id 'com.github.johnrengelman.shadow' version '7.0.0'
  id 'maven-publish'
}

publishing {
  publications {
    minifiedDistribution(MavenPublication) {
      artifact minifiedDistZip
    }
  }
}
```

This adds a handful of publishing tasks, including `publishMinifiedDistributionPublicationToMavenLocal`. We can run it and inspect the output:
```
$ ./gradlew echo:publishMinifiedDistributionPublicationToMavenLocal
$ tree ~/.m2/repository/mutual/aid/echo/
~/.m2/repository/mutual/aid/echo/
├── 1.0
│   ├── echo-1.0-minified.zip
│   └── echo-1.0.pom
└── maven-metadata-local.xml
```

We even get a pom file, so we could resolve this artifact from a repository by referencing its Maven coordinates `mutual.aid:echo:1.0`.

## Wrapping up

This post is meant as a minimal example of how you might build a small Kotlin app with Gradle, which also had the requirements that it should have zero dependencies, be as small as possible, and be publishable for ease of access by potential users. A lot more is possible if you're willing to read the docs and experiment.

As a reminder, the full source code is available [here](https://github.com/autonomousapps/blog-minified-kotlin-app).

<a name="endnotes"/>
## Endnotes
<sup>1</sup> What I mean is that the Kotlin app is very fast because it exploits knowledge of the system, and uses regex to parse build scripts to derive metadata about them. By contrast, the Gradle task is intelligent in that it uses Gradle's own introspection capabilities. Unfortunately, it's 400x slower. Our compromise is to run the task in CI to verify the correctness of the Kotlin app, which is used daily by our developers. [<small>up</small>](#source-1)
<sup>2</sup> I might tell that story in a few months after getting corporate comms approval. Stay tuned! [<small>up</small>](#source-2)
<sup>3</sup> [PRs welcome](https://github.com/autonomousapps/blog-minified-kotlin-app). [<small>up</small>](#source-3)

## References
1. Application plugin: https://docs.gradle.org/current/userguide/application_plugin.html
2. Distribution plugin: https://docs.gradle.org/current/userguide/distribution_plugin.html
3. Shadow plugin: https://github.com/johnrengelman/shadow
4. Proguard task: https://www.guardsquare.com/manual/setup/gradle
5. Proguard JVM application example: https://github.com/Guardsquare/proguard/blob/master/examples/gradle/applications.gradle