A crash course in classpaths: Run!
Tags: android, java, kotlin
Published: 2021-06-14 23:39:45

---
title: A crash course in classpaths: Run!
published: true
description: Understanding classpaths on the JVM and ART
tags: android, java, kotlin
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/f9zi0gb8jxccdey8ri8l.jpg
---

Over the [past](https://dev.to/autonomousapps/build-compile-run-a-crash-course-in-classpaths-f4g) [three](https://dev.to/autonomousapps/a-crash-course-in-classpaths-build-l08) [posts](https://dev.to/autonomousapps/abusing-gradle-s-class-loader-hierarchy-for-fun-and-profit-oca), we have explored classpaths and class loading on the JVM and Android. Till now, we have focused on the build: we've learned how to inspect, influence, and even manipulate build classpaths to achieve our software goals. 

In this post, we're going to take this a step further and look at the classpaths involved in compiling and running our applications themselves. These are distinct from the classpaths that influence our build environment, and are critical for understanding our apps. Knowing how they work will make you a more effective JVM/ART (Android Runtime) programmer; it will help you solve problems and avoid the hyperbolic curve of desperation that is reading ancient answers on Stack Overflow.

By the end of this post, the next time you see a [`NoClassDefFoundError`](https://docs.oracle.com/javase/8/docs/api/java/lang/NoClassDefFoundError.html), you will feel like a superhero rather than an innocent bystander.

## Foundations

Let's start with a simple Kotlin program, which we'll use to probe a bit what we mean when we talk about compile-time vs runtime classpaths.

> In these examples, we'll be using `kotlinc` (for compilation of Kotlin source), and `kotlin` (for running Kotlin programs). ([official docs](https://kotlinlang.org/docs/command-line.html))

```kotlin
// Main.kt
import lib.Library

fun main() {
  Library().truth()
}

// lib/Library.kt
package lib

class Library {
  fun truth() = println("All billionaires are bad.")
}
```

In order to compile this program, we'll need to ensure all the classes involved are on the compile classpath: both `Main` and our library, `Library`. That looks like the following:

```
$ cd lib && kotlinc Library.kt && cd ..
```

This produces a class file, `lib/Library.class` (the path is important). We need this for the next step:

```
$ kotlinc -classpath lib Main.kt -d main.jar
```

This compiles `Main.kt`. The program `kotlinc` will look for a class file at `lib/Library.class` (based on the import statement, `import lib.Library` in `Main.kt`!). Including `-classpath lib` ensures that the library class file is available for compiling `Main.kt`.

Finally, we have `-d main.jar`, which tells `kotlinc` to bundle its output into a jar with that name. It is illuminating to inspect the jar that results from this command:

```
$ jar tf main.jar
META-INF/MANIFEST.MF
META-INF/main.kotlin_module
MainKt.class
```

We see the usual jar cruft (manifest, kotlin_module); more importantly, we see our compiled class, `MainKt.class`. Note that one thing we _don't_ see is `Library.class`. We _compiled against_ it, but it does not get bundled into our jar file. When we run our program, we'll have to provide that class file another way, on the _runtime_ classpath. Let's try to figure out how to do that. We'll start naively:

```
$ kotlin MainKt
error: could not find or load main class MainKt
```

Whoops, that didn't work. Let's try again:

```
$ kotlin -classpath main.jar MainKt
Exception in thread "main" java.lang.NoClassDefFoundError: lib/Library
...
```

Arrgh. One last time:

```
$ kotlin -classpath lib:main.jar MainKt
All billionaires are bad.
```

What's going on here? Well, the first command didn't work because `MainKt` was not on the classpath for our `kotlin` invocation (`kotlin` could not find the class `MainKt`, which we told it to run). We fixed that by telling `kotlin` about `main.jar`, but the command still failed, this time with a `NoClassDefFoundError` for `lib/Library`. We then fixed _that_ by updating our classpath to include the directory that contains the library class file (class files and jars are specified on a `:`-delimited list). This last command finally succeeded, and in an even more positive turn of events, taught us an important truth!

We now have a better grasp of the difference between the compile-time and runtime classpaths. In particular, we know that specifying the compile-time classpath (as we did with `kotlinc`) is not sufficient to also _run_ our program (with `kotlin`). For that, we need to provide a separate (but possibly identical) _runtime_ classpath.

## The runtime would like a word

What about when the runtime differs from the compile-time? Let's explore.

```kotlin
// Main.kt same as before
// lib/Library.kt same as before

// lib-alt/Library.kt
package lib

class Library {
  fun truth() = println("Some billionaires are very fine people.")
}
```

We've already compiled `Main.kt` and `lib/Library.kt`. Let's compile `lib-alt/Library.kt`.

```
$ cd lib-alt && kotlinc Library.kt && cd ..
```

And now run our program again, but with a different runtime classpath!

```
$ kotlin -classpath lib-alt:main.jar MainKt
Some billionaires are very fine people.
```

We did _not_ recompile `Main.kt`. We simply ran it in another context &mdash; a different classpath that provided a different implementation of the `truth()` function (`lib-alt/Library` and `lib/Library` are binary-compatible). We also learned another important truth, albeit one in a meta-ironic mode.

### On the perils of being too clever

I want to really hammer home the power of the runtime, so let's work through one more example. We'll start by updating our alternative library:

```kotlin
// lib-alt/Library.kt
package lib

class Library {
  fun truth() = println("Some billionaires are very fine people.")
  fun lie() = println("Actually, billionaires earned their wealth.")
}
```

Sneaky! I wonder what that's for.

```kotlin
// Main.kt
import lib.Library

fun main(args: Array<String>) {
  val lib = Library()

  val arg = args.firstOrNull() ?: "truth"
  lib.javaClass
    .getDeclaredMethod(arg)
    .invoke(lib)
}
```

Huh... where's this going? Note that we must recompile `Main.kt` because we have changed it this time, unlike in our past examples. We also use a new flag, `-include-runtime`, which bundles the Kotlin stdlib (the "runtime") into our jar, so we can use the stdlib function `firstOrNull()` without having to manually put it on the classpath later.

```
$ kotlinc -classpath lib Main.kt -include-runtime -d main.jar
$ cd lib-alt && kotlinc Library.kt && cd ..
```

Note that the order of compilations above does not matter, nor the fact that we first compile `Main.kt` against `lib/Library`, and later run it against `lib-alt/Library`. The Kotlin compiler only needs some way to resolve all the symbols, and in this case, that is only `import lib.Library`.

Ok! Time to run our new program.

```
$ kotlin -classpath lib-alt:main.jar MainKt lie
Actually, billionaires earned their wealth.
```

See, this is why reflection is bad.

## Connecting this to Gradle

Since we don't generally use the SDK tools directly, but rather via a build tool such as Gradle, let's take a moment to translate the above into Gradle terms. Imagine a project with a structure like the following:
```
.
├── app
│   ├── build.gradle
│   └── src
│       └── main
│           └── Main.kt
├── lib
│   ├── build.gradle
│   └── src
│       └── main
│           └── lib
│               └── Library.kt
├── lib-alt
│   ├── build.gradle
│   └── src
│       └── main
│           └── lib
│               └── Library.kt
└── settings.gradle
```
You already know what the source files look like. `settings.gradle` is simple:

```groovy
include ':app', ':lib', ':lib-alt'
```

The only other interesting file is `app/build.gradle`:
```groovy
plugins {
  id 'org.jetbrains.kotlin.jvm' version '1.5.10'
}
dependencies {
  // Two choices:
  // 1. :lib is on the compile and runtime classpaths of :app
  implementation project(':lib')
  
  // 2. :lib is on the compile classpath, while :lib-alt is
  //    on the runtime classpath.
  compileOnly project(':lib')
  runtimeOnly project(':lib-alt')
}
```
Gradle's docs have a nice graphic describing the relationship between the various "configurations" [here](https://docs.gradle.org/current/userguide/java_library_plugin.html#sec:java_library_configurations_graph).

## Use-cases
### Writing maximally-compatible Gradle plugins

Let's say you want to write a Gradle plugin that is compatible across a range of Android Gradle Plugin (AGP) versions, say 4.2 through 7.0. There are at least two things you should do:

1. Compile your plugin against the _minimum_ version you intend to support. In this case, 4.2.0.
2. Declare this dependency as a `compileOnly` dependency, _not_ as `implementation` or `api`.

Here's what that looks like:
```groovy
dependencies {
  // 4.2.0 is the minimum version we support
  compileOnly 'com.android.tools.build:gradle-api:4.2.0'
}
```

Doing this ensures that the dependency is available at compile-time, and also that it is _not_ exposed to users of your plugin at runtime. (For an excellent discussion of Gradle "configurations" like compileOnly, etc., see [this post](https://mbonnin.medium.com/actual-footage-of-different-kinds-of-gradle-configurations-9678bd681793) by Martin Bonnin.) Because the dependency is available at compile-time, you can see all its types in your IDE, enabling a pleasant development experience. And because the dependency is not automatically supplied at runtime, you're requiring your users to ensure it's on their build-runtime classpath &mdash; which they normally will have anyway, if they're using a plugin (yours) that is meant to enhance AGP in some way.

<!--
We discussed build classpaths in detail in [earlier](https://dev.to/autonomousapps/build-compile-run-a-crash-course-in-classpaths-f4g) [posts](https://dev.to/autonomousapps/a-crash-course-in-classpaths-build-l08) in this series, but it's worth reiterating and expanding on a couple of points here. Gradle has a powerful dependency resolution engine, which does its best to ensure that the correct dependency, its variant and its version, are on the right classpath, for your builds. When it comes to Gradle plugins, though, things can get a bit wonky. For example, if you know that all of yours users exclusively use the `plugins {}` block for declaring dependencies, then the only thing you need to do to ensure that a compatible version of AGP is on the classpath is to use the minimum supported version. By normal dependency resolution, Gradle will choose the latest version of AGP that is being requested: this will generally be the later of yours and your user's. However, if instead you are using the legacy `buildscript { dependencies { classpath ... } }` approach, then the result can differ greatly depending on the order of declaration. Consider:

(Reminder: in these scenarios below, imagine your plugin is using `implementation 'com.android.tools.build:gradle-api:4.2.0'`, rather than `compileOnly`.)

```groovy
// build.gradle
buildscript {
  dependencies {
    classpath 'com.example:plugin:1.0'
    classpath 'com.android.tools.build:gradle-api:4.2.0'
  }
}
```

In this case, the build will use whatever version of AGP is supplied by the plugin. If you invert the order, however...

```groovy
// build.gradle
buildscript {
  dependencies {
    classpath 'com.android.tools.build:gradle-api:4.2.0'
    classpath 'com.example:plugin:1.0'
  }
}
```

Now the build will use AGP 4.2.0. **This is because `classpath` dependencies don't participate in normal dependency resolution!** They are _literally_ putting jars on the classpath, and [as we know](https://dev.to/autonomousapps/build-compile-run-a-crash-course-in-classpaths-f4g), order matters with classpaths.

So, this is why you should declare dependencies like AGP on `compileOnly`!
-->

### Mysterious `RuntimeException`s: Stub!

At work, we were seeing unit test failures, but _only_ in the IDE. The command line, and CI, worked fine. (In fact, this is how the IDE error got distributed to the whole company: green build on CI.)

```
Caused by: java.lang.RuntimeException: Stub!
	at android.os.Looper.getMainLooper(Looper.java:7)
	at flow.Preconditions.getMainThread(Preconditions.java:55)
	at flow.Preconditions.<clinit>(Preconditions.java:22)
```

Naturally, we were suspicious that this was an IDE bug (spoiler alert: it was), but since app devs generally prefer to run tests from their IDEs, we couldn't just ignore it. After all, my team is named "Mobile Developer Experience," not "It's Green on CI."

If you have a bit of experience with Android, you may already be thinking that we've somehow messed up and are trying to unit test something that ought to be instrument-tested instead. If so, you'd be _kind of_ on to something. We had recently added a `compileOnly` (!) dependency on `'com.google.android:android:4.1.1.4'` &mdash; these are the maven coordinates for "android.jar," that is, the Android runtime. We wanted to compile against them because some code touched the `Parcelable` class, but didn't actually _use_ it. Therefore, we knew we'd be safe at runtime because we'd never try to invoke anything relating to a `Parcelable` &mdash; it just had to be available at compile-time.

To reiterate, this worked as expected from the command line, but when invoking tests from Android Studio, they'd always fail (at least it was deterministic). Turns out the reason was, and I quote (emphasis mine) "there are **classpath** issues running the tests inside the IDE because the whole setup is a mess," and "unit tests have some many issues before [Arctic Fox], it is surprising they worked at all." Basically, Android Studio before Arctic Fox is using the **wrong runtime classpath**, leading to runtime failures.

The fix for this turned out to be very simple, once the problem was understood. First, we removed `compileOnly 'com.google.android:android:4.1.1.4'`. We added a new module which had hand-written stubs for `Parcelable` and `Parcel`, and we put _that_ module on the compile classpath instead. Now all our tests pass, from both CLI and (old, broken) Android Studio!

### Understanding `NoClassDefFoundError`s

Maybe you've seen something like the following in your career as an Android or JVM dev:

> Instrumentation test failed due to uncaught exception in thread [main]:
`java.lang.NoClassDefFoundError`: Failed resolution of: `Lcom/example/ThisClassShouldDefinitelyBeHere;`

"Huh," you think. "ThisClassShouldDefinitelyBeHere should definitely be there. What's going on?"

What's going on is a runtime failure because the ART/JVM tried to resolve a class and couldn't find it anywhere: it was not available on the runtime classpath. 

"But my code compiled fine!" you say. "That class is in dependency 'foo', which is declared on the 'implementation' configuration. Why isn't it available at runtime?"

These are excellent questions. You've already identified that the class _should_ be there: you declared it on the right configuration, `implementation`, and you know that such dependencies are &mdash; or ought to be &mdash; available at _both_ compile-time and runtime. So what you've found, actually, is a bug in one of your build tools. I recently encountered [just such an issue](https://issuetracker.google.com/issues/187353303) with AGP 4.2.0, which occurs with project dependencies declared on `androidTestImplementation` and which contain mixed Java/Kotlin source. The bug is already fixed and will be released as 4.2.2, but we've [monkey patched](https://dev.to/autonomousapps/abusing-gradle-s-class-loader-hierarchy-for-fun-and-profit-oca) it locally while we wait for the official release.

The takeaway here is that, once you have a sufficient understanding of what can cause problems like `NoClassDefFoundError`, you can already eliminate a whole class of possible causes and narrow your focus considerably.

Well, that's it for the crash course in classpaths. I hope you've found it interesting and useful. Happy coding!

Thanks once again to [César Puerta](https://twitter.com/CesarDielo) for his typically thorough reviews. If at any point you find this post confusing, know that it means I ignored César's advice. Thanks as well to [Nate Ebel](https://twitter.com/n8ebel) for his early encouragement.