Dependency Hell
Tags: java, dependency, bazel, sbt
Published: 2019-12-02 04:08:01

---
title: Dependency Hell
published: True
description: How conflicting dependencies can cause bugs you'll miss until you hit production.
tags: java,dependency,bazel,SBT
---

Resolving dependency conflicts is not fun. I personally have not had to pleasure of dealing with conflicts until more recently. `Bundler` and the entire Ruby community does an amazing job making sure I didn't have to. In the JVM world, things are less clear. For example, with SBT, the latest version of a dependency always wins, but if the latest version is higher by a major version, things will most likely break. SBT won't tell you when it replaces dependencies, you have to ask! I covered more details of the strange behavior in a [previous post]( https://dev.to/devkaoru/surprising-behaviors-of-sbt-1mp9). What I didn't go over is how we can get into this situation in the first place.

# Setting the stage
The best way I learn is by doing, tweaking things, and breaking it, then fixing it...maybe. Let's come up with a contrived example (follow along by cloning [this repo](https://github.com/KaoruDev/jvm_dependency_hell)). Below we have 3 files describing classes depending on each other Life > Person > Cat.

```java
// com/kaoruk/Life.java =============================
package com.kaoruk;

import com.kaoruk.Person;

class Life {
  public static void main(String args[]) {
    Person person = new Person("Kaoru");
    if (args.length > 0) {
    	person.adoptCat();
    }
    System.out.println(person.sayHello());
  }
}


// com/kaoruk/Person.java =============================
package com.kaoruk;

import com.kaoruk.Cat;
import java.util.Objects;

class Person {
  private final String name;
  private Cat cat;

  public Person(String name) {
    this.name = name;
  }

  public void adoptCat() {
    this.cat = new Cat();
  }

  public String sayHello() {
    if (Objects.isNull(cat)) {
      return this.name + ": Life has no meaning without a cat";
    } else {
      return this.name + ": Oh hai, I haz kitty! " + cat.sayHello();
    }
  }
}


// com/kaoruk/Cat.java =============================
package com.kaoruk;

public class Cat {
  public String sayHello() {
    return "meow!";
  }
}
```

Now let's compile each file and place them in a jar:

```bash
#!/bin/bash

set -e

mkdir jars || true

echo "Compling Cat..."
javac com/kaoruk/Cat.java
jar -cvf jars/Cat.1.0.0.jar com/kaoruk/Cat.class
rm com/kaoruk/Cat.class

echo "Compling Person..."
javac -cp jars/Cat.1.0.0.jar com/kaoruk/Person.java
jar -cvf jars/Person.1.0.0.jar com/kaoruk/Person.class
rm com/kaoruk/Person.class

echo "Compling Life..."
javac -cp jars/Person.1.0.0.jar com/kaoruk/Life.java
jar -cvf jars/Life.1.0.0.jar com/kaoruk/Life.class
rm com/kaoruk/Life.class
```

If all goes well we should be able to run Life:

```bash
$ java -cp jars/Life.1.0.0.jar:jars/Person.1.0.0.jar:jars/Cat.1.0.0.jar com.kaoruk.Life

Kaoru: Life has no meaning without a cat

$ java -cp jars/Life.1.0.0.jar:jars/Person.1.0.0.jar:jars/Cat.1.0.0.jar com.kaoruk.Life Foo

Kaoru: Oh hai, I haz kitty! meow!
```

Great! Okay so let's say we want to teach our `Cat` how to say "Can I haz job?" but also we decide that the method name "sayHello" doesn't really accurately portray the message. We change `Cat.java` look like:

```java
// com/kaoruk/Cat.java =============================
package com.kaoruk;

public class Cat {
  public String beg() {
    return "Can I haz job?";
  }
}
```

Great, let's compile it, replace `Cat.1.0.0.jar` and run it:

```bash
$ javac com/kaoruk/Cat.java
$ jar -cvf jars/Cat.2.0.0.jar com/kaoruk/Cat.class

$ java -cp jars/Life.1.0.0.jar:jars/Person.1.0.0.jar:jars/Cat.2.0.0.jar com.kaoruk.Life

Kaoru: Life has no meaning without a cat


$ java -cp jars/Life.1.0.0.jar:jars/Person.1.0.0.jar:jars/Cat.2.0.0.jar com.kaoruk.Life foo

Exception in thread "main" java.lang.NoSuchMethodError: com.kaoruk.Cat.sayHello()Ljava/lang/String;
	at com.kaoruk.Person.sayHello(Person.java:22)
	at com.kaoruk.Life.main(Life.java:11)
```

Oh hai, dependency hell!


# Detection is difficult
You might be thinking to yourself, well duh `Life` is broken because you updated a method in `Cat` and replaced it with a new JAR. This is _exactly_ what build tools do! To detect the problem above, _before_ running the code, we would need to _recompile_ `Person.1.0.0.jar`. But more importantly, we would need to recompile it using `Cat.2.0.0.jar`, ignoring its explicit reliance on `Cat.1.0.0.jar`. As a build tool author, not that I am one, I would imagine my users would not expect the tool to recompile the entire world, because it would take too long. 

The logic to recompile the entire world is no easy task. First, you'd have to list all dependencies and transitive dependencies, override dependencies if there is a higher version, create a graph, find the leaves, and the compile JARs all the way back. I'm not even sure this would work, I'm just spitballing!

# Tests are not thou Savior
In our example, if our tests only cover running the `main` method without an argument, we wouldn't catch this bug, one of our poor users will. So that means we have to have tests for every single branch of our code? Yes, regardless if it doesn't work in this scenario! Having a test for every branch might not work because you'd also have to test every branch of your dependencies and transitive dependencies! In our example, if we depended on `Life`, our tests probably would not have hit the bug, because why would we test every branch of `Life`?

# It ain't so bad...right?
Yes, _if_ we follow a standard. In this case, the standard is semantic versioning, or [`semver`](https://semver.org/). If a library's major version has changed, we know that something will break, we don't have to go digging into the source code of the library or every dependee of the library. Unfortunately, _not all_ library maintainers follow `semver`. For example, Scala Akka, does not follow this pattern. Their minor versions introduce breaking changes that are incompatible with older minor versions. The reality is we have to keep track of which library follows `semver` and which follows a different pattern.

# Okay so what can we do about it?
`Bundler` solves this problem by allowing engineers to specify an acceptable range. For example, `gem 'thin', '~> 1.1'` means that `Bundler` is allowed to install newer versions of `thin` `< 2.0`. If I bring in a `gem` which requires `thin 2.0`, `Bundler` throws an exception forcing me to deal with the problem. `Maven` also has this [functionality](https://cwiki.apache.org/confluence/display/MAVENOLD/Dependency+Mediation+and+Conflict+Resolution#DependencyMediationandConflictResolution-DependencyVersionRanges), so Coursier shouldn't be too far behind? But it still doesn't solve the problem if library maintainers ignore `semver`. So I'm not entirely sure there's a solid solution to this problem yet. I think `Bazel`, [`rules_jvm_external`](https://github.com/bazelbuild/rules_jvm_external) to be specific, at least, makes a step towards the right direction informing us when there is a conflict.