How to simply get your dependencies with Nix
Tags: nix, devops, bash, functional
Published: 2020-01-19 09:33:02

> If you haven't hear of Nix yet - that's perfect! If you have looked into it and came off it a bit bewildered... I understand. Nix can seem very unusual at first, but it can also be of tremendous value for your software projects. I hope we can start fresh for this post. 🙂

# Managing dependencies and creating reproducible environments is a Very Hard™ problem

Managing dependencies in software development can be a major pain, and we created a plethora of solutions for it: We have apt, npm, brew, pip, gem, rpm, just to name a few... We also use containers like Docker to manage software packages and create relatively reproducible environments.

All of those solutions have their limitations. Either they are specific to some language (`pip` for Python, `gem` for Ruby, `stack` for Haskell...), to some OS (`apt` for Debian/Ubuntu, `rpm` for Fedora & co. ...), to some part of our stack (backend/frontend), or they come with extra complexity (containers). They can be more or less reliable (`npm`), their results are more or less reproducible. Few solutions are able to sensibly manage multiple versions of the same package.

# The solution to all dependency management pains

The solution is [obvious](https://xkcd.com/927/): We need another package manager to rule them all!

[Nix](https://nixos.org/) is getting remarkably close to being an ideal cross-OS, cross-language, cross-stack package manager, while being extremely reliable and reproducible. Of course, it's not perfect and it has its own set of trade-offs and limitations. There is a lot to explain on Nix, how there is a functional programming language called `Nix`, a package repository called `Nixpkgs`, a whole operating system called `NixOS`... and a lot more. But for this post, I would like to show you a simple, minimal use-case that might already be very valuable to you!

# Simple example: Managing development dependencies with Nix

Let's say that for hacking on one of our projects we need [`curl`](https://curl.haxx.se/), [`jq`](https://stedolan.github.io/jq/), [`entr`](http://eradman.com/entrproject/) and [`pg_tmp`](http://eradman.com/ephemeralpg/). `pg_tmp` in turn depends on the [Postgres](https://www.postgresql.org/) binaries. This is how to get all that with Nix:

```bash
# Install Nix
curl https://nixos.org/nix/install | sh

# Create a `shell.nix` file
cat > shell.nix << EOF
let
  pkgs = import <nixpkgs> {};
in
pkgs.stdenv.mkDerivation {
  name = "my-env";

  buildInputs =
    [
      pkgs.curl
      pkgs.jq
      pkgs.entr
      pkgs.ephemeralpg
    ];
}
EOF

# Run `nix-shell`
nix-shell

# Have all dependencies that we need ready to go on your $PATH! 🎉
```

That's it!

Installing Nix as above is a relatively invasive change to your system, for example you will find a new '/nix/' directory in your root file system. I would recommend to try this out in a virtual machine or Docker container first.

I guess it makes sense for me to explain a little bit more on what is happening here and how you can use it. 

# Parts of the Nix ecosystem that we use in this example

We are using three parts of the Nix ecosystem.

## 1/3: The Nix programming language

It's a very small language, the syntax can be described on [one page](https://nixos.wiki/wiki/Nix_Expression_Language). If you are familiar with functional programming languages (especially from the ML family, Haskell, Elm), you will be right at home. If you are not, it might seem quite alien and restrictive (no loops?!). 

Everything in the Nix language is an expression, that is everything you write evaluates to a value. The language is mostly pure: Most function calls have no side effects and will always return the same result for the same inputs. The remaining impurities are mostly taken care of with cryptographic hashes. Based on this, the results of any Nix expression are highly reproducible.

## 2/3: The Nix packages repository

[Nixpkgs](https://github.com/NixOS/nixpkgs) is one huge, but well organized Nix expression that defines how over 40,000 software packages can be built, including *all* their dependencies. It's currently over 17 mb in total.

The Nix programming language keeps working with this huge expression efficient by being lazy and only evaluating the parts of it that you currently need. 

Nixpkgs is where a lot of the complexity that you might encounter with Nix comes from. Abstractions like `mkDerivation`, `callPackage` etc. are great for taking out repetition and keeping Nixpkgs maintainable, but their abstractions can be quite difficult to understand. Thinking too much about some of them makes my head hurt.

## 3/3: The `nix-shell` utility

The [`nix-shell`](https://nixos.org/nixos/nix-pills/developing-with-nix-shell.html) binary loads a Nix expression from a file (by default from the files `shell.nix` or, if that one doesn't exist, `default.nix`), evaluates that expression and then drops us in a shell where all dependencies defined in the expression are (magically) available.

# More explanations and details

With that said, let's go through the example above in more detail.

## The `shell.nix` file explained

Let's have another look at the `shell.nix` file that we created earlier:

```nix
let
  pkgs = import <nixpkgs> {};
in
pkgs.stdenv.mkDerivation {
  name = "my-env";

  buildInputs =
    [
      pkgs.curl
      pkgs.jq
      pkgs.entr
      pkgs.ephemeralpg
    ];
}
```

The `let [DEFINITIONS] in [EXPRESSION]` expression allows us to define variables in a local scope in the first part after `let`, and then evaluates the second part after `in` with that scope. Accordingly, the variable `pkgs` defined in the first part can be used in the second part.

`import <nixpkgs> {}` on the second line is how we import the Nixpkgs expression that is available in our Nix installation by default. I'll explain in a separate post how to 'pin' the Nixpkgs to one specific, reproducible version.

`pkgs.stdenv.mkDerivation` is a [complex function](https://nixos.org/nixos/nix-pills/fundamentals-of-stdenv.html) that is used all over Nixpkgs to define how different packages can be built. All we need to know about it for now, is that it takes a set (`{...}`) as an argument and returns a derivation (that is 'something that can be built' in Nix parlance). The build inputs will be able on our path if we run this expression in `nix-shell`.

With `name = "my-env";` we set the `name` attribute of the set we use as an argument to `mkDerivation`. It doesn't really matter what we put here.

`buildInputs` is the most important part: We set it to a list of derivations that we want to have available in our `nix-shell`. The items of our list are separated by spaces. We simply use attributes from the `pkgs` value. Remember that `pkgs` resulted from importing the huge Nixpkgs Nix expression, so it contains all of the over 40,000 packages in its attributes. We just pick out the right ones.

Finding the right attributes on Nixpkgs is not always simple, but a bit of googling goes a long way. For example, Google told me that `pg_tmp` is part of the `ephemeralpg` package in Nix.

## The `nix-shell` utility explained

A lot of wonderful magic happens when we run `nix-shell`. Let's go through the most significant steps it performs:

1. `nix-shell` will look for a `shell.nix` file in the current directory, and immediately find it in our example.
2. It evaluates the Nix expression contained in the file. In most cases, the expression is quite huge as the whole Nixpkgs expression with over 40,000 packages is being imported. Keep in mind that Nix keeps this reasonably efficient by only evaluating the parts that we actually need.
3. Now that the Nix expression is evaluated, `nix-shell` knows exactly which dependencies are needed. It checks in the local cache located in the `/nix/store` directory, if the individual dependencies are already built. If not, it will try to get them from a binary cache at `cache.nixos.org`. Only as a last resort will it actually build the required packages itself. 
4. When all dependencies are successfully cached in `/nix/store`, `nix-shell` puts together a `$PATH` environment variable that ties all the requested dependencies together. It then launches a new shell with that path set.

If you only want to run a single command instead of being dropped into a shell, you can use the `--run` flag. For example, `nix-shell --run "curl http://google.com/"` will query Google using the curl version that was installed by Nix.

As soon as you leave the Nix shell again with `Ctrl-D`, any of the changes that `nix-shell` made to our environment will disappear. It will look like the `entr` package that we defined as a dependency was never installed, with exception of the cached binary in `/nix/store`. The latter can be cleaned up with `nix-collect-garbage` or kept for a faster start up of `nix-shell` the next time.

# More to come

Congratulations on surviving your first toe-dipping with Nix! With the example above, you already have a usable first piece that can easily be extended with additional packages. In an upcoming post, I'd like to show you how to pin your version of Nixpkgs in order to make your dependencies reproducible across time and different systems!