The one thing I do not like about the Nix package manager (and a fix for it)
Tags: nix, linux, tutorial, packages
Published: 2024-01-16 16:51:41

---
title: The one thing I do not like about the Nix package manager (and a fix for it)
published: true
description: What'd you do if you had to compile a package for 50 minutes? That's what I almost had to do!
tags: nix, linux, tutorial, packages 
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4umces537mwefmrsi9s1.png
---

The [nix](https://nixos.org) package manager is an awesome package manager for linux and macos, which focuses on declarative packages. This means that you can dump out all the packages you want into a file, and nix will go out and fetch them for you.

This package manager builds itself on the concept of reproducibility, and it boasts a collection of over 80,000 packages, second only to the AUR! But this blog post is not about why nix-os is great, you'll find many other [blog posts](https://serokell.io/blog/what-is-nix) and [videos](https://www.youtube.com/channel/UC_zBdZ0_H_jn41FDRG7q4Tw) if you want to learn why.

Many nix package definitions include compiling from source. This can be tedious, since compiling takes a long time and wastes more bandwidth downloading build dependencies. Hence, the nix team has a binary cache, in which they build binaries for all the systems that nix supports for most of the common packages out there, like web browsers, desktop environments, and many more.

But what if your desired package is not in the binary cache? And what if it takes a long time to compile? This is what happened to me, and this is the one thing I don't like about nix.

The AUR provides binary packages, alongside source packages, for example, [yay](https://aur.archlinux.org/packages/yay) and [yay-bin](https://aur.archlinux.org/packages/yay-bin), but nix only provides source packages for most of its packages. Granted you'll see some exceptions, like [firefox](https://search.nixos.org/packages?channel=23.11&show=firefox&from=0&size=50&sort=relevance&type=packages) and [firefox-bin](https://search.nixos.org/packages?channel=23.11&show=firefox-bin&from=0&size=50&sort=relevance&type=packages), but those are pretty rare.

## The fix

In this article, I'll show you how you can create a binary package for your desired program. I wanted to download the [SurrealDB](https://surrealdb.com/) package, but the [package on nix](https://search.nixos.org/packages?channel=unstable&show=surrealdb&from=0&size=50&sort=relevance&type=packages&query=surrealdb) was a source package, meaning that I had to spend over 50 minutes waiting for a stupid package to compile.

Surreal provides binaries over at its [GitHub releases](https://github.com/surrealdb/surrealdb/releases), which I could've downloaded and ran, but I'd have to manually update the package, and as a 10x developer (I'm not one), I'd never manually do anything, but spend 10x the time trying to automate it. Doing this also defeats the reproducibility of nix, hence it's better to create a nix package so that anyone would be able to download your dependencies without having to do anything extra.

## Getting set up

First, make sure you have in hand a name for your package, and its version. Generally, binary packages end with `-bin`, so I'll call my package `surrealdb-bin`, and I'll be downloading the binary for the latest version as of the time of writing, which is `v1.1.0`.

Since I use NixOS, and since I want to install `surrealdb` globally, I'll create my `surrealdb` package in `/etc/nixos`, which is the folder which holds my `configuration.nix`. You may choose to co-locate your package in the directory with your `flake.nix`, for example, or even push it to GitHub/Lab so you can make your nix config truly reproducible.

This approach will contain two `nix` files, one for the package itself, and another for an overlay. An [overlay](https://nixos.wiki/wiki/Overlays) gives you the ability to modify `nixpkgs` without having to publish your package to the `nixpkgs` repository, and without having to mess around with `inputs`. Overlays make it very easy to use your package.

### Creating the package

Generally, I put my custom packages in the `packages/` subfolder, and custom overlays in the `overlays/` subfolder, and I name both these files with the name of the package, in this case `surrealdb-bin.nix`. I'd recommend you follow the same strategy to avoid clutter, but if you're only going to create one package, you can just keep everything in one directory.

I shall refer to `packages/surrealdb-bin.nix` as the package declaration. Now, put this code in your package declaration:

```nix
{ stdenv, fetchzip, autoPatchelfHook, glibc, gcc-unwrapped }: stdenv.mkDerivation rec {
  pname = "PACKAGE_NAME";
  version = "PACKAGE_VERSION";

  src = fetchzip {
    url = "PATH_TO_YOUR_PACKAGE'S_TARBALL_HERE";
    hash = "A_HASH_OF_YOUR_PACKAGE'S_CONTENTS";
  };

  nativeBuildInputs = [ autoPatchelfHook ];
  buildInputs = [
    glibc gcc-unwrapped
    # Any other system binaries your app may need
  ];

  installPhase = ''
    runHook preInstall
    mkdir -p $out/bin
    install -m755 PACKAGE_BINARY_FILE_NAME $out/bin
    runHook postInstall
  '';
}
```

Let's go over this code step-by-step. If you're familiar with nix syntax, you'll recognise that we're creating a function which destructures its first argument to accept some fields, and returns the output of the function `stdenv.mkDerivation`. `mkDerivation` is the helper function that you use to define `nix` packages. We give it an argument which defines the basic metadata of our package. Here are the fields it defines:

- **`pname`:** This is the name of the package. You could also use `name`, but you'll have to also specify the package's version in the `name`. Using `pname` makes nix automatically generate the `name` for us.

- **`version`:** This is the version of your package. Make sure you're fetching the correct binary!

- **`src`:** Now here's the meat of the package. We use nix's [fetchers](https://ryantm.github.io/nixpkgs/builders/fetchers/) to fetch our package's binary from the internet. There are two mainly used fetchers for binary packages: `fetchurl` and `fetchzip`. Let's take a closer look at them. 

**`fetchurl`:** This fetcher directly downloads the file provided to it by the `url` field.

**`fetchzip`:** This file downloads the archive provided to it by the `url` field, and unarchives it. The fetcher may be named `fetchzip`, but it also works for other archives like `.tar.gz`.

I'll be using the `fetchzip` fetcher to download the tarball of the correct SurrealDB version using a direct link pointing to its GitHub release. You can use nix's string interpolation to generate a proper link. This is what mine would look like:

```plain
https://github.com/surrealdb/surrealdb/releases/download/v${version}/surreal-v${version}.linux-amd64.tgz
```

I'll replace `PATH_TO_YOUR_PACKAGE'S_TARBALL_HERE` with the above link.

Now for the `hash` field. You must specify a checksum hash for the binary that you're downloading so that nix can verify that the correct file is being downloaded. The easiest way to set this value is to set `hash` to something random, install the package, and set the hash to whatever it says in the error that gets thrown.

If you'd like to write `hash` yourself, the syntax for it will be:

```plain
hashingAlgorithm-base64encodedhash
```

Many hashing algorithms are supported, but the most commonly used ones are `md5` and `sha256`. Make sure to `base64` encode your hash value, since `nix` only accepts that! Don't give it a `hex` string.



Now let's get back to our package derivation. The rest of the code is instructing `patchelf` to automatically patch the binary to make it run with nix. Since nix doesn't work like other package managers, binaries which expect shared libraries to be at one particular location will not work, hence we need to use `patchelf` to update these locations. Thankfully we won't have to manually run `patchelf`, since nix provides us with the `autoPatchelf` package. This package is defined in the `nativeBuildInputs`.

The additional libraries which the package depends on must be specified in the `buildInputs` array. The best way to find this out, would again be to install the package and add the dependencies it lists out in the error, but generally the two packages I've included should suffice.

Finally, in the `installPhase`, we define a shell script that runs. The `hooks` that are called are all related to `patchelf`, so just make sure they're present in your code. In between the two hook calls, we write code to create a `bin/` directory in our package's nix-store path, and we transfer any binaries the package provides to the `bin/` folder. Do update the `PACKAGE_BINARY_FILE_NAME` variable with the name of the binary that gets downloaded by the fetcher. In my case, that'd be `surreal`.

The `install` command is actually just some syntactic sugar for the `cp` command, they both have the same function. The only extra thing is the `-m` flag, which sets `chmod` permissions on the binary to make it readable, writable and executable by the user (the `7`), and only readable and executable by the user's group and everyone else (the two `5`s).

Finally, this is how your package derivation should look like:

```nix
# /etc/nixos/packages/surrealdb-bin.nix 
{ stdenv, fetchzip, autoPatchelfHook, glibc, gcc-unwrapped }: stdenv.mkDerivation rec {
  pname = "surrealdb-bin";
  version = "1.1.0";

  src = fetchzip {
    url = "https://github.com/surrealdb/surrealdb/releases/download/v${version}/surreal-v${version}.linux-amd64.tgz";
    sha256 = "2611de5eb7779dfe3b32bb47833fee2e3e168e39e43d76b47ea649b2f8c407fa";
  };

  nativeBuildInputs = [ autoPatchelfHook ];
  buildInputs = [ glibc gcc-unwrapped ];

  installPhase = ''
    runHook preInstall
    mkdir -p $out/bin
    install -m755 surreal $out/bin
    runHook postInstall
  '';
}

```

### Creating the overlay

Now we need to create an overlay to be able to add our package to the existing list of `nixpkgs`. I'll call my overlay derivation `surrealdb-bin.nix`, and place it in the `overlays/` folder in the `/ext/nixos` directory.

If you're using a different path than me, be sure to update the relevant imports.

Add these lines to your overlay definition:

```nix
self: super: {
  surrealdb-bin = super.callPackage ../packages/surrealdb-bin.nix {};
}
```

Now this may seem a little scarier, if you're new to the `nix` language. Since nix functions can only accept one argument, we use nested functions to declare multiple arguments. `self` and `super`, as they're commonly called, or more recently `final` and `prev`, are both instances of `nixpkgs`. It's just that `super`/`prev` is the version of `nixpkgs` before the overlay is applied, and `self`/`final` is the version of `nixpkgs` after all overlays are applied.

You should generally only use the `super` argument. Read [the wiki](https://nixos.wiki/wiki/Overlays) if you want to learn more about overlays.

The overlay functions return type is a set of packages which will be merged into `nixpkgs`. Here you can see that I'm defining one package called `surrealdb-bin`, and I'm calling the `super.callPackage` function and giving it the location of my `surrealdb-bin` package derivation as the argument. The `callPackage` function ensures that the package derivation is called with the proper arguments supplied. The blank argument set at the end is just to specify that I don't want to extend the list of arguments passed to the package derivation any further.

### Using the overlay

The only thing left to do now, is to actually use the overlay and extend the `nixpkgs` definition. Now this depends on where you plan to use the overlay, i.e. in a `shell.nix`, `flake.nix`, the nixos `configuration.nix`, or in a home-manager configuration. The steps are different for all of those.

Check [the wiki](https://nixos.wiki/wiki/Overlays#Applying_overlays_manually) for instructions pertaining to your specific use case. Since I want to install `surreal` globally, I'll add the overlay to my `configuration.nix`. All I have to do is add the below line somewhere, and then add `surrealdb-bin` to the list of environment/user packages.

```nix
nixpkgs.overlays = [ (import ./overlays/surrealdb-bin.nix) ];
```

> Don't forget to update the path to the overlay if you're using a different path!

And with a small `nixos-rebuild switch`, I have access to the `surreal` command in my environment! Great success! Now if there's a new surreal version published, all I have to do is change the `version` in my package derivation and rebuild!

You should push your package derivation and overlay to a centralised place like GitHub, GitLab, or any other place that provides direct download links (GitHub Gist / GitLab snippets would be a great place), so that you can download this overlay in any nix configuration and maintain reproducibility, instead of possibly having to duplicate your overlay and maintain two sources of truths.

## What if I don't have access to binaries

If your package only has the source available, with no binaries, and still takes a long time to compile, then you should probably leave the compilation to a GitHub action.

Just create a GitHub repository, create an action to build the package, and then upload it as an artifact/release once done.

Then use Nix to fetch that artifact in your package.

Do note that GitHub-provided hosted action runners have a maximum runtime of 6 hours, so if you have a package that takes longer than that (e.g. a web browser), then you need to use self-hosted runners, which have a max runtime of 35 days!

It is risky to host a self-hosted runner on your own machine, since your machine may go down, and you'll have to restart the build process again

Instead it'd be best to use a cloud VPS service like [Digital Ocean](https://m.do.co/c/371591aa3027). If you use that link, or [this one](https://m.do.co/c/371591aa3027), you can get `$200` in credit for upto 60 days, that's literally giving you free access to a 16GiB ram + 8 vCPU instance to build all your hopes and dreams on! (You will have to request access to these machines first though).

If you use [my link](https://m.do.co/c/371591aa3027), I also get some credits, so it helps me out too! Thanks for using [my link](https://m.do.co/c/371591aa3027).

Anyway, custom GitHub actions are out of scope of this article, but do let me know if you want me to create another article on that topic in the comments below!