How to set up ad hoc (developer) environments
Tags: productivity, development, nix
Published: 2022-04-04 10:00:37

Have you had a task to fix a supposedly simple thing in a project, a code or configuration change, only to spend most of the time to set up the environment so you can do the fix in the first place?

The setup you just did for this now clutter your setup, and conflicts with other projects? In order to get back to your old working environment, you have to do some re-installs of software packages?


One approach is to use containers from Docker images to package and run the specific tools. However, that can sometimes be cumbersome to set up and combine with tools that you already have.

The approach here does not use containers, and it can just be a matter of a one-line command to create a new **shell environment**. The extra software needed is in place as long as the shell is active only.

First, I will show an example, then look at how to set it up and get started.

For a project, we want to install Node.js 14 + Yarn + git + ripgrep, plus cowsay for good measure. First, just check what is on the machine before. After that, install the required tools and verify that they are available in our new shell. When we are done, we exit the shell and get the environment back to its old state.

```bash
bash-4.4# # First let us create a work directory
bash-4.4# mkdir work && cd work
bash-4.4# pwd
/work
bash-4.4# # We want to have Node.js 14.x, yarn, git, ripgrep and cowsay
bash-4.4# which node yarn git rg cowsay
which: no node in (/root/.nix-profile/bin:/nix/var/nix/profiles/default/bin:/nix/var/nix/profiles/default/sbin)
which: no yarn in (/root/.nix-profile/bin:/nix/var/nix/profiles/default/bin:/nix/var/nix/profiles/default/sbin)
/root/.nix-profile/bin/git
which: no rg in (/root/.nix-profile/bin:/nix/var/nix/profiles/default/bin:/nix/var/nix/profiles/default/sbin)
which: no cowsay in (/root/.nix-profile/bin:/nix/var/nix/profiles/default/bin:/nix/var/nix/profiles/default/sbin)
bash-4.4#       
bash-4.4# # None of the tools are installed 
bash-4.4# # Now we will set up the tools in our ad hoc environment
bash-4.4# nix-shell -p nodejs-14_x yarn git ripgrep cowsay
these 25 paths will be fetched (71.04 MiB download, 327.01 MiB unpacked):
  /nix/store/1l6jgfsgjmsrbrciz8r714dnkyngvpkb-zlib-1.2.11-dev
  /nix/store/1v0pmb9f8jmn126nvbs2k2varzqfqczq-icu4c-69.1
  /nix/store/2dd4n8cca16pa8yic1gh079hxb2x0ji4-yarn-1.22.10
  /nix/store/3k69hbxg04sdxlgi1236ddggs346sxf3-stdenv-linux
  /nix/store/4wf2bnddgwg17m465rg66j67gmcwy83q-bash-interactive-4.4-p23-info
  /nix/store/4xkhd51k7y6g9vi2h9kfvcxd67k3v3f1-gawk-5.1.0
  /nix/store/5ddb4j8z84p6sjphr0kh6cbq5jd12ncs-binutils-2.35.1
  /nix/store/5wvf6qy2pjijd93n4dn6vvbn45ij9fdr-linux-headers-5.12
  /nix/store/7al99yyrc2ix9c5idq2kf8zgksscaf9z-bash-interactive-4.4-p23-doc
  /nix/store/88ghxafjpqp5sqpd75r51qqg4q5d95ss-gcc-wrapper-10.3.0
  /nix/store/bjyyiipkwxgn8awrpp5aj3yp75l043i9-nodejs-14.17.6
  /nix/store/cr23xzsbbvjgxaf1mchn2drhh6yvsq0q-cowsay-3.03+dfsg2
  /nix/store/dr5yvz4dv2rq6mqr8pq5ix2g9wn4h5wp-icu4c-69.1-dev
  /nix/store/k6sslkn2f764mgnd2a6q0wfwlnfwpybi-ripgrep-12.1.1
  /nix/store/llywalnma9vv74z0xivwc87kxs6crk9b-binutils-wrapper-2.35.1
  /nix/store/mbc7y2k96nhgazi7w7hls8y4pbma1y82-diffutils-3.7
  /nix/store/p5lnl4zr45n7mf9kz9w8yz3rqh001b5c-bash-interactive-4.4-p23-dev
  /nix/store/q141hd8jl7in5223jmf7kmx9h517km4p-glibc-2.32-54-dev
  /nix/store/sa90ixnafl08k8d9wwmzhr7br2dyjxp2-expand-response-params
  /nix/store/sf6w5cyyksw7kacvhhw0aw0m1x74r524-ed-1.17
  /nix/store/shcm9y3r9xlx6pksg3kkkpi56vm5cgqg-libuv-1.41.0
  /nix/store/sjhz1j2d1ssn59f66kqp92xj9mpsww2d-gcc-10.3.0
  /nix/store/wfzdk9vxayfnw7fqy05s7mmypg5a8lyr-gnumake-4.3
  /nix/store/xfc0ffwz6s87mnz9x2k2qj6ykwxgwjxk-patchelf-0.12
  /nix/store/ywkvpy8va67s7z8i1m32mklar9mii8qd-patch-2.7.6
copying path '/nix/store/7al99yyrc2ix9c5idq2kf8zgksscaf9z-bash-interactive-4.4-p23-doc' from 'https://cache.nixos.org'...
copying path '/nix/store/p5lnl4zr45n7mf9kz9w8yz3rqh001b5c-bash-interactive-4.4-p23-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/4wf2bnddgwg17m465rg66j67gmcwy83q-bash-interactive-4.4-p23-info' from 'https://cache.nixos.org'...
copying path '/nix/store/5ddb4j8z84p6sjphr0kh6cbq5jd12ncs-binutils-2.35.1' from 'https://cache.nixos.org'...
copying path '/nix/store/cr23xzsbbvjgxaf1mchn2drhh6yvsq0q-cowsay-3.03+dfsg2' from 'https://cache.nixos.org'...
copying path '/nix/store/mbc7y2k96nhgazi7w7hls8y4pbma1y82-diffutils-3.7' from 'https://cache.nixos.org'...
copying path '/nix/store/sf6w5cyyksw7kacvhhw0aw0m1x74r524-ed-1.17' from 'https://cache.nixos.org'...
copying path '/nix/store/sa90ixnafl08k8d9wwmzhr7br2dyjxp2-expand-response-params' from 'https://cache.nixos.org'...
copying path '/nix/store/4xkhd51k7y6g9vi2h9kfvcxd67k3v3f1-gawk-5.1.0' from 'https://cache.nixos.org'...
copying path '/nix/store/wfzdk9vxayfnw7fqy05s7mmypg5a8lyr-gnumake-4.3' from 'https://cache.nixos.org'...
copying path '/nix/store/1v0pmb9f8jmn126nvbs2k2varzqfqczq-icu4c-69.1' from 'https://cache.nixos.org'...
copying path '/nix/store/shcm9y3r9xlx6pksg3kkkpi56vm5cgqg-libuv-1.41.0' from 'https://cache.nixos.org'...
copying path '/nix/store/dr5yvz4dv2rq6mqr8pq5ix2g9wn4h5wp-icu4c-69.1-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/5wvf6qy2pjijd93n4dn6vvbn45ij9fdr-linux-headers-5.12' from 'https://cache.nixos.org'...
copying path '/nix/store/ywkvpy8va67s7z8i1m32mklar9mii8qd-patch-2.7.6' from 'https://cache.nixos.org'...
copying path '/nix/store/q141hd8jl7in5223jmf7kmx9h517km4p-glibc-2.32-54-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/xfc0ffwz6s87mnz9x2k2qj6ykwxgwjxk-patchelf-0.12' from 'https://cache.nixos.org'...
copying path '/nix/store/llywalnma9vv74z0xivwc87kxs6crk9b-binutils-wrapper-2.35.1' from 'https://cache.nixos.org'...
copying path '/nix/store/sjhz1j2d1ssn59f66kqp92xj9mpsww2d-gcc-10.3.0' from 'https://cache.nixos.org'...
copying path '/nix/store/k6sslkn2f764mgnd2a6q0wfwlnfwpybi-ripgrep-12.1.1' from 'https://cache.nixos.org'...
copying path '/nix/store/88ghxafjpqp5sqpd75r51qqg4q5d95ss-gcc-wrapper-10.3.0' from 'https://cache.nixos.org'...
copying path '/nix/store/1l6jgfsgjmsrbrciz8r714dnkyngvpkb-zlib-1.2.11-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/3k69hbxg04sdxlgi1236ddggs346sxf3-stdenv-linux' from 'https://cache.nixos.org'...
copying path '/nix/store/bjyyiipkwxgn8awrpp5aj3yp75l043i9-nodejs-14.17.6' from 'https://cache.nixos.org'...
copying path '/nix/store/2dd4n8cca16pa8yic1gh079hxb2x0ji4-yarn-1.22.10' from 'https://cache.nixos.org'...

[nix-shell:/work]# 

[nix-shell:/work]# which node yarn git rg cowsay
/nix/store/bjyyiipkwxgn8awrpp5aj3yp75l043i9-nodejs-14.17.6/bin/node
/nix/store/2dd4n8cca16pa8yic1gh079hxb2x0ji4-yarn-1.22.10/bin/yarn
/nix/store/cpx9hzqcxci468qjwx1yk9k8rwmhlqxl-git-2.31.1/bin/git
/nix/store/k6sslkn2f764mgnd2a6q0wfwlnfwpybi-ripgrep-12.1.1/bin/rg
/nix/store/cr23xzsbbvjgxaf1mchn2drhh6yvsq0q-cowsay-3.03+dfsg2/bin/cowsay

[nix-shell:/work]# git init
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint: 
hint: 	git config --global init.defaultBranch <name>
hint: 
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint: 
hint: 	git branch -m <name>
Initialized empty Git repository in /work/.git/

[nix-shell:/work]# cowsay Now we have the tools for our environment in place
 _______________________________ 
/ Now we have the tools for our \
\ environment in place          /
 ------------------------------- 
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

[nix-shell:/work]# exit
exit
bash-4.4# which node yarn git rg cowsay
which: no node in (/root/.nix-profile/bin:/nix/var/nix/profiles/default/bin:/nix/var/nix/profiles/default/sbin)
which: no yarn in (/root/.nix-profile/bin:/nix/var/nix/profiles/default/bin:/nix/var/nix/profiles/default/sbin)
/root/.nix-profile/bin/git
which: no rg in (/root/.nix-profile/bin:/nix/var/nix/profiles/default/bin:/nix/var/nix/profiles/default/sbin)
which: no cowsay in (/root/.nix-profile/bin:/nix/var/nix/profiles/default/bin:/nix/var/nix/profiles/default/sbin)
bash-4.4# 
bash-4.4# # We have left the shell we created with nix-shell, and the tools are not there anymore 
bash-4.4# # One more time...
bash-4.4# 
bash-4.4# nix-shell -p nodejs-14_x yarn git ripgrep cowsay

[nix-shell:/work]# cowsay All packages cached locally, so quicker this time
 _________________________________________ 
/ All packages cached locally, so quicker \
\ this time                               /
 ----------------------------------------- 
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

[nix-shell:/work]# exit
exit
bash-4.4# 
```

So what is this **nix-shell**? It is a command-line tool which is part of a group of tools, under the name **Nix**. Nix is a few different things:

* A package manager
* A domain specific language for building, packaging and deploying software
* A Linux distribution using the Nix package manager, which is called NixOS.

Nix is very much about creating reproducible and declarative deployments and system configurations, with tooling to support these tasks. One outcome of this focus is to have i*mmutable configurations* - you are not destroying any particular resources, just adding to what is already there. 

You do not have to run NixOS to take advantage of other features of Nix. In fact, the tools work on other Linux distributions, on macOS or in WSL 2 (Windows Subsystem for Linux) on Windows machines. You just install Nix to get a couple of command-line tools.

The Nix project has been around for a while, about 19 years. However, a large portion of that time, its usage was quite small. It has only grown more significantly in the last few years.

A good starting point is the [NixOS website](https://nixos.org). The site covers Nix and a tool and you do not need to set up NixOS to get started with it. It outlines some of [the use cases for Nix](https://nixos.org/explore.html), of which ad hoc environments are just one of them.

You find the [installation instructions for Nix here](https://nixos.org/download.html#download-nix). This also includes running it as a container via Docker, which is a good way to get an idea of what it is all about, before actually installing the software itself. The example above was running in an instance of this docker image, in fact. The recommendation is to do a multi-user installation if you use Linux or macOS, while use a single-user installation under Windows WSL2.

It is essentially running a shell command on your local computer to download and run an installation script:

For macOS:
```bash
sh <(curl -L https://nixos.org/nix/install)
```

For Windows (WSL2), single-user installation:
```bash
sh <(curl -L https://nixos.org/nix/install) --no-daemon
```

For Linux, single user installation is same as above. For multi-user installation:
```bash
sh <(curl -L https://nixos.org/nix/install) --daemon
```

If you want to run the docker image instead and have Docker desktop installed, you can run:
```bash
docker run -it nixos/nix
```

or run it to expose a selected directory (workdir) on your local computer:
```bash
docker run -it -v $(pwd)/workdir:/workdir nixos/nix
```

The first time you run the command it will take a bit of time, because it will download the and setting up the specified packages from **NixPkgs,** *which have over 80000 packages at this point in time*. If you re-run the command again later, it will be faster, since the packages have been cached locally.

## Tweak and persist the package information

For the simplest use cases, just specifying the package labels works fine. However, tweak it a bit more, depending on which packages to install and any additional settings that may be needed. For this purpose, there is also the Nix domain specific language, which is geared towards repeatable and/or reproducible setups.

Let us create a directory **work**, and in that directory, create a file **mysetup.nix**. The content will look like this:

```nix
let
    getpkgs = import <nixpkgs>;
    pkgs = getpkgs {};
in
    pkgs.mkShell {
        buildInputs = [
            pkgs.git
            pkgs.nodejs-14_x
            pkgs.yarn
            pkgs.ripgrep
            pkgs.cowsay
        ];
        shellHook = ''
            cowsay "here is your shell environment"
        '';
    }  
```

What this piece of code will do is to retrieve information about all available packages in the latest update of **NixPkgs**. Using the retrieved data, it will create a shell environment (mkShell), where there packages we want to install/build are inputs. *As a bonus here, we also add an execution of cowsay once the environment is ready*.

Now we can set up the environment using the command

```bash
nix-shell mysetup.nix
```

In the **work** directory, we should see something like this: 

```bash
bash-4.4# nix-shell mysetup.nix 
 ________________________________ 
< here is your shell environment >
 -------------------------------- 
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

[nix-shell:/work]# 
```

This makes for a *repeatable* setup, but is *not reproducible*. This is because the latest versions of the different packages might be different if the latest versions of these packages are updated. To get the same versions each time, we need to be specific about which release of NixPkgs we are using. Nix has multiple ways to get a specific release of NixPkgs. Since NixPkgs information is available through a repository in GitHub, we can fetch a specific release and git commit from that repository. In that case, we will *always get the same releases for the packages, as well as the same dependencies for all these packages*. We cannot choose specific versions for each package, since there is an infinite number of combinations with these packages, including their dependencies in that case. **Repeatable and reproducible environments are of higher importance here**. *We will get the specific versions of the packages and their dependencies that are in that specific release of NixPkgs*.

Let us create a new file, this time called **shell.nix**. The content is adjusted slightly, so that we fetch the package data from the NixPkgs repository in GitHub, with a specific tag or branch and commit that is associated with that. One way to get the specific commit is to use the **git ls-remote** command:

```bash
git ls-remote --tags https://github.com/nixos/nixpkgs/ 21.11
506445d88e183bce80e47fc612c710eb592045ed        refs/tags/21.11
```

Instead of a general reference to "\<nixpkgs\>" we make a more specific reference, using the **fetchGit** function. The other parts of the code stay the same.

```bash
let
    nixpkgs = (fetchGit {
        name = "nixos-release-21.11";
        url = "https://github.com/nixos/nixpkgs/";
        ref = "refs/tags/21.11";
        rev = "506445d88e183bce80e47fc612c710eb592045ed";
    });
    getpkgs = import nixpkgs;
    pkgs = getpkgs {};
in
    pkgs.mkShell {
        buildInputs = [
            pkgs.git
            pkgs.nodejs-14_x
            pkgs.yarn
            pkgs.ripgrep
            pkgs.cowsay
        ];
        shellHook = ''
            cowsay "here is your shell environment"
        '';
    }  
```

Now, because we have named the file **shell.nix**, we do not need to specify the filename when running, we can simply run **nix-shell**. Again, the first time you run the command, it will take some time to fetch the package information (be patient!), but if that is run again, it will be cached and execution is much faster.

## Final words

This was a quick introduction to a specific tool within Nix, the nix-shell. It is a nice way to create ad hoc environments in a repeatable and/or reproducible way. It is only scratching the surface of Nix, though, and there is a lot more to explore there. 

I must stay though that the documentation around Nix and its usage leaves a bit to be desired. It can sometimes be a bit confusing to figure out what you can do and how to do it. Right now I hope to make some examples and descriptions as I learn and explore more of Nix, which I think is very promising, despite some challenges.