Reproducible data science with Nix, part 2 -- running {targets} pipelines with Nix
Tags: datascience, nix, rstats
Published: 2023-07-20 09:10:00

<p>This is the second post in a series of posts about Nix. Disclaimer: I’m a super
beginner with Nix. So this series of blog posts is more akin to notes that I’m
taking while learning than a super detailed tutorial. So if you’re a Nix expert
and read something stupid in here, that’s normal. This post is going to focus on
R (obviously) but the ideas are applicable to any programming language.</p>
<p>So in <a href="https://www.brodrigues.co/blog/2023-07-13-nix_for_r_part1/">part 1</a> I
explained what Nix was and how you could use it to build reproducible
development environments. Now, let’s go into more details and actually set up
some environments and run a <code>{targets}</code> pipeline using it.</p>
<p>Obviously the first thing you should do is install Nix. A lot of what I’m
showing here comes from the <a href="https://nix.dev/tutorials/">Nix.dev</a> so if you want
to install Nix, then look at the instructions
<a href="https://nix.dev/tutorials/install-nix">here</a>. If you’re using Windows, you’ll
have to have WSL2 installed. If you don’t want to install Nix just yet, you can
also play around with a NixOS Docker image. NixOS is a Linux distribution that
uses the concepts of Nix for managing the whole operating system, and obviously
comes with the Nix package manager installed. But if you’re using Nix inside
Docker you won’t be able to work interactively with graphical applications like
RStudio, due to how Docker works (but more on working interactively with IDEs in
part 3 of this series, which I’m already drafting).</p>
<p>Assuming you have Nix installed, you should be able to run the following command
in a terminal:</p>
<pre><code>nix-shell -p sl</code></pre>
<p>This will launch a Nix shell with the <code>sl</code> package installed. Because <code>sl</code> is
not available, it’ll get installed on the fly, and you will get “dropped” into a
Nix shell:</p>
<pre><code>[nix-shell:~]$</code></pre>
<p>You can now run <code>sl</code> and marvel at what it does (I won’t spoil you). You can quit
the Nix shell by typing <code>exit</code> and you’ll go back to your usual terminal. If you
try now to run <code>sl</code> it won’t work (unless you installed on your daily machine).
So if you need to go back to that Nix shell and rerun <code>sl</code>, simply rerun:</p>
<pre><code>nix-shell -p sl</code></pre>
<p>This time you’ll be dropped into the Nix shell immediately and can run <code>sl</code>.
So if you need to use R, simply run the following:</p>
<pre><code>nix-shell -p R</code></pre>
<p>and you’ll be dropped in a Nix shell with R. This version of R will be different
than the one potentially already installed on your system, and it won’t have
access to any R packages that you might have installed. This is because Nix
environment are isolated from the rest of your system (well, not quite, but
again, more on this in part 3). So you’d need to add packages as well (exit the
Nix shell and run this command to add packages):</p>
<pre><code>nix-shell -p R rPackages.dplyr rPackages.janitor</code></pre>
<p>You can now start R in that Nix shell and load the <code>{dplyr}</code> and <code>{janitor}</code>
packages. You might be wondering how I knew that I needed to type
<code>rPackages.dplyr</code> to install <code>{dplyr}</code>. You can look for this information
<a href="https://search.nixos.org/packages">online</a>. By the way, if a package uses the
<code>.</code> character in its name, you should replace that <code>.</code> character by <code>_</code> so to
install <code>{data.table}</code> write <code>rPackages.data_table</code>.</p>
<p>So that’s nice and dandy, but not quite what we want. Instead, what we want is
to be able to declare what we need in terms of packages, dependencies, etc,
inside a file, and have Nix build an environment according to these
specifications which we can then use for our daily needs. To do so, we need to
write a so-called <code>default.nix</code> file. This is what such a file looks like:</p>
<pre><code>{ pkgs ? import (fetchTarball &quot;https://github.com/NixOS/nixpkgs/archive/e11142026e2cef35ea52c9205703823df225c947.tar.gz&quot;) {} }:

with pkgs;

let
  my-pkgs = rWrapper.override {
    packages = with rPackages; [dplyr ggplot2 R];
  };
in
mkShell {
  buildInputs = [my-pkgs];
}</code></pre>
<p>I wont discuss the intricate details of writing such a file just yet, because
it’ll take too much time and I’ll be repeating what you can find on the
<a href="https://nix.dev/">Nix.dev</a> website. I’ll give some pointers though. But for
now, let’s assume that we already have such a <code>default.nix</code> file that we defined
for our project, and see how we can use it to run a <code>{targets}</code> pipeline. I’ll
explain how I write such files.</p>
<div id="running-a-targets-pipeline-using-nix" class="section level2">
<h2>Running a {targets} pipeline using Nix</h2>
<p>Let’s say I have this, more complex, <code>default.nix</code> file:</p>
<pre><code>{ pkgs ? import (fetchTarball &quot;https://github.com/NixOS/nixpkgs/archive/8ad5e8132c5dcf977e308e7bf5517cc6cc0bf7d8.tar.gz&quot;) {} }:

with pkgs;

let
  my-pkgs = rWrapper.override {
    packages = with rPackages; [
      targets
      tarchetypes
      rmarkdown
    (buildRPackage {
      name = &quot;housing&quot;;
      src = fetchgit {
        url = &quot;https://github.com/rap4all/housing/&quot;;
        branchName = &quot;fusen&quot;;
        rev = &quot;1c860959310b80e67c41f7bbdc3e84cef00df18e&quot;;
        sha256 = &quot;sha256-s4KGtfKQ7hL0sfDhGb4BpBpspfefBN6hf+XlslqyEn4=&quot;;
      };
    propagatedBuildInputs = [
        dplyr
        ggplot2
        janitor
        purrr
        readxl
        rlang
        rvest
        stringr
        tidyr
        ];
      })
    ];
  };
in
mkShell {
  buildInputs = [my-pkgs];
}</code></pre>
<p>So the file above defines an environment that contains all the required packages
to run a pipeline that you can find on <a href="https://github.com/b-rodrigues/nix_targets_pipeline">this Github
repository</a>. What’s
interesting is that I need to install a package that’s only been released on
Github, the <code>{housing}</code> package that I wrote for the <a href="https://raps-with-r.dev/packages.html">purposes of my
book</a>, and I can do so in that file as
well, using the <code>fetchgit()</code> function. Nix has many such functions, called
<em>fetchers</em> that simplify the process of downloading files from the internet, see
<a href="https://ryantm.github.io/nixpkgs/builders/fetchers/">here</a>. This function takes
some self-explanatory inputs as arguments, and two other arguments that might
not be that self-explanatory: <code>rev</code> and <code>sha256</code>. <code>rev</code> is actually the commit
on the Github repository. This commit is the one that I want to use for this
particular project. So if I keep working on this package, then building an
environment with this <code>default.nix</code> will always pull the source code as it was
at that particular commit. <code>sha256</code> is the hash of the downloaded repository. It
makes sure that the files weren’t tampered with. How did I obtain that? Well,
the simplest way is to set it to the empty string <code>""</code> and then try to build the
environment. This error message will pop-up:</p>
<pre><code>error: hash mismatch in fixed-output derivation &#39;/nix/store/449zx4p6x0yijym14q3jslg55kihzw66-housing-1c86095.drv&#39;:
         specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
            got:    sha256-s4KGtfKQ7hL0sfDhGb4BpBpspfefBN6hf+XlslqyEn4=</code></pre>
<p>So simply copy the hash from the last line, and rebuild! Then if in the future
something happens to the files, you’ll know. Another interesting input is
<code>propagatedBuildInputs</code>. These are simply the dependencies of the <code>{housing}</code>
package. To find them, see the <code>Imports:</code> section of the
<a href="https://github.com/rap4all/housing/blob/fusen/DESCRIPTION">DESCRIPTION</a> file.
There’s also the <code>fetchFromGithub</code> fetcher that I could have used, but unlike
<code>fetchgit</code>, it is not possible to specify the branch name we want to use. Since
here I wanted to get the code from the branch called <code>fusen</code>, I had to use
<code>fetchgit</code>. The last thing I want to explain is the very first line:</p>
<pre><code>{ pkgs ? import (fetchTarball &quot;https://github.com/NixOS/nixpkgs/archive/8ad5e8132c5dcf977e308e7bf5517cc6cc0bf7d8.tar.gz&quot;) {} }:</code></pre>
<p>In particular the url. This url points to a specific release of <code>nixpkgs</code>, that
ships the required version of R for this project, R version 4.2.2. How did I
find this release of <code>nixpkgs</code>? There’s a handy service for that
<a href="https://lazamar.co.uk/nix-versions/?channel=nixpkgs-unstable&amp;package=r">here</a>.
So using this service, I get the right commit hash for the release that install
R version 4.2.2.</p>
<p>Ok, but before building the environment defined by this file, let me just say
that I know what you’re thinking. Probably something along the lines of: <em>damn
it Bruno, this looks complicated and why should I care? Let me just use
{renv}!!</em> and I’m not going to lie, writing the above file from scratch didn’t
take me long in typing, but it took me long in reading. I had to read quite a
lot (look at <a href="https://www.brodrigues.co/blog/2023-07-13-nix_for_r_part1/">part
1</a> for some nice
references) before being comfortable enough to write it. But I’ll just say this:</p>
<ul>
<li>continue reading, because I hope to convince you that Nix is really worth the effort</li>
<li>I’m working on a package that will help R users generate <code>default.nix</code> files like the one from above with minimal effort (more on this at the end of the blog post)</li>
</ul>
<p>If you’re following along, instead of typing this file, you can clone
this <a href="https://github.com/b-rodrigues/nix_targets_pipeline">repository</a>.
This repository contains the <code>default.nix</code> file from above, and a <code>{targets}</code>
pipeline that I will run in that environment.</p>
<p>Ok, so now let’s build the environment by running <code>nix-build</code> inside a terminal
in the folder that contains this file. It should take a bit of time, because
many of the packages will need to be built from source. But they <strong>will</strong> get
built. Then, you can drop into a Nix shell using <code>nix-shell</code> and then type R,
which will start the R session in that environment. You can then simply run
<code>targets::tar_make()</code>, and you’ll see the file <code>analyse.html</code> appear, which is
the output of the <code>{targets}</code> pipeline.</p>
<p>Before continuing, let me just make you realize three things:</p>
<ul>
<li>we just ran a targets pipeline with all the needed dependencies which include not only package dependencies, but the right version of R (version 4.2.2) as well, and all required system dependencies;</li>
<li>we did so WITHOUT using any containerization tool like Docker;</li>
<li>the whole thing is <strong>completely</strong> reproducible; the exact same packages will forever be installed, regardless of <em>when</em> we build this environment, because I’m using a particular release of <code>nixpkgs</code> (8ad5e8132c5dcf977e308e7bf5517cc6cc0bf7d8) so each piece of software this release of Nix installs is going to stay constant.</li>
</ul>
<p>And I need to stress <em>completely reproducible</em>. Because using {renv}+Docker,
while providing a very nice solution, still has some issues. First of all, with
Docker, the underlying operating system (often Ubuntu) evolves and changes
through time. So lower level dependencies might change. And at some point in the
future, that version of Ubuntu will not be supported anymore. So it won’t be
possible to rebuild the image, because it won’t be possible to download any
software into it. So either we build our Docker image and really need to make
sure to keep it forever, or we need to port our pipeline to newer versions of
Ubuntu, without any guarantee that it’s going to work exactly the same. Also, by
defining <code>Dockerfile</code>s that build upon <code>Dockerfile</code>s that build upon
<code>Dockerfile</code>s, it’s difficult to know what is actually installed in a particular
image. This situation can of course be avoided by writing <code>Dockerfile</code>s in such
a way that it doesn’t rely on any other <code>Dockerfile</code>, but that’s also a lot of
effort. Now don’t get me wrong: I’m not saying Docker should be canceled. I
still think that it has its place and that its perfectly fine to use it (I’ll
take a project that uses <code>{renv}</code>+Docker any day over one that doesn’t!). But
you should be aware of alternative ways of running pipelines in a reproducible
way, and Nix is such a way.</p>
<p>Going back to our pipeline, we could also run the pipeline with this command:</p>
<pre><code>nix-shell /path/to/default.nix --run &quot;Rscript -e &#39;setwd(\&quot;/path/to\&quot;);targets::tar_make()&#39;&quot;</code></pre>
<p>but it’s a bit of a mouthful. What you could do instead is running the pipeline
each time you drop into the nix shell by adding a so-called <code>shellHook</code>. For
this, we need to change the <code>default.nix</code> file again. Add these lines in the
<code>mkShell</code> function:</p>
<pre><code>...
mkShell {
  buildInputs = [my-pkgs];
  shellHook = &#39;&#39;
     Rscript -e &quot;targets::tar_make()&quot;
  &#39;&#39;;
}</code></pre>
<p>Now, each time you drop into the Nix shell in the folder containing that
<code>default.nix</code> file, <code>targets::tar_make()</code> get automatically executed. You can
then inspect the results.</p>
<p>In the next blog post, I’ll show how we can use that environment with IDEs like
RStudio, VS Code and Emacs to work interactively. But first, let me quickly talk
about a package I’ve been working on to ease the process of writing
<code>default.nix</code> files.</p>
</div>
<div id="rix-reproducible-environments-with-nix" class="section level2">
<h2>Rix: Reproducible Environments with Nix</h2>
<p>I wrote a very early, experimental package called <code>{rix}</code> which will help write
these <code>default.nix</code> files for us. <code>{rix}</code> is an R package that hopefully will
make R users want to try out Nix for their development purposes. It aims to
mimic the workflow of <code>{renv}</code>, or to be more exact, the workflow of what Python
users do when starting a new project. Usually what they do is create a
completely fresh environment using <code>pyenv</code> (or another similar tool). Using
<code>pyenv</code>, Python developers can install a per project version of Python and
Python packages, but unlike Nix, won’t install system-level dependencies as
well.</p>
<p>If you want to install <code>{rix}</code>, run the following line in an R session:</p>
<pre class="r"><code>devtools::install_github(&quot;b-rodrigues/rix&quot;)</code></pre>
<p>You can then using the <code>rix()</code> function to create a <code>default.nix</code> file like so:</p>
<pre class="r"><code>rix::rix(r_ver = &quot;current&quot;,
         pkgs = c(&quot;dplyr&quot;, &quot;janitor&quot;),
         ide = &quot;rstudio&quot;,
         path = &quot;.&quot;)</code></pre>
<p>This will create a <code>default.nix</code> file that Nix can use to build an environment
that includes the current versions of R, <code>{dplyr}</code> and <code>{janitor}</code>, and RStudio
as well. Yes you read that right: you need to have a per-project RStudio
installation. The reason is that RStudio modifies environment variables and so
your “locally” installed RStudio would not find the R version installed with
Nix. This is not the case with other IDEs like VS Code or Emacs. If you
want to have an environment with another version of R, simply run:</p>
<pre class="r"><code>rix::rix(r_ver = &quot;4.2.1&quot;,
         pkgs = c(&quot;dplyr&quot;, &quot;janitor&quot;),
         ide = &quot;rstudio&quot;,
         path = &quot;.&quot;)</code></pre>
<p>and you’ll get an environment with R version 4.2.1. To see which versions are
available, you can run <code>rix::available_r()</code>. Learn more about <code>{rix}</code> on its
<a href="https://b-rodrigues.github.io/rix/">website</a>. It’s in very early stages, and
doesn’t handle packages that have only been released on Github, yet. And the
interface might change. I’m thinking of making it possible to list the packages
in a yaml file and then have <code>rix()</code> generate the <code>default.nix</code> file from the
yaml file. This might be cleaner. There is already something like this called
<a href="https://github.com/luispedro/nixml/tree/main">Nixml</a>, so maybe I don’t even
need to rewrite anything!</p>
<p>But I’ll discuss this is more detail next time, where I’ll explain how you can
use development environments built with Nix using an IDE.</p>
</div>
<div id="references" class="section level2">
<h2>References</h2>
<ul>
<li>The great <a href="https://nix.dev/tutorials/install-nix">Nix.dev</a> tutorials.</li>
<li>This <a href="https://rgoswami.me/posts/rethinking-r-nix/">blog post: Statistical Rethinking and Nix</a> I referenced in part 1 as well, it helped me install my <code>{housing}</code> package from Github.</li>
<li><a href="https://github.com/luispedro/nixml/tree/main">Nixml</a>.</li>
</ul>
<p>
Hope you enjoyed! If you found this blog post useful, you might want to follow
me on <a href="https://fosstodon.org/@brodriguesco">Mastodon</a> or <a href="https://www.twitter.com/brodriguesco">twitter</a> for blog post updates and
<a href="https://www.buymeacoffee.com/brodriguesco">buy me an espresso</a> or <a href="https://www.paypal.me/brodriguesco">paypal.me</a>, or buy my <a href="https://www.brodrigues.co/about/books/">ebooks</a>.
You can also watch my videos on <a href="https://www.youtube.com/c/BrunoRodrigues1988/">youtube</a>.
So much content for you to consoom!
</p>