Data science and notebooks = databooks: a love story
Published: 2022-02-02 10:00:20

---
title: Data science and notebooks = databooks: a love story
published: true
date: 2022-02-02 04:00:00 UTC
tags: 
canonical_url: https://dataroots.io/research/contributions/data-science-and-notebooks-databooks-a-love
---

If you're not new to Python and data science, you probably heard of [Jupyter notebooks](https://jupyter.org/). But if you haven't, here's the gist: it's an interactive environment, meaning you can run little bits of code and see the output, store variables in memory, etc.

That makes notebooks a good tool for experimentation, reporting and visualizations. And because of that, it's a popular tool of choice for data science in general. And this is why you see a lot of notebooks in places like [Kaggle](https://www.kaggle.com/).

Personally I think notebooks are a good tool for figuring out what you want (think of experiments and proofs-of-concept), and once you do, I'd move everything to scripts and modules (think production code). But that's where the Python community gets divided. Because some people _hate_ notebooks. Regardless of the stage of development you're in.

> "I'd rather roll around in shattered glass naked than work with notebooks" - Someone, somewhere... probably

And honestly, I get it. As a machine learning engineer, I've had my pains with notebooks. That's because as great as they may be for quick prototyping and experimentation, as soon as you want to work with someone else, it gets equally as painful. Notebooks are filled with metadata (execution counts, kernel things, and other stuff that you just don't care about) that can cause git conflicts and leaving you to open a lot of nonsensical JSON files to resolve. As a quick example, just by opening someone else's notebook you will change some metadata, because notebooks include some kernel information (kernel specs and language to name a few).

Calls were heard, prayers answered. A family of tools emerged with the purpose of relieving those pains. And recently, it got a new family member: [databooks](https://datarootsio.github.io/databooks).

## `databooks --help`

`databooks` is a `black`-like tool for sharing and caring about notebooks. It has (currently) two functions: clearing metadata and resolving git conflicts.

### `databooks meta --help`

Much like `black`, the only thing you **need** to pass is a path. We have sensible defaults to do the rest.

```
databooks meta path/to/notebooks
```

"What are these defaults?", you ask? Great question. For each notebook in the path, by default:

- It will remove execution counts
- It **won't** remove cell outputs
- It will remove metadata from all cells (like cell tags or ids)
- It will remove all metadata from your notebook (including kernel information)
- It **won't** overwrite files for you

But the user is king, and as such you can still change all that. You could choose to remove cell outputs by passing `--rm-outs`. Or if there is some metadata you'd like to keep, such as cell tags, you can do so by passing `--cell-meta-keep tags`. Also, if you do want to save the clean notebook you can either pass a prefix (`--prefix ...`) or a suffix (`--suffix ...`) that will be added before writing the file, or you can simply overwrite the source file (`--overwrite`).

If you don't like typing every time you want to run the command, you can specify any argument values in the configuration file, which is a `pyproject.toml` in the root of your project. `databooks` finds this file by default, but you can also specify the config via `--config path/to/your/config/pyproject.toml`.

Make sure to check the docs for more info on the [configuration](https://datarootsio.github.io/databooks/latest/usage/configuration/) and all the [arguments](https://datarootsio.github.io/databooks/latest/CLI/) that you can pass in the CLI.

### `databooks fix --help`

As one would know, (git) conflicts are, unfortunately, part of life. No matter how much we try to avoid them. If in `meta` we try to avoid git conflicts, in `fix` we are fixing these conflicts after they have appeared. Similarly to `databooks meta ...`, the only **required** argument here is a `path`.

```
databooks fix path/to/notebooks
```

What will it do? For each notebook in the path _that has git conflicts_:

- It will keep the metadata from the notebook in `HEAD`
- For the conflicting cells, it will wrap some special cells around the differences, like in normal git conflicts

![](https://dataroots.ghost.io/content/images/2022/01/databooks_conflict_resolver_example.jpeg)

And as you could've guessed, you can still change the behavior. Similarly to what we saw above, this can be done with passing a configuration `pyproject.toml` file or specifying the CLI arguments. You could, for instance, keep the metadata from the notebook in `BASE` (as opposed to `HEAD`). If you know you only care about the notebook cells in `HEAD` or `BASE`, then you could pass `--cells-head` or `--no-cells-head` and not worry about all that.

You can also pass a special `--cell-fields-ignore` parameter, that will remove the cell metadata from both versions fo the conflicting notebook before comparing them. This is because depending on your `Jupyter` version you may have an `id` field, that will be unique for each cell. That is, all the cells will be considered different even if they have the same `source` and `outputs` as their `id`s are different. By removing `id` and `execution_count` (we'll do this by default) we only compare the actual code and outputs to determine if the cells have changed or not.

There is one tiny catch that if you commit a conflicted notebook without fixing it, `databooks` will **not** consider this a conflicted file anymore (just like `git` wouldn't). And since you've read this far, I'll share a little secret 🤫: we don't really "fix" the conflict, but rather we retrieve the previous file versions and compare them.

Once again, make sure the check the [docs](https://datarootsio.github.io/databooks) for more details!

## Automate it

Now, as nice as this all may be, we can still do better. Having the tools is one thing, but that does't mean we'll always remember to use them.

![](https://dataroots.ghost.io/content/images/2022/01/pebkac.png)

Luckily, this is not a new problem. We can enforce standards via [CI/CD](https://en.wikipedia.org/wiki/CI/CD). Or even better, we could use [pre-commit hooks](https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks).

With that in mind, `databooks` has some examples in the docs on how to integrate the tool into the normal workflow, using [GitHub Actions](https://datarootsio.github.io/databooks/latest/usage/cicd/#github-actions) and the [`pre-commit` package](https://datarootsio.github.io/databooks/latest/usage/precommit/#pre-commit-package). Feel free to check them out!  
  
And of course, all relevant info can be found on the [repo](https://github.com/datarootsio/databooks/)! Thanks for reading! 🚀