~/.dotfiles
Tags: devops, bash, productivity, unix
Published: 2021-04-27 16:42:38

# Why

Suppose you are receiving your brand new computer. You have a bunch of softwares to install (your favorite browser, code editor, windows management tool, etc.) and some configurations and shortcuts tweaks. How long does it take you to be fully productive once unboxed? One hour, one day, one week?

If you do this manually, you'll make mistakes. Because, we are all human beings.

You can end up doing it successfully with a well written todo-list but, wait, do you make all the downloads, clicks, configuration by hand? You work in computer science, use computers for doing repetitive and boring tasks.

Now we suppose your laptop is ready for work. You're happy with it and want the same work environment on your desktop, which has a different operating system, should you start again from a blank page?

That's why you need installation automation. They are often called "Dotfiles". Let's deep dive into it.

# What

We want to put all our configuration for various environments in a place that is easy to share or reuse, and ideally idempotent.

## Context

The following guidelines and explanations work on an [Unix-like operating systems](https://en.wikipedia.org/wiki/Unix-like). Sorry Microsoft's users, since I've not used this system for a while, I can't talk about it. It would work with [Windows Subsystem for Linux](https://fr.wikipedia.org/wiki/Windows_Subsystem_for_Linux) because it's a Unix system under the hood.

## Content

We have two things to take care of: installation of softwares and their configurations.

* For the installation part, thankfully, most operating-systems have a built-in or mainstream packages manager. Aptitude, Brew, Pacman, you name it. It's a good starting point but not every tool or software are available on it or on each architecture. You may need to download a binary or an archive and to put it at the right place.

* For the configuration part, it's more ~complicated~ diverse. Some tools rely on files, others on environment variables and others are simply not configurable easily, e.g. hidden or binary files.

Note that [Brew](https://brew.sh), the package manager for macOS, is not built-in and must be installed manually first.

## Storage

It looks like we are starting to talk about code, notably infrastructure as code. Code nowadays is in a version control software (VCS). The most broadly use is [Git](https://git-scm.com). In order to make it shareable to everyone, you must store your Git repository on GitHub, GitLab, Bitbucket, etc. They are easy to use, well adopted by the community and easy to find.

Git has many features, the one we'll focus on is branching. With branches, you can handle the `architecture x operation-system` matrix with ease. A branch for "personal-macos", a branch for "work-ubuntu", a branch for "home-raspbian", etc. You can also add conditions in your code for doing things based on hostnames, environment variables, etc.

My Dotfiles are in place for personal and professional contexts, on `amd64` and `arm64` architectures, on macOS and Linux operating systems. They are all in one repository with only two branches (`main` and `work`). I normalize many differences of each operating system.

# How

## Bash

You can find [many tools on GitHub](https://github.com/webpro/awesome-dotfiles) (or elsewhere) to bootstrap your Dotfiles. Some people choose to rely on Ansible, others on some tools you have to install. But how to install a tool that install the tools? Manual installation is not an option. It's like a chicken-egg problem.

Obviously, you need a starting point and ensure that it's available everywhere. On Unix-like systems, [Bash](https://en.wikipedia.org/wiki/Bash_%28Unix_shell%29) is the common denominator. This responsibility, being the common denominator, is hard when you are a thirty years old software.

For example, in the latest macOS version, the shipped version of `bash` is `3.2.57`, a version from 2007, [because of a licensing change](https://apple.stackexchange.com/a/238281). I'll not explain all the differences between v3 and v5 (current one in 2021) but you can see below a difference.

```bash
MY_VARIABLE="macOS"

# Lower case, works only with bash 4+
echo "${MY_VARIABLE,,}"

# Lower case, the old way, bash 3 compliant
echo "${MY_VARIABLE}" | tr "[:upper:]" "[:lower:]"
```

So, _Bash_ is the broader compatible software already installed on most systems. We know how to run our code, but how can we retrieve it then?

## Zip & Curl

We saw before that the best way to store our Dotfiles is in Git, but `git` must be installed for cloning the repository, that will install `git`. Chicken-egg problem again. Fortunately, solutions like GitHub or Gitlab provide a way to download a repository in a [Zip](<https://en.wikipedia.org/wiki/ZIP_(file_format)>) archive. `unzip`, like `bash`, is present natively on most distributions.

For fetching datas, we also need a tool. [curl](https://en.wikipedia.org/wiki/CURL) is the common way to make a request on the Internet from a terminal and, guess what, it's also built-in on most distributions.

## The golden triangle

Thank to [Bash piping](https://linuxhint.com/bash_pipe_tutorial/), we can `curl` a bootstrap script, pipe it to `bash` and start installation by `unzip`-ing archive of the repository. All tools are present after installation (tested on macOS, Manjaro, Debian 10).

```bash
curl "https://my.bootstrap.script" | bash
```

⚠️ When doing this kind of command, in any case, always check what you are going to run, especially when some "one-liner install scripts" require `sudo`. [The Internet](https://www.youtube.com/watch?v=iDbyYGrswtg) is a cool thing, but not everyone acts as a cool person on it.

In the bootstrap script, you can choose where the Dotfiles will be (in my case, I place it in `${HOME}/code/dotfiles`). I would bring your attention on how the `curl` options are written, in their long format. When writing a script, you're not in a terminal trying to be quick and limiting your keyboard input. You write code that will be shared and must be comprehensive by other humans. Be verbose.

```bash
#!/usr/bin/env bash

set -o nounset -o pipefail -o errexit

main() {
  local INSTALL_PATH="${HOME}/code"
  local GITHUB_USER="ViBiOh"
  local DOTFILES_NAME="dotfiles"
  local DOTFILES_BRANCH="main"
  local ARCHIVE_FILENAME="${INSTALL_PATH}/dotfiles.zip"

  mkdir -p "${INSTALL_PATH}"

  curl \
    --disable \
    --silent \
    --show-error \
    --location \
    --max-time 60 \
    --output "${ARCHIVE_FILENAME}" "https://github.com/${GITHUB_USER}/${DOTFILES_NAME}/archive/${DOTFILES_BRANCH}.zip"
  unzip "${ARCHIVE_FILENAME}" -d "${INSTALL_PATH}"
  rm -f "${ARCHIVE_FILENAME}"

  rm -rf "${INSTALL_PATH:?}/${DOTFILES_NAME}"
  mv "${INSTALL_PATH}/${DOTFILES_NAME}-${DOTFILES_BRANCH}" "${INSTALL_PATH}/${DOTFILES_NAME}"

  (
    cd "${INSTALL_PATH}/${DOTFILES_NAME}"
    "./init" -a

    git init
    git remote add origin "http://github.com/${GITHUB_USER}/${DOTFILES_NAME}.git"
    git fetch origin
    git checkout --force "${DOTFILES_BRANCH}"
  )
}

main
```

The bootstrap script does the "first download" of all needed files. Once done, the `init` script takes over. The `init` is the script that installs and configures all the things, you'll run it multiple times in the future.

At the end of the script, you can see that there are `git` commands. After a successful installation, Git will be installed, and I reconcile the archive downloaded (without `.git` folder) with the Git upstream.

# Structure

The first thing we need in our Dotfiles repository is the two scripts described before: `bootstrap` and `init`. You'll need also at least 3 folders more: `install`, `symlinks` and `sources`. We are gonna dive into them.

```bash
> tree -L 1
.
├── bootstrap
├── init
├── install/
├── sources/
└── symlinks/
```

## Install

Softwares are most of the time available as a binary and maybe on your favorite package manager. But there are also tasks that are not "software installation" in proper words but need to be run at least once on your computer (e.g. disabling unwanted features of your os, generating configuration file that cannot be symlinked, etc.). From my point of view, it's part of the installation scripts.

Among the softwares I use, all are not available on the package manager. Package managers often install optional softwares I don't want and install them globally, which can interfere with other users. I tend to download binaries directly from GitHub and put them in my `${PATH}` instead of relying on the package manager. It's also easier to stick to a defined version and binaries tend to update faster than package upstreams.

In an ideal world, Dotfiles should not run any command in `sudo`. You install _your_ configuration on a machine, keep everything in your `${HOME}` if you can. For example, I create an `${HOME}/opt/` folder for putting my stuff in it (GOPATH, Python packages, etc). I add the `${HOME}/opt/bin` in my `${PATH}`. If I delete my `opt/` folder, I'm the only person impacted.

In the `install/` folder, you can separate concerns of your installation by splitting things in different files. This way, you can disable a script based on an environment variable. e.g. On my server, I don't need to install my code editor. I set an environment variable `DOTFILES_NO_EDITOR` and the install script will not try to run the `install/editor` file.

I see the installation phase in 3 steps:

- `clean`. Installation must be **idempotent**, it can require cleaning before installing or simply "resetting" Dotfiles installation.
- `install`. Installation as you may think of it: running script that download from package manager, put binaries in appropriate folders, etc.
- `credentials`. Retrieving secrets from your password-manager and put them in your configuration file. See the [Secrets](#secrets) section in `sources/` for more details. This phase must be done after all installations (you have to install the password manager first).

For running each phase separately, we can rely on the presence of a function in a file. Sourcing a file is easy in Bash and checking that a function is available too.

If we condense all we have talked before in one function, it will be like the following code, put in the `init` script.

```bash
browse_install() {
  while IFS= read -r -d '' file; do
    local BASENAME_FILE
    BASENAME_FILE="$(basename "${file}")"

    local UPPERCASE_FILENAME
    UPPERCASE_FILENAME="$(printf "%s" "${BASENAME_FILE}" | tr "[:lower:]" "[:upper:]")"
    local DISABLE_VARIABLE_NAME="DOTFILES_NO_${UPPERCASE_FILENAME}"

    if [[ ${!DISABLE_VARIABLE_NAME:-} == "true" ]]; then
      continue
    fi

    if [[ -r ${file} ]]; then
      for action in "${@}"; do
        unset -f "${action}"
      done

      source "${file}"

      for action in "${@}"; do
        if [[ $(type -t "${action}") == "function" ]]; then
          printf "%s - %s" "${action}" "${BASENAME_FILE}"
          "${action}"
        fi
      done
    fi
  done < <(find "${CURRENT_DIR}/install" -type f -print0 | LC_ALL=C sort --zero-terminated)
  # ${CURRENT_DIR} is the root of the Dotfiles repository
}

browse_install clean install
browse_install credentials
```

## Symlinks

The easiest tools to configure are the ones that rely on a single file on your home directory, that starts with a dot. That's where the repository takes its name: the famous "dotfiles". You know them: the `.bashrc`, the `.vimrc` or the `.gitconfig`. In order to keep the file under version control system (Git), but not committing your entire `${HOME}`, the easy trick is using a [symlink](https://en.wikipedia.org/wiki/Symbolic_link). The file exists in the `${HOME}` folder, so your tool can read it, but the content remains on the folder under version control. The best of both worlds.

```bash
~/ > ls -la
.bashrc -> /Users/macbook/code/dotfiles/symlinks/bashrc
.curlrc -> /Users/macbook/code/dotfiles/symlinks/curlrc
.gitconfig -> /Users/macbook/code/dotfiles/symlinks/gitconfig
.ignore -> /Users/macbook/code/dotfiles/symlinks/ignore
.inputrc -> /Users/macbook/code/dotfiles/symlinks/inputrc
.tmux.conf -> /Users/macbook/code/dotfiles/symlinks/tmux.conf
.vimrc -> /Users/macbook/code/dotfiles/symlinks/vimrc
```

They are simple to install: put all your `.file` in a directory (e.g. symlinks) and link every file in that folder to your `${HOME}` folder. Note that my files are named without a dot (e.g. `bashrc`) and I add the "dot" during the symlink phase. A dotfile is by default a hidden file on most operting systems. I don't want it to be hidden in code, only when used.

Creating the symlinks can be done with the following snippet, put in the `init` script.

```bash
create_symlinks() {
  while IFS= read -r -d '' file; do
    local BASENAME_FILE
    BASENAME_FILE="$(basename "${file}")"

    if [[ -n ${FILE_LIMIT} ]] && [[ ${BASENAME_FILE} != "${FILE_LIMIT}" ]]; then
      continue
    fi

    rm -f "${HOME}/.${BASENAME_FILE}"
    ln -s "${file}" "${HOME}/.${BASENAME_FILE}"
  done < <(find "${CURRENT_DIR}/symlinks" -type f -print0)
  # ${CURRENT_DIR} is the root of the Dotfiles repository
}
```

A gentle reminder on symlinks: they are living "content". When you pull the new version of your Dotfiles repository, the configuration is changed instantaneously.

👍 It has good side: you pull your code and have nothing to do.

👎 It has down side: if you have a Git conflict on your `symlinks/gitconfig` file, your Git can be broken because your `~/.gitconfig` is invalid 😅.

Chicken-egg problem, always.

## Sources

For configuring you shell environment, the music is a little bit different, in other words, complex! The common way for configuring your shell is the `~/.bashrc` (or `~/.zshrc`, `.you-name-it-rc`). That's what we just configured before with symlinks. But putting the configuration of every tools used in the same file makes the file unreadable, with a lot of comments for separating purpose of code. Not a clean way to do it.

Fortunately, it's up to you to split it correctly. My `~/.bashrc` sources every file contained in the `dotfiles/sources/` folder.

I don't want to hardcode the path of my Dotfiles folder. Thanks to symlinks already being linked to a folder inside my repository, I'm able to find where my files really are with the `${BASH_SOURCE[0]}` trick.

Sourcing the `sources` folder can be done with the following snippet, put in the `symlinks/.bashrc` file.

```bash
source_all() {
  local SCRIPT_DIR
  SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

  for file in "${SCRIPT_DIR}/../sources/"*; do
    [[ -r ${file} ]] && [[ -f ${file} ]] && source "${file}"
  done
}
```

In `sources/` folder, you put every terminal configuration you need, e.g. environment variable, function, aliases etc. It's time for a good reminder.

> _Rule #1 of the Dotfiles Club_
>
> You don't put secrets in your Dotfiles.
>
> _Rule #2 of the Dotfiles Club_
>
> YOU DO NOT PUT SECRETS IN YOUR DOTFILES.

![](https://media.giphy.com/media/1AfLx8dLTVoHK/source.gif)

If you want to have secrets in your environment variables (e.g. a token, a password), put them in the `~/.localrc`. Not symlinked anywhere! As its name stands, it remains "local".

```bash
if [[ -e "${HOME}/.localrc" ]]; then
  source "${HOME}/.localrc"
fi
```

### Secrets

If you have read the two rules of the _Dotfiles Club_, you may wonder how to configure automatically your computer without putting secrets in a repository. The secret of the secrets management is: a password manager.

Because we are in Unix environments, I personally use [pass](`https://www.passwordstore.org`) that is easily scriptable, relies on the solid [PGP encryption](https://en.wikipedia.org/wiki/Pretty_Good_Privacy) and stores everything in Git. I need to import my GPG Public Key (relying on a [Yubikey](https://www.yubico.com) from the private) and it's fine. Security is a serious topic. You can use any other password manager, install the CLI with `install/` scripts and you should be able to retrieve your credentials and to push them in your configuration files during the `credentials` phase.

## Tips and tricks

Keep in mind that your dotfiles are expected to work on various environments at various stages of installation. Always be backward compatible and lenient on errors.

When sourcing, you should not use Bash flags `-o nounset -o pipefail -o errexit`, because if an error happens, it will crash Bash startup and you can end up being locked outside of your terminal.

Also, if you use a tool that is not built-in, always check its presence with the following snippet. You may have disabled its installation, it may not be available on your architecture, etc.

```bash
if command -v git >/dev/null 2>&1; then
  # do `git` related stuff
fi
```

### Bashrc

Using a `.bashrc` for automatically loading your environment is a good thing, but you don't need it when performing a headless operation (such as `rsync`). In that case, it can slow you down. Fortunately, there is this trick for disabling the loading of the `.bashrc`.

```bash
[[ -z ${PS1:-} ]] && return
```

macOS has a particular case here. By default, it doesn't look for the `~/.bashrc` file, but the `~/.bash_profile`. You can fix it by creating this file and pointing it to your `.bashrc`.

```bash
#!/usr/bin/env bash

if [[ -f "${HOME}/.bashrc" ]]; then
  source "${HOME}/.bashrc"
fi
```

### Order

Browsing files must be done in a certain order because you may have dependencies between files (e.g. changing the `${PATH}` variable before checking if a software is installed, installing Python before using `pip`). You don't have this kind of problems in a single file, because you simply put the line at the right place.

When iterating on a folder with `for` or `find`, Bash relies on [the alphabetic order](https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html#Filename-Expansion), defined by the [`LC_ALL / LC_COLLATE`](https://www.gnu.org/software/bash/manual/html_node/Bash-Variables.html#index-LC_005fALL) environment variables. You can control order of sourcing by ensuring your files are in alphabetical order. Pro-tips: underscore is before any lower case character.

In order to properly configure your locale, you can add the following snippet before the `source_all`, that gracefully sets locale if it's present.

```bash
set_locale() {
  local LOCALES=("en_US.UTF-8" "en_US.utf8" "C.UTF-8" "C")
  local ALL_LOCALES
  ALL_LOCALES="$(locale -a)"

  for locale in "${LOCALES[@]}"; do
    if [[ $(echo "${ALL_LOCALES}" | grep --count "${locale}") -eq 1 ]]; then
      export LC_ALL="${locale}"
      export LANG="${locale}"
      export LANGUAGE="${locale}"

      return
    fi
  done

  return 1
}
```

## All together

I've made the simplest possible dotfiles repository in [this Gist](https://gist.github.com/ViBiOh/e38aeb07d6d56dfa78256b61781478c5). It's not possible to create folder on a Gist, so for example `symlinks_bashrc` should be `symlinks/bashrc`.

You'll find a snippet for installing Brew and changing the default shell of user on macOS. If you install Bash with Brew, you have two `bash`: one in `/bin/bash` (3.2.57) and one in `/usr/local/bin/bash` (5+).

{% github ViBiOh/dotfiles %}

My [personal Dotfiles](https://github.com/vibioh/dotfiles) have more syntactic sugar but are the source of inspiration for writing this article. It's a "template repository", feel free to use it. With near 2k commits, achieving my Dotfiles was a journey and I just wanted to share it. Hope you enjoyed the reading ride!

PS: English is not my native language and it's my first post here, be kind, I'm open to advice for improving =)

[cover image from Unsplash](https://unsplash.com/photos/pUAM5hPaCRI)