Perfect Elixir: Development Workflows
Tags: elixir, tutorial, development, webdev
Published: 2024-06-17 12:25:00

Today we'll explore the tools and workflows essential for our daily development. Our goal is to create a streamlined onboarding experience and establish efficient mechanisms for code changes. Let's dive into some solutions to see how it all works out.

**Table of Contents**
* [A Reflection on Workflows](#a-reflection-on-workflows)
    * [Goals for Our Workflows](#goals-for-our-workflows)
    * [Rethinking Branches](#rethinking-branches)
* [Bootstrapping](#bootstrapping)
    * [Be Careful with System Dependencies](#be-careful-with-system-dependencies)
    * [Maximize Trust](#maximize-trust)
    * [The First Step](#the-first-step)
    * [All The Steps](#all-the-steps)
* [Daily Workflows](#daily-workflows)
    * [In Defence of Shell Scripting](#in-defence-of-shell-scripting)
    * [Doctor](#doctor)
    * [Update](#update)
    * [Shipit](#shipit)
    * [Check All the Things](#check-all-the-things)
* [Test Automating Our Workflows](#test-automating-our-workflows)
    * [Testability](#testability)
    * [Mocking System Calls](#mocking-system-calls)
    * [Expect](#expect)
    * [bats! 🦇](#bats)
* [Conclusion](#conclusion)

&nbsp;<br>
## A Reflection on Workflows
![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/k2rdk3gxvoz9obsue5cf.png)

It's not uncommon that a team's workflow is "_go clone the repo_" and "_get your pull-requests reviewed_". While not inherently bad, this kind of vaguely defined workflow can be difficult to improve for several reasons:

1. **Team-wide Pain Points:** If running various upgrade commands causes groans, how can we turn these observations into solutions when there's no code to iteratively improve? I've seen teams argue **against** committing frequently because running the required upgrade-commands was too cumbersome. I've seen developers **not want to pull** out of fear it might mess up their setup. These anti-patterns are hard to uproot once established.

2. **Onboarding Complexity:** Without local workflows, onboarding often ends up just a list of manual steps. If new hires encounter a pain point they can probably update the guide, but those changes are likely to just rot themselves. Without code, it's hard to see which steps are redundant or combinable to reduce complexity.

To avoid this, we'll aggressively adopt **local workflows**. Runnable code is easier to iterate on and keep correct through test automation.

> ℹ️ _BTW I've worked on projects that took more than a **week** to get started 😱. This was a shocking waste of time, with senior developers debugging dependencies for hours. We can and must do better._

&nbsp;<br>
### Goals for Our Workflows
To determine our direction, let's consider the [DORA research on software delivery](https://dora.dev). This research identifies what software delivery patterns lead to the best outcomes, and for this article we'll focus on two key metrics that are part of a statistically meaningful pattern that is likely to **cause** improvements to organizational performance:

1. **Minimal time from code committed to that code running in production**, ideally no more than an hour.
2. **Frequent deploys**, ideally each commit resulting in a deployment.

We'll align with these principles to create workflows that enable our team to **continuously pull and push code changes with minimal delay**.

> ℹ️ _BTW for more on the DORA research, check out my articles: [Introduction to "Accelerate", the scientific analysis of software delivery](https://dev.to/jonlauridsen/an-introduction-to-accelerate-its-dora-metrics-30lh) and [The Software Delivery Performance Model](https://dev.to/jonlauridsen/the-software-delivery-performance-model-dora-metrics-2nf4). Their book, [Accelerate: The Science of Lean Software and DevOps](https://www.amazon.com/Accelerate-Software-Performing-Technology-Organizations/dp/1942788339), is highly recommended._

&nbsp;<br>
### Rethinking Branches
This research leads us to a choice: To achieve high-frequency changes, branches are not ideal. Why? Because pushing commits anywhere other than `main` introduces latency. If we're serious about an ideal workflow, we **shouldn't use branches**.

For some, this is a shocking statement. How else can changes land safely? If you rely on branches read on for more details, I promise it's possible to do away with them.

> ℹ️ _BTW for more on trunk-based development, see the [DORA research](https://dora.dev/devops-capabilities/technical/trunk-based-development/) and my [Beginners Intro to Trunk Based Development](https://dev.to/jonlauridsen/beginners-intro-to-trunk-based-development-3158)._

Let's start experimenting!

&nbsp;<br>
## Bootstrapping
![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/oxs36zv70572ud6f28qr.png)

The most extreme onboarding will be just a single command with no prior dependencies or requirements. Here’s the simplest way to run a remote script (on Mac and Linux):

```sh
$ curl -Ssf https://…/script | sh
```

Imagine an onboarding guide that's just that one line 😍.

But one small constraint: We'll need to inspect the user's configuration (e.g. to check if [pkgx](https://dev.to/jonlauridsen/perfect-elixir-environment-setup-1145#pgkx) is installed), and to do that we can't pipe to `sh` because that spawns a new shell. Instead, we need to `source` the script. And because we can't pipe to `source` we need to slightly change our ideal invocation:

```sh
$ curl -fsSL https://…/script > /tmp/script && source /tmp/script
```

But that's fine, this is still promising to be an extremely simple onboarding one-liner.

&nbsp;<br>
### Be Careful with System Dependencies
A word of caution before we get to coding: installing system dependencies affects the user's computer as a whole, and it's **not** wise to try to fully automate their installation:

1. It's **invasive**: Some developers have strong preferences, and our script could disrupt their setup. We're asking them to trust a script they don't know, so we should write code that can't cause damage.

2. It's **brittle**: We can't account for everyone's different system setups, so the more sophisticated our solutions are the more we risk our code will fail.

3. It's **unmaintainable**: We invite pointless sophistication where developers with different preferences will add their choice to the automation, and we end up with a mess of sophistication to maintain.

And for what? **pkgx** already has a slick installation process, so no amount of automation is saving much time! Let's instead just **identify missing dependencies** and let the user handle the installation.

&nbsp;<br>
### Maximize Trust
Let's make it clear that our script only suggests actions:

```sh
$ URL="https://raw.githubusercontent.com/gaggle/perfect-elixir/main/bootstrap"
$ curl -fsSL $URL > /tmp/bootstrap && source /tmp/bootstrap
This script bootstraps our development environment by suggesting
what dependencies need to be installed and configured.

To be clear: This script never changes or affects your system, 
it only ever inspects and makes suggestions.

Ok to proceed? [y/n]:
```

> {% details 🖥️ Terminal %}
> <!-- 🎬 bootstrap_intro -->
> ![Bootstrap script introducing itself and prompting the user to proceed](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3nma54fdnsyzj3ornb03.gif)
> {% enddetails %}

That should establish trust right from the start.

> ℹ️ _BTW I'm not showing bootstrap code here because it's mostly a simple script that outputs the above text. But if you'd like to follow the details [you're welcome to inspect the full bootstrap script here](https://raw.githubusercontent.com/gaggle/perfect-elixir/main/bootstrap)._

&nbsp;<br>
### The First Step
Let's first check if the developer has **pkgx** installed by verifying the exit code of `which pkgx`. If not installed, instruct the user:

```sh
Ok to proceed? [y/n]: y
• Checking for pkgx… ✓
• pkgx is not installed x

User action required: Install pkgx
──────────────────────────────────
You need to install pkgx. Source this script again afterwards.

pkgx can be installed in various ways, depending on your 
preferences:

• Via Homebrew:
    $ brew install pkgxdev/made/pkgx

• Via cURL:
    $ curl -Ssf https://pkgx.sh | sh

For other ways to install see: 
  https://docs.pkgx.sh/run-anywhere/terminals

pkgx is the package manager that handles system dependencies, 
and it is not currently installed. The installation is simple,
and via Homebrew does not require sudo or other forms of 
elevated permissions.

Read more about pkgx on https://pkgx.sh

Source this script again after pkgx has been installed.
```
> {% details 🖥️ Terminal %}
> <!-- 🎬 bootstrap_pkgx -->
> ![Proceeding with bootstrap script, it checks for pkgx and fails, outputting detailed instructions to the user for how to install pkgx](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vjzvoio0gyosuzwl4qa2.gif)
> {% enddetails %}

This way the user can proceed at their own pace, but are offered easy copy-pasteable choices.

&nbsp;<br>
### All The Steps
To complete onboarding we'll do three more requirements:

1. Verify pkgx's **shell integration**.
2. Ensure the user has **cloned** our repository.
3. Confirm pkgx provides its **developer environment** (e.g., Elixir, Erlang).

Skipping details for brevity, the final bootstrapping script ends up running like this:

```console
Ok to proceed? [y/n]: y
• Checking for pkgx… ✓
• pkgx is installed ✓
• Checking pkgx shell integration… ✓
• Shell integration is active ✓
• Checking repository is cloned… ✓
• Repository is available ✓
• Checking development environment is active… ✓
• Development environment is active ✓

Good to go

Bootstrapping is done:
✓ pkgx is installed
✓ pkgx shell integration is active
✓ The repository is cloned and ready
✓ All system dependencies are available

This system has been bootstrapped and can now hook into our project 🎉

• Run this command to continue onboarding:

    $ bin/doctor
```
> {% details 🖥️ Terminal %}
> Here is the full flow from factory-reset machine to ready to work on the project:
> <!-- 🎬 bootstrap_all_steps -->
> ![Running bootstrap script on a factory reset machine, pasting in each of the suggested commands until bootstrapping completes successfully](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uf2e6c3v10np9dfjwhid.gif)
> {% enddetails %}

And with that, we’ve unlocked a simple onboarding solution. And the simplicity of the code should invite incremental improvements by the whole team. Nice!

Now let's explore the daily development workflow scripts developers will use regularly.

> ℹ️ _BTW this bootstrapping script may not suit enterprise requirements but can be extended to cover various cases. Keep in mind after its initial "**pkgx is installed**" check the full pkgx ecosystem is available, enabling powerful tools like GitHub CLI and entire programming languages. Bootstrapping can evolve significantly based on needs!_

&nbsp;<br>
## Daily Workflows
![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/myfdtkc2us95f9v91ns4.png)

To enable developers to quickly pull and push code changes, we need to decide on the scripting language for our workflows. With pkgx, we can use any language, but should we?

&nbsp;<br>
### In Defence of Shell Scripting
Shell scripts are the industry standard for scripting, and are widely used and understood by almost everyone. While they may not be the most elegant choice, they are practical and often low-maintenance. We don't earn any money from writing workflow scripts, so probably our best choice is to avoid unnecessary complexities and go with what is most simple: shell scripting.

&nbsp;<br>
### Doctor
Our first workflow component will be a script to keep our development environment up-to-date, ensuring vital preconditions are met (e.g., the local database is running, migrations are applied, Mix dependencies installed, etc.).

> ℹ️ _BTW I've gotten used to calling this script `doctor` because it verifies the health of our environment. You can choose whatever name you feel is most fitting._

First, let's catch if the user has fallen out of the pkgx ecosystem. By overlapping with where bootstrap left off we provide a fallback for unforeseen errors:

```bash
$ cat bin/doctor
#!/usr/bin/env bash
set -euo pipefail
command which pkgx && which erl && which elixir ||\
(echo "Missing system dependencies, \
run 'source bin/bootstrap'" && exit 1)
```

We can simulate an issue by turning off the development environment:

```console
$ dev off
env -erlang.org=26.2.4 -elixir-lang.org=1.16.2 -postgresql.org=15.2.0 

$ bin/doctor
/usr/local/bin/pkgx
Missing system dependencies, run 'source bin/bootstrap'

$ echo $?
1
```

Re-enabling the environment makes the check pass:

```console
$ dev on
env +erlang.org=26.2.4 +elixir-lang.org=1.16.2 +postgresql.org=15.2.0 

$ bin/doctor
/usr/local/bin/pkgx
/Users/cloud/.pkgx/erlang.org/v26.2.4/bin/erl
/Users/cloud/.pkgx/elixir-lang.org/v1.16.2/bin/elixir

$ echo $?
0
```

This directs users back to bootstrapping if the pkgx system isn't activated, quite nice.

This implementation is pretty noisy though, and mixes low-level shell implementation with high-level goals. We can improve on that by introducing an abstraction layer via a shell helper function called `check`:

```bash
$ cat bin/doctor
#!/usr/bin/env bash
set -euo pipefail
source "$(dirname "$0")/.shhelpers"
check "Check system dependencies" \
  "command which pkgx && which erl && which elixir" \
  "source bin/bootstrap"
```
```console
$ bin/doctor
• Check system dependencies ✓
```
> {% details 🖥️ Terminal %}
> <!-- 🎬 doctor_one_check -->
> ![Running bin/doctor and it shows "Checking system dependencies" with a green checkmark after it](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/re5hhgn5uesuz7m1o0j1.gif)
> {% enddetails %}

It's worth taking care of our terminal output to not go blind to all the mindless muck we can otherwise end up printing.

> ℹ️ _BTW the `.shhelpers` code is not directly relevant to this article, but [you can find the full script here](https://github.com/gaggle/perfect-elixir/blob/perfect-elixir-3-development-workflows-%26-processes/bin/.shhelpers) if you'd like. They're inspired by workflows introduced to me by [Eric Saxby](https://github.com/sax) and [Erik Hanson](https://github.com/eahanson) of [Synchronal](https://github.com/synchronal)._

Let's check if our local database is running next:

```sh
$ git-nice-diff -U0 .
/bin/doctor
L#7:
+check "Check PostgreSQL server is running" \
+ "pgrep -f bin/postgres" \
+ "bin/db start"

$ bin/doctor
• Check system dependencies ✓
• Check PostgreSQL server is running x
> Executed: pgrep -f bin/postgres
Suggested remedy: bin/db start
(Copied to clipboard)

$ bin/db start
• Creating /Users/cloud/perfect-elixir/priv/db ✓
• Initializing database ✓
• Database started:
waiting for server to start.... done
server started
↳ Database started ✓

$ bin/doctor
• Check system dependencies ✓
• Check PostgreSQL server is running ✓
```
> {% details 🖥️ Terminal %}
> <!-- 🎬 doctor_two_checks -->
> ![Running bin/doctor and it shows the database in need of starting, then running `db/start`, then running bin/doctor again and the database-checking step now passes](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/em6go0gbeqwec1n2wkxo.gif)
> {% enddetails %}

The **doctor** pattern is now clear: Check for a condition, suggest a fix. Simple to extend, easy to understand.

> ℹ️ _BTW the `bin/db` script abstracts logic away from the doctor script and provides a handy way for developers to manage their database. Its implementation isn't directly relevant but [you can read the full script here](https://github.com/gaggle/perfect-elixir/blob/perfect-elixir-3-development-workflows-%26-processes/bin/db)._

Let's skip to having added all necessary checks for our app to start:

```sh
$ bin/doctor
Running checks…

• Check system dependencies ✓
• Check developer environment ✓
• Check PostgreSQL server is running ✓
• Check PostgreSQL server has required user ✓
• Check mix hex ✓
• Check mix dependencies ✓
• Check PostgreSQL database exists ✓

✓ System is healthy & ready
```
> {% details 🖥️ Terminal %}
> <!-- 🎬 doctor_system_is_healthy -->
> ![Running `bin/doctor` showing 6 green checkmarks, reporting the system is healthy and ready](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kdboy6mjuezmkh6y688p.gif)
> {% enddetails %}

And now we can start our app 🎉:

```sh
$ iex -S mix phx.server
[info] Running MyAppWeb.Endpoint with Bandit 1.4.2 at 127.0.0.1:4000 (http)
[info] Access MyAppWeb.Endpoint at http://localhost:4000
[watch] build finished, watching for changes...
Erlang/OTP 26 [erts-14.2.4] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1] [dtrace]

Interactive Elixir (1.16.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)>
```
> {% details 🖥️ Terminal %}
> <!-- 🎬 start -->
> ![Running `iex -S mix phx.server` now results in the successful starting of the Phoenix server](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uhogz1nlskvq3stykr1x.gif)
> {% enddetails %}

We now have `bin/doctor` ensuring our system is ready. And new developers can go from a factory-reset machine to having our product running locally in just a handful of minutes by running **bootstrap** & **doctor**.

&nbsp;<br>
### Update
Next, we'll create a script to easily get the latest code. This will be a **replacement** for `git pull`, as it will pull down the latest changes **and** run necessary commands to apply changes correctly.

> ℹ️ _BTW especially teams that use trunk-based development can generate several dozens of commits per day, so there's good need for a script like this._

First, let's run `git pull`, and then ensure mix dependencies are up-to-date and compiled. The `.shhelpers` library from before has a `step` function that runs a command but hides the output unless an error occurs, which is perfect for this:

```console
$ cat bin/update
#!/usr/bin/env bash
set -euo pipefail
source "$(dirname "$0")/.shhelpers"
check "Check branch is main" \
  '[ "$(git rev-parse --abbrev-ref HEAD)" = "main" ]' \
  "git checkout main"
step "Pulling latest code" \
  "git pull origin main --rebase"
step "Installing dependencies" "mix deps.get"
step "Compiling dependencies" "mix deps.compile"
bin/doctor

$ bin/update
• Check branch is main ✓
• Pulling latest code ✓
• Installing dependencies ✓
• Compiling dependencies ✓
Running checks…
• Check system dependencies ✓
• Check PostgreSQL server is running ✓
• Check PostgreSQL server has required user ✓
• Check mix hex ✓
• Check mix dependencies ✓
• Check PostgreSQL database exists ✓

✓ System is healthy & ready
```
> {% details 🖥️ Terminal %}
> ![Running bin/update, resulting in latest changes being pulled down, dependencies installed and compiled, and the system checked](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/x3lp4jqqm8zi8hudzuka.gif)
> {% enddetails %}

This script makes it easy to integrate the latest changes, and because it ends with running `doctor` we're constantly ensuring our system is in a good state. And we'll add more steps to this script as needed whenever we discover additional tasks that should be run after pulling new code.

> ℹ️ _BTW usually `update` would also apply migrations, but we don't have any yet so I've skipped that for now._

&nbsp;<br>
### Shipit
The final workflow script, `shipit`, is crucial because it will let us safely ship changes. It must ensure our code is in a shippable state by running test automation and other quality gates before pushing the code.

Our needs are simple right now as we don't have much code: we just need to run unit-tests and formatting checks. Here's how we can do that:

```bash
$ cat bin/shipit
#!/usr/bin/env bash
set -euo pipefail

source "$(dirname "$0")/.shhelpers"

bin/update
step --with-output "Run tests" "mix test"
check "Check files are formatted" "mix format --check-formatted" "mix format"
step "Pushing changes to main" "git push origin main"
cecho "\n" -bB --green "✓" --green " Shipped! 🚢💨"
```

And the result is:

```console
$ bin/shipit
Integrating changes…
• Check active branch ✓
• Pulling latest code ✓
• Installing dependencies ✓
• Compiling dependencies ✓

Running checks…
• Check system dependencies ✓
• Check PostgreSQL server is running ✓
• Check PostgreSQL server has required user ✓
• Check mix hex ✓
• Check mix dependencies ✓
• Check PostgreSQL database exists ✓

✓ System is healthy & ready

Checking code…
• Run tests:
.....
Finished in 0.1 seconds (0.05s async, 0.06s sync)
5 tests, 0 failures

Randomized with seed 297141
↳ Run tests ✓
• Check files are formatted ✓
• Pushing changes to main ✓

✓ Shipped! 🚢💨
```
> {% details 🖥️ Terminal %}
> <!-- 🎬 shipit -->
> ![Shipit script running, showing all checks passing and ending up pushing the code](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/h7g5whj3cz1hnzcxetdk.gif)
> {% enddetails %}

This provides a safe and quick way to ship code by first updating the code to ensure the environment is in sync, then running tests and checking for any issues, and finally pushing to main. This workflow maximizes Continuous Integration (CI) and Continuous Delivery (CD) by constantly integrating changes and pushing code to production with minimal latency.

All that's left now is to practice shipping frequently, and continuously engage customers for feedback!

> ℹ️ _BTW it's beneficial to adopt these scripts **while they're still raw and simple**. Waiting for them to be "perfect" is IMO a mistake because the clarity and ease of iteration of the initial versions is what builds trust in the workflows. The initial scripts should cover essential needs, and then be allowed to naturally expand. This engages the team and maximizes collective involvement._

&nbsp;<br>
### Check All the Things
We've established workflows that let us continuously integrate changes with `bin/update` and push changes with `bin/shipit` (replacing `git pull` and `git push`). While these scripts can be improved and made more robust by adding more quality gates (e.g., run Dialyzer, prevent compiler warnings, run security scans), there's one aspect we can't automate: **The review**.

Code is improved by multiple pairs of eyes, but how can that be done without adding branches and latency? The answer is simple yet impactful:

**Reviewing must also happen locally**.

Some developers may resist this idea, but it aligns with modern development practices: Discrete reviews, often from code-changes that have been hidden away in branches for hours and days, add latency and friction. We should instead aim for **continuous code reviewing**.

So when a commit is ready, get it reviewed immediately. Don't wait, don't delay, and don't start other work until the current work is reviewed. And to further reduce disruptions just code it together: share a workstation (or use screen sharing remotely) and develop the code collaboratively. This way, changes flow to main without obstacles, enabling true continuous integration and continuous delivery.

Then, practice taking many more much smaller steps, shipping dozens of times an hour.

**Now** we're achieving real continuous integration and continuous delivery 🤩.

> ℹ️ _BTW there is extensive literature on pair and whole-team programming. While negative pairing can be exhausting, positive pairing is very enjoyable 😊. Articles like [Pair Programming by Martin Fowler](https://martinfowler.com/articles/on-pair-programming.html) explain the dos and don'ts, and [Dave Farley's videos](https://www.youtube.com/watch?v=aItVJprLYkg) explore the topic insightfully. Additionally, [Woody Zuill's Mob Programming: A Whole Team Approach](https://www.youtube.com/watch?v=SHOVVnRB4h0) offers insights beyond pairing. For continuous improvement, [Many More Much Smaller Steps by GeePaw Hill](https://www.geepawhill.org/2021/09/29/many-more-much-smaller-steps-first-sketch/) provides excellent inspiration._

&nbsp;<br>
## Test Automating Our Workflows
![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hbevtocvzl1oc9fqkv5d.png)

Testing scripts is often considered too hard, but without tests, iterating on our scripts becomes increasingly difficult. Let's tackle this challenge step-by-step.

&nbsp;<br>
### Testability
To make scripts testable, we need to mock external calls. Running the script in a real environment for every test isn't feasible, so we need a way to simulate these calls.

&nbsp;<br>
### Mocking System Calls
We'll use a `call` command that by default simply wraps external calls, but it also allows itself to be mocked during tests. Here's a basic implementation we can wire into the start of our scripts, which defines `call` only if it doesn't already exist:

```bash
# Define call if 'call' is not already a command
if ! type call >/dev/null 2>&1; then
  call() { "$@"; }
fi
```

Under test conditions, we can replace `call` with a mock implementation:

```console
$ call() { echo "called with: $*" > call.log; return 1; }
$ source bin/bootstrap
…
Ok to proceed? [y/n]: y
• Checking for pkgx… ✓
• pkgx is not installed x
…

$ cat call.log
called with: which pkgx
```

What we see above is that our mocked `call` gets invoked when we run bootstrap, allowing us to control its behavior and verify that our script responds correctly.

To put it into our bootstrapping context, we can wrap it all into an easier-to-use script that lets us configure and create a mocked `call` command, like this:

```console 
$ export MOCK=$(test/mcall configure "which pkgx|1|")
$ source test/mcall
$ source bin/bootstrap
…
Ok to proceed? [y/n]: y
• Checking for pkgx… ✓
• pkgx is not installed x
…
$ test/mcall assert
All mocks used ✓
```

And that's it. That's one fully tested use case right there! 

Well, sort of: I did have to manually answer `y` to bootstrap's prompt. How do we automate that?

> ℹ️ _BTW it's possible this complexity in testing shell scripts should be a reason to switch away from shell scripting. Maybe if we were writing all this in a more comprehensive programming language, we could use their mature test-runners to more easily achieve test-automation? It's worth considering, although for now I'll stay the course to see if this is even possible to solve._

&nbsp;<br>
### Expect
Next, we need to interact with our scripts and assert their outputs. For this, we'll use `expect`, a tool for automated interaction with programs:

```console
$ expect -c 'spawn echo "foo"; expect "foo"; expect eof'
spawn echo foo
foo
```

To test our bootstrap script, we can create an `expect` script:

```console
$ cat test/expect.exp
#!/usr/bin/env expect
set timeout 3
expect_before {
    timeout { puts "timeout"; exit 2 }
}
spawn bash
send "source bin/bootstrap\r"
expect "Ok to proceed? [y/n]:"
send "n\r"
send "exit\r"
expect eof
```

Running this script simulates user interaction and validates the output:

```console
$ ./test/expect.exp
…
Ok to proceed? [y/n]: n
exit
bash-5.2$ exit
exit

$ echo $?
0
```

Cool.

And note: `expect` is **quite** extensible because it's based on the `Tcl` language (pronounced "tickle"). It has a great history dating back to the late 80s, and is a proper language with procedures and conditionals and much more.

> ℹ️ _BTW the full `expect` script I ended up with takes additional arguments and pretty-prints some details. [It's available here](https://github.com/gaggle/perfect-elixir/blob/perfect-elixir-3-development-workflows-%26-processes/test/run-expect-scenario) if you're curious._
> {% details 🖥️ Terminal %}
> <!-- 🎬 run-expect-scenario -->
> ![Animated gif of a terminal showing the script `run-expect-scenario` being run with two parameters: An Expect script that sources bootstrap and answers no to the prompt, and shell invocation parameter that specifies `zsh -f`. The output of the run is of bootstrap script running and answering no at the prompt](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/41sd3jwcfh3jx62qz0y7.gif)
> {% enddetails %}

&nbsp;<br>
### bats! 🦇
To manage our tests, ensuring they all get run and tracking which ones pass and which fail, we'll turn to the **[Bash Automated Testing System](https://bats-core.readthedocs.io)** (bats). It lets us define tests in a bootstrap.bats file like this:

```bash
@test "pkgx not installed" {
  run_mocked_scenario '
which pkgx|1|pkgx not found
' \ '
send "source bin/bootstrap\n"
exp "Ok to proceed?"
send "y\n"
exp "pkgx is not installed"
exp "User action required: Install pkgx"
exp_prompt
'
}

$ bats test/bootstrap.bats
bootstrap.bats
 ✓ pkgx not installed

1 test, 0 failures
```

> ℹ️ _BTW here I'm skipping over some details to not extend this article even more. The above snippet actually calls a bats helper function that orchestrates `mcall` and `expect`. The [full bootstrap.bats is available here](https://github.com/gaggle/perfect-elixir/blob/perfect-elixir-3-development-workflows-%26-processes/test/bootstrap.bats) if you'd like to follow all the details._

Finally, we can write tests for all possible use cases to ensure our script behaves as expected:

```console
$ bats test/bootstrap.bats
bootstrap.bats
 ✓ no to proceed
 ✓ pkgx not installed
 ✓ pkgx is too old
 ✓ pkgx is missing dev integration
 ✓ pkgx is missing env integration
 ✓ folder is not a repository
 ✓ remote is not expected repository
 ✓ no erl so should activate dev
 ✓ no elixir so should activate dev
 ✓ no psql so should activate dev
 ✓ good to go with git remote
 ✓ good to go with https remote

12 tests, 0 failures
```
> {% details 🖥️ Terminal %}
> <!-- 🎬 bats-bootstrap -->
> ![Terminal showing the command `bats test/bootstrap.bats` being run, resulting in 12 tests being run each with a checkmark](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/q4yf3efgd81fkoejz0q1.gif)
> {% enddetails %}

These tests help guard against regressions and ensure our scripts work across different shells. They've been very helpful in driving out several small bugs that would have otherwise plagued these scripts.

If you have simpler methods for testing interactive scripts, please share them!

&nbsp;<br>
## Conclusion
![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ioqfasxbgrieypjffnyc.png)

We've covered a lot today, and have come away with a set of workflow scripts that streamline onboarding and daily development tasks. These scripts support rapid code iteration and align with best scientific practices, and avoid relying on latency-adding workflows such as branches and pull-requests. I think they will form a great foundation for many projects, to foster a culture of efficiency, quality, and continuous improvement.