Predictable Code in Elixir: Expressions as Reducers and Macros
Tags: elixir
Published: 2022-08-30 11:51:52

In the [first part of this series on maintainable Elixir code](https://blog.appsignal.com/2022/07/19/writing-predictable-elixir-code-with-reducers.html), we started by applying rules for code predictability in our code design. We immediately saw an emerging pattern: a series of transformations on state. In this part, we'll explore this pattern further.

We'll first learn how to write expressions as reducers. Then we'll use metaprogramming to make use of reducers and enforce code style seamlessly.

Finishing up, we'll see an example where all the pieces fit together.

Let's get going!

## Quick Recap of Part One and What We'll Do

In the first part, we experimented with applying code styles to this snippet:

```elixir
with %Payment{} = payment <- fetch_payment_information(params),
     {:ok, user} <- Session.get(conn, :user),
     address when !is_nil(address) <- fetch_address(user, params),
     {:ok, order} <- create_order(user, payment, address) do
  conn
  |> put_flash(:info, "Order completed!")
  |> render("checkout.html")
else
  {:error, :payment_failed} ->
    handle_error(conn, "Payment Error")

  %Store.OrderError{message: message} ->
    handle_error(conn, "Order Error")

  error ->
    handle_error(conn, "Unprocessable order")
end
```

By the end of this post, we'll have all the tools to rewrite it like this:

```elixir
case Checkout.execute(%{params: params}, options) do
  {:ok, %{order: order}} ->
    conn
    |> put_flash(:info, "Order completed!")
    |> redirect(to: Routes.order_path(conn, order))

  {:error, error}
    conn
    |> put_flash(:error, parse_error(error_description))
    |> render("checkout.html")
end
```

While writing this post, I created [ex_pipeline](https://github.com/msramos/ex_pipeline), an open source library that allows developers to create pipelines while enforcing code style easily.

All code examples in this post are simplified versions of what exists in that project.

## Chain of Transformations

One of my favorite hobbies is woodworking — I'm not good at it, by all means, but it's something that I do enjoy. Before starting a new project, I always write down all the materials and tools I'll need, as well as the steps I'll follow. That's how I go from pieces of wood to a finished table, from an initial state to a finished state.

Let's think about how a table is made: you get some raw materials — wood, nails, glue — and a few tools. Then you execute a set of actions to modify the shape and appearance of the materials — cutting, assembling, polishing, and painting.

You apply a _chain of transformations_ on an initial state (raw materials) until you get the desired state (the table).

We already talked about the emerging pattern of a _pipeline_. That's what we are going to explore over the next paragraphs.

## Writing Expressions as Reducers in Elixir

Reducers are one of the key concepts of functional programming. They are very simple to understand: you take a list of values, apply a function to each element, and accumulate the results.

In Elixir, we can work with reducers using the [`Enum.reduce/3`](https://hexdocs.pm/elixir/1.12/Enum.html#reduce/3)
function and its variants — [`Enum.reduce/2`](https://hexdocs.pm/elixir/1.12/Enum.html#reduce/2) and [`Enum.reduce_while/3`](https://hexdocs.pm/elixir/1.12/Enum.html#reduce_while/3).

For example, we can use a reducer to filter all even numbers from a list:

```elixir
numbers = [1, 2, 3, 4, 5, 6]
starting_value = []

Enum.reduce(numbers, starting_value, fn current_number, filtered_values ->
  if rem(current_number, 2) == 0 do   # is the current number even?
    [number | filtered_values]        # if so, add it to the accumulator
  else                                # otherwise...
    filtered_values                   # just ignore it and return the accumulator
  end
end)
```

This code will return the list `[2, 4, 6]`.

To execute different actions on a value, we can transform the list of values into a list of functions. Then we pass the value as the initial value for the accumulator:

```elixir
transformations = [
  fn x -> x + 1 end, # add 1
  fn x -> x * 2 end  # multiply by two
]
starting_value = 10

Enum.reduce(transformations, starting_value, fn transformation, current_value ->
  apply(transformation, [current_value])
end)
```

This looks like a complicated way to write `(10 + 1) * 2`, right? But by expressing each transformation as a function, we can attach an arbitrary number of transformations. They all must accept the same number of parameters and return the next updated state. In other words, these functions must look the same and be small.

In the end, we want to write the previous example as something like this:

```elixir
defmodule MyModule do
  use Pipeline

  def sum_step(value, _), do: {:ok, value + 1}
  def double_step(value, _), do: {:ok, value * 2}
end
```

In the next part, we can do just that using a bit of magic from macros!

## Sewing Reducers Into Features with Macros

Now that we know how to use reducers let's use them to write pipelines.

The goal is to make the process of writing, reading, and maintaining pipelines as easy as possible. So, instead of adding a [DSL](https://opensource.com/article/20/2/domain-specific-languages) or using complex configs, we just want to write our functions using the same pattern of name and arity. We can then automate the process of building the pipeline by detecting which functions follow this pattern.

We create a new pipeline by _writing code on the same pattern_.

## The State: Track What's Happening in Elixir

Before we dive into macros, let's take a step back and talk a little bit about the _state_ of the pipeline.

When executing a pipeline, we need to know a few things: the initial state, the current state, if it's still valid, and if there are any errors. All this information will help us debug and troubleshoot any problems we might find when writing our code.

Also, by abstracting state updates in this module, we can enforce a very important rule on reducers: the return value must be an ok/error tuple.

The starting point of a module that manages state should be something like this:

```elixir
defmodule Pipeline.State do
  defstruct [:initial_value, :value, :valid?, :errors]

  def new(value) do
    %Pipeline.State{valid?: true, initial_value: value, value: value, errors: []}
  end

  def update(state, {module, function_name} = _function_definition, options \\ []) do
    case apply(module, function_name, [state.value, options]) do
      {:ok, value} ->
        %Pipeline.State{state | value: value}

      {:error, error_description} ->
        %Pipeline.State{state | valid?: false, errors: [error_description | state.errors]}

      bad_return ->
        raise "Unexpected return value for pipeline step: #{inspect(bad_return)}"
    end
  end
end
```

The `new/1` function will create a new valid and clean `%Pipeline.State{}` struct based on the initial value we give to the pipeline executor.

The `update/3` function will update or invalidate the given state by calling `function` with the state's `value` and the given `options`.

If the given `function` returns an `{:ok, <updated_value>}`, then the state's `value` is updated, and we are good to go. However, if the `function` returns an `{:error, <error>}` value, the state is invalidated. The returned error is added to the list of errors in the state. If the function returns anything that's different from an ok/error tuple, it will raise an exception.

[See the full implementation of this module](https://github.com/msramos/ex_pipeline/blob/090200bda9e3aeb43dd8bdc9892cde7200e6a1ab/lib/pipeline/state.ex).

## Pipeline Steps and Hooks

Each function that transforms the state will be called a _step_. Here are some guidelines to follow:

- A _step_ is a function whose name ends with `_step` and accepts two parameters.
- Steps execute in the order they are declared in their module.
- If one step fails, then all remaining steps are ignored.

Sometimes, we still want to execute code, even during failures — write a log, update a trace, publish a message, etc. We have a special type of step for these cases: a _hook_. Let's list the guidelines for hooks:

- A _hook_ is a function whose name ends with `_hook` and accepts two parameters.
- Hooks execute in the order they are declared, just like _steps_.
- They are always executed at the end of the pipeline, after all steps execute.

Good! But how do we detect that steps and hooks exist? The answer is
[metaprogramming](https://blog.appsignal.com/2021/09/07/an-introduction-to-metaprogramming-in-elixir.html)! Luckily for us, Elixir has powerful metaprogramming tools to make all this possible.

## Setting Up the Required Macros in Elixir

We'll need two things: [macros](https://elixir-lang.org/getting-started/meta/macros.html) and [compile callbacks](https://hexdocs.pm/elixir/1.12/Module.html#module-compile-callbacks) to read information about the functions of a module, and store what a step and a hook are.

Starting with the base module `Pipeline`, we want to `use Pipeline` in other modules. For that to happen, we declare a special macro, `__using__/1`. This macro is called with the keyword `use` in Elixir.

In this macro, we add a compile hook to the target module so that we can inject code into it. The `@before_compile` hook does exactly that by calling the given function when a compiler is about to generate the underlying bytecode. It accepts either a module or a tuple with a module and function name. We'll go with the second option to make things simpler.

```elixir
defmodule Pipeline do
  defmacro __using__(_options) do
    quote do
      # Calls Pipeline.build_pipeline/1 before generating the final bytecode, allowing us to analyze, generate and
      # inject code whenever we call `use Pipeline`.
      @before_compile {Pipeline, :build_pipeline}
    end
  end

  defmacro build_pipeline(env) do
    # inject code!
  end
end
```

Anything called inside the `quote` block will be injected into the caller module — including the result of the `@before_compile` hook.

[Check out the official docs for more information about `quote` and `unquote` blocks](https://elixir-lang.org/getting-started/meta/quote-and-unquote.html).

Now, whenever we want to create a pipeline, we call `use`:

```elixir
defmodule MyPipeline do
  use Pipeline # <- code will be injected!
end
```

### Detecting Steps and Hooks

We have everything in place to inject code, but we still need to generate the code that will be injected. Let's expand the `build_pipeline/1` macro:

```elixir
defmacro build_pipeline(env) do
  # Fetch all public functions from the caller module
  definitions = Module.definitions_in(env.module, :def)

  # Filter functions that are steps and hooks
  steps = filter_functions(env.module, definitions, "_step", 2)
  hooks = filter_functions(env.module, definitions, "_hook", 2)

  # Generate code on the caller module
  quote do
    # returns all steps and hooks
    def __pipeline__, do: {unquote(steps), unquote(hooks)}

    # Syntatic sugar so we can execute directly from the module that defines it
    def execute(value, options \\ []), do: Pipeline.execute(__MODULE__, value, options)
  end

  # Filter functions from the given module based on their suffix and arity
  defp filter_functions(module, definitions, suffix, expected_arity) do
    functions =
      Enum.reduce(definitions, [], fn {function, arity}, acc ->
        # Detect if the function name ends with the desired suffix: _step or _hook
        valid_name? =
          function
          |> Atom.to_string()
          |> String.ends_with?(suffix)

        # Detect if the function arity matches the expected arity
        has_expected_args? = arity == expected_arity

        cond do
          valid_name? and has_expected_args? ->
            # In this case, the function does end with the desired suffix and has the correct arity.
            # We include it in the accumulator along with the line where it was declared so we can
            # order them by their position in their module
            {_, _, [line: line], _} = Module.get_definition(module, {function, arity})
            [{module, function, line} | acc]

          valid_name? ->
            # If the function has the desired suffix but the arity is not correct, we raise this error
            # because we don't want people trying to figure out why their steps or hooks are not
            # being executed.
            raise(PipelineError, "Function #{function} does not accept #{expected_arity} parameters.")

          true ->
            # If the function doesn't match our filters, it's not a part of the pipeline and can be
            # just ignored
            acc
        end
      end)

    # At this point we have filtered the functions that match the filter criteria but they
    # are not in order. We use the line number information to sort them and then drop it
    # once we don't need it anymore.
    functions
    |> Enum.sort(fn {_, _, a}, {_, _, b} -> a <= b end) # order by line number
    |> Enum.map(fn {m, f, _l} -> {m, f} end)            # drop line number
  end
end
```

This macro receives the compiler `env`, [which has lots of context information](https://hexdocs.pm/elixir/1.13.4/Macro.Env.html). Here, the important bit is the _caller module_, accessible via the `env.module` attribute.

Then we use the [module.definitions_in/2](https://hexdocs.pm/elixir/1.13.0/Module.html#definitions_in/2) special function to get a list of all functions declared on the module `env.module` with the `def` keyword.

We call the `Pipeline.filter_functions/4` function to filter these definitions by their suffix and arity, essentially detecting our steps and hooks! The `Pipeline.filter_functions/4` function is kind of big, so I've added comments to help you navigate through it.

As said before, anything inside the `quote` block is injected directly into the caller module. So whenever we call `use Pipeline`, the module will have two extra functions: `__pipeline__/0`, and `execute/2`.

The `Pipeline` module uses the first function to execute our custom pipeline, while the `execute/2` function is just a convenience function that will execute `Pipeline.execute/3`. It allows us to execute our pipeline by calling `MyPipeline.execute(value, options)`.

Speaking of `Pipeline.execute/3`, it's time to define it. This function is the core of this engine, and it's responsible for actually calling a reducer that will power our pipeline engine:

```elixir
defmodule Pipeline do
  # ...

  def execute(module, value, options) do
    # Build a new initial state of the pipeline
    initial_state = State.new(value)

    # Fetch steps and hooks from the module
    {steps, hooks} = apply(module, :__pipeline__, [])

    # Run each step from the pipeline and build the final state
    final_state =
      Enum.reduce(steps, initial_state, fn reducer, current_state ->
        State.update(current_state, reducer, options)
      end)

    # Run each hook with the final state
    Enum.each(hooks, fn {module, function} ->
      apply(module, function, [final_state, options])
    end)

    # Transform the state value into an ok/error tuple
    case final_state do
      %State{valid?: true, value: value} ->
        {:ok, value}

      %State{errors: errors} ->
        {:error, errors}
    end
  end
end
```

[Check out the complete version of this module here](https://github.com/msramos/ex_pipeline/blob/f657556df3c48ed6fa33c9be892d8844a58cd654/lib/pipeline.ex).

**Note**: In the complete version of this module, there is also another kind of hook, called an _async hook_. Async hooks work just like the hooks presented here, but are executed in parallel rather than sequentially. They will not be covered in this post, since they are essentially a hook [executed with a Task](https://hexdocs.pm/elixir/1.13/Task.html).

### Building and Executing a Pipeline

By simply calling `use Pipeline` and writing function names with the `_step` or `_hook` suffix, we are now able to build our pipelines. Let's rewrite our last example from the first part of the post with this new mechanic:

```elixir
defmodule Checkout do
  use Pipeline

  def fetch_payment_information_step(value, options) do
    # ...
  end

  def fetch_user_step(value, options) do
    # ...
  end

  def fetch_address_step(value, options) do
    # ...
  end

  def create_order_step(value, options) do
    # ...
  end

  def report_checkout_attempt_hook(state, options) do
    # ...
  end
end
```

We can execute this new pipeline like this:

```elixir
case Checkout.execute(%{params: params}, options) do
  {:ok, %{order: order}} ->
    conn
    |> put_flash(:info, "Order completed!")
    |> redirect(to: Routes.order_path(conn, order))

  {:error, error}
    conn
    |> put_flash(:error, parse_error(error_description))
    |> render("checkout.html")
end
```

No more `with` blocks, and no need to manually handle errors. The controller is now focused on handling the HTTP layer, and the checkout feature has an official place to live. Adding a new step to the checkout process just requires that we write a function in the right place!

## Wrap Up

In this post, we learned how to express a set of transformations as a reducer. Then, using the power of macros, we automatically created pipelines by detecting the signature of functions at compile time.

The features that use pipeline mechanics are explicit and contained — we know exactly what to expect and where to go when modifying them. We expect all steps to look and behave in the same way. The basics of error handling are already in place, you just need to decide what to do. These pipelines are, indeed, _predictable_.

I hope you've found this [two-part series about keeping your Elixir code maintainable](https://blog.appsignal.com/category/maintainable-code) helpful. Until next time, happy coding!

**P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, [subscribe to our Elixir Alchemy newsletter and never miss a single post](/elixir-alchemy)!**