Under the Hood of Macros in Elixir
Tags: elixir
Published: 2021-10-12 11:52:21

Welcome back to part two of this [series on metaprogramming in Elixir](https://blog.appsignal.com/category/under-the-hood-of-metaprogramming.html). In [part one](https://blog.appsignal.com/2021/09/07/an-introduction-to-metaprogramming-in-elixir.html), we introduced metaprogramming and gave a brief overview of macros.

In this part, we will explore the inner workings and behaviors of macros in more depth.

As discussed in the previous post, macros are compile-time constructs in Elixir. So, before diving into how macros work, it is important to understand where macros lie within Elixir's compilation process.

## Stages of Elixir's Compilation Process

We can boil down the Elixir compilation process to the following basic stages:

![Compilation process of an Elixir program](https://blog.appsignal.com/images/blog/2021-10/compilation_process.png)

(Note: the actual compilation process of an Elixir program is more intricate than above.)

The compilation process can be broken down into the following phases:

1. **Parsing** — The Elixir source code (program) is parsed into an AST, which we will call the **initial AST**.
2. **Expansion** — The initial AST is scanned and macro calls are identified. Macros are executed. Their output (AST) is injected and expanded into the callsite. Expansion occurs recursively, and a **final AST** is generated.
3. **Bytecode generation phase** — After the final AST is generated, the compiler performs an additional set of operations that eventually generate and execute BEAM VM bytecode.

As you can see, macros sit at the expansion phase, right before code is converted into bytecode. Therefore, a good knowledge of the expansion phase helps us understand how macros work.

## Expansion Phase

Let's start by first examining the expansion phase on a general level.

The compiler will expand macros (as per `Macro.expand`) to become part of the program's pre-generated AST. Macro expansion occurs recursively, meaning that Elixir will continue expanding a macro until it reaches its most fundamental AST form.

As macros expand right before bytecode is generated, they can modify a program's behavior during compile-time.

If we dig a little deeper, we will find that the compiler first injects the output AST of a macro at its callsite. Then the AST is expanded recursively.

We can observe this behavior as follows:

```elixir
defmodule Foo do
  defmacro foo do
    quote do
      IO.inspect("hello")
    end
  end
end

iex(1)> require Foo
iex(2)> ast = quote do: Foo.foo
{{:., [], [{:__aliases__, [alias: false], [:Foo]}, :foo]}, [no_parens: true],
 []}
iex(3)> ast |> Macro.expand(__ENV__)
{{:., [],
  [
    {:__aliases__, [counter: -576460752303423391, alias: false], [:IO]},
    :inspect
  ]}, [], ["hello"]}
```

`ast` represents the initial AST generated by the compiler before expansion. It holds a reference to the macro `Foo.foo` but it is not expanded as the macro has not been evaluated yet.

When we call `Macro.expand` on the given AST, the compiler begins by injecting the behavior of the macro into the callsite. We can expand the AST one step at a time using [`Macro.expand_once`](https://hexdocs.pm/elixir/1.12/Macro.html#expand_once/2).

## Contexts in Macros

Now that we understand the basics of the expansion phase, we can investigate the parts of a macro.

Macros contain two contexts — a macro context and a caller context:

```elixir
defmacro foo do
  # This is the macro's context, this is executed when the macro is called

  # This is the return value of the macro (AST)
  quote do
    # This is the caller's context, this is executed when the callsite is called
  end
end
```

As you can see, the macro's context is any expression declared before the `quote`.

The caller's context is the behavior declared in the `quote`. The `quote` generated AST is the macro's output and is injected into and expanded at the callsite.

The behavior defined under the caller's context 'belongs' to the caller, not the module where the macro is defined. The following example, taken from [Metaprogramming Elixir](https://pragprog.com/titles/cmelixir/metaprogramming-elixir/), illustrates this:

```elixir
defmodule Mod do
  defmacro definfo do
    IO.puts "definfo :: Macro's context #{__MODULE__}"

    quote do
      IO.puts "definfo :: Caller's context #{__MODULE__}"

      def friendly_info do
        IO.puts "friend_info :: Module name #{__MODULE__}"
      end
    end
  end
end

defmodule MyModule do
  require Mod
  Mod.definfo
end

iex(1)> c "context.exs"
definfo :: Macro's context Elixir.Mod
definfo :: Caller's context Elixir.MyModule
[Mod, MyModule]
iex(2)> MyModule.friendly_info
friend_info :: Module name Elixir.MyModule
:ok
```

As you can see, the module that executes the macro's context is `Mod`. But the module that executes the caller's context is `MyModule` — the callsite where the macro is injected and expanded.

Similarly, when we declare `friendly_info`, we inject this function into the callsite of the macro, which is `MyModule`. So the function now 'belongs' to `MyModule`.

But why does there need to be two different contexts? What exactly makes the macro context and caller context different from one another?

## Order of Evaluation in Macro and Caller Contexts

The key difference between the macro context and the caller context is that behavior is evaluated at different times.

Let's look at an example:

```elixir
defmodule Foo do
  defmacro foo do
    IO.puts("macro")
    quote do
      IO.puts("caller")
    end
  end
end

iex(1)> require Foo
iex(2)> ast = quote do: Foo.foo
{{:., [], [{:__aliases__, [alias: false], [:Foo]}, :foo]}, [no_parens: true],
 []}

iex(3)> ast |> Macro.expand(__ENV__)
macro
{{:., [],
  [{:__aliases__, [counter: -576460752303423293, alias: false], [:IO]}, :puts]},
 [], ["caller"]}
```

When we expand the AST of a macro call, the macro context is evaluated and the caller context is injected and expanded into the callsite.

However, we can go a level deeper.

Let's look again at the first component of the expansion phase: *Macros are executed. Their output (AST) is injected and expanded into the callsite*.

For a compiler to know what AST needs to be injected into the callsite, it has to retrieve the output of the macro during compilation (when the expansion phase occurs). The macro call is parsed as an AST during the parsing phase. The compiler identifies and executes these macro call ASTs prior to the expansion phase.

If we think of macros as regular functions, the macro context is the function body and the caller context is the result of the function. During compilation, a macro is executed and evaluates the macro context. Then the `quote` is evaluated, returning the results of the function. The caller context is injected and expanded into the callsite of the macro.

The macro context is evaluated during compile-time and treated as a regular function body. It executes within and is 'owned' by its containing module.

The caller context is injected into the callsite, so it is 'owned' by the caller. It is evaluated whenever the callsite is evaluated.

The example above showcases what happens when a macro is invoked at the module level. But what if we attempt to invoke the macro in a function? When do the macro and caller contexts evaluate?

Well, we can use another example here:

```elixir
defmodule Foo do
  defmacro foo do
    IO.puts("macro")

    quote do
      IO.puts("caller")
    end
  end
end

defmodule Bar do
  require Foo

  def execute do
    IO.puts("execute")
    Foo.foo
  end
end

macro
iex(1)> Bar.execute
execute
caller
```

The caller context evaluates when the function is called (as we established earlier).

However, something interesting happens with our macro context — it evaluates when the module compiles. This evaluation only happens once when `Bar` compiles, as evidenced by the lack of "macro" in our output when we call `Bar.execute`. Why is this the case?

Well, we only need to evaluate the macro once to retrieve its output (caller context) and inject it into the callsite (which is a function in this case). The caller context behavior evaluates every time the function is called.

This difference in the order and time of evaluation helps guide us on when to use the macro and the caller contexts.

We use the macro context when we want the behavior to be evaluated during compile-time. This is regardless of when the caller context is evaluated or where the macro is called in the code.

We use the caller context when we want to invoke behavior injected into the callsite at evaluation.

Now that we have a better grasp of the Elixir compilation process, macros, and the order of evaluation, we can revisit `unquote`.

## Revisiting `unquote`

In part one of this series, we established that `unquote` evaluates a given expression and injects the result (as an
AST) into the AST built from `quote`. This is only a piece of the puzzle.

Let's dig deeper to understand the behavior of `unquote` during compilation and the necessity of using it.

While the rest of the `quote` body is evaluated at the same time as the callsite, `unquote` is evaluated (immediately) during compile-time — when the macro is evaluated. `unquote` aims to evaluate and inject the result of a given expression. This expression might contain information that is only available during the macro evaluation, including variables that are initialized during this process. `unquote` must be evaluated during compile-time along with the macro, so that the AST of the result injects into the `quote` that we build.

But why do we need to `unquote` the expression to inject it into the AST? To answer this, let's compare the expanded AST of a macro using `unquote` against one that does not:

```elixir
defmodule Foo do
  defmacro foo(x) do
    quote do
      IO.inspect(x)
    end
  end
end

defmodule Bar do
  defmacro bar(y) do
    quote do
      IO.inspect(unquote(y))
    end
  end
end

iex(1)> require Foo
iex(2)> require Bar
iex(3)> ast_foo = quote do: Foo.foo(1 + 2 * 3)
iex(4)> ast_bar = quote do: Bar.bar(1 + 2 * 3)
iex(5)> ast_foo |> Macro.expand(__ENV__)
{{:., [],
  [
    {:__aliases__, [counter: -576460752303423448, alias: false], [:IO]},
    :inspect
  ]}, [], [{:x, [counter: -576460752303423448], Foo}]}

iex(6)> ast_bar |> Macro.expand(__ENV__)
{{:., [],
  [
    {:__aliases__, [counter: -576460752303423384, alias: false], [:IO]},
    :inspect
  ]}, [],
 [
   {:+, [context: Elixir, import: Kernel],
    [1, {:*, [context: Elixir, import: Kernel], [2, 3]}]}
 ]}
```

Observe that the expanded `Foo.foo` AST is vastly different from the `Bar.bar` AST even though they are both given the same variable. This is because Elixir is quite literal with variable references. If a variable is referenced without `unquote`, an AST of that variable reference injects into the AST.

Using `unquote` ensures that the underlying AST of the variable's value injects into the `quote` body.

Now you may ask: What is the difference in variable scoping between the evaluation of macros and the execution of the callsite? Why does it matter?

The scoping of variables in macros can be a confusing subject, so let's demystify it.

## Variable Scoping in Macros

Now that we understand how macros are evaluated and expanded, we can look at the scoping of variables in macros, and when to use the options `unquote` and `bind_quoted` in `quote`.

Due to [function clause scoping](https://elixir-lang.readthedocs.io/en/latest/technical/scoping.html#function-clause-scope), the arguments of a variable are initialized and 'in scope' during the macro evaluation of a function. 

Similarly, variables declared and assigned within a function body remain in scope until the function ceases. The same behavior applies to macros.

When the macro context is evaluated, its arguments and any initialized variables are 'in scope.' This is why `unquote` can evaluate variable references declared as arguments of the macro or any variables initialized in the macro context.

Any evaluation of variable initialization in the caller context will initialize these variables within the callsite during execution.

To understand this difference better, let's look at a few examples:

```elixir
defmacro foo do
  quote do
    x = 1 + 1
    def bar, do: IO.inspect(unquote(x))
  end
end
```

In this first example, `unquote` will not work. The variable `x` has not yet been initialized, but should have been initialized during the execution of the callsite. The immediate evaluation of `unquote` runs too early, so we cannot reference our variable `x` when we need to. When `unquote` evaluates during compile-time, it attempts to evaluate the variable reference expression of `x` and finds that it is not in scope.

How can we fix this? By disabling unquoting. This means disabling the immediate evaluation of `unquote`. We only want `unquote` to evaluate when our caller context evaluates. This ensures that `unquote` can properly reference a variable in scope (`x`) as variable initialization would have occurred
during the evaluation of the callsite.

```elixir
defmacro foo do
  quote unquote: false do
    x = 1 + 1
    def bar, do: IO.inspect(unquote(x))
  end
end
```

This example highlights the impact of scoping in macros. If we attempt to access a variable that is available during the evaluation of the macro context, `unquote` as-is is perfect for us.

However, suppose we try to access a variable that is only available during the evaluation of the callsite. In that case, we must disable the immediate unquoting behavior to initialize variables in scope before `unquote` attempts to reference them.

Let's apply this understanding to two other examples.

```elixir
defmacro foo(opts) do
  quote bind_quoted: [opts: opts] do
    x = Keyword.get(opts, :x)
    def bar, do: IO.inspect(unquote(x))
  end
end
```

In this example, we have initialized the variable `x` from a keyword list. As the keyword list is initialized during compile-time (along with the evaluation of the macro context), we first have to bind it to the caller context to:

1. Generate an initialization of the variable during the evaluation of the callsite, and
2. Disable unquoting behavior.

We have to bind `opts` to the caller context, as the variable is no longer in scope during the evaluation of the callsite.

Finally, we have:

```elixir
defmacro foo(x) do
  quote do
    def bar, do: IO.inspect(unquote(x))
  end
end
```

In this last example, `x` remains a variable in scope during the evaluation of the macro context — i.e. when the macro is called. The immediate evaluation of `unquote` works in our favor. It renders `unquote(x)` valid, as `x` is in scope when `unquote` is evaluated.

## Macro Hygiene in Elixir

While we are on the topic of scopes in macros, let's discuss macro hygiene.

According to [tutorialspoint.dev](https://tutorialspoint.dev/language/c/hygienic-macros-introduction):

> Hygienic macros are macros whose expansion is guaranteed not to cause the accidental capture of identifiers.

This means that if we inject and expand a macro into the callsite, we need not worry about the macro's variables (defined in the caller context) conflicting with the caller's variables.

Elixir ensures this by maintaining a distinction between a caller variable and macro variable. You can explore this further using an example from the [official tutorial] (https://elixir-lang.org/getting-started/meta/macros.html#macro-hygiene).

A macro variable is declared within the body of `quote`, while a caller variable is declared within the callsite of the macro.

```elixir
defmodule Foo do
  defmacro change do
    quote do: a = 13
  end
end

defmodule Bar do
  require Foo

  def go do
    a = 1
    Foo.change
    a
  end
end

Bar.go
# => 1
```

In this example, `a` referenced in `change` is not the same variable `a` in the scope of `go`. Even when we attempt to change the value of `a`, the value of `a` in the scope of `go` remains untouched.

However, there may be a time where you have to reference a variable from the caller's scope in the macro's scope.
Elixir provides the macro `var!` to bridge the gap between these two scopes:

```elixir
defmodule Foo do
  defmacro change do
    quote do: var!(a) = 13
  end
end

defmodule Bar do
  require Foo

  def go do
    a = 1
    Foo.change
    a
  end
end

Bar.go
# => 13
```

This distinction ensures no unintended changes to a variable due to changes within a macro (whose source code we may not have access to).

You can apply the same hygiene to [aliases and imports](https://hexdocs.pm/elixir/1.6.1/Kernel.html#alias!/1).

## Understanding `require`

In the code examples shown in this section, we have always used `require <module>` before we invoke the macros within the module. Why is that?

This is the perfect segue into how the compiler resolves modules containing macros — particularly the order in which modules are compiled.

In Elixir, modules compile [in parallel](https://elixir-lang.org/blog/2012/04/24/a-peek-inside-elixir-s-parallel-compiler/), and usually — for regular modules — the compiler is smart enough to compile dependencies of functions in the proper order of use. The parallel compilation process pauses the compilation of a file until the dependency is resolved. This behavior is replicated when handling modules that contain macro invocations.

However, as macros must be available during compile-time, the module these macros belong to must be compiled beforehand.
Here's where `require` comes into the picture. `require` explicitly informs the compiler to [compile and load the module containing the macro first] (https://www.theerlangelist.com/article/macros_2).

We can use an example to illustrate this behavior:

```elixir
defmodule Foo do
  IO.puts("Compiling Foo")

  defmacro foo do
    IO.puts("Foo.foo := macro")
    quote do
      IO.puts("Foo.foo := caller")
    end
  end
end

defmodule Bar do
  IO.puts("Compiling Bar")

  require Foo

  IO.puts("Bar := before Foo.foo")
  Foo.foo()
end

Compiling Foo
Foo.foo := macro
Compiling Bar
Bar := before Foo.foo
Foo.foo := caller
```

(Note that this is just an approximation of the actual compilation process, but it aims to paint a clearer picture of how `require` works.)

Let's try to understand why the outputs are in this order.

Firstly, `Bar` tries to compile. The compiler scans and finds a `require` for `Foo` before evaluating any module-level expressions within the module (such as `IO.puts`). So it pauses the compilation of `Bar` and compiles the `Foo` module first. As `Foo` is compiled, module-level code — like `IO.puts` — is evaluated, and the compiler prints the first line of the output.

Once `Foo` is compiled, the compiler returns to `Bar` to resume compilation. `Bar` is parsed, macro calls are executed, and the macro context is evaluated. Even though `Foo.foo` is called after `IO.puts("Bar := before Foo.foo")`, the evaluation of the macro call takes precedence over the evaluation of module-level code.

During expansion, `Foo.foo`'s caller context is injected and expanded into the callsite in `Bar`. It then behaves just like a regular module-level function call, printing the last three output lines in order of declaration.

In essence, `require` instructs the compiler on the order of compilation that each module should go through if there are macro dependencies. This ensures that the macros are available during compile-time.

## Escaping Macros in Elixir

Before explaining what `Macro.escape` does, let's look at an example:

```elixir
iex(1)> x = {1, 2, 3}
iex(2)> quote do: unquote(x)
{1, 2, 3}
iex(3)> ast = quote do: IO.inspect(unquote(x))
{{:., [], [{:__aliases__, [alias: false], [:IO]}, :inspect]}, [], [{1, 2, 3}]}
iex(4)> Code.eval_quoted(ast)
** (CompileError) nofile: invalid quoted expression: {1, 2, 3}

Please make sure your quoted expressions are made of valid AST nodes. If you would like to introduce a value into the
AST, such as a four-element tuple or a map, make sure to call Macro.escape/1 before
    (stdlib 3.15) lists.erl:1358: :lists.mapfoldl/3
    (elixir 1.11.3) lib/code.ex:706: Code.eval_quoted/3
```

That is a strange error. Based on our understanding of `unquote` and macros, the code should work as intended, but it doesn't. Why is that?

Well, the answer is found on `iex(2)`. When we attempt to unquote `x`, we return not an AST, but a tuple — the same one initially assigned to `x`. The error then points to the fact that the tuple is an invalid quoted expression.

When we `unquote(x)` as-is, we inject a raw tuple into the AST, which cannot be evaluated as it is not a valid AST and throws an error.

So, how do we fix it?

We need to convert the raw tuple referenced by `x` into a valid AST. This can be achieved by escaping this value using `Macro.escape`. Let's understand what `Macro.escape` does:

```elixir
iex(1)> a = {1, 2, 3}
{1, 2, 3}
iex(2)> Macro.escape(a)
{:{}, [], [1, 2, 3]}
iex(3)> quote do: unquote(a)
{1, 2, 3}
iex(4)> quote do: unquote(Macro.escape(a))
{:{}, [], [1, 2, 3]}
```

In `iex(2)`, we see that `Macro.escape(a)` returns an AST of the tuple, not the raw tuple — and this is exactly what we are looking for. By combining `Macro.escape`'s behavior with `unquote`, we can inject the AST of the tuple into the `quote` as seen in `iex(4)`.

Let's test this:

```elixir
iex(1)> x = {1, 2, 3}
{1, 2, 3}
iex(2)> quote do: unquote(Macro.escape(x))
{:{}, [], [1, 2, 3]}
iex(3)> ast = quote do: IO.inspect(unquote(Macro.escape(x)))
{{:., [], [{:__aliases__, [alias: false], [:IO]}, :inspect]}, [],
 [{:{}, [], [1, 2, 3]}]}
iex(4)> Code.eval_quoted(ast)
{1, 2, 3}
{{1, 2, 3}, []}
```

As you can see, the code works just as intended because we escape the tuple.

Often when working with data structures like tuples and dictionaries, you may find that the injected data from `unquote` does not inject a valid AST. In these cases, you should use `Macro.escape` before `unquote`.

## Guard Clauses and Pattern Matching

Finally, it's worth mentioning that, much like regular functions defined through `def`, macros can use [guard clauses](https://inquisitivedeveloper.com/lwm-elixir-24/) and [pattern matching](https://elixirschool.com/en/lessons/basics/pattern-matching/):

```elixir
defmacro foo(x) when x == 4 do
  IO.inspect("macro with x being 4")
end

defmacro foo(_) do
  IO.inspect("macro with any other value")
end
```

## Up Next: Applications of Macros in Elixir

Congratulations, you've made it to the end of this part! You should now have a better grasp of how macros work internally.

In part three, we will look at the many applications of macros in Elixir.

Happy coding!

**P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, [subscribe to our Elixir Alchemy newsletter and never miss a single post!](https://blog.appsignal.com/category/elixir-alchemy.html#elixir-alchemy)**

*Jia Hao Woo is a developer from the little red dot — Singapore! He loves to tinker with various technologies and has been using Elixir and Go for about a year. Follow his programming journey at [his blog](https://woojiahao.github.io/blog) and on [Twitter](https://twitter.com/woojiahao_).*