On Elixir Metaprogramming
Tags: elixir, softwareengineering
Published: 2020-11-24 11:30:05

---
title: On Elixir Metaprogramming
published: true
date: 2020-11-23 17:00:00 UTC
tags: Elixir,SoftwareEngineering
canonical_url: https://www.christianblavier.com/on-elixir-metaprogramming/
---

_Metaprogramming_ is a scary word: it sounds like voodoo for programmers, and in some manners it is. But it is used by most popular Elixir libraries (Ecto, Phoenix and Elixir itself) and as I leveled up as an Elixir programmer, I needed to understand how it worked under the hood.

I personally felt intimidated by the topic, and I owned the [_Metaprogramming Elixir_ book](https://pragprog.com/titles/cmelixir/metaprogramming-elixir/) for more than a year before actually reading it 😱

![On Elixir Metaprogramming](https://www.christianblavier.com/content/images/2020/11/tempImageemj1Me.gif)<figcaption>My Elixir personal library </figcaption>

But for any seasoned Elixir programmer, it is an excellent handbook and very pleasant to read. It is also pragmatic and I concur 100% with Chris McCord’s introductory advice:

> Rule 1 : Don't Write Macros - Chris McCord

Be cautious with _metaprogramming_ as it is hardly maintainable and a pain to debug; don't even think about writing your application core features with it. It is more relevant for technical libraries, custom DSL and any other specific subjects.

## It's a kind of magic

As many of you, I came from a Ruby on Rails development background and I remember that while learning Rails, I was so impressed by the language expressivity and its magic features. Coming then from a Java background, it was a blessing to me!

```
irb> 2.years.from_now + 5.days.from_now 
#=> Mon, 28 Nov 2022 13:38:36 UTC

irb> Person.find_by_user_name_and_password(user_name, password)
#=> #<Person id: 10, user_name: "Ryan">
```
<figcaption>Ruby magic in action</figcaption>

Rails made indeed an intensive usage of Ruby dynamic features to extend the language at runtime: the `find_by_user_name_and_password` call  just above is an example of dynamically generated class method based on the `Person` model attributes.

![On Elixir Metaprogramming](https://www.christianblavier.com/content/images/2020/11/90c9a1015850dff7dece6249e4a25eef.jpg)<figcaption>How I felt when learning Ruby on Rails</figcaption>

But there are caveats to Ruby magic:

- [too much magic](https://stackoverflow.com/questions/441717/whats-wrong-with-magic) may hurt as you lack the understanding of what really happens under the hood. Same logic also applies to Elixir _metaprogramming_ (remember Chris McCord's motto? "_Don't Write Macros"_ 😅)
- Ruby magic is **dynamic** by nature which means it is performed at runtime, at the expense of performance. Whereas **Elixir macros are evaluated at compile time** and inserted in your bytecode as if it were manually written code.

## What is metaprogramming?

Before going deeper in the subject, it’s worth reminding that _metaprogramming_ is the art of writing code that will generate code. _Metaprogramming_ features are first class members of the Elixir language which provides macro API to rapidly extend the core language.

Did you know that _if-then-else_ or _unless_ block syntax are written in Elixir through its macro mechanism? It provides syntaxic sugar over a binary case pattern matching structure.

```
defmacro macro_unless(clause, do: expression) do
  quote do
    if(!unquote(clause), do: unquote(expression))
  end
end
```
<figcaption>unless block written with an Elixir macro</figcaption>

You could also be using meta-programmed features everyday without knowing it:

- [ExUnit](https://github.com/elixir-lang/elixir/tree/master/lib/ex_unit) test definitions leverages on macros
- [Ecto querying DSL](https://github.com/elixir-ecto/ecto/blob/master/lib/ecto/query.ex) can perfectly mimic SQL language thanks to _metaprogramming_
- Of course Chris McCord’s Phoenix framework uses macro a lot, like the [Routing DSL](https://hexdocs.pm/phoenix/routing.html)

**By the way, did you know that more than 90% of the Elixir sourcecode is written in Elixir? Meta enough for you? 🤯**

## Elixir Macros 101

Your first Elixir macros will be a bit tedious to write. I won’t give a full course about Elixir _metaprogramming_ (buy [Chris McCord’s book](https://pragprog.com/titles/cmelixir/metaprogramming-elixir/)!) but I will instead give you a few keys to understand it.

![On Elixir Metaprogramming](https://www.christianblavier.com/content/images/2020/11/CleanShot-2020-11-21-at-08.29.12@2x.png)<figcaption>Can't blame <a href="https://twitter.com/matt_chabert">Matthieu</a>, happened to me as well!</figcaption>

The main thing you need to understand is that all code you write in Elixir is represented in memory as an _Abstract Syntax Tree_ (AST). The compiler will parse your code and turn it into an AST, then the BEAM will evaluate the AST in order to run it.

![On Elixir Metaprogramming](https://www.christianblavier.com/content/images/2020/11/Untitled-2020-11-20-0920.png)<figcaption>Abstract Syntax Tree (AST)</figcaption>

_Metaprogramming_, which we previously defined as « code to generate code » could be achieved by generating raw Elixir code (as strings) but it would be unefficient and hardly maintainable.

That’s why Elixir provides API’s to generate and evaluate AST. Here comes `quote`: an Elixir primitive which can convert any Elixir code into its AST.

```
iex(1)> quote do 5 + 8 * 3 end
#=> {:+, [context: Elixir, import: Kernel],
#=> [5, {:*, 
#=> [context: Elixir, import: Kernel], 
#=> [8, 3]
#=> ]}]
#=> }
```

This code can be evaluated the same way you would have evaluated code in a string (think `eval` in [Javascript](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/eval))

```
iex(1)> Code.eval_string("5 + 8 * 3")
#=> {29, []}
iex(2)> Code.eval_quoted(quote do 5 + 8 * 3 end)
#=> {29, []}
```

Now what happens if you try to evaluate code with dynamic parts?

```
iex(1)> a = 5
iex(2)> Code.eval_string("3 * a")
#=> ** (CompileError) nofile:1: undefined function a/0
```

Of course `a` is unknown to your evaluated code and a simple way to fix this would be to use text interpolation.

```
iex(1)> a = 5
iex(2)> Code.eval_string("3 * #{a}")
#=> {15, []}
```

Well the good news is you can interpolate within your quoted code as well: use the `unquote` primitive.

```
iex(1)> a = 5
iex(2)> Code.eval_quoted(quote do 3 * unquote(a) end)
#=> {15, []}
```

Simple as that! So whenever you think you are lost between your `quote` / `unquote` blocks, just look at it as if it were text interpolation.

Then you need to know about `defmacro` which you probably already encountered in your Phoenix project. It's a special construct able to produce code, from quoted expressions and parameters.

Here is an example that generates a few functions :

```
defmodule MyMacro do
  defmacro generate_functions(name_and_values) do
    for {name, value} <- name_and_values do
      quote do
        def unquote(name)(), do: unquote(value)
      end
    end
  end
end

defmodule MyModule do
  require MyMacro
  
  MyMacro.generate_functions([{:one, 1}, {:two, 2}, {:three, 3}])
end

iex(1)> MyModule.two + MyModule.three 
#=> 5
```

There are quite a few other things to know when it comes to Elixir _metaprogramming_, but `quote`, `unquote` and `defmacro` are the basics and can already help you to build powerful stuff.

## Ok, but what for?

I will end this post with some examples of the way we use _metaprogramming_ in our Elixir projects.

### Extends Ecto query API

Ecto provides a fully featured querying API that mimics SQL. It relies on Elixir Macros and will be able to trigger compilation errors whenever your query is malformed.

```
iex(1)> query = from u in User, where: u.age > 0, select: u.name
iex(2)> Repo.all(query)
```
<figcaption>Typical Ecto query</figcaption>

But every now and then you'll want to use some specific database functions not available in Ecto API. You can then use `fragment` Ecto function which let you insert raw SQL in your Ecto Query:

```
query = 
  from u in User,
  where: is_nil(fragment("?->>?", u.metadata, "phone"))
```
<figcaption>Using fragment to access a field within a Postgres JSONB column</figcaption>

But the proper way is to leverage on macros to extend the Ecto query API:

```
defmodule MyQueryMacros do
  defmacro jsonb_get(column, key) do
    quote do
      fragment("?->>?", unquote(column), unquote(key))
    end
  end
 end
 
 query = 
   from u in User,
   where: is_nil(jsonb_get(u.metadata, "phone"))
```
<figcaption>Same Ecto query using our homemade jsonb_get macro</figcaption>
### Aliases & Imports

As your Elixir codebase grows, you can see that you repeat over and over the same `alias` and `import` blocks at the head of your files. Not really [DRY](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself)...

Phoenix already offers a solution you can use through the `use` / ` __using__ ` macro. To declare a _controller_ or a _view,_ phoenix suggests a bunch of default aliases and imports declared in `my_app_web.ex` file. Here is an example of [changelog.com](https://changelog.com) `*_web.ex` file : [changelog.com/changelog_web.ex file]
(https://github.com/thechangelog/changelog.com/blob/master/lib/changelog_web.ex)

You can then use the macro with the following snippet, which will _alias_ and _import_ a few Phoenix modules and even generate some authorization related functions:

```
defmodule MyAppWeb.MyController do
  use MyAppWeb, :controller
  
  # ...
end
```

In our own project, we also brewed our own _aliasing_ system that we can use like this to prevent us from declaring the same aliases over and over.

```
defmodule MyApp.Contracts.SomeContractService do
  use MyApp,
    aliases: [:campaigns, :contracts, :users],
    private_aliases: [:contracts]
    
  # ...
end   
```

I may write further about these specific macros in another post 😉

### changix

A final example of how we use macros in our main Elixir application is our **application changelog**. We present new features, bug fixes and improvement to our users with a changelog sidebar (and a blinking badge to draw their attention).

![On Elixir Metaprogramming](https://www.christianblavier.com/content/images/2020/11/CleanShot-2020-11-23-at-17.16.53@2x-2.png)<figcaption>A changelog sidebar embedded in our webapp</figcaption>

Instead of feeding this changelog from database, we use _markdown_ files with a [YAML front matter](https://jekyllrb.com/docs/front-matter/) header.

![On Elixir Metaprogramming](https://www.christianblavier.com/content/images/2020/11/CleanShot-2020-11-23-at-17.29.19@2x-1.png)<figcaption>Our markdown changelog files, commited right along source code</figcaption>

At first, we used to parse these files at runtime everytime a user triggered the sidebar, then we cached the HTML output in memory in order to improve performance.

But after reading Chris McCord's book, we realized that these _markdown_ files could be considered as source code and be compiled as well (or at least processed at compile-time). Here comes [**changix**](https://github.com/cblavier/changix): a library relying on macros to provide compile-time changelog features.

[cblavier/changix on Github](https://github.com/cblavier/changix)

## Final words

I hope that this post will help you feel more confortable with Elixir _metaprogramming_ and give you enough courage to hack yours; but remember Chris's advice and always be cautious 😅