A "Hello, World!" of Lisp Macros
Tags: lisp, clojure, computerscience
Published: 2020-10-30 15:07:04

# "Hello, macros!"
## TL;DR
**Common Lisp:**
{% gist https://gist.github.com/boolpath/40bd4f263aca797640e64e51bfbc1065 file=hello-macros-tldr-defmacro-pass-args.lisp %}
**Clojure:**
{% gist https://gist.github.com/boolpath/40bd4f263aca797640e64e51bfbc1065 file=hello-macros-tldr-defmacro-pass-args.clj %}

## Context
As a Lisp beginner, I think the most succinct program capable of portraying the beauty of Lisp is `(+)`. I first saw this expression in Paul Graham's ANSI Common Lisp (page 8) [1], and despite its apparent simplicity I found it shockingly powerful. In this post, I use the `(+)` expression to explore the notion of "code as data" in Common Lisp, and try to figure out what the "Hello, World!" of Lisp macros is, a challenge proposed by the book's author in [this tweet](https://twitter.com/paulg/status/1260138502974570497).
{% twitter 1260138502974570497 %}

## Content
### The (quote ...) operator
The key to understanding Lisp macros lies in the concept of **protecting expressions from evaluation**. Lisp programs are expressed as lists, and programs are executed by evaluating their list representations. Protecting lists from evaluation means doing something to avoid their interpretation as programs (i.e. an operator followed by its arguments), or simply to *prevent their execution* when they actually are programs. 

The `quote` operator (abbreviated as `'`) protects its arguments from evaluation. Clearly this is useful for building lists of data without triggering the *evaluation rule* of function calls, but why would we want to prevent actual programs from being evaluated as such? In Lisp, the answer is straightforward: so that we can **manipulate programs as data** and transform them into other programs. This is where macros come into play. But first, let's go back to `(+)`.

### The (+) program
The `(+ 1 2 3)` expression is a list representation of a Lisp program that adds the numbers 1 to 3 together. It uses _prefix notation_ to express that the arguments `1`, `2` and `3` should be passed to the `+` function.
``` lisp
> (+ 1 2 3)
6
```
This notation makes the `+` operator appear only once, whereas _infix notation_ would make it appear twice (1 + 2 + 3). The elegance of this approach is that it holds for an arbitrary number of arguments (e.g. `(+ 1 2 3 4 5 ...)`), but what I find most surprising is that it holds for *zero arguments* as well. We can thus write a program that adds *zero numbers* together:
``` lisp
> (+)
0
```

This is so beautiful, and even more so that it also holds for `(*)`, `(and)` and `(or)`. Admittedly, a program that adds zero numbers together is not useful at all, unless we can somehow *modify it* to take more arguments **before it is evaluated**. One way of doing this modification is by *appending* expressions to the list representation of the program. Let's try to use the built-in `append` function to concatenate the `(+)` program with the list `(1 2 3)`:
``` lisp
> (append (+) (list 1 2 3))
           |
  (append  0  (list 1 2 3))
...TYPE-ERROR...
The value 0 is not of type LIST
```
Unfortunately, trying to evaluate this expression causes an error, because **arguments passed to function calls are always evaluated**. Thus the `(+)` program will evaluate to `0` before we try to append  `(1 2 3)`, and the `append` function only takes lists as arguments. What we really need  is a way to concatenate arguments to the `(+)` list.

### The '(+) list
Let's try to protect the `(+)` program from evaluation before passing it to the `append` function. We can do so with the `quote` operator (here in abbreviated form):
``` lisp
> '(+)
(+)
```
The result of preventing the evaluation of the `(+)` program is the `(+)` list. Even though _they look exactly the same_, the former is **code**, while the latter is **data**. This is the distinctive notion of "code as data" in Lisp. We can now use the `append` function to concatenate the lists `(+)` and `(1 2 3)`:
``` lisp
> (append '(+) '(1 2 3))
(+ 1 2 3)
```
Notice that both arguments are written with the `'` operator, so as to prevent the evaluation of the `(+)` expression as a program (which would return `0`), as well as the evaluation of the `(1 2 3)` expression as a function call (which would cause an error because `1` is not a function). However, the resulting expression `(+ 1 2 3)` **is data, not code**, so we still need to find a way to make Common Lisp treat it as code. Would defining a function solve this problem?
``` lisp
(defun pass-args (code &rest args)
  (append code args))

> (pass-args '(+) 1 2 3)
(+ 1 2 3)
```

Unfortunately not. This `pass-args` function evaluates the expression `(append code args)`, where the `code` parameter must be a list representation of a program, and `&rest args` will build a list of zero or more arguments to be passed to `code`. But whether we call the `append` function directly or define a function that calls `append` internally, the final expression still needs to be evaluated as code. What else could we try then? Enter `(eval ...)`:
``` lisp
> (eval (append '(+) '(1 2 3)))
6
> (eval (pass-args '(+) 1 2 3))
6
```
These expressions finally produce the desired result. However, calling `eval` is not the best way to _cross the line between data and code_ [1] \(page 161) mainly due to efficiency reasons (run-time vs compile-time). As we shall see next, macros are in fact the best way to manipulate code as data and evaluate data as code.

### Hello, (defmacro ...)!
Macros are programs that write programs [2]. They do so by handling programs as data and evaluating their transformed representations as code. This is possible because, unlike functions, **macros do not evaluate their arguments** when they are passed. Thus, in order to modify the `(+)` program, we can redefine the `pass-args` function as a macro and call it with an _unquoted_ expression of the `(+)` program:
``` lisp
(defmacro pass-args (code &rest args)
  (append code args))
  
> (pass-args (+) 1 2 3)
6
```
We finally get `6` instead of just `(+ 1 2 3)`. Notice that, in contrast to the `pass-args` function call, there is no need to use the quote `'` operator before the `(+)` program because **macros operate on the unevaluated expressions for the arguments** [3]. But once inside the body of a macro, we do need a way to specify that we want to treat some data as code again before returning the value of the transformed expression.

The backquote `` ` `` operator **turns evaluation off** like the regular quote `'` operator, but we can use `,` and `,@` within a backquoted expression to **turn evaluation back on** [1] \(page 163). The `,@` is especially useful because it inserts the elements of a list into a template, so given a term `,@args`, if `args` is the list `(1 2 3)`, then its elements `1 2 3` will be inserted *without the enclosing parentheses*. The same would apply to a term `,@code` where `code` is `(+)`, only the `+` operator would be inserted. This allows us to rewrite the `pass-args` macro without the `append` function:

``` lisp
(defmacro pass-args (code &rest args)
  `(,@code ,@args))

> (pass-args (+) 1 2 3)
6
```
Here too we get the expected result, but something different happened under the hood: a process known as *macro expansion*, which turns the template `(,@code ,@args)` into actual code by inserting the required argument expressions. We can visualize the result of this process by applying the `macroexpand` function to a quoted expression containing the macro call that we want to expand:
``` lisp
> (macroexpand '(pass-args (+) 1 2 3))
(+ 1 2 3)
T

> (macroexpand '(pass-args (+ 1) 2 3))
(+ 1 2 3)
T
```
This is extremely useful when designing macros, since it allows us to see what code will be generated when passing specific arguments to a macro call. Notice that both expansions generate the same expression `(+ 1 2 3)`, even though the former passes the `1 2 3` arguments to the `(+)` program, while the latter passes the `2 3` arguments to the `(+ 1)` program. Therefore, both macro calls will ultimately evaluate to `6` as expected.

### Hello, World!

Now let's use the `pass-args` macro and the `format` function to finally write a "Hello, World!" of Lisp macros:
``` lisp
> (macroexpand '(pass-args (format) t "Hello, World!"))
(FORMAT T "Hello, World!")
T

> (pass-args (format) t "Hello, World!")
Hello, World!
NIL
```
This macro call passes the arguments `t` and `"Hello, World!"` to the `(format)` program, which expands to `(format t "Hello, World!")`. Then it prints the string `Hello, World!` and finally returns `NIL`. Notice that the `(format)` program is not a valid Common Lisp expression because `format` is a two-argument function. Thus, unlike the `(+)` program that could be executed with no arguments, trying to evaluate `(format)` causes an error:
``` lisp
> (format)
...ERROR...
invalid number of arguments: 0
```
This means that the `pass-args` macro is also useful for **modifying programs that would otherwise fail to evaluate**. Trying to evaluate the `(format t)` expression would fail too, but if we pass it to the macro along with the `"Hello, World"` argument, it will expand to the same valid expression as the `(format)` program with the `t` and `"Hello , World"` arguments, and thus produce the same results:

``` lisp
> (macroexpand '(pass-args (format t) "Hello, World!"))
(FORMAT T "Hello, World!")
T

> (pass-args (format t) "Hello, World!")
Hello, World!
NIL
```
And that's it! A "Hello, World!" of Lisp macros. 

### Clojure
In Clojure, this macro looks almost identical to Common Lisp, we just have to replace `()` for `[]` in the list of arguments, `&rest` for `&` to collect the optional parameters, and `,@` for `~@` to splice them into the template:
``` clojure
(defmacro pass-args [code & args]
  `(~@code ~@args))
  
> (macroexpand '(pass-args (+) 1 2 3))
(+ 1 2 3)

> (pass-args (+) 1 2 3)
6
```

### Goodbye, reader!

Of course there is much more to macros than just appending arguments to a list representation of a program, but when I started learning Lisp I found this thought process helpful to understand this feature of the language. Hopefully this will help other learners see the power that comes from the ability to pass code as arguments, transform it as data and finally treat it as code again. This is what makes Lisp the ultimate _programmable programming language_.

## References
1. [ANSI Common Lisp](https://www.amazon.com/-/es/gp/product/0133708756).
2. [Beating the Averages](http://www.paulgraham.com/avg.html).
3. [GNU Emacs Manual](https://www.gnu.org/software/emacs/manual/html_node/elisp/Macros.html)