How To Write A Programming Language
Tags: programming, c, computerscience
Published: 2023-02-03 00:13:58

---
title: How To Write A Programming Language
published: true
description: Straight to the point article about writing your own programming language with a simple toy language as an example
tags: #programming #C #computerscience
cover_image: https://www.itprotoday.com/sites/itprotoday.com/files/styles/article_featured_retina/public/Screen%20Shot%202018-12-08%20at%204.45.18%20PM_0.png?itok=UvQJWfWh
# Use a ratio of 100:42 for best results.
# published_at: 2022-12-09 11:59 +0000
---

* [The Grammar](#TheGrammar)
* [Composing Grammar Rules](#ComposingGrammaticsRules)
* [The $$ Operator And Empty Grammar Rules](#TheOperatorAndEmptyGrammarRules)
* [Rule Types](#RuleTypes)
* [Parsing The Language](#ParsingTheLanguage)
* [AST](#AST)


Recently I wrote a very simple programming language, [here is the repo](https://github.com/0000marcell/square), the language is extremely small, my objective was to write a programming language that can do fibonacci, that's it.
fib written in square
```
[>:fib n:
  [:if n < 3:
    return n - 1
  ]
  return [:fib n - 1] + [:fib n - 2]
]
[:fib 10]

[:print n]
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3phcrvf4juhx42ngjdm7.gif)

it works!!!

It uses one of the most popular tools to write the grammar, the AST is very minimal and is executed in C, today I'm gonna explain the main components of a programming language trying to be as concise as possible. 

<a name='TheGrammar'></a>
## The Grammar

This is the heart of the language, This is how you interact with the language, the grammar rules define how to write code for the language, I used flex and [bison](https://www.gnu.org/software/bison/) to write the grammar, the grammar of the language is written on [square.y](https://github.com/0000marcell/square/blob/master/square.y#L127) skim through those rules just to get a idea of how they work. 

Here's a grammar rule from the language

```
ID EQ NUM
```
and it defines how to interpret assignments

```
n = 2
```

You can think about the grammar rules as a way of organizing regex blocks, each `token` (ID, EQ, NUM) is a regex block

```
identifier  ([a-z]+)  // this is ID 
equal       ([=])     // this is EQ
numbers     ([0-9])+  // this is NUM
```

You organize the blocks in a way that you want the blocks to be matched in the language, so when the parser sees 
```
x = 99
```
it's going to first transform that in the tokens `ID EQ NUM` then search for a rule that matches that, after finding the rule, the parser is going to execute the body of the rule

```
ID EQ NUM {
  struct scope * number = (struct scope *) 
  malloc(sizeof(struct scope));
  (number)->type = "number";
  (number)->value = $3;
```

Think about the body of the rule as a function that is going to be executed when the parser finds that match, you can access the value in the tokens using `$1, $2, $3`, here is where you build your AST, more on that later 

<a name='ComposingGrammaticsRules'></a>
## Composing Grammar Rules

You can compose grammar rules, for example, in the language we have function calls that work like this `[:add n]` but you can also call the function passing the value directly instead of a variable `[:add 10]` instead of creating two different grammar rules for each case, we create a grammar rule with these two options and use it in the function call grammar rule.

``` C
OPBRA IDFUNC fcallparam CLBRA {
```

```C
fcallparam: ID {
              ..body 
          } 
          | NUM {
              ..body
          }
;
```

OPBRA is `[` IDFUNC is the name of the function `:add` in this case, fcallparam can be a `ID` identifier, basically a variable, or NUM, NUM is a number, you're free to compose the rules how many times you want.

<a name='TheOperatorAndEmptyGrammarRules'></a>
## The $$ Operator And Empty Grammar Rules

The `$$` operator can be used to save the result of the composing grammar rule to be used in another grammar rule, it's easier to see the utility in action, in the last example 
we can save anything on `$$` than use it in the composing grammar rule
```C
fcallparam: ID {
  $$ = "Hello There!!" 
}

OPBRA IDFUNC fcallparam CLBRA {
  printf($3) // $3 here is "Hello There!!"  
}
```
remember that we can access the tokens with $1, $2, $3 etc...
So now $3 is "Hello There!!".

If You just want to use the token directly you don't need the body, example

```C
fcallparam: ID | NUM
```

```C
OPBRA IDFUNC fcallparam CLBRA {
  // $3 will be either a identifier or a number depending on 
  // the match   
}
```
the default behavior of any rule without a body is ```$$ = $1```

<a name='RuleTypes'></a>
## Rule Types

A rule also has a type, to return a string like we did in the `fcallparam` example, the rule `fcallparam` needs to be a `string`, we define this in the beginning of the parser when we connect the parser with the lexer, you can see all definitions [here](https://github.com/0000marcell/square/blob/master/square.y#L106)

In the toy language `fcallparam` is of type node which is a node of the AST
`%type <node> fcallparam`

<a name='ParsingTheLanguage'></a>
## Parsing The Language

With the information that we have above we can begin to understand how to parse the language, and how to create the AST, for example we could create a `print` like function called `output` in the language.

In the source file…
```C
  output("hello world")
```
In the parser...
We parse the contents with the right rules
```
OUTPUT OPPAR STR CLPAR {
  // here we just print the content inside the parenteshis
  printf($3)
}
```
for very simple rules we could do something like that and execute the code that we created, the problem is that programming languages have concepts like functions that postpone code execution, you cannot execute what you see directly because we have definitions that are going to be executed when the program tells you to, for example if `output` was inside another function we can not execute it right away, this is one of the reasons why we need a AST, the AST is a data structure that represents the code but can be executed easily, we don't need to parse anything, just traverse the data structure and execute it.

``` C
function test() {
  output("hello world")
}
``` 
in this case we cannot execute `output` as soon as we identify it in the parser, we need to wait for a function call that calls test(), then we execute what's inside test's body, to solve this problem we need to breakdown the parsing and the executing in to different things, that's where the AST comes in.

<a name='AST'></a>
## AST

Abstract Syntax Three is just a data structure like an linked list or binary three, you can use nested array as an AST, and some languages do use it, but we're going to use a linked list instead, this is the data structure that I used in the language
```C
struct scope {
  char * type;
  char * extra;
  struct arg * args;
  int value;
  struct scope * scopes;
  struct scope * next; 
  struct scope * prev; 
  int return_value;
};
```
This is essentially a [doubly linked list](https://en.wikipedia.org/wiki/Doubly_linked_list) after I finished the language I realized that it could have been much easier but I didn't have the patience to change everything so that's what we have right now :P, it's easier to understand what it represents with an example
``` C
{
  .type = "function",
  .extra = "global",
  .scopes = {
    .type = "body",
    ...
    .scopes = {
      .type = "function",
      .extra = "fib",
      .args = {
        .key = "n", 
        .value = 0,
      }
      .scopes = {
        .type = "body",
        .scopes = {
          .type = "if"
          ...
        }
      }
    }
  }
}
```
The root node is a function called global, same thing as main in other languages like C, this node has a body and the body 
has another node function called fib which is the declaration of the fib function, this node has a if node and so on..., I hope you get the idea, the AST is just another way of representing the code, this is what we called a IR or [Intermediate Representation](https://en.wikipedia.org/wiki/Intermediate_representation) the reason why we do this is to make it easier to do something with that data, specially traverse it, it's very hard to traverse source code, so instead of doing that we transform the source code in to a different data structure and work with that instead
so this 
```
[:fib n
  [:if n < 3
  ]
]
``` 
becomes this
```
{
  .type = "function",
  .extra = "fib",
  .scopes = {
    .type = "body",
    ...
    .scopes = {
      .type = "if",
      ...
```

The next thing that we need to do is execute the AST, a lot of popular languages like Python, Ruby and Java have an additional IR representing the code as bytecode and executing the bytecode in a virtual machine for optimization, but since this is a toy language we just execute the AST itself

##Executing The AST

Executing the AST is pretty straighforward we pass the AST to an `exec` function and this function traverse and execute the AST following the rules of the language, this is the file that creates the AST and calls exec passing the AST [square.y](https://github.com/0000marcell/square/blob/master/square.y) , the whole thing is less then 400 lines, and it could be much simpler as I said before, for example, the languages accepts evaluating statement in returns, so you can do things like this 
```
return [:fib n - 1] + [:fib n - 2]
```
but that's not strictly necessary, you can check what the exec function does [here](https://github.com/0000marcell/square/blob/master/exec.c#L277) but essentially we just traverse the AST check the node type like `function` `if` `function calls` etc.. and do what needs to be done, the most challenging part after building the AST is implementing scope but that can be done with pointers, each node holds the value of the variables used inside it and we can use `.next` and `.prev` to jump between scopes, I'm not gonna explain how I execute the AST here because you can simply follow the code [here](https://github.com/0000marcell/square/blob/master/exec.c#L279), it's very simple, and I think I already cover the meat of building a language, which is the parser and the AST, after you have the AST it's just a matter of executing it which is an entire different thing outside the scope of showing how a language can be implemented, if you want to build something specific I think the toy language that I've build is a good start, since a lot of the work is the actual set up.