Creating Terraform-like configuration languages with HCL and Go
Tags: go, terraform, hcl, devops
Published: 2022-08-06 20:58:51


![PCL: The Pizza Configuration Language](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pxhy1ecmdup5nvgoys75.png)

In the past year and a half, I've been working on [Atlas](https://atlasgo.io), a database schema management tool that we're developing at [Ariga](https://github.com/ariga). As part of this effort, I worked on implementing the infrastructure for the [Atlas DDL](https://atlasgo.io/atlas-schema/sql-resources), a data definition language that is the basis for Atlas's declarative style workflow for managing database schemas. 

The Atlas language is based on [HCL](https://github.com/hashicorp/hcl), a toolkit for creating configuration languages with a neat and simple [information model and syntax](https://github.com/hashicorp/hcl#information-model-and-syntax). HCL was created at HashiCorp and used in popular tools such as [Terraform](https://www.terraform.io/language/syntax/configuration) and [Nomad](https://www.nomadproject.io/docs/job-specification). We chose HCL as the basis of our configuration language for multiple reasons: 
* It has a base syntax that is clear and concise, easily readable by humans and machines. 
* Popularized by Terraform and other projects in the DevOps / 
  Infrastructure-as-Code space, we thought it would feel familiar to practitioners 
  which are the one of the core audiences for our tool.
* It's written in Go, making it super easy to integrate with the rest of our codebase 
  at [Ariga](https://github.com/ariga).
* It has great support for extending the basic syntax into a full-blown DSL using
 functions, expressions and context variables.

## PCL: The Pizza Configuration Language

In the rest of this post, we will demonstrate how to create a basic configuration language using HCL and Go. To make this discussion entertaining, let's imagine that we are creating a new PaC (Pizza-as-Code) product that lets users define their pizza in simple HCL-based configuration files and send them as orders to their nearby pizza place. 

### Orders and Contacts

Let's start building our PaC configuration language by letting users define where they want their pizza delivered and who is the hungry contact waiting for the pizza to arrive. We're aiming for something like:
```hcl
contact {
  name  = "Sherlock Holmes"
  phone = "+44 20 7224 3688"
}
address {
  street  = "221B Baker St"
  city    = "London"
  country = "England"
}
```
To capture this configuration, we will define a Go struct `Order` with sub-structs for capturing the `Contact` and `Address`:
```go
type (
	Order struct {
		Contact *Contact `hcl:"contact,block"`
		Address *Address `hcl:"address,block"`
	}
	Contact struct {
		Name  string `hcl:"name"`
		Phone string `hcl:"phone"`
	}
	Address struct {
		Street  string `hcl:"street"`
		City    string `hcl:"city"`
		Country string `hcl:"country"`
	}
)
```
The Go HCL codebase contains two packages with a fairly high-level API for decoding HCL documents into Go structs: `hclsimple` ([GoDoc](https://pkg.go.dev/github.com/hashicorp/hcl/v2/hclsimple))
and `gohcl` ([GoDoc](https://pkg.go.dev/github.com/hashicorp/hcl/v2/gohcl)).  Both packages rely on the user supplying Go struct field tags to map from the configuration file to the struct fields.

We will start the example by using the simpler one, with the surprising
name, `hclsimple`:

```go
func TestOrder(t *testing.T) {
	var o Order
	if err := hclsimple.DecodeFile("testdata/order.hcl", nil, &o); err != nil {
		t.Fatalf("failed: %s", err)
	}
	require.EqualValues(t, Order{
		Contact: &Contact{
			Name:  "Sherlock Holmes",
			Phone: "+44 20 7224 3688",
		},
		Address: &Address{
			Street:  "221B Baker St",
			City:    "London",
			Country: "England",
		},
	}, o)
}
```

### Pizza sizes and toppings (using static values)

Next, let's add the ability to order actual pizzas in our PaC application. To describe a pizza in our configuration language users should be able to do
something like:
```hcl
pizza {
  size = XL
  count = 1
  toppings = [
    olives,
    feta_cheese,
    onions,
  ]
}
```
Notice that to make our API more explicit, users do not pass string values to the `size` or `toppings` field, and instead they use pre-defined, static identifiers (called "variables" in the HCL internal API) such as `XL` or `feta_cheese`.

To support this kind of behavior, we can pass an `hcl.EvalContext` ([GoDoc](https://pkg.go.dev/github.com/hashicorp/hcl/v2#EvalContext)),
which provides the variables and functions that should be used to evaluate an expression. 

To construct this context we'll create this `ctx()` helper function:

```go
func ctx() *hcl.EvalContext {
	vars := make(map[string]cty.Value)
	for _, size := range []string{"S", "M", "L", "XL"} {
		vars[size] = cty.StringVal(size)
	}
	for _, topping := range []string{"olives", "onion", "feta_cheese", "garlic", "tomatoe"} {
		vars[topping] = cty.StringVal(topping)
	}
	return &hcl.EvalContext{
		Variables: vars,
	}
}
```

To use it we need to add the `pizza` block to our top level `Order` struct:
```go
type (
    Order struct {
        Contact *Contact `hcl:"contact,block"`
        Address *Address `hcl:"address,block"`
        Pizzas  []*Pizza `hcl:"pizza,block"`
    }
    Pizza struct {
        Size     string   `hcl:"size"`
		Count    int      `hcl:"count,optional"`
		Toppings []string `hcl:"toppings,optional"`
    }
	// ... More types ...
)
```
Here's our `pizza` block read using `ctx()` in action:

```go
func TestPizza(t *testing.T) {
	var o Order
	if err := hclsimple.DecodeFile("testdata/pizza.hcl", ctx(), &o); err != nil {
		t.Fatalf("failed: %s", err)
	}
	require.EqualValues(t, Order{
		Pizzas: []*Pizza{
			{
				Size: "XL",
				Toppings: []string{
					"olives",
					"feta_cheese",
					"onion",
				},
			},
		},
	}, o)
}
```

### How many pizzas to order? (Using functions in HCL)

The final conundrum in any pizza delivery order is of course, how many pizzas to order. To help our users out with this riddle, let's level up our DSL and add the `for_diners` function that will take a number of diners and calculate for the user how many pizzas should be ordered. This will look something like:
```text
pizza {
  size     = XL
  count    = for_diners(3)
  toppings = [
    tomato
  ]
}
```
Based on the universally accepted heuristic that one should order 3 slices per diner and round up, we can register the following function into our `EvalContext`:

```go
func ctx() *hcl.EvalContext {
    // .. Variables ..
	
    // Define a the "for_diners" function.
    spec := &function.Spec{
        // Return a number.
        Type: function.StaticReturnType(cty.Number),
        // Accept a single input parameter, "diners", that is not-null number.
        Params: []function.Parameter{
            {Name: "diners", Type: cty.Number, AllowNull: false},
        },
        // The function implementation.
        Impl: func (args []cty.Value, _ cty.Type) (cty.Value, error) {
            d := args[0].AsBigFloat()
            if !d.IsInt() {
                return cty.NilVal, fmt.Errorf("expected int got %q", d)
            }
            di, _ := d.Int64()
            neededSlices := di * 3
            return cty.NumberFloatVal(math.Ceil(float64(neededSlices) / 8)), nil
        },
    }
    return &hcl.EvalContext{
        Variables: vars,
        Functions: map[string]function.Function{
          "for_diners": function.New(spec),
        },
    }
}
```
Testing the `for_diners` function out:
```go
func TestDiners(t *testing.T) {
    var o Order
    if err := hclsimple.DecodeFile("testdata/diners.hcl", ctx(), &o); err != nil {
      t.Fatalf("failed: %s", err)
    }
	// For 3 diners, we expect 2 pizzas to be ordered.
    require.EqualValues(t, 2, o.Pizzas[0].Count)
}
```

### Wrapping up

With these features, I think we can call it a day for this prototype of the world's first Pizza-as-Code product. As the source code for these examples is available on [GitHub](https://github.com/rotemtam/hcl-examples) under an Apache 2.0
license, I truly hope someone picks this up and builds this thing!

In this post we reviewed some basic things you can do to create a configuration language for your users using HCL. There's a lot of other cool features we built into the Atlas language (such as [input variables](https://atlasgo.io/atlas-schema/input-variables), [block referencing](https://atlasgo.io/concepts/ddl#references) and [block polymorphism](https://github.com/ariga/atlas/blob/3ee44a20e9cc0a5837d82b1c27276165c6206d6c/schemahcl/extension.go#L61)), so if you're interested so if you're interested in reading more about it feel free to ping me on [Twitter](https://twitter.com/_rtam).