Take a cue to supercharge your configurations
Tags: productivity, cue, configuration, opensource
Published: 2022-03-04 16:51:28

Do you have a love/hate relationship with YAML or JSON for configurations?

Are your configurations getting messed up because of simple typos, or problems with white space, or confusion between strings and numbers?

Do you try to manage some templating solution on top of YAML?

Is it hard to manage complex overrides across multiple configuration files?

If any of these questions resonate with you, then this article may interest you!



## Configuration complexity ##

In today's configuration landscape, YAML is pretty much everywhere. It is relatively easy for humans to read and also fairly compact format. It is also easy to mess up with some idiosyncrasies and white-space/indentation dependent structure. JSON is a simpler format in its structure, but more limited and not really intended for human consumption. TOML is another format which tries to strike a balance for readability and maintainability.

Neither of them have any built-in capabilities for elaborate validation of data or feature to facilitate organising (complex) configurations. It gets messy, in particular with more complex configurations.

Can we do better? Yes, I believe we can. There is one effort in this space that I think is promising, and that is [CUE (Configure Unify Execute)](https://cuelang.org), an open-source configuration language. 

There is a lot to like about CUE, and I will explore a few bits and pieces of it in this article. It is only scratching the surface of what you can do with CUE though.

We are going to focus on a bit of data validation and simple conversions:

* Convert a YAML configuration file to CUE
* Step-by-step define schema and validation rules for that configuration file


## Install CUE ##

The [CUE website has installation instructions](https://cuelang.org/docs/install/) for how to install CUE. There are binary downloads you can download for macOS, Windows, and Linux distributions, or you can install via Homebrew on macOS and Linux.

```bash
brew install cue-lang/tap/cue
```

If you have the Go language installed, you can also download and build CUE directly easily, since CUE is written in Go.

You do not have to install CUE now if you do not want to follow the rest of the article, but if you want to play around yourself, you should do that.

You can also have a look at the [CUE Playground](https://cuelang.org/play/#cue@export@cue). This is only for writing CUE and possibly converting it to JSON or YAML, so that is different use case than what we are covering in this article.




## Read and convert YAML configuration ##

We are going to start with a relatively small configuration file in YAML. This configuration describes a network configuration (VPC - Virtual Private Cloud) in Amazon Web Services (AWS). *It is not important exactly what it is though; we are only using this for illustration*. Our YAML configuration file has the name **network\_config.yaml** and looks like this:

```yaml
account: "123456789012"
region: eu-north-1
network:
  vpc_cidr: 10.100.0.0/16
  az_count: 2
  nat_gateway_count: 1
  public_mask: 25
  private_subnets:
    - name: private1
      mask: 21
  isolated_subnets:
    - name: isolated1
      mask: 21
```

AWS has then notion of organising cloud resources into accounts and regions, so this is something that is relevant for provisioning cloud resources. The rest of the structure has a mix of networking related information, essentially telling us how a network configuration should be set up.

The configuration has a mix of single values, some in a hierarchy, some in lists/arrays. There is a mix of strings and numbers. There are restrictions on what may be valid values for several fields. All of that is invisible to us in the YAML file, though.

Let us start by converting this YAML into CUE. We can do this with the cue import command:

```bash
cue import network_config.yaml 
```

This will produce a **network\_config.cue** file, with the same content, which looks like this:

```cue
account: "123456789012"
region:  "eu-north-1"
network: {
        vpc_cidr:          "10.100.0.0/16"
        az_count:          2
        nat_gateway_count: 1
        public_mask:       25
        private_subnets: [{
                name: "private1"
                mask: 21
        }]
        isolated_subnets: [{
                name: "isolated1"
                mask: 21
        }]
}
```

As you can see, it looks fairly similar. CUE is a superset of JSON, any valid JSON is also valid CUE. However, there is a lot more to CUE, and it has a more readable syntax than plain JSON and does not have white-space/indentation formatting of YAML. It does not require commas between key-value pairs either.

One thing to note here also is that string values are quoted, always - like JSON and unlike YAML.

*We will use this generated as a starting point to define data validation for our YAML configuration.*



## Define data validation ##

The CUE language has an unusual property which sets it apart from many other languages and formats - types and values are the same. *Not the same as only same type of structure, like JSON and JSON Schema,* __*but actually the same*.__

Our CUE file is just as much a schema definition as it is actual data values. The CUE command-line tool has the command **cue vet** to perform validation. You can provide both CUE and non-CUE files (e.g. JSON or YAML) to validate. So if we want to validate our YAML configuration towards the CUE data, we can run the command.

```bash
❯ cue vet network_config.cue network_config.yaml 

```

There is no output, which means it validated ok. You might think, how do I really know it has performed some validation here? Let us just change a value in the CUE file, for example, the "eu-north-1" value to "eu-west-1", then run the cue vet command again.

```bash
❯ cue vet network_config.cue network_config.yaml
region: conflicting values "eu-west-1" and "eu-north-1":
    ./network_config.cue:2:10
    ./network_config.yaml:2:10
```

As you can see, it now reports that there is a conflict, and points out where the data is conflicting in these files. So the validation works.

It is meaningless, though, to only have a hard-coded configuration to validate against. So let us generalise this a bit! 

1. For the region parameter, let us decide that we only allow the values "eu-north-1" and "eu-west-1"
2. For other values, let us define general string and integer types. 

The updated CUE file looks like below then. We have not added **string** and **int** types, and we have with the "|" character told CUE that either "eu-west-1" or "eu-north-1" are valid values, but nothing else.

```cue
account: string
region:  "eu-west-1" | "eu-north-1"
network: {
	vpc_cidr:          string
	az_count:          int
	nat_gateway_count: int
	public_mask:       int
	private_subnets: [{
		name: string
		mask: int
	}]
	isolated_subnets: [{
		name: string
		mask: int
	}]
}
```

If we run **cue vet** on this CUE file and the YAML file, we can see that it is still valid. In CUE, we can mix actual values with type definitions, that is all the same. 


## Reusable definitions and lists ##

We can see in the structure that we have two lists/array structures under private\_subnets and isolated\_subnets. The data structures in each of these are the same for both as well. Right now, if we would add another element in either of these lists in the YAML configuration, the validation would fail. So how can we make a reusable definition for the structure in the lists and also allow an arbitrary number of them in each list?

In CUE, a reusable description of a structure is called a **Definition**, and it looks like any other structure, only that *the name itself is prefixed with "#"* (or "\_#", but that is out of scope for now). We can in the list use a spread operator "..." to define a variable number of elements in the list. Our new CUE file looks like this now:

```cue
account: string
region:  "eu-west-1" | "eu-north-1"
network: {
	vpc_cidr:          string
	az_count:          int
	nat_gateway_count: int
	public_mask:       int
	private_subnets: [...#Subnet]
	isolated_subnets: [...#Subnet]
}

#Subnet: {
	name: string
	mask: int
}
```

You can run **cue vet** to check that it is still valid. In this case, we put the **#Subnet** definition after the rest, we just as well put it first as well. *The order does not matter to CUE*. This property becomes important when we organise our data in different ways.



## Adding constraints ##

Next step is to add additional constraints on some fields. The integer fields we use cannot have any integer value. We have a few constraints we would like to impose on these fields:

1. The **az\_count** fields refers to the number of *Availability Zones* that our AWS network setup should include. In most AWS regions, this would be a number between 1 and 3.
2. The **nat\_gateway\_count** field refers to the number of *NAT Gateways* provisioned. This is an infrastructure component that may be the same value as the number of availability zones, but could also be fewer. It should not be larger than the number of availability zones.
3. The **mask** and **public\_mask** fields can only have integer values between 16 and 28. This is a hard limitation set by AWS.

Besides types and values, we can also add constraints to our fields. The above rules can be described in the example below:

```cue
account: string
region:  "eu-west-1" | "eu-north-1"
network: {
	vpc_cidr:          string
	az_count:          int & >=1 & <=3
	nat_gateway_count: int & >=1 & <=az_count
	public_mask:       #Netmask
	private_subnets: [...#Subnet]
	isolated_subnets: [...#Subnet]
}

#Netmask: int & >=16 & <=28

#Subnet: {
	name: string
	mask: #Netmask
}
```

We added another reusable definition for the mask value, since that is used in two different fields. The constraints can be combined with the type data, or actual values. Multiple types/constraints/values for a field can be combined with the "&" symbol.

The syntax is concise, and also easy to read.

Now let us look at the strings. There are two fields, **account** and **vpc\_cidr**, which we should impose some constraints on. An AWS account is a string value that always comprises 12 digits. The VPC CIDR value is a combination of an IP address, a slash, and a net mask value. One way to approach the constraints here is to use regular expressions to define what is allowed. If you are not familiar with regular expression syntax, this will be a bit like keyboard noise to you. Fear not, you are not alone! It is useful to define restrictions on string patterns, and we use that to define the constraints on **account** and **vpc\_cidr**.

```cue
account: string & =~"^[0-9]{12}$"
region:  "eu-west-1" | "eu-north-1"
network: {
	vpc_cidr:          #CIDR
	az_count:          int & >=1 & <=3
	nat_gateway_count: int & >=1 & <=az_count
	public_mask:       #Netmask
	private_subnets: [...#Subnet]
	isolated_subnets: [...#Subnet]
}

#CIDR: string & =~"^10\\.(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\\.){2}(0|16|32|64|128)(\\/(2[0-8]|1[6-9]))$"

#Netmask: int & >=16 & <=28

#Subnet: {
	name: string
	mask: #Netmask
}
```

You can run **cue vet** and see that it still validates. Try also to change any of the values that we have put constraints on to see what the result will be if we violate the constraints and run **cue vet**.

Let us do one more modification here, set a default value. For the **vpc\_cidr** field, if no value is specified, with will use the string "10.0.0.0/16" as the default value. We use an asterisk to denote a default value:

```cue
account: string & =~"^[0-9]{12}$"
region:  "eu-west-1" | "eu-north-1"
network: {
	vpc_cidr:          *"10.0.0.0/16" | #CIDR
	az_count:          int & >=1 & <=3
	nat_gateway_count: int & >=1 & <=az_count
	public_mask:       #Netmask
	private_subnets: [...#Subnet]
	isolated_subnets: [...#Subnet]
}

#CIDR: string & =~"^10\\.(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\\.){2}(0|16|32|64|128)(\\/(2[0-8]|1[6-9]))$"

#Netmask: int & >=16 & <=28

#Subnet: {
	name: string
	mask: #Netmask
}
```

We have dipped our toes into how to define schemas and constraints with CUE. There is more to explore here, however, these basics can get you a good start. 

## Enforcing validation ##

We have validated our YAML configuration, and this has worked fine. However, if we add another dummy entry to our YAML configuration and run cue vet on that, it will still work fine.

```yaml
account: "123456789012"
region: eu-north-1
stuff: 333  ################### Our extra stuff here ###########
network:
  vpc_cidr: 10.100.0.0/16
  az_count: 2
  nat_gateway_count: 1
  public_mask: 25
  private_subnets:
    - name: private1
      mask: 21
  isolated_subnets:
    - name: isolated1
      mask: 21
```

```bash
❯ cue vet network_config.cue network_config.yaml  

```

As long as CUE finds valid and non-conflicting data structures, it is all good. If we want to restrict the validation to only accept specific fields, we need to be explicit about what to validate. Let us just add a layer in our schema definition that includes the three top level fields we have, **account**, **region** and **network**. 

```cue
#Config: {
	account: string & =~"^[0-9]{12}$"
	region:  "eu-west-1" | "eu-north-1"
	network: {
		vpc_cidr:          *"10.0.0.0/16" | #CIDR
		az_count:          int & >=1 & <=3
		nat_gateway_count: int & >=1 & <=az_count
		public_mask:       #Netmask
		private_subnets: [...#Subnet]
		isolated_subnets: [...#Subnet]
	}
}

#CIDR: string & =~"^10\\.(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\\.){2}(0|16|32|64|128)(\\/(2[0-8]|1[6-9]))$"

#Netmask: int & >=16 & <=28

#Subnet: {
	name: string
	mask: #Netmask
}
```

By defining reusable definition, we can also be specific in what should be included in the non-CUE data we validate when we run **cue vet**:

```bash
❯ cue vet --schema '#Config' network_config.cue network_config.yaml
field not allowed: stuff:
    ./network_config.yaml:3:2
    ./network_config.cue:1:1
    ./network_config.cue:1:10
```

Now our extra stuff is not accepted. Sometimes, though, we want only partial validation and allow for extra items to be present. If our schema definition work is a work in progress, that may very well be the case. In those cases, we can add "..." to our definition, which makes the structure ***open*** *to other fields*, as opposed to the default **closed** structure definition.

```cue
#Config: {
	account: string & =~"^[0-9]{12}$"
	region:  "eu-west-1" | "eu-north-1"
	network: {
		vpc_cidr:          *"10.0.0.0/16" | #CIDR
		az_count:          int & >=1 & <=3
		nat_gateway_count: int & >=1 & <=az_count
		public_mask:       #Netmask
		private_subnets: [...#Subnet]
		isolated_subnets: [...#Subnet]
	}
	...
}

#CIDR: string & =~"^10\\.(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\\.){2}(0|16|32|64|128)(\\/(2[0-8]|1[6-9]))$"

#Netmask: int & >=16 & <=28

#Subnet: {
	name: string
	mask: #Netmask
}
```

Running **cue vet** with this schema definition will accept our extra stuff.

## Final words ##

This has been an introduction to CUE, a quite versatile configuration language which also can play well with existing tooling. This has only scratched the surface of CUE though, and there is a lot more to it!

There are some neat material on CUE and some good presentations, at different levels - a few of these can be found here:

<https://fosdem.org/2022/schedule/event/cue_pratical_guide/>

<https://cuelang.org>

<https://cuetorials.com>

{% youtube https://www.youtube.com/watch?v=jSRXobu1jHk %}