How we define and load configuration from files.
Tags: go, microservices, webdev, devops
Published: 2022-05-09 13:50:19

## Background

When we write applications, we basically use configuration files, from various `shell`s to `nginx`, etc., all have their own configuration files. Although this is not too difficult, the configuration items are relatively cumbersome, and parsing and verifying them can be troublesome. In this article, I will tell how we simplify the definition and parsing of configuration files.

## Scenario

If we want to write a `Restful API` service, the configuration items will probably have the following contents.

* `Host`, the listening `IP`, if not filled, the default is `0.0.0.0`
* `Port`, the port to listen to, required, can only be a number, greater than or equal to 80, less than 65535
* `LogMode`, logging mode, only `file` or `console` can be selected
* `Verbose`, see whether to output detailed logs, optional, default is `false`
* `MaxConns`, the maximum number of concurrent connections allowed, default `10000`
* `Timeout`, timeout setting, default `3s`
* `CpuThreshold`, sets the threshold for `CPU` usage to trigger load shedding, default `900`, `1000m` means `100%`

Previously we used `json` for the configuration file, but there was a problem with `json` that we couldn't add comments, so we switched to `yaml` format.

Now let's see how easy it is to define and parse such a configuration file with `go-zero` ~

## Defining the configuration

First, we need to define the above configuration requirements into a Go structure, as follows.

```go
RestfulConf struct {
    Host string `json:",default=0.0.0.0"`
    Port int `json:",range=[80,65535)"`
    LogMode string `json:",options=[file,console]"`
    Verbose bool `json:",optional"`
    MaxConns int `json:",default=10000"`
    Timeout time.Duration `json:",default=3s"`
    CpuThreshold int64 `json:",default=900,range=[0:1000]"`
}
```

As you can see, we have certain definitions and restrictions for each configuration item, some of which are defined as follows.

* `default`, the default value is used if the configuration is not filled, you can see that `3s` in it will be automatically resolved into `time.Duration` type
* `optional`, you can leave the item blank, if not, use the zero value of its type
* `range`, which restricts the number type to the given range
* `options`, which restricts the configured value to only one of the given values

And, some attributes can be used together, such as.

* `default` and `range` can be used together to both add a range restriction and provide a default value
* `default` and `options` can be used together to add option restrictions and provide a default value

## Configuration files

Because we give a lot of default values when defining the configuration, as well as using `optional` to specify as optional, so our configuration file has relatively few configuration items that can use the default values without writing explicitly, as follows.

```yaml
# Because many of them have default values, you only need to write the ones
# that need to be specified and the ones that don't have default values
Port: 8080
LogMode: console
# You can read the values of environment variables
MaxBytes: ${MAX_BYTES}
```

Here is a note, if the `value` of the configuration chars are all numbers, and you define the configuration type is `string`, for example, some people often use `123456` to test the password, but the password will generally be defined as `string`, the configuration should be written as follows (just an example, passwords are generally not recommended to be written in the configuration file)

```yaml
Password: "123456"
```

Here the double quotes can not be missed, otherwise `type mismatch` will be reported, because `yaml` parser will parse `123456` as `int`.

## Loading configuration files

We have the configuration definition (`config.go`) and the configuration file (`config.yaml`), the next step is to load the configuration file, which can be done in three ways.

* must be loaded successfully, otherwise the program exits, we generally use so, if the configuration is not correct, the program will not be able to continue

```go
// If error, exit the program directly
var config RestfulConf
conf.MustLoad("config.yaml", &config)
```

The default code generated by `go-zero`'s  `goctl` tool also uses `MustLoad` to load the configuration file

* Load the configuration and handle the `error` on your own

```go
// handle errors on your own
var config RestfulConf
// For simplicity, LoadConfig will be changed to Load later,
// LoadConfig has been marked as Deprecated.
if err := conf.LoadConfig("config.yaml", &config); err ! = nil {
    log.Fatal(err)
}
```

* Load the configuration and read the environment variables

```go
// Automatically read environment variables
var config RestfulConf
conf.MustLoad(configFile, &config, conf.UseEnv())
```

Here why we need to explicitly specify `conf.UseEnv()`, because if default to use, you may need to `escape` when writing specific characters in the configuration, so the default does not read the environment variables.

## Implementation principle

We usually use `encoding/json` or the corresponding `yaml` library when implementing `yaml/json` parsing, but for `go-zero`, we need to have more precise control over `unmarshal`, which leads us to customize `yaml/json` parsing ourselves, the complete code is here.

Configuration file code: [https://github.com/zeromicro/go-zero/tree/master/core/conf](https://github.com/zeromicro/go-zero/tree/master/core/conf)

`yaml/json` parsing code: [https://github.com/zeromicro/go-zero/tree/master/core/mapping](https://github.com/zeromicro/go-zero/tree/master/ core/mapping)

This is also a good example of how `reflect` can be used and how unit tests can be used to ensure correctness in complex scenarios.

## Summary

I always recommend the idea of `Fail Fast`, we do the same on loading configuration files, once there is an error, immediately exit, so that `ops` will find the problem in time when deploying services, because the process can not get up running.

All the configuration items of `go-zero` services are loaded and automatically verified in this way, including the configuration of many of the tools I wrote are also based on this, hope it helps!

## Project address

[https://github.com/zeromicro/go-zero](https://github.com/zeromicro/go-zero)

Welcome to use `go-zero` and **star** to support us!