Doctoring your application configuration
Tags: python
Published: 2022-01-01 02:15:00

## Opinion: Code for configuration handling is ugly
If you've ever written a statement that looks something like `my_setting = config.value if config.value else some_default` and hated it, then I expect we can agree that handling application configuration is a miserable thing to code for. Its boring to write, ugly to read, and annoying to ensure that all user configuration options have sane default values.

![Oscar the Grouch - I'll be miserable here in my trash can](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/u17p2cpia2j16hao3cnw.gif)

The `argparse` library can go a long way to making things better, but if you need to set dozens of options, or if your application supports plugins with configuration requirements unknown to the main program, it becomes harder to simply `add_argument` our way back to sanity.

![Mad scientist (black and white) - We'll see if I'm crazy or not!](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/iw9kj8qn2yxout70uujl.gif)

In these cases, it makes more sense to opt for a configuration file. Also, it would be nice if our application code didn't care about the configuration file at all, where we had a single source of truth to count on being correct.

## Getting Classy
When I've hit the point in my development process where I have the basic functionality working and I've settled on the basics of how the program will flow, I like to create a new python file `constfig.py` and define the class `_C` where the 'C' stands for both "constants" and "config" (🎶 and that's good enough for me. Both config and constants start with C 🎶). By convention, I lead the with an underscore to signal to the other developers on the project that the class isn't intended to be implemented directly, and at the end of the file I create an instance of `_C()` called `C` which can be imported and will contain all the information needed by the application.

For example, let's implement a simple dice rolling API endpoint using Flask which returns a JSON formatted string. For this type of application we would want to be able to easily configure the IP address and port that the service listens on, so let's define our `_C` class, establish the variable names, pre-populate the variables with some reasonable default values, and then create an instance of our class named `C` which the user is meant to import from our file.

```python
class _C(object):
    def __init__(self):
        self.LISTEN_IP = '0.0.0.0'
        self.LISTEN_PORT = 8080

C = _C()
```

We also have certain values which will never change at runtime, but if our specification changes later, we don't want to have to hunt down all instances of that value in our code, so let's also add our constant values.

```python
class _C(object):
    def __init__(self):
        # Constant values
        self.JSON_RESPONSE_KEYWORD_D6ROLL = 'd6_roll'

        # User configurable values
        self.LISTEN_IP = '0.0.0.0'
        self.LISTEN_PORT = 8080

C = _C()
```

Now that we have our constant values and our configurable variables with sane defaults, let's import `C` (the instance of `_C`) into our main Flask application.

```python
from constfig import C  # Our constants + config = constfig
from random import randint
from flask import Flask, jsonify

app = Flask(__name__)

@app.route('/rolld6')
def roll_d6():
    value = {C.JSON_RESPONSE_KEYWORD_D6ROLL: randint(1,6)}
    return jsonify(value)

if __name__ == '__main__':
    app.run(host=C.LISTEN_IP, port=C.LISTEN_PORT)
```

We can now give it a quick run:
```bash
user@host:~ $ python3 roll.py 
 * Serving Flask app "roll" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
INFO:werkzeug: * Running on http://0.0.0.0:8080/ (Press CTRL+C to quit)
```

...and test to make sure all looks good so far.

```bash
user@host:~ $ curl -s localhost:8080/rolld6 | python3 -m json.tool
{
    "d6_value": 3
}
```

## 🧑‍⚕️ Operate on your `self`

Now that we have our application and our `_C` class, we're ready for it to poke at its own guts.

![Characters from the TV show Scrubs in an operating room. One of them is asking what is going on](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/09jhssp7r21n4woz0izi.gif)

Let's add the function `load_config` which will open up our configuration file `config.yaml`, and then use the `setattr` function to update our own values at startup (when Python runs `from constfig import C`). Python's ability to alter its own state is knows as "reflection" or "reflective programming".

```python
import yaml
import logging


class _C(object):
    def __init__(self):
        # Constant values - and a "gotcha!"
        self.JSON_RESPONSE_KEYWORD_D6ROLL = 'd6_roll'

        # User configurable values
        self.LISTEN_IP = '0.0.0.0'
        self.LISTEN_PORT = 8080

        # Load user config (override defaults above)
        self.load_config()

    def load_config(self):
        try:
            config_file = open('config.yaml', 'r')
            config_string = config_file.read()
            config_file.close()

            configuration = yaml.load(config_string, Loader=yaml.SafeLoader)  # Don't handle badly formatted YAML. Let the parser inform the user of the error.
            if isinstance(configuration, dict):
                for variable_name, value in configuration.items():
                    setattr(self, variable_name, value)
            else:
                raise yaml.scanner.ScannerError(f'The file config.yaml should be structured as type dict, but got type {type(configuration)}')
        except FileNotFoundError:
            logging.warning('Configuration file config.yaml is missing. Using default values.')


C = _C()
```

We can now create our `config.yaml` file containing key:value pairs where the key matches the name of our configuration items

```yaml
LISTEN_IP: 127.0.0.1
LISTEN_PORT: 32000
```

Let's fire up the service again...

```bash
user@host:~ $ python3 roll.py 
 * Serving Flask app "roll" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:32000/ (Press CTRL+C to quit)
```

...and this time we see that the default values have been overridden by those in the configuration file. Specifically, we're now listening on the loopback IP, and that our port number has changed as expected.

## Gotta' catch that "gotcha!"
The problem with this method is that we open ourselves up to having the configuration file change values that should not be changed. For example, if we set `JSON_RESPONSE_KEYWORD_D6ROLL: this is bad` in `config.yaml` then make a request to our endpoint, we see that we've indirectly altered our application's response.

```bash
user@host:~ $ curl -s http://127.0.0.1:32000/rolld6 | python3 -m json.tool
{
    "this is bad": 4
}
```

![American Psycho - I know my behavior can be erratic sometimes](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gbn0etfi9pgpegsvi0aj.gif)

While the fix here isn't hard, this bad behavior underscores the importance of the order in which we set out values. Let's move our constant values to where they are being set _after_ the values in our configuration file by adding a `finally` clause to our try/except block on `load_config()` and call our new method `set_constants()`.

```python
import yaml
import logging


class _C(object):
    def __init__(self):
        # User configurable values
        self.LISTEN_IP = '0.0.0.0'
        self.LISTEN_PORT = 8080

        # Load user config (override defaults above)
        self.load_config()

    def set_constants(self):
        # Constant values
        self.JSON_RESPONSE_KEYWORD_D6ROLL = 'd6_value'


    def load_config(self):
        try:
            config_file = open('config.yaml', 'r')
            config_string = config_file.read()
            config_file.close()

            configuration = yaml.load(config_string, Loader=yaml.SafeLoader)  # Don't handle badly formatted YAML. Let the parser inform the user of the error.
            if isinstance(configuration, dict):
                for variable_name, value in configuration.items():
                    setattr(self, variable_name, value)
            else:
                raise yaml.scanner.ScannerError(f'The file config.yaml should be structured as type dict, but got type {type(configuration)}')
        except FileNotFoundError:
            logging.warning('Configuration file config.yaml is missing. Using default values.')
        finally:
            self.set_constants()


C = _C()
```

...then launch again using our "bad" (which tries to set `JSON_RESPONSE_KEYWORD_D6ROLL`) configuration file.

```bash
user@host:~ $ python3 roll.py 
 * Serving Flask app "roll" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:32000/ (Press CTRL+C to quit)
```

...and of course test the result.

```bash
user@host:~ $ curl -s http://127.0.0.1:32000/rolld6 | python3 -m json.tool
{
    "d6_value": 5
}
```

Tada! A single source of truth for your Python application!

![The Yellow Ranger from the 90s TV show Power Rangers does a dance move and then a yellow explosion happens in the background (...and frankly, it looks like she farted). I have probably accidentally spent an hour of my life watching this gif while editing this post](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/g1ray01bjhblp10j7grl.gif)

## You hate this, but I need validation
Yes, I hear you bemoan, "I've got a children's song about cookies stuck in my head, I'm half way through a pack of Oreos, and this seems like an abstraction that makes it harder to see how the configuration is loaded."

Yeah, this solution is probably not for everyone, but I've hidden the superpower of this method behind a mild mannered alter-ego.

![Actor Christopher Reeve playing Clark Kent takes off his glasses, only to realize that he might be giving himself away as Superman](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ife9ar06iqdh8oryrb4p.gif)

The real power is automatic validation of your configuration simply by importing `C`. To do this, let's define the function `validate_config()`, write some basic assertions to validate the configuration, and then call the function after we have successfully loaded the yaml file and set the constant values in our class.

```python
import yaml
import logging


class _C(object):
    def __init__(self):
        # Default values for user configurable items
        self.LISTEN_IP = '0.0.0.0'
        self.LISTEN_PORT = 8080

        # Load user config (override defaults above)
        self.load_config()

    def set_constants(self):
        # Constant values
        self.JSON_RESPONSE_KEYWORD_D6ROLL = 'd6_value'

    def load_config(self):
        try:
            config_file = open('config.yaml', 'r')
            config_string = config_file.read()
            config_file.close()

            configuration = yaml.load(config_string, Loader=yaml.SafeLoader)  # Don't handle badly formatted YAML. Let the parser inform the user of the error.
            if isinstance(configuration, dict):
                for variable_name, value in configuration.items():
                    setattr(self, variable_name, value)
            else:
                raise yaml.scanner.ScannerError(f'The file config.yaml should be structured as type dict, but got type {type(configuration)}')
        except FileNotFoundError:
            logging.warning('Configuration file config.yaml is missing. Using default values.')
        finally:
            self.set_constants()
            self.validate_config()  # Validate our config file

    def validate_config(self):
        # Validate LISTEN_IP
        assert isinstance(self.LISTEN_IP, str), 'LISTEN_IP is not a string value'
        assert len(self.LISTEN_IP.split('.')) == 4, 'LISTEN_IP has an unexpected number of octets'
        assert all([ip.isnumeric() for ip in self.LISTEN_IP.split('.')]), 'LISTEN_IP is not a valid IP address.'
        assert all([0 <= int(ip) <= 255 for ip in self.LISTEN_IP.split('.')]), 'LISTEN_IP is not a valid IP address.'

        # Validate LISTEN_PORT
        if isinstance(self.LISTEN_PORT, str):
            assert self.LISTEN_PORT.isnumeric(), 'LISTEN_PORT must be a whole number.'
            self.LISTEN_PORT = int(C.LISTEN_PORT)
        assert 999 < self.LISTEN_PORT < 65536, 'LISTEN_PORT is outside expected range.'


C = _C()
```

...and just as an example, let's put a deliberate typo in our configuration file.

```yaml
LISTEN_IP: 127.0.0.1x  # My fingers are fat :-(
LISTEN_PORT: 32000
```

Now, have a couple of opportunities to validate our configuration.

For example, at runtime...
```bash
user@host:~ $ python3 roll.py 
Traceback (most recent call last):
  File "roll.py", line 1, in <module>
    from constfig import C  # Our constants + config = constfig
  File "/Users/adrong/PycharmProjects/constfig/constfig.py", line 33, in <module>
    validate_config()
  File "/Users/adrong/PycharmProjects/constfig/constfig.py", line 14, in validate_config
    assert all([a.isnumeric() for a in C.LISTEN_IP.split('.')]), 'LISTEN_IP is not a valid IP address.'
AssertionError: LISTEN_IP is not a valid IP address.
```

... or run `constfig.py` directly to validate your configuration in your test or deployment pipelines.
```bash
user@host:~ $ python3 constfig.py 
Traceback (most recent call last):
  File "constfig.py", line 33, in <module>
    validate_config()
  File "constfig.py", line 14, in validate_config
    assert all([a.isnumeric() for a in C.LISTEN_IP.split('.')]), 'LISTEN_IP is not a valid IP address.'
AssertionError: LISTEN_IP is not a valid IP address.
```

## Small, but mighty
![The cartoon character Mighty Mouse eating a hole into a wheel of cheese, crawling into it, then eating it so fast it becomes a whirlwind that transforms into the hero mouse](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/q02hnor9agpbcqvdb3il.gif)

This is a relatively small amount of code that, with some tweaks, can source configuration from a database, from environment variables, from command line arguments, and can have validation code that can reconcile configuration from any combination of those sources. This pattern for handling configuration has enabled me to quickly create configuration handling in a standardized way, across multiple tools , that members of other teams have found approachable and easy to manage.