ML Configuration Management
Tags: mlops, dynaconf, configuration, ai
Published: 2022-07-28 10:40:00

This post follows our blog describing our [ML Ops manifest](https://dev.to/artlist/lessons-learned-on-the-road-to-mlops-22lj). In this post we will dive into the our configuration management within our ML projects    

## Background

Working in a real-world business environment, requires moving codes between research/development, test and production environments, which are crucial for development velocity. While doing so, it is important to allow for a common language & standards between various AI & Development teams, for frictionless deployment of codes. Additionally configuration management assist in:  

* ML work with many parameters, 
* hyper params etc. , 
* we want to separate config from code (12 factor app)


These days, it comes without a surprise that there are several OpenSource (OS) configuration frameworks that can be utilized for this. After reviewing several options (including [Hydra](https://hydra.cc/)), we decided on [dynaconf](https://www.dynaconf.com/), since it fulfilled our requirements of being:  

* Python based
* Simple
* Easily configurable and extendable
* Allow for overriding and cascading


![Moving from Dev to Prod](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bljmsryuxo7o4vyp6q2k.jpeg)

### Development Environment

[Artlist](https://www.artlsit.io) runs in multiple cloud environments, however currently most of the ML workloads run on GCP. Following GCP best practices, we have set up different projects for each environment, thus allowing for strict isolation between them, in addition to enabling billing segmentation. This separation needs to be easily propagated into the configuration, for seamless code execution.


In this post we review our configuration in relation to
1. [Basic implementation](#basic-implementation)
2. [Advance Templating](#advance-templating)  
3. [Simple Overriding](#simple-overriding)
4. [Project vs. Module settings](#project-vs.-module-settings)  
5. [Updating Configurations](#updating-configurations)


Now let’s see how `dynaconf` can help out with this.  

### Basic Implementation  

We decided to work with configuration settings that are stored in external `toml` files, which are easily readable and are becoming one of the [de-facto standards](https://peps.python.org/pep-0680/) in python.  

A code snippet from our basic configuration file is as follows:

```toml
[default]
PROJECT_ID = "@format artlist-{this.current_env}"
BASE_NAME = “my_feature_name”
BASE_PIPELINE_NAME_GCP = "@jinja {this.BASE_NAME | replace('_', '-')}"
BUCKET_NAME = "@format {this.BASE_NAME}--{this.current_env}"

[dev]
SERVICE_ACCOUNT="service-account1@artlist-dev.iam.gserviceaccount.com"

[tst]
SERVICE_ACCOUNT="service-account2@artlist-tst.iam.gserviceaccount.com"

[prd]
SERVICE_ACCOUNT="service-account3@artlist-prd.iam.gserviceaccount.com"
```

Now let's break it down.  
Whenever *dynaconf* runs - it runs in a specific environment. The default environment is called DEVELOPMENT. However, since we wanted to move easily between the environments (and GCP resources), we changed the naming convention of the environments (to a  3 letter acronym = dev, tst, prd), so we can readily reference the relevant GCP project while specifying the environment.  
Using the [`env_swithcher`](https://www.dynaconf.com/configuration/#env_switcher), we can indicate to *dynaconf* which configuration to load and what GCP project to access with the following line: 
{% raw %} 
> PROJECT_ID = "@format artlist-{this.current_env}"  
{% endraw %}

Using the `@format` as a prefix to the string, we can parse the parameter that is within the curly brackets. For example, if the current environment is set to ‘dev’ the PROJECT_ID variable will be `artlist-dev`, thus accessing only the resources from the `dev` project,  whereas if the environment is set to ‘prd’ the PROJECT_ID will be `artlist-prd`.  

Accessing the rest of the relevant variables is based on the [various sections in the toml](https://github.com/toml-lang/toml) file.  
For example, referencing the *Production Service Account* (SA) will be by accessing the `SERVICE_ACCOUNT` variable which is under the [prd] section

### Advance Templating  

*Dynaconf* includes the ability to work with [`Jinja` templating](https://jinja.palletsprojects.com/) - this can be useful for manipulating strings. GCP has a quark that requires naming containers within the GCP registry so as not to have ‘\_’ (underscore) as separators, but rather ‘\-’ (hyphen). And since we wanted to sync our registry and the artifacts coming out of [*Vertex AI pipelines*](https://cloud.google.com/vertex-ai/docs/pipelines/introduction) (that are stored within buckets / Cloud Storage), we were able to keep the python naming convention of ‘_’ , while converting the strings to the GCP convention when required.  
Using the *jinja*’s [text replace](https://jinja.palletsprojects.com/en/3.0.x/templates/?highlight=replace#jinja-filters.replace) method we can easily alter the text as necessary:

{% raw %}  
> BASE_PIPELINE_NAME_GCP = "@jinja {this.BASE_NAME | replace('_', '-')}"  
{% endraw %}

### Simple Overriding  

Another useful feature of `dynaconf` is that you can easily override the configuration using local settings. This is very convenient since local settings for development doesn’t need to be checked into source control, while the general settings should be synced to the entire team.  
All that is required to differentiate between the settings is to add the `.local` [suffix](https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.suffix) to the file name, see example below:

* General Settings - settings.toml  
* Local Settings - settings.local.toml  
  
Whenever `dynaconf` identifies the suffix `.local.toml` it will overwrite the variables configuration that exists in the `settings.toml` with the loaded `settings.local.toml` file.

![overriding example](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ps21dh0nqxqdkgsqeyfi.png)
An example for overwriting with local credentials

### Project vs. Module settings  

Our ML framework is [KubeFlow](https://www.kubeflow.org/) (hosted by *GCP VertexAI pipelines*), which requires various configurations: some at the component level (which are reused independently in various pipelines), while others are at the pipeline/project/cross-component level. To load both settings, we can use another feature of `dynaconf` which can define a specific file name template that will automatically be loaded by `dynaconf`. Here is our implementation practice:  

* Any configurations that are at the project level will be written in the project settings - `settings_prj.toml` (see [dynaconf settings_files](https://www.dynaconf.com/configuration/#settings_file-or-settings_files) configuration)
* Any configurations that are at the component level will be written in the component settings - `settings.toml` .

### Updating Configurations  

Sometimes there is a need to update the configuration during runtime, this can be challenging since the entire configuration is loaded immediately when the library is called. To do so, we can use a decorator to update the configuration. Assuming the `cfg` is the configuration settings, we can write the following decorator:  

```python  
@input_to_config(cfg)
def input_to_config(config, sequence_override=True):
   """[decorator] override config parameters with function inputs
   Args:
       config: Dynaconf configuration / settings to be updated
       wrapped_func ([function]): [the function to capture it's input and push to the config]
       sequence_override: configures the option for overriding keys or merging the values
   """
 
   def decorator(wrapped_func):
       @wraps(wrapped_func)
       def inner(*args, **kwargs):
           _override_config(kwargs, config, sequence_override=sequence_override)
           return wrapped_func(*args, **kwargs)
 
       return inner
 
   return decorator
```


# Summary

In this blog post, we have laid out our configuration implementation using `dynacof` library. We saw how we

1. Used the basic setup of `dynaconf` configuration
2. Synced our GCP project with `dynaconf` environments
3. Worked with the advanced `dynaconf` settings

In our next posts, we will extend our description of the various elements that have been incorporated into our ML project workflow, while developing our internal base library, which include standardization of:  

* Logging
* Accessing our secrets (using [GCP Secret Manager](https://cloud.google.com/secret-manager))
* Conduct our experiment tracking (using [ClearML](https://clear.ml/) )

####### Image copyright 
The banner image was co-created using [DALLE2](https://openai.com/dall-e-2/)