Kubernetes Configuration Management with Ansible (Part 1)
Tags: ansible, kubernetes, automation, devops
Published: 2020-06-22 16:52:04

Since day 0 I started to administrate Openshift last year, one of the core thing I wanted we tackle with the team was how to manage all customization we would have to do. I'm used to automation software, from bare shell scripts (yes this is automation :) to things like Puppet, CFengine or Ansible and I can't imagine now managing an application, a fleet of nodes, a cluster without being able to automate the deployment of the configuration.


So I share here - not in details - what we did in our team. I'll split the explanation in a couple of posts. This is quite simple but effective, we started recently to run regularly this job from AWX in check mode to observe configuration drift and may run it in run mode sooner or later.

### Set a plan 

We started to make some status & requirements with the colleagues (this is ALWAYS important to do so)
* We will administrate several clusters.
* A big part of the clusters configuration will be the same for all clusters.
* We must be able to deploy for each cluster some specific manifests
* The same manifests will deployed on several cluster but may differ slightly (quota values for instance), and we don't want to store all these manifests just for the sake of different values inside. So we must be able have placeholder in the manifests which would be filled with each cluster values.
* We must be able to add manifests but also to remove manifests from the cluster.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/5kbc773hwytxuqjdk8g6.jpg)

### Directory structure

Here is an example of the structure we choose
```
project-dir
├── config
│   ├── common.yml
│   ├── dev.yml
│   └── prod.yml
└── manifests
    ├── common
    │   ├── 05_authentication
    │   ├── 10_feature_foo
    │   └── 30_feature_baz
    ├── dev
    │   ├── 40_feature_dev_buz
    │   └── 60_feature_dev_buu
    └── prod
        ├── 50_feature_prod_bar
        └── 70_feature_prod_boo
```

Directory `config` contains a yaml file for each cluster, with the connection details and all specific variables, the file `common.yml` contains common and default variables.

```yaml
# prod config file
connection:
  - url: https://prd.prj.domain.tld
    token: !vault....

ldap_connection:
  - bind_user: cn=ldap-prd-user,...
    bind_password: !vault...
 
authorized_groups:
  -  group-dev-1
  -  group-dev-2
  -  group-dev-3

# other prod specific variables
```

### File naming Convention

We think after about a naming convention for the manifest files, not that it would enforce the content in the file (exception made to the `status` field), but to permit to easily spot a manifest file on the disk. We came up with convention `XX_kind_name_namespace_status.yml` where

* `XX` is a number to which help applying manifest in a specific order.
* `kind` is the kind of the manifest (`deployment`, `configMap`, ...).
* `name` is the value of the file `metadata.name` of the object to be modified.
* `namespace` is the namespace the object belongs to. If the object is not namespaced, like namespace or clusterrole, we set the value to `global`.
* `status` has either value `present` to ensure it exists or `absent` to ensure it does not exist. This value in the name of the file will be to the module `k8s` and will enforce the state of the manifest.

__few examples__

To better understand how it works, let have a look
* a manifest file to create a namespace `foobar` would be named `10_namespace_foobar_global_present.yml`
* a deployment named fluentd in namespace `cluster-logging` would be named `30_deployment_fluentd_cluster-logging_present.yml`
* a manifest to delete a `configMap` named `my-config` in namespace `dev-hideout` would be named `XX_configmap_my-config_dev-hideouts_absent.yml` 

Now we had an overview of the structure of the project, the next post will explain the ansible playbook.