Let's get started with Salt
Tags: salt, saltstack, devops, iac
Published: 2020-01-07 23:12:54

Configuration management is an essential piece of any computer system consisting of multiple components. If implemented correctly, it can make maintenance a lot easier and predictable for anyone involved. 

This becomes especially powerful when the system is managed with machine-readable configuration files instead of using various interactive tools. Spinning up and configuring each component should be automatic and consistent.

[SaltStack](https://www.saltstack.com/) (also known by the less searchable name Salt) is one such configuration management system. It doesn't only support configuration management, but also remote execution and event driven automation but let's focus on getting started with Salt configuration management.

# Simple architectural overview

Before we dive into a small example, let's take a look at a very brief and simplistic overview of how Salt works so you'll know what it's all about.

At its core Salt operates by setting up a small agent, called a *minion*, on all the system components you want to configure or operate (e.g. server). This minion talks to a Salt *master* which knows how the minion should be configured based on some configuration files written in YAML.

![Simple overview of the Salt architecture](https://thepracticaldev.s3.amazonaws.com/i/gnbeney3rlfxowwwrgf9.png)

In the configuration management corner of Salt, these YAML configuration files define the *state* the component should find itself in. The minion itself then makes sure it is in the desired state by doing some Python-fu.

So basically to get Salt working, all you need to do is to set up a Salt master and define the configuration in a YAML file. Then you just install the minion on the system components you want the Salt master to control.

# Example

To see it all in action let's set up a small Docker-based example. I'm fully aware that it's kind of weird to use Salt to configure Docker images because the *Dockerfile* is essentially a configuration file providing consistency, but Docker just makes it simpler to showcase the simplicity and for you to try out Salt.

We're going to create one Debian image with Salt master and create configuration files to control one Ubuntu image and one Alpine image.

First, let's create our two minions.

## Alpine minion

The only thing you have to do really, is to install the Salt minion agent, it will automatically be configured with sane defaults although we'll change them slightly.

Installing Salt minion on Alpine is as easy as just running:

    apk add salt-minion

*(in Docker we'll add the `--no-cache` flag because we want to clean up after ourselves)*

We have to make one small change to make the configuration files for the states easier. We're going to change something called the *minion id*.

Normally when the salt minion agent starts up it will by default take the hostname and set it as a *minion id*. The id is how the Salt master will recognize the different minions.

In the configuration management setup we need to *target* our minions which will allow us to define different states for different machines. There are many ways to *target* minions but we're going to target ours using the minion ids.

In Docker the hostnames are going to be a random string (e.g. `71665beda883`) which makes predefined targeting difficult. What we're going to do is prefix the minion id with a string that makes it easier for us to target the machine.

For the Alpine example, we're going to use the prefix `locomotive`. To do that we'll create a docker entrypoint file which echoes `"locomotive-$HOSTNAME"` into the file where the salt minion stores its id, `/etc/salt/minion_id`. Afterwards we'll start up the salt minion.

So this is our docker-entrypoint.sh for alpine:

```sh
echo "locomotive-$HOSTNAME" > /etc/salt/minion_id
salt-minion
```

Now all we have to do to create the Alpine Salt minion, is install the Salt minion, copy our docker entrypoint file into the image, and set the command to run the docker entrypoint file.

```Dockerfile
FROM alpine

RUN apk add --no-cache salt-minion

COPY docker-entrypoint.sh /

CMD ["sh", "/docker-entrypoint.sh"]
```

Let's build the Docker image as `alpine-minion` (assuming we're in a folder where we have the alpine Dockerfile):

    docker build -t alpine-minion .

Alright, onwards to the next minion.

## Ubuntu minion

The Ubuntu minion is going to be almost exactly the same. The difference is going to be that we use apt to install the salt minion instead of apk, and we're going to use the prefix `cow` instead of `locomotive` (because we want to target the minions differently).

So the docker-entrypoint.sh is going to be like this:

```sh
echo "cow-$HOSTNAME" > /etc/salt/minion_id
salt-minion
```

The Dockerfile is going to look like this:

```Dockerfile
FROM ubuntu

RUN apt-get update && \
    apt-get install -y salt-minion

COPY docker-entrypoint.sh /

CMD ["sh", "/docker-entrypoint.sh"]
```

Let's build this one as `ubuntu-minion` (again assuming we're now in the ubuntu minion's directory):

    docker build -t ubuntu-minion .

That's enough of the minions, let's move on to the states for our minions.

## Salt states

You may be wondering why I chose the `locomotive` and `cow` prefix for the alpine and ubuntu images, respectively. That's because we're going to install two different software packages based on the prefix.

As I said before, we're going to target the minions (based on their minion id) and install the different packages based on the prefix. When compiling the state for a specific minion, Salt always starts looking in a file called `top.sls`. The `sls` file extension stands for SaLt State file, but don't let that fool you, it's just a YAML file. All state definition files should have the `sls` file extension.

We're going to create three state files. One to install two software packages: `cowsay` and `fortune`. One to install software packages `sl` and `fortune` (we install fortune in both this way for the example). Lastly we create the `top.sls` file where Salt starts its journey in.

### Make sure a package is installed

Let's start by making sure the packages `cowsay` and `fortune` are installed. A Salt state file can contain multiple states, all defined by state names as the root elements of the state file. We're just going to have one in a file called `cow.sls` to make sure we install `cowsay` and `fortune`.

We can call the state what we want, but to be explicit let's call it *install fun cow tools*. The only thing we have to make sure when giving these names is that the for any given minion, no two states can have the same name.

The state element in the YAML file is a dictionary of the states we want. Salt comes with a lot of built in state modules we can use (that's the beauty of Salt) but the Salt state module we want to use is called `pkg`.

We use the `pkg` state module to define that the minions where this state applies need to have packages `installed` (a function in the `pkg` module called `installed`). The `pkg.installed` function can take in a keyword argument `pkgs` which for its value is a list of packages to install, in our case: `fortune` and `cowsay`.

The `cow.sls` file will look like this:

```yaml
install cow tools:
  pkg.installed:
    - pkgs:
      - fortune
      - cowsay
```

The locomotive file is going to be similar, but instead of installing cowsay, we're going to install `sl` in addition to `fortune`. We'll also use the name `install locomotive tools` for the state. So the file `locomotive.sls` will look like this:

```yaml
install locomotive tools:
  pkg.installed:
    - pkgs:
      - fortune
      - sl
```

As you can see, the files use the same salt modules and functions even if we know that behind the scenes one will be executed on an alpine machine and the other on an ubuntu machine. Salt will use the appropriate package manager.

### top.sls

Let's tie this all together in the `top.sls` and target the machines based on their prefix. Salt allows use of an asterisk/glob when targeting so it's fairly easy to target the minion ids with these prefixes.

First we must define an environment we use. We'll use the default Salt environment called `base`. That's going to be the root element and the value is a dictionary. The key elements of that dictionary are the targeting patterns. Their values are what state files it should apply. So something like this:

```yaml
<environment>:
  <target pattern>:
    - <state file>
    - <state file>
```

The state files are given as the name without the file extension. The state files can be in subdirectories but then we have to use the Python convention where directories and files are separated by a period. So a file `cow.sls` in the subdirectory `farm` is included as `farm.cow`.

Now this may seem complicated but it's really not. It's just hard to describe. If we assume the files are in the same directory as the top.sls file, our top.sls will look like this:

```yaml
base:
  locomotive-*:
    - locomotive
  cow-*:
    - cow
```

This means any minion id that starts with `locomotive-` is going to apply the states found in the `locomotive.sls` file in the same folder as the `top.sls` file.

This is the gist of how we build up and define the state of a system with Salt, using YAML files. Now we all we need is the master that can tell the minions what to do.

## Salt master

Let's use Debian for the Salt mater. Just like with the minions, The only thing we have to do to get Salt up and running with sane defaults, is to install the salt-master package. So it's going to be a simple Docker image we need (we'll do the magic that we need to make our Salt system operational, when we mount configuration files):

```Dockerfile
FROM debian:10-slim

RUN apt-get update && \
    apt-get install -y salt-master

CMD salt-master
```

Let's build this image as `salt-master` (assuming we're in the same directory as the Salt master Dockerfile):

    docker build -t salt-master .

Now with all images we need let's make it operational!

## Salt master configuration

First off, we're going to need a basic configuration file that will tell the Salt master where our state files are.

Thanks to Salt's sane defaults, the configuration file is super-simple. We only have to define a file root where our state environment can be found. It is possible to manage a lot of different Salt environments with different states using the same Salt master and configuration file but we're going to stick to the default `base` environment.

The configuration file just needs to point to the directory with the root of our state files for a `base` Salt environment. We'll create a file called `master.config` with only three lines:

```yaml
file_roots:
  base:
    - /srv/salt/
```

Our state files (those we created earlier) need to be made available at `/srv/salt` and we need to mount our `master.config` in our Salt master as the file `/etc/salt/master`. 

## Docker compose environment

To make all of this easier we'll use Docker compose to set up all of our services. Before we create the Docker compose file let's first have a look at how our local file system should look when we've created our Docker compose file (assuming the Dockerfiles we used earlier are stored elsewhere):

```
salt-example/
  |
  |- docker-compose.yml
  |- master.config
  `- states/
       |
       |- cow.sls
       |- locomotive.sls
       `- top.sls
```

Now we can create our Docker compose file.

There are a few things we have to make sure we do. Two I have already mentioned: one is to mount the `master.config` as `/etc/salt/master` in the Salt master service, the other to mount our `states/` directory as `/srv/salt/` in the Salt master service.

The third thing I haven't mentioned, but it's very important. By default minions will look for the Salt master at the domain name `salt`. This is configurable but we are using defaults, so we have to make sure the Salt master service in the Docker compose file is called `salt` (which makes it available as `salt` to our minions who share the docker compose network).

So the `docker-compose.yml` file will define three services, salt (with files and directories mounted properly), our alpine minion and the ubuntu minion:

```yaml
version: "3"
services:
  salt:
    image: salt-master
    volumes:
      - ./master.config:/etc/salt/master
      - ./states:/srv/salt
  alpine-minion:
    image: alpine-minion
  ubuntu-minion:
    image: ubuntu-minion
```

That is all. We're ready to start everything up. Just type:

    docker-compose up -d

Congratulations! If there haven't been any typos, you now have your first Salt configuration management environment up and running. But it actually doesn't do anything.

## Accepting minions

If Salt would just start handing out states to any machine that contacts the Salt master and fits into a specific minion id pattern, we'd have a problem on our hands. To avoid this we need to accept what minions we want the Salt master to manage.

When the minion contacts the Salt master it sends its public key. If we haven't accepted the minion before, we need to accept it (that is to say, if we trust it) using a tool installed on the Salt master called `salt-key`.

Before we continue, let's open up a shell on the Salt master docker container:

    docker-compose exec salt bash

Now in the Salt master bash shell you can list all minion keys that are accepted, denied, rejected and unaccepted, by running `salt-key` with the `-l all` option:

    salt-key -l all

You should see an output similar to this one:

```
Accepted Keys:
Denied Keys:
Unaccepted Keys:
cow-7d7f4faf9802
locomotive-1fbb4f5c409d
Rejected Keys:
```

As you can see there should be two unaccepted keys (one with a cow prefix and another with a locomotive prefix).

If we do not accept them we will not be able to manage their configurations so let's go ahead and accept all the keys we've gotten using the `-A` flag:

    salt-key -A

You'll be asked to confirm if you want to accept the two listed keys. Just enter `Y` if all looks good (or simply press Enter because accepting is the default). Now you should see something like:

```
Key for minion cow-7d7f4faf9802 accepted.
Key for minion locomotive-1fbb4f5c409d accepted.
```

Now we can control our minions. They may have shutdown because it took some time to accept their keys (they don't want to wait forever). Let's just be sure they're up by exiting the Salt master:

    exit

Then we run docker compose up again:

    docker-compose up -d

Then we can attach to the Salt master again:

    docker-compose exec salt bash

If you want to be sure we've accepted our keys and they're still there just type:

    salt-key -l all

You should see something like:

```
Accepted Keys:
cow-7d7f4faf9802
locomotive-1fbb4f5c409d
Denied Keys:
Unaccepted Keys:
Rejected Keys:
```

## Applying state

The states still haven't been applied to our minions. They don't know what packages they should install so they haven't installed anything. To trigger the configuration we will have to execute a `state.apply` on the minions we want to apply the state.

To do this we target our minions and we can safely just target all of our minions using the glob (asterisk) and tell it to apply the states. To do that we type the following command:

    salt '*' state.apply

Notice the quotes around the glob. This is needed because we're passing the string with the targeting pattern.

You should see a lot of stuff on your screen but near the end of it all, you'll see a summary like:

```
Summary for locomotive-1fbb4f5c409d
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:   5.633 s
```

This tells you all states were run successfully and one change was made (it installed the packages) on the locomotive machine (Alpine). You'll see a similar report for the cow machine (Ubuntu)

You can run the `state.apply` function as often as you like. Because it is a state, it defines *how* your minion should look so what it actually does before installing packages in our case, is to check if all the packages are installed. If they aren't or some are missing, it goes ahead and installs the missing ones.

The state apply does the magic of making sure that the state is fulfilled. Running `state.apply` again should result in a report for (one for the Alpine machine and another for the Ubuntu machine) like this one: 

```
locomotive-1fbb4f5c409d:
----------
          ID: install locomotive tools
    Function: pkg.installed
      Result: True
     Comment: All specified packages are already installed
     Started: 23:53:42.277146
    Duration: 723.556 ms
     Changes:   

Summary for locomotive-03f0f2574592
------------
Succeeded: 1
Failed:    0
------------
Total states run:     1
Total run time: 723.556 ms
```

This is awesome. Salt tells us everything is great, but is it? Let's check. First exit the Salt master

    exit

Alright now we're ready to check if Salt did what we wanted it to.

### Does it work?

The cowsay package in Ubuntu is installed in `/usr/games/cowsay` so to see it in action we'll run this:

    docker-compose exec ubuntu-minion /usr/games/cowsay I think I need some Salt

You'll see a bovine ascii art telling you it needs some Salt. Let's make see if we have fortune installed (also in /usr/games/):

    docker-compose exec ubuntu-minion /usr/games/fortune

Enjoy the wisdom!

Alpine should also have some packages (in this case we don't have to access it at `/usr/games/` because it is already in the path):

    docker-compose exec alpine-minion fortune

Woah! More wisdom! Wait, maybe we installed the cow on Alpine as well:

    docker-compose exec alpine-minion cowsay Salt told me not to

Nope. You should get an error telling you that cowsay isn't found in the path. But sl is:

    docker-compose exec alpine-minion sl

Here's a joke because you got this far: What sound does a sick train make? -- *Ah-Ah-Ah Choo choo!*

So these were the basics of Salt. In future posts I want to dive deeper into Salt but I needed to get the basics out of the way.

---

<a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-sa/4.0/80x15.png" /></a> This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>.

*Cover image by Jorge Royan: [Four salt shakers on a table with salt spilled from one of them](https://commons.wikimedia.org/wiki/File:Saleros_-_5394.jpg)*