Grains and pillars in Salt
Tags: salt, saltstack, devops, iac
Published: 2020-01-16 23:13:14

The [first article in this series](https://dev.to/trickvi/let-s-get-started-with-salt-55hb) went over an example of how to set up the [Salt](https://saltstack.com) configuration management tool to automate a computer infrastructure.

Salt is very powerful and in this next section of the series, we'll take a look at two important components, *grains* and the *pillar*. At a first glance grains and the pillar seem to serve the same purpose but there is an important distinction between them.

# What are grains and the pillar?

Grains and the pillar are used by Salt to store data about the minion it is controlling. They are both in the end a key value store. In fact they are what is called in Python a *dictionary* (if you're into Perl or Ruby, they're known as a *hash*).

For example, the following is an example of some grains on the alpine minion we created in the first article in the series:

```
ipv4:
    - 127.0.0.1
    - 172.25.0.3
ipv6:
kernel:
    Linux
kernelrelease:
    4.15.0-65-generic
```

If you want to check out all of the grains, you can execute the following command in the salt master docker container (to get a shell, type this in the same directory as you keep your docker compose file: `docker-compose exec salt bash`):

    salt 'locomotive-*' grains.items

The pillar data looks the same except the keys and values are different. However if you run the command to list the pillar items on your salt master, you won't get a similar result now:

    salt 'locomotive-*' pillar.items

You'll get nothing as a result:

```
locomotive-03f0f2574592:
    ----------
```

That's because while many grains are automatically generated when the salt minion starts up, the pillar data isn't. In fact the salt minion doesn't ever generate the pillar data. That's the big distinction between grains and the pillar.

Grains are generated and/or stored on the minions themselves. Pillar data are generated and/or stored on the salt master.

This distinction allows the pillar data to contain sensitive data you don't want lying around on the minion. It could be minion configurations or variables you want to manage in a central place, well or just any other data.

Grains on the other hand are useful for information you won't necessarily know when you configure the minion like IP addresses, operating system, CPU information etc. It can also be arbitrary data, like a role you want to configure and store on the minion itself.

For example you could on the machine create a grain called `role` with value `webserver`. Then the salt master could pick up that grain and from there decide how to configure it, without changing anything in the salt master configuration files. The pillar data for that machine may then include information such as software licenses you need for a particular application, users allowed to log into web servers etc.

One good thing to keep in mind is that grains are fairly static, they change infrequently because they're just information about the system (either automatically generated or custom data). The pillar on the other hand is better suited for data that will change more frequently.

# Configuring grains

Most of the grains you normally need are automatically generated. However, if you ever find yourself in a position where you want to configure them you can store them in the minion configuration file on the machine. Location depends on operating system, for example on Unix systems you find it at `/etc/salt/minion` except in FreeBSD where it's stored at `/usr/local/etc/salt/minion`. On Windows machines the configuration is found at `C:\salt\conf\minion`.

The configuration file is a YAML file and you can add `grains` as a root element in the configuration file with the grains you want to store. For example you could add this to the minion config:

```yaml
grains:
  roles:
    - webserver
```

This would make the grain `roles` available with a list as its value, containing the webserver role (it doesn't have to be a list of strings but I try to choose a data structure applicable to the context and a server could have multiple roles so that's why I'd use a list for the roles).

Another place where you can store grains is in `/etc/salt/grains`. I like explicit so I like this location more than the minion config. It's a YAML file but if you want to use `/etc/salt/grains` you don't need the root `grains` element. So the equivalent `/etc/salt/grains` file to the one above is:

```yaml
roles:
  - webserver
```

Let's adapt the example from the first post in this series and configure grains for the two salt minions.

Let's create two grains files, one for our alpine container and another for the Ubuntu container, defining roles for the machines. Create a file called `alpine.grains` in the same location as your `docker-compose.yml` file (we're going to mount it later on). Here are the contents of the `alpine.grains` file:

```yaml
roles:
  - locomotive
  - fortuneteller
```

Create a similar file for the Ubuntu container called `ubuntu.grains` with these contents:

```yaml
roles:
  - cow
  - fortuneteller
```

Now let's update our `docker-compose.yml` We just have to mount the two files we created at `/etc/salt/grains` on both machines so the `docker-compose.yml` file should look like this:

```yml
version: "3"
services:
  salt:
    image: salt-master
    volumes:
      - ./master.config:/etc/salt/master
      - ./states:/srv/salt
  alpine-minion:
    image: alpine-minion
      - ./alpine.grains:/etc/salt/grains
  ubuntu-minion:
    image: ubuntu-minion
    volumes:
      - ./ubuntu.grains:/etc/salt/grains
```

Let' get all of them up and running using these new grains and therefore their new roles. Run this command from the same directory:

    docker-compose up -d

## Still running first post containers?

If you shut down and removed the containers you created in the first post, you'll just have to accept keys like you did in the first post.

If you are still running the same containers, you should see the minions being updated (recreated), but not the salt master (it's up to date). 

If you try to check things out with salt master now, you'll notice that it doesn't manage your salt minions any longer. The reason is that because we recreated the docker containers they now have a new minion id (it's a new machine basically). So if you're not following this guide adamantly and want to play around, you'll have to accept the keys as shown in the first blog post in the series.

Now we're ready to use those newly defined roles somehow.

# Usefulness of grains

## Targeting minions

Grains are very useful for targeting minions. Remember in the first article how we targeted the machines using the minion id? We can also target them using grains.

Let's update the configuration file for the salt master to target based on grains instead of minion ids. First let's create a special state file for the `fortune` program. Remove the line in both `cow.sls` and `locomotive.sls` where we tell salt to install the `fortune` package. Then create a similar state file called `fortune.sls`:

```yaml
install fortuneteller tools:
  pkg.installed:
    - pkgs:
      - fortune
```

Alright. Now we can change the `top.sls` file to target based on grains instead of minion ids. The pattern for it is to say `key:value`. Then you can, before you list what state files to include, begin by telling salt what you want to match with your pattern (in our case the grain).

So to apply the `apache.sls` file to machines with the role `webserver` you would write something like this:

```yaml
'roles:webserver':
  - match: grain
  - apache
```

Notice the `'` around `roles:webserver`. This is so that the YAML file gets processed correctly. There is a shorthand version to achieve the same. You could instead write this:

```yaml
'G@roles:webserver':
  - apache
```

Let's modify the top file so we target each role specifically and apply the appropriate state files. The top file could look like this then (I choose the more explicit targeting pattern):

```yaml
base:
  'roles:cow':
    - match: grain
    - cow
  'roles:fortuneteller':
    - match: grain
    - fortune
  'roles:locomotive':
    - match: grain
    - locomotive
```

## Logic in state files

Grains can also be useful in the state files themselves. For example, we can check the operating system and do different things based on the operating system. You'll see more on this when we go through templating in the Salt files but to give you a taste, let's solve the problem we had in the first article of the series, where the Ubuntu machine installed the packages in `/usr/games/`.

To solve that problem we'll put an if statement that checks the `os` grain to see if it is an Ubuntu operating system. If it is, then we make a symbolic link to `/usr/bin`. We'll need to change all state files because Ubuntu does this with all packages.

Before we change the files, let's first see how we look up the grain value in the state file. The grain we're after to check the operating system is the `os` grain. It becomes available as `grains['os']` (if you know Python, you'll understand that this is just a dictionary lookup).

So the trick is to add a Python `if` statement inside `{% %}` then include the state configurations you want before you end with a `{% endif %}`. I'll cover this better in a later post but to start `cow.sls` should have this content now:

```
install cow tools:
  pkg.installed:
    - pkgs:
      - cowsay

{% if grains['os'] == 'Ubuntu' %}
make cowsay available on Ubuntu:
  file.symlink:
    - name: /usr/bin/cowsay
    - target: /usr/games/cowsay
{% endif %}
```

To dissect this a little bit, first you see the state we defined in the first post of the series and modified a few minutes ago (to remove `fortune`). Then we create our if statement which checks if the `os` grain is equal to `Ubuntu`. If it is, then we create a state called *make cowsay available on Ubuntu*. This states uses the file state module to create a symlink to `/usr/games/cowsay` (the target) at `/usr/bin/cowsay` (defined in name of the symlink). Then we close the if statement.

`fortune.sls` is going to be similar

```
install fortuneteller tools:
  pkg.installed:
    - pkgs:
      - fortune

{% if grains['os'] == 'Ubuntu' %}
make fortune available on Ubuntu:
  file.symlink:
    - name: /usr/bin/fortune
    - target: /usr/games/fortune
{% endif %}
```

So will `locomotive.sls`

```
install locomotive tools:
  pkg.installed:
    - pkgs:
      - sl

{% if grains['os'] == 'Ubuntu' %}
make sl available on Ubuntu:
  file.symlink:
    - name: /usr/bin/sl
    - target: /usr/games/sl
{% endif %}
```

## Trying out the grain based configuration

Let's test our new grain based configuration. First let's stop and remove all of our docker containers because we want a fresh slate (this way we'll also get around the new minion ids discussed earlier and easily remove the older containers.

Type in these docker-compose commands to get stop containers, remove them, start up new ones and finally enter bash shell on the salt master (answering yes when prompted):

    docker-compose stop
    docker-compose rm
    docker-compose up -d
    docker-compose exec salt bash

Once you've entered the bash shell on the new salt master, accept the minion keys with this command (pressing Enter for yes when prompted):

    salt-key -A

Now we're ready to test our grain based targeting in the `top.sls`. Let's apply the states to all machines (targeting them with the glob `*`):

    salt '*' state.apply

This will install all the programs, based on the roles defined in the grains and create the symlinks on the Ubuntu machines thanks to the if statement in the states. If you look at the output report, you'll see 2 states succeeded in the Alpine container while four changed in the Ubuntu container.

## Targeting in the command line

Another place you can use to target based on grains is on the command line. Instead of using the glob or minion id when you run the `salt` command on the salt master, you can target based on grain by using the `-G` option.

As an example, let's run the fortune command on all fortuneteller minions (both Ubuntu and Alpine containers). To run a command via the `salt` command on the salt master you type `cmd.run` instead of the `state.apply` we've used earlier and then you tell salt which command to run (in our case `fortune`). Go ahead and type this in your salt master's bash shell:

    salt -G roles:fortuneteller cmd.run fortune

As you can see, we give salt the `-G` option followed by the grain key and value we want to target (much like we targeted them in the `top.sls` file. Then we tell salt to run the `fortune` command with the `cmd.run fortune`. The output should be some nice words of wisdom:

```
cow-e54f99c12817:
    Don't feed the bats tonight.
locomotive-68af4abb8a98:
    Put no trust in cryptic comments.
```

We can also only ask Alpine minions to deliver fortunes:

    salt -G os:Alpine cmd.run fortune

This returns only a single wisdom because the grain only matched a single minion:

```
locomotive-68af4abb8a98:
    The meek shall inherit the earth -- they are too weak to refuse.
```

OK. That's enough about grains for now, let's quickly turn to the pillar.

# Configuring the pillar

As I said previously. The pillar offers the same data structure as grains but instead of being generated and hosted on the salt minions, the pillar data are generated on the salt master and handed to the minion.

That means that one big difference between the pillar and grains is that the pillar needs targeting, so that the pillar data can be handed to the right minion. Besides that, they're just the same.

Let's move the exact programs we need to install for each role into the pillar. That makes it more flexible because we don't have to hard code the programs to install on each minion based on the role. It's going to simplify maintenance and our state code.

Configuring the pillar is a mix between states and grains. We have a `top.sls` file for the pillar where we do the targeting. There we include files, which should return a data structure similar to grains.

## Pillar configuration files

We begin by creating a folder called `pillar`. This folder we're going to mount on our salt master. This is the folder that will hold our pillar data. Let's first create files for each role. The root element in each file will be `packages` (could be anything we want but we want it to be explicit right?). Followed by the name of the program as a key with a value which is just a description of the package we install. Something like this:

```yaml
packages:
  package-name: package description
```

We won't really use the package description, we're only interested in the package name. However, salt will automatically merge dictionaries in a smart way (which is useful and cool) but unfortunately not append to lists. That's fine though because this gives us a nice place to put a description or comment with an Easter egg.

So first create a file called `cow.sls` with this content:

```yaml
packages:
  cowsay: A program with a cow
```

The second file called `fortuneteller.sls` with this content:

```yaml
packages:
  fortune: A program for a fortune teller
```

Lastly create the third file called `locomotive.sls` with this content:

```yaml
packages:
  sl: Tech Model Railroad Club
```

Finally we need our `top.sls` file. It's going to be exactly the same as our state file (you can copy it if you want). First we define the environment we want it to be used in, then we target the roles and include the sls files we just created. So go ahead and create the `top.sls` with this content:

```yaml
base:
  'roles:cow':
    - match: grain
    - cow
  'roles:fortuneteller':
    - match: grain
    - fortune
  'roles:locomotive':
    - match: grain
    - locomotive
```

As you can see we can match grains when targeting our pillar. That's pretty cool. We use the `roles` grain to match the roles.

Be careful, because matching grains in the pillar comes with a caveat. When you use grains to match because they can be configured on the minion itself which means a minion can send a grain to get data it shouldn't get. In our case we're just declaring packages to install, so that's not so harmful.

## Configuring the master

Next we'll reconfigure our salt master to read from our newly created pillar directory. For that we have to mount the pillar directory with docker compose and configure the master to read the pillar files from that directory. We begin by changing the master configuration.

We only need to add a new configuration variable called `pillar_roots` which is similar to the preexisting `file_roots`. It tells in what folder we keep the pillar files for a particular environment (in our case `base`).

We will mount the directory with the pillar data at `/srv/pillar`. Update the `master.config` file and add the `pillar_root` and point the base environment to `/srv/pillar`, so your `master.config` file should look like this:

```yaml
file_roots:
  base:
    - /srv/salt/
pillar_roots:
  base:
    - /srv/pillar
```

Then we need to change our `docker-compose.yml` file to mount our new `pillar` directory at `/srv/pillar`.

```yaml
version: "3"
services:
  salt:
    image: salt-master
    volumes:
      - ./master.config:/etc/salt/master
      - ./states:/srv/salt
      - ./pillar:/srv/pillar
  alpine-minion:
    image: alpine-minion
    volumes:
      - ./alpine.grains:/etc/salt/grains
  ubuntu-minion:
    image: ubuntu-minion
    volumes:
      - ./ubuntu.grains:/etc/salt/grains
```

That's it. That's all you need.

# Do we have pillar data?

Let's have a look at the pillar data we've created. Again, we want to start with a clean slate so we stop our docker containers and remove them (press y when prompted if we're sure we want to remove them). Then we spin them back up again and attach to the bash shell of salt master:

    docker-compose stop
    docker-compose rm
    docker-compose up -d
    docker-compose exec salt bash

Once we're attached to the salt master, we accept the salt keys for the new minions (press enter to accept them when prompted):

    salt-key -A

To have a look at the pillar data we just created type the following command

    salt '*' pillar.items

You should now see the packages listed based on the roles of the machines. Something like this:

```
cow-e9e01577bbd8:
    ----------
    packages:
        ----------
        cowsay:
            A program with a cow
        fortune:
            A program for a fortune teller
locomotive-023378ead2d9:
    ----------
    packages:
        ----------
        fortune:
            A program for a fortune teller
        sl:
            Tech Model Railroad Club
```

We still aren't using this pillar data. To do that properly, we'll have to dive into the next part of this series: *templating* (I've teased it a little bit in this article though).

Little by little we'll get a better understanding of many of Salt's features.

---

<a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-sa/4.0/80x15.png" /></a> This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>.

*Cover image by Yair Aronshtam: [Salt pillar at the Dead Sea, Israel](https://commons.wikimedia.org/wiki/File:Salt_pillar_at_the_Dead_Sea,_Israel_(35253925685).jpg)*