Encrypt Status Communication Messages in Consul and Nomad
Tags: hashicorp, nomad, consul
Published: 2020-06-08 18:12:48

[Consul](https://learn.hashicorp.com/consul), the service management and service discovery tool, and [Nomad](https://nomadproject.io/), the job scheduler, are central tools in my infrastructure at home project. Both tools run as daemons on all nodes. They exchange data with the [Gossip](https://www.consul.io/docs/internals/gossip.html) protocol, and API endpoints via HTTP and RPC. There is one last requirement to fulfill: Securing these communications by encrypting all traffic. The particular Infrastrucure Security requirements are:

* INFSEC01: Secure Consul internal communication
* INFSEC02: Secure Nomad internal communication

This article shows you the essential steps to secure Nomad and Consul. There are excellent documentation on the [Consul homepage](https://learn.hashicorp.com/consul/security-networking/certificates) and [Nomad homepage](https://learn.hashicorp.com/nomad/transport-security/enable-tls#overview), my approach compresses the rich information from these pages for you.

_This article originally appeared at [my blog](https://admantium.com/blog/ic12_encrypt_consul_and_nomad/)_.

## Consul: Encrypt the Gossip Protocol

The Gossip protocol is used to constantly exchange information about your cluster. Enabling encryption is a two-step process: Generate a secret, and then set the nodes to use encrypted communication.

1. On any node in your cluster, run the command `consul keygen` and note down the password
2. On all nodes, add the following configuration options, then restart consul
  ```yaml
  {
    encrypt = "SECRET"
    encrypt_verify_incoming = false
    encrypt_verify_outgoing = false
  }
   ```
3. On all nodes, set `encrypt_verify_outgoing = true`, then restart consul
  ```yaml
  {
    encrypt = "SECRET"
    encrypt_verify_incoming = false
    encrypt_verify_outgoing = true
  } 
  ``` 
4. On all nodes, set `encrypt_verify_incoming = true`, then restart consul
  ```yaml
  {
    encrypt = "SECRET"
    encrypt_verify_incoming = true
    encrypt_verify_outgoing = true
  }  
  ```

Now all consul traffic is encrypted.

## Consul: TLS Encryption

To enable TLS encryption, the following steps are necessary:

1. Generate root certificate
2. Generate server and client certificates
3. Enable TLS encryption on the servers
4. Enable TLS encryption on the clients

Let’s detail these steps.

### Generate Certificates

TLS certificates all start with a valid root certificate, then intermediate certificates, and finally the certificate that is used by your servers/clients. You can create your own root certificate with tools like OpenSSL, or you can use Nomads built-in commands. We will do the later.

First, we generate the root certificate as follows:

```shell
>> consul tls ca create
==> Saved consul-agent-ca.pem
==> Saved consul-agent-ca-key.pem
```

Second, we generate the server certificates. This certificate will be signed with the key `Saved consul-agent-ca-key.pem`. Be careful to set the correct name of your data center.

```shell
>>consul tls cert create -server -dc=infra

==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
==> Saved infra-server-consul-0.pem
==> Saved infra-server-consul-0-key.pem
```

### Enable TLS Encryption

Similar to the Gossip encryption, we need to add new configuration options and restart the server/clients one at a time. Perform the following steps.

1. Server: Copy the certificates and add the following configuration options.
   ```yaml
   {
     "verify_incoming": false,
     "verify_outgoing": false,
     "verify_server_hostname": false,
     "ca_file": "consul-agent-ca.pem",
     "cert_file": "dc1-server-consul-0.pem",
     "key_file": "dc1-server-consul-0-key.pem"
   }
   ```
2. Restart the server nodes - be sure that they are running before continuing.
3. Clients: Copy the certificates, add the following configuration options.
   ```yaml
   {
     "verify_incoming": true,
     "verify_outgoing": true,
     "verify_server_hostname": true,
     "ca_file": "consul-agent-ca.pem",
     "cert_file": "dc1-server-consul-0.pem",
     "key_file": "dc1-server-consul-0-key.pem"
   }
   ```
4. Restart the client nodes
5. Server: set the following configuration options
    ```yaml
    {
      "verify_incoming": true,
      "verify_outgoing": true,
      "verify_server_hostname": true
    }
    ```  
6. Restart the server nodes

Congratulations! All traffic within your consul cluster is now encrypted.

## Nomad: Encrypt the Gossip Protocol

Encrypting the Gossip protocol in Nomad is simpler because you only need to modify the servers. The steps are similar: Create a secret, change the config file, then restart the servers.

1. Create a secret with `nomad operator keygen`
2. Add this secret to the server’s configuration files, and restart all servers
  
```yaml
server {
  encrypt = SECRET
}
```  

## Nomad: TLS Encryption

The encryption process in Nomad is very similar to Consul: You create certificates, configure and restart the server, and then configure and restart the clients.

### Generate Certificates

The Nomad documentation suggests to use the tool [cfssl](https://cfssl.org/). I could not get it working, so I reused the Consul commands.

First, generate the root certificates. Be sure to set the correct `domain`

```shell
consul tls ca create -domain=nomad -name-constraint
==> Saved nomad-agent-ca.pem
==> Saved nomad-agent-ca-key.pem
```

Second, generate certificates for the server. Be sure to set the correct `domain`. Also, unless configured otherwise, the nomad agents will talk to the servers by using the datacenter `global` - add this as a flag to the command.

```shell
>> consul tls cert create -server -domain nomad -dc=global

==> Saved global-server-nomad-1.pem
==> Saved global-server-nomad-1-key.pem
```

Third, generate certificates for the clients.

```shell
>> consul tls cert create -client -domain nomad -dc=global

==> Using nomad-agent-ca.pem and nomad-agent-ca-key.pem
==> Saved global-client-nomad-1.pem
==> Saved global-client-nomad-1-key.pem
```

### Enable TLS Encryption

These steps are similar to configuring TLS for Nomad.

1. Server: Copy the certificated, then add the following configuration options:
 ```yaml
tls {
  http = true
  rpc  = true

  ca_file = "certs/nomad-agent-ca.pem"
  cert_file = "certs/global-server-nomad-0.pem"
  key_file = "certs/global-server-nomad-0-key.pem"

  verify_server_hostname = true
  verify_https_client    = false
 }
 ```
2. Restart the server
3. Clients: Copy the certificates, then add the following configuration options:
```yaml
  tls {
    http = true
    rpc  = true

    ca_file = "certs/nomad-agent-ca.pem"
    cert_file = "certs/global-client-nomad-0.pem"
    key_file = "certs/global-client-nomad-0-key.pem"

    verify_server_hostname = true
    verify_https_client    = true
  }
  ```
4. Restart all clients

Congratulations, all Nomad traffic is now encrypted.

One more thing: If you want to access the Nomad HTTP UI, and have `verify_https_client = true`, you need to add the client certificate to your browser. If you are using a proxy server, you also need to add the Hostname with which you access Consul as a Subject Alternative Name (SAN) when creating the client certificates.

## Conclusion

This article showed how to enable complete encryption for all client/server communication in Nomad and Consul. Gossip encryption protects the status updates between the nodes. It is enabled by setting a secret key and by adding configuration options to use encrypted incoming/outgoing messages. TLS encryption secures all HTTP and RPC endpoints of the clients/servers. You first create a root certificate and the client/server certificates. Then you add configuration options to all nodes, starting with the servers, continuing with the clients. After each change you do a rolling update of all nodes. With the final update, all traffic is encrypted.

You can verify the encryption by starting an agent from the command line. Take a look at the last line in the following log.

```shell
/usr/local/bin/consul agent -config-dir=/etc/consul/ -bind=192.168.2.201 -client=192.168.2.201
==> Starting Consul agent...
           Version: 'v1.7.1'
           Node ID: '6f859439-c48f-f3af-5282-26b9f9dccad2'
         Node name: 'raspi-3-1'
        Datacenter: 'infra' (Segment: '')
            Server: false (Bootstrap: false)
       Client Addr: [192.168.2.201] (HTTP: 8500, HTTPS: -1, gRPC: -1, DNS: 8600)
      Cluster Addr: 192.168.2.201 (LAN: 8301, WAN: 8302)
           Encrypt: Gossip: true, TLS-Outgoing: true, TLS-Incoming: true, Auto-Encrypt-TLS: true
```