Using Let's Encrypt with the Puppet Enterprise console
Tags: puppet, ssl, nginx, tutorial
Published: 2023-03-30 22:28:40

Had an itch I've been meaning to scratch for a while.  I build my [Puppet](https://puppet.com) environment using [Terraform](https://terraform.io), which makes it nice and easy to tear things down and rebuild them.  That is great, but it does leave me with an issue when it comes to the console SSL certificates.

Puppet will generate self-signed certs for the console, which work *fine*, but it was always a niggle that the certs couldn't be automagically coaxed into being valid.

Since moving over to [kubernetes](https://github.com/albatrossflavour/k3s-cluster) for my home lab, I've come to expect managed SSL certificates for any public facing services, without me having to do anything.

Finally set some time aside to look at the options and thought I'd publish the details of the journey as well as where I ended up.

# Step 1 - Let's Encrypt

Obviously I wasn't going to reinvent the wheel.  If I wanted to manage and use Let's Encrypt on a Puppet Enterprise server, I'd be using the [Let's Encrypt module](https://forge.puppet.com/modules/puppet/letsencrypt).  The module makes it very simple to get the relevant packages installed and configured.

```puppet
class { 'letsencrypt':
  config     => {
    email  => 'certs@albatrossflavour.com',
  },
  config_dir => '/etc/letsencrypt',
}
```

One of the beauties of this module is that it also sets up a `cron` job to renew the generated certs, so you don't need to keep an eye on it.

Classify your server with that and you'll end up with `certbot` and it's dependencies.  Great start and feeling confident about the future!

Then we need to generate a certificates:

```puppet
letsencrypt::certonly { 'puppet.gcp.albatrossflavour.com':
  domains       => ['puppet.gcp.albatrossflavour.com'],
  manage_cron   => true,
  plugin        => 'webroot',
  webroot_paths => ['/var/www],
}
```

Annnnnnnd that's where things start to go south.

When you request a cert, the most simple method of validating you are who you say you are is to have a web server on the host respond to a query sent to port `80`.  Sure we say, easy, couldn't take much to do!

By default, the Puppet Enterprise `nginx` config does a `301` redirect for `http` requests to `https`.  This means any queries from Let's Encrypt to validate the requests will end up as `https` requests and the cert request will fail.

# Step 2 - `nginx`

Let me introduce you to `puppet_enterprise::profile::console::proxy::http_redirect::enable_http_redirect`.  This parameter controls if that redirect is in place.  So step .... 6(?) was to use `hiera` to disable the `http` redirect:
```yaml
❯ cat data/role/role::pe::master.yaml
puppet_enterprise::profile::console::proxy::http_redirect::enable_http_redirect: false
```
Once we run this through a `puppet` run, the redirect gets removed!  Score!

Only problem is, without the redirect, the `nginx` server doesn't listen on port `80`.  OK, we can fix that easily.  We could use the `pe_nginx::directive` type, but I found it to be a bit of an overkill for what I needed.  Instead I opted for a simple template:

```puppet
  file { '/etc/puppetlabs/nginx/conf.d/certs.conf':
    ensure  => file,
    owner   => 'root',
    group   => 'root',
    mode    => '0644',
    content => template("${module_name}/cert_vhost.conf.erb"),
    notify  => Exec['pe_nginx'],
  }
```

and the template is just:

```erb
server {
  listen       80;
  server_name  <%= @fqdn %>;
  index        index.html;
  location /.well-known {
    root /var/www;
  }
  location / {
    return 301 https://$server_name$request_uri;
  }
}
```

(Sorry, still using `erb`, I will **at some point** rewrite everthing in `epp`)

The template keeps the `301` redirect in place for anything *other* than a request to `/.well-known`, which is where Let's Encrypt looks for the validation info.

Run this through and the `nginx` vhost gets created.  However the `letsencrypt::certonly` call still fails on the first run.  The `notify` to the `pe_nginx` service, which is done when we create the `cert_vhost.conf`, doesn't happen in the order we need.  The Let's Encrypt module is trying to get a response *before* we've setup the vhost.  Now this would work on subsequent runs, but **Golden Rule #1** is to make sure, whenever possible, that you get a clean puppet run in one pass.  Plus this was a challenge.

Before I worked on fixing that, I wanted to make sure I could make the rest of it work.  Let's sum up where we are:

* I've got the Let's Encrypt client installed and configured
* I've got a new `nginx` vhost running that allows the Let's Encrypt web validation queries through and redirects any other `http` traffic.
* After a couple of puppet runs, I've got valid SSL certs in `/etc/letsencrypt`

# Step 3 - The console certs

It'd been a while since I played with the Puppet console SSL certs, so I checked in with [the source of truth](https://www.puppet.com/docs/pe/latest/use_a_custom_ssl_cert_for_the_console.html), which outlines the steps we need to go through to use custom SSL certs with the console:

1. Retrieve the custom certificate and private key.
1. Move the certificate to `/etc/puppetlabs/puppet/ssl/certs/console-cert.pem`, replacing any existing file named `console-cert.pem`.
1. Move the private key to `/etc/puppetlabs/puppet/ssl/private_keys/console-cert.pem`, replacing any existing file named `console-cert.pem`.

So we can just create a `file` resource that takes the Let's Encrypt cert/key and places them into the console SSL directory structure.

```puppet
  file { '/etc/puppetlabs/puppet/ssl/certs/console-cert.pem':
    ensure    => file,
    owner     => 'pe-puppet',
    group     => 'pe-puppet',
    links     => 'follow',
    mode      => '0640',
    source    => "/etc/letsencrypt/live/${facts['puppet_server']}/cert.pem",
    backup    => '.puppet_bak',
    notify    => Service['pe-nginx'],
    subscribe => Letsencrypt::Certonly[$facts['puppet_server']],
  }

  file { '/etc/puppetlabs/puppet/ssl/private_keys/console-cert.pem':
    ensure    => file,
    owner     => 'pe-puppet',
    group     => 'pe-puppet',
    links     => 'follow',
    mode      => '0644',
    source    => "/etc/letsencrypt/live/${facts['puppet_server']}/privkey.pem",
    backup    => '.puppet_bak',
    notify    => Service['pe-nginx'],
    subscribe => Letsencrypt::Certonly[$facts['puppet_server']],
  }
```

Sure enough, when I try this on the Puppet server, life is good and we have a console with valid SSL certs.

Time to crack open a bottle of red and tick an item off my to do list.

Damn it, **Golden Rule #2** raises it's head.  It's not finished until you know it works fine from scratch... ***WITHOUT*** breaking **Golden Rule #1**.

# Step 4 - Rebuild

Nuked the Puppet server and rebuilt it.  Lots of failures on the first run (we kinda expected that), however they didn't go away.  The `nginx` service never gets restarted as the various dependencies can be resolved, but not in the way we need.

This is the *fun* (?) of declarative configuration management.  I can only ***manage*** things once, which includes only being able to `notify` the `pe-nginx` service once.  Then the compiler will figure out when the service is restarted, based on all of the dependencies in the catalog. 

I played around with a lot of options, some *waaaaay* hackier than I wanted to go with.  Plus this was becoming a fun exercise.

All I needed to be able to do was to inject a restart of the `pe-nginx` service twice in a single run.

To get there, I bent a few of the future **Golden Rules** and came up with:

```puppet
exec { 'restart_nginx':
  command     => '/bin/systemctl restart pe-nginx',
  refreshonly => true,
}
```

This `exec` resource will allow me to do a restart of `pe-nginx` outside of, and before, the `notify => Service['pe-nginx']` parameters.

Yes, it's a hack, but it's a hack on the side of the angels.  Not only will it solve the issue of the process not working at all, but it will also allow the certs to be generated, and installed, in a single run.

# Step 5 - Quick Robin, to the `pdk` mobile

I did a fair bit of testing and found the end result to be far more reliable and useful than I thought it would be.

I first created a fully parameterised profile to manage the certs.  Once I got that working, it was a no-brainer to create a module to share the love.

That's how I ended up with the [pe_console_letsencrypt](https://forge.puppet.com/modules/albatrossflavour/pe_console_letsencrypt/readme) module.  You can also just [check out the code](https://github.com/albatrossflavour/pe_console_letsencrypt/blob/main/manifests/init.pp).

I've done a fair bit of testing, but I'm certainly not saying it's bulletproof.

# Step 6 - [Profit](https://en.wikipedia.org/wiki/Gnomes_(South_Park))?

I'm still scratching my head to think of another way I could get the certs generated and in place in a single run **without** the `exec` hack.  That being said, I've had hacks a lot worse go live in the past!


# Note

While researching the links for this post, I came across a [blog less frequently updated than mine](http://blog.graypockets.com/2017/10/lets-encrypt-puppet-enterprise-console.html) which has a post from 2017 showing a similar way of achieving the same thing, but without the hack to make it work in a single run.