Encrypted Remote Terraform State with Postgres and LUKS
Tags: devops, security, postgres, terraform
Published: 2024-01-03 12:33:07

Either if you have already decided to use a PostgreSQL remote state backend for Terraform, or stumbled on this guide out of curiosity, let me help you set everything up in a more secure manner 😁!

> edit 2024-01-20: Added a tip on workspaces.

---

## Table of Contents

* [Why?](#why)
* [Prerequisites](#prerequisites)
* [What Versions of Software are Used?](#what-versions-of-software-are-used)
* [Set Up a Postgres Server](#set-up-a-postgres-server)
 * [Prepare Storage](#prepare-storage)
 * [Configure Postgres](#configure-postgres)
* [Configure a Terraform Client](#configure-a-terraform-client)
* [What to Do After a Reboot?](#what-to-do-after-a-reboot)
* [The Result](#the-result)

## Why?
When you work with Terraform by yourself, don't store sensitive data in state, and have a couple of projects, it's OK to use the [*local*](https://developer.hashicorp.com/terraform/language/settings/backends/local) state backend.

But as soon as the number of projects increases, you start working in a team, and suddenly realize there're things like passwords and API keys in your state file, uhmm... Things start to get tricky 🫠.

A number of questions arise:
* How do I protect sensitive data stored in state?
* How can I collaborate with my colleagues? The state must be somehow shared in real time.

That's where a remote state backend comes in handy. It can be encrypted, allows to team up on the same projects simultaneously by having a SSOT for state, and supports locking to prevent collisions.

### But Why PostgreSQL?
It's quicker and easier to set up than other remote state backends. It doesn't require a ton of resources. The thing just works, yet highly flexible.

## Prerequisites
What you need is a single server and a way to obtain a TLS certificate for it. For me, it's a VM on Proxmox and a self-signed certificate.

> If you want to use self-signed certs for this, check out my old article on how to generate them via openssl 👇.

> {% embed https://dev.to/deathroll/trusted-self-signed-tls-certificates-for-dummies-w-thorough-explanations-included-da7 %}

Play around and find the exact amount of resources suitable for your specific case. Maybe less cores and RAM, more storage, etc.

Ubuntu was chosen as one of the most popular server distros. The configuration can be adjusted for any OS.

| OS | CPU | RAM | Disk |
| --- | --- | --- | --- |
| Ubuntu Server | 2 | 2 GB | 8 GB + 1 GB |

> 📒 NOTE: For this demo, 1 GB disk is dedicated for a Postgres database cluster. Be aware, in production such disk size may be inappropriately small.

## What Versions of Software are Used?
* Ubuntu 22.04.3 LTS
* PostgreSQL 14.10
* cryptsetup 2.4.3
* Terraform v1.5.7 (on a separate host)

{% details What is LUKS, btw 🤔? %}
Check out *cryptsetup*'s README to learn more: https://gitlab.com/cryptsetup/cryptsetup/
{% enddetails %}

## Set Up a Postgres Server
### Prepare Storage
#### Create a LUKS Device
First, we need to find out our disk's block device name. In my case, it's */dev/sdb*.

```console
$ lsblk
NAME    MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0     7:0    0  63.5M  1 loop /snap/core20/2015
loop1     7:1    0 111.9M  1 loop /snap/lxd/24322
loop2     7:2    0  40.8M  1 loop /snap/snapd/20092
sda       8:0    0     8G  0 disk 
├─sda1    8:1    0   7.9G  0 part /
├─sda14   8:14   0     4M  0 part 
└─sda15   8:15   0   106M  0 part /boot/efi
sdb       8:16   0     1G  0 disk 
sr0      11:0    1     4M  0 rom  
```

Now, create a new partition table and a partition on this block device.

I prefer to use GPT instead of MBR even for smaller disks most of the time.

```console
# fdisk /dev/sdb
```

{% details fdisk example %}
```console
Welcome to fdisk (util-linux 2.37.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0x20d6b116.

Command (m for help): g
Created a new GPT disklabel (GUID: 0B0CD5AB-337A-CF47-A2BF-F377A158D0FE).

Command (m for help): n
Partition number (1-128, default 1): 
First sector (2048-2097118, default 2048): 
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-2097118, default 2097118): 

Created a new partition 1 of type 'Linux filesystem' and of size 1023 MiB.

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
```
{% enddetails%}

When the partition is created, LUKS can be initialized using *cryptsetup*.

Considering that block device naming in Linux is not persistent between reboots, we won't rely on device names. Instead, we'll use labels for both a LUKS partition and a filesystem on it later.

```console
# cryptsetup luksFormat /dev/sdb1 --type luks2 --label tfstate-pg-luks
```

> 👆 LUKS version 2 is used because it supports labels. Use any label of your liking. Don't forget to adjust all the commands and configurations.

#### Configure crypttab and fstab
Opening an encrypted disk partition requires providing a password or a key file. We could automate this process, but as always, there's a trade-off between security and convenience.

*/etc/crypttab* and */etc/fstab* — these two files can greatly simplify the experience of opening and mounting an encrypted disk manually after reboots.

##### crypttab
Open */etc/crypttab* with your editor of choice and add the line below:

```crypttab
tfstate-pg-luks LABEL=tfstate-pg-luks   none    luks,noauto
```

> 📒 NOTE: 👆 The `noauto` option prevents automatic opening and mapping of the partition.

##### fstab
Do the same thing with */etc/fstab*.

> 📒 NOTE: Don't forget to change the filesystem name here if you don't use Btrfs 👇.

You can use a single mount point, and mount the device straight to */var/lib/postgres*. It's all a matter of preference. I like my devices to be mounted under */media*, and then bind-mounted to other places.

```fstab
LABEL=tfstate-pg        /media/tfstate-pg       btrfs   defaults,noauto 0       0
/media/tfstate-pg       /var/lib/postgres       none    bind,noauto     0       0
```

#### Create a Filesystem
Choose whatever filesystem you're comfortable with. Don't forget to update *fstab* accordingly.

To create a filesystem on the encrypted disk partition, connect to it first.

```console
# cryptdisks_start tfstate-pg-luks
```

Create the filesystem itself with the label specified in *fstab*:

```console
# mkfs.btrfs --label tfstate-pg 
```

#### Mount and Prepare the Filesystem
At this point, we're halfway there.

Create directories for the mount points specified in */etc/fstab*, and mount the LUKS-encrypted partition.

```console
# mkdir /media/tfstate-pg /var/lib/postgres
# mount /media/tfstate-pg/
# mount /var/lib/postgres/
```

To check if everything is appropriately mounted, issue *findmnt*.

```console
$ findmnt
```

{% details Example output %}
```console
$ findmnt
***OUTPUT TRUNCATED***
├─/var/lib/postgres                           /dev/mapper/tfstate-pg-luks btrfs       rw,relatime,space_cache=v2,subvolid=5,subvol=/
└─/media/tfstate-pg                           /dev/mapper/tfstate-pg-luks btrfs       rw,relatime,space_cache=v2,subvolid=5,subvol=/
```
{% enddetails %}

The directory where the database cluster will be located must be owned by the *postgres* user and group.

```console
# chown -R postgres: /var/lib/postgres
```

### Configure Postgres
#### Install
```console
# apt install postgresql
```

#### Disable and Stop the Service
Since the database cluster will be located on the encrypted partition that must be manually connected after a reboot, the Postgres service should be disabled.

```console
# systemctl disable --now postgresql
```

> 💡 TIP: 👆 The `--now` option stops the daemon.

#### Copy a TLS Certificate and Key
Encrypted storage is useless if the network traffic between the server and its remote clients is not encrypted.

Grab a TLS certificate and its key, and copy them to your server. You can place them wherever you want — filenames don't matter either, as long as the *postgres* user has access.

```console
# mv deathroll-internal.pem /etc/ssl/certs/
# mv deathroll-internal-key.pem /etc/ssl/private/
# chown postgres: /etc/ssl/private/deathroll-internal-key.pem
```

{% details File permissions %}
```console
$ stat --printf '%n\nMode: %a (%A)\nOwner: %u (%U)\nGroup: %g (%G)\n' /etc/ssl/certs/deathroll-internal.pem
/etc/ssl/certs/deathroll-internal.pem
Mode: 644 (-rw-r--r--)
Owner: 1000 (t_chuchkanov)
Group: 1000 (t_chuchkanov)
$ sudo -u postgres stat --printf '%n\nMode: %a (%A)\nOwner: %u (%U)\nGroup: %g (%G)\n' /etc/ssl/private/deathroll-internal-key.pem
/etc/ssl/private/deathroll-internal-key.pem
Mode: 600 (-rw-------)
Owner: 113 (postgres)
Group: 122 (postgres)
```
{% enddetails %}

#### Configure
##### Settings
Instead of editing the config file manually, it's easier and quicker to use *pg_conftool*.

```console
# pg_conftool set ssl_cert_file /etc/ssl/certs/deathroll-internal.pem
# pg_conftool set ssl_key_file /etc/ssl/private/deathroll-internal-key.pem
# pg_conftool set listen_addresses localhost,tfstate-pg.deathroll.internal
```

To check if all the values are correct:

```console
$ for P in listen_addresses ssl_cert_file ssl_key_file; do pg_conftool show $P; done
```

##### Access Control
Configure access control for your Postgres installation. Append the line below to the */etc/postgresql/14/main/pg_hba.conf* file:

> 📒 NOTE: 👆 Pay attention to the Postgres version. It may differ in your case.

```pg_hba.conf
hostssl all             all             172.17.0.0/24            scram-sha-256
```

> 📒 NOTE: 👆 Change the allowed network to whatever you need. It may even be `0.0.0.0/0`.

#### Start
```console
# systemctl start postgresql.service
```

If your configuration is correct, the Postgres systemd unit must be in the *running* state, the Postgres cluster *online*, and the daemon listening on the port *5432*. Congrats 🥳.

```console
$ systemctl list-units postgres*.service
```

{% details Example %}
```console
$ systemctl list-units postgres*.service
  UNIT                       LOAD   ACTIVE SUB     DESCRIPTION               
  postgresql.service         loaded active exited  PostgreSQL RDBMS
  postgresql@14-main.service loaded active running PostgreSQL Cluster 14-main

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
2 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
```
{% enddetails %}


```console
$ pg_lsclusters
```

{% details Example %}
```console
$ pg_lsclusters
Ver Cluster Port Status Owner    Data directory              Log file
14  main    5432 online postgres /var/lib/postgresql/14/main /var/log/postgresql/postgresql-14-main.log
```
{% enddetails %}


```console
# ss -tlpn | grep postgres
```

{% details Example %}
```console
# ss -tlpn | grep postgres
LISTEN 0      244      172.17.0.40:5432      0.0.0.0:*    users:(("postgres",pid=19728,fd=7))       
LISTEN 0      244        127.0.0.1:5432      0.0.0.0:*    users:(("postgres",pid=19728,fd=6))       
LISTEN 0      244            [::1]:5432         [::]:*    users:(("postgres",pid=19728,fd=5))
```
{% enddetails %}

Almost there! The last few steps of Postgres configuration are left.

#### Configure a Database
Now we need to:
1. Change the *postgres* user password
1. Create a Postgres user for Terraform clients
1. Create a database for Terraform state
1. Grant some privileges on this database to the Terraform user

Let's roll! 

```console
$ sudo -u postgres psql
```

> 📒 NOTE: 👆 Depending on your *$PWD*, psql may throw a warning about its inability to access this directory. Ignore it.

```console
postgres=# ALTER USER postgres ENCRYPTED PASSWORD '******';
postgres=# CREATE USER terraform ENCRYPTED PASSWORD '******';
postgres=# CREATE DATABASE terraform_backend;
postgres=# GRANT CREATE,CONNECT,TEMPORARY ON DATABASE terraform_backend TO terraform;
```

{% details Example %}
![Postgres shell example](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xk9ce7l8ebt2i5r89eu6.png)
{% enddetails %}

#### Configure a Firewall
Don't forget to configure a firewall on your system.

```console
# ufw allow 22,5432/tcp
# ufw enable
```

To check the status:

```console
# ufw status
```

That's it! It's time to test the connection from another host and configure a Terraform client 😎.

#### Check Connection from Another Host
Install *psql* and issue the command below:

```console
$ psql postgres://terraform@tfstate-pg.deathroll.internal/terraform_backend
```

> 💡 TIP: The connection string scheme is: `postgres://<user>@<host>/<database_name>`

<br/>

> 💡 TIP: On Ubuntu, the package is called "postgresql-client".

You can quit out of the psql client prompt right after it establishes the connection, because there's nothing to do in it apart from testing connectivity.

{% details Example %}
```console
$ psql postgres://terraform@tfstate-pg.deathroll.internal/terraform_backend
Password for user terraform: 
psql (15.5 (Debian 15.5-0+deb12u1), server 14.10 (Ubuntu 14.10-0ubuntu0.22.04.1))
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, compression: off)
Type "help" for help.

terraform_backend=> \dn
  List of schemas
  Name  |  Owner   
--------+----------
 public | postgres
(1 row)

terraform_backend=>
```
{% enddetails %}

## Configure a Terraform Client
> 💡TIP: When you need to work with multiple terraform projects, use [workspaces](https://developer.hashicorp.com/terraform/cli/workspaces), since all of the freshly initialized projects have only the *default* workspace. Hence, there will be remote state conflicts, if a single workspace is used for several independent states.

In a Terraform project's main file, create a `backend` block inside of `terraform`. Provide a *string* value for *conn_str*.

It is really [that simple](https://developer.hashicorp.com/terraform/language/settings/backends/pg#example-configuration).

```hcl
terraform {
  backend "pg" {
    conn_str = "postgres://terraform@tfstate-pg.deathroll.internal/terraform_backend"
  }
}
```

Before issuing any Terraform commands, export the Postgres password as an environment variable: 

```console
$ read -s PGPASSWORD
$ export PGPASSWORD
```

For a fresh project, `terraform init` is sufficient. For an existing project, add the `-reconfigure` flag to this command.

```console
$ terraform init

Initializing the backend...

Successfully configured the backend "pg"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Finding telmate/proxmox versions matching "2.9.11"...
- Installing telmate/proxmox v2.9.11...
- Installed telmate/proxmox v2.9.11 (unauthenticated)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

╷
│ Warning: Incomplete lock file information for providers
│ 
│ Due to your customized provider installation methods, Terraform was forced to calculate lock file checksums locally for the following providers:
│   - telmate/proxmox
│ 
│ The current .terraform.lock.hcl file only includes checksums for linux_amd64, so Terraform running on another platform will fail to install these providers.
│ 
│ To calculate additional checksums for another platform, run:
│   terraform providers lock -platform=linux_amd64
│ (where linux_amd64 is the platform to generate)
╵

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
```

> If you want to learn how to use Terraform providers in isolated environments, I have a yet another article for you 😉.
> {% embed https://dev.to/deathroll/using-terraform-providers-in-isolation-17n0 %}

## What to Do After a Reboot?
1. Open and map the LUKS-encrypted partition
1. Mount its filesystem
1. Start the Postgres service

As simple as three to four commands!

![A set of commands to do after a reboot](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ox23q4ey42zzcs3eynv1.png)

## The Result
As the result, we have a PostgreSQL server with a database cluster located on a LUKS-encrypted partition.

This Postgres installation is used for storing remote Terraform state, and the network traffic between it and its remote clients is encrypted.

Sounds great, innit 🤌?

![A drawn diagram representing the resulting architecture](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/w6tdm79o9p8du0df3ob4.png)

---

Scroll down to the comments section to get some enhancement tips.