This is how you want to manage your Terraform modules
Published: 2023-01-08 07:21:57

---
title: "This is how you want to manage your Terraform modules"
coverAlt: "Photo by Kadarius Seegars on Unsplash"
published: true
canonical_url: https://devopsian.net/posts/terraform-modules-management/
description: "What if I told you there is a way to manage all your private terraform modules, in a mono-repo, with independent versioning, without using git tags? After researching for a proper      open-source tool, I found the right one for the job."
keywords: ["terraform", "terraform registry", "terraform modules", "terraform private registry", "terraform private modules", "infrastructure as code"]
---
What if I told you there is a way to manage all your private terraform modules, in a mono-repo, with independent versioning, without using git tags? After researching for a proper open-source tool, I found the right one for the job.

## Intro
_A private registry is needed._

I looked for a way to manage private terraform modules like public ones.
In my company, we write tailor-maid modules that describe our infrastructure. These modules have to be private.
Terraform [recommends](https://developer.hashicorp.com/terraform/registry/modules/use#module-versions) each module has its own git repository, yet, this has the burden of managing and syncing        multiple repositories.

The second option suggested is to use a mono-repo, and reference the module's version using [git tags](https://developer.hashicorp.com/terraform/language/modules/sources#selecting-a-revision). At    first, this seemed alright,
but there was a drawback — you had to give up the terraform version syntax. You no longer can use the convenient `version = ~> 1.0.0` module parameter.

It seemed to me that there had to be a better way.

## The problem
When it comes down to managing my infrastructure, I prefer the mono-repo approach. Since our terraform modules are small configuration blocks,
it makes sense. It is much easier to find a module in a single repository rather than searching multiple repositories for a module.
Our module release workflow looked something like this

- Our repository was structured with sub-directories per module
- We used Git tags for module versioning in the form of _**moduleName-vX.Y.Z**_
- Updates to module versions were _hand-delivered_ to clients (even minor patches)

Our root modules (where we execute terraform plan and apply) reference modules using git tags:

```
terraform {
  source = “https://github.com/my-org/my-repo?ref=s3-v1.0.0"

  .. module inputs ..
}
```

This has worked well for a while. But once we had released a new module version and wanted to use it, there was no convenient way to apply that.

We had to go through our root modules using this module and update their references. (We could write a simple script, but we decided not to. More on that later)

The problem we had with this approach is, you can’t use the `version` argument in the terraform block.
This argument lets you specify a [range of acceptable versions](https://developer.hashicorp.com/terraform/language/expressions/version-constraints) instead of a hard-coded one.

For example, you can provide version constraints such as `version = “~> 1.2.0, < 2.0”` which allows incrementing the “patch” automatically every time you run `terraform init`.
 No need to manually update a patch release, and no need to write a bash script. Terraform can manage that reliably for us,
with a better API, if we can only discover a way to use this feature.

Manually managing our tags was **chaotic**. It is not an easy task to manage independent module versions in a mono-repo using the Git tags approach.

A simpler approach is to release a version for the whole repository. That means every new release includes all the modules together.
This couples the modules into a single artifact, which is easier to manage, but at the cost of development velocity.
You cannot release minor patches just for your module, no matter how small the code change is.

## Towards a better future
The structure and workflows we had in place worked; just as not as well or easily as we wanted to.
What we needed was to be able to use terraform versioning syntax, while maintaining our mono-repo.
To achieve that, we need to treat our terraform _modules as artifacts_; something we can archive, version, and release independently.

After doing some research, we found an open-source project called **[Terralist.](https://github.com/valentindeaconu/terralist)**

Terralist is a private Terraform registry for providers and modules following the published HashiCorp protocols.
It provides a secure way to distribute your confidential modules and providers. That looked like a project that might help us to solve our problem, so we decided to try it out.

So we maintain our mono-repo as it is. We created a job in our CI system that archives a single terraform module and uploads it to Terralist.
Since it's a private registry, clients need to authenticate to be able to download the modules. After authentication (using terraform login \<terralist-url\>)
we could use our private modules just as we use the public modules. Yes, it means we can use the versioning syntax I was talking about.

 Here is how we use our modules nowadays
```code
 terraform {
     source = “https://my-terralist.com/my-org/s3/aws”
     version = “~> 1.0.0”

     .. module inputs ..
 }
 ```

This client code will retrieve automatic patch updates on every execution from our CI system (if you execute terraform locally, you would need to re-run `terraform init`).
If we make larger code changes to one of our modules, for example, something that might break the API
we would increment the **major or minor** version, so it wouldn't impact our clients.

### Terralist

_Terralist_ has an API that lets you upload a module from a git repository. It means you don't need to archive the module yourself,
just point Terralist to its location.

Terralist will clone the repository and create the artifact for you. Let's walk through an example.

I have a `demo` module, which resides in my `terraform.git` repository under the `modules/demo` directory. In the module itself, we keep a `version.tf`
with the **major and minor** versions of the module. These values change only _when we introduce a change that breaks our existing API_
(e.g, adding a new mandatory parameter without defaults). In such case we update the major or minor **manually**.

A **patch** number in the semver represents a safe change, such as a bug fix or non-breaking changes to the module.
This value is _calculated based on the build number_ of our CI. We don't really care about its value, because the version constraint
applied is in the form of `version = "~> 1.0.0"`. This automatically updates to the latest patch on every execution.

As part of our module build process, we _read_ the values of `version.tf` and append the patch. Then we upload the module to Terralist using this API call:

```bash
 curl -X POST registry.example.com/v1/api/modules/demo/aws/1.0.10/upload \
      -H "Authorization: Bearer x-api-key:$TERRALIST_API_KEY" \
      -d '{ "download_url": "https://github.com/example-org/terraform/archive/refs/heads/master.zip//modules/demo" }'
 ```

The module version is composed of `major.minor` coming from the `version.tf` file, and the `.patch` is the build ID, which is incremented with every run and guaranteed to be unique.

Pay close attention to the **double //** -- this instruct Terralist to:

1. Download an archived repository
2. Extract it locally
3. Make an archive only from the path `modules/demo`
4. Upload it to the registry (basically upload it to S3 and update the registry database)

It's an elegant solution: I tell Terralist where my module is, and what version it is tagged with, and it takes care for everything else.

## Summary
There are various ways to manage your infrastructure code.

It depends on multiple factors, such as team size, how much you are willing to spend on 3rd party tools, and your company policy, to name a few.
As of today, I've been using Terraform for more than 3 years, relying solely on the open-source ecosystem. That means my team and I manage everything related to our infrastructure (it might change   soon, as our infrastructure size has grown).

_Terralist_ was a helpful addition to our stack; it solved a problem we had in our existing workflow, with minimal effort and no code changes to our existing modules.
We just had to upload and version them to the new registry.
Now, we can release each module independently and decide if we want our clients to automatically upgrade their module version.

 **Happy Terraforming.**

For further reading, HashiCorp documentation contains a lot of good information. Check out the links below
- [Terraform Module Blocks](https://developer.hashicorp.com/terraform/language/modules/syntax)
- [Finding and using modules](https://developer.hashicorp.com/terraform/registry/modules)
- [Reuse configuration with modules (tutorial)](https://developer.hashicorp.com/terraform/tutorials/modules)