Migrating to Terraform: a retrospective
Tags: devops, terraform, aws
Published: 2021-04-09 16:17:39

###Summary:
At [15Five](http://15five.com/) we migrated to Terraform in 8 months with a 3-person team. It went smoothly and gave us more confidence in our AWS infrastructure. When it came to setup a new VPC for a customer, the hours we invested returned to us in a speedy and simple setup, a process that would have been a massive pain if we had done it manually. 

###The Stats:
* Lines of code: ~16,000
* Number of commits: 903
* PR's: 514
* The team: Paul (principal engineer), James (devops manager), Caleb (SRE)

###The Timeline:
2019.09 - I played around with Terraform for the first time in a company hackathon.
2019.09 - I used Terraform to do my first AWS change.
2019.11 - James joins team & starts work on terraform. Repo is split into a main terragrunt repo and a terraform repo for our various modules. At this point we only have two modules. CI is setup.
2020.01 - we migrated our first environment
2020.04.28 - preview environment migration attempt 1
2020.04.29 - preview migration attempt 2 ✔️
2020.05 - staging migration
2020.06 - solitary customer VPC migration
2020.06 - production migration

###Takeaways:

####Caleb:
Pain points:
* Having two different repos is a bit awkward. I feel that having a monorepo or more CI would have been better, as the lack of CI feedback when changing code in the modules repo slowed things down. We'll probably be moving to a monorepo in the future.

What went well:
* Hackathons are good for inspiring change.
* Checklists are great. It simply feels good to check things off, for one. It also makes you think of all the items to do ahead of time and makes sure that you don't forget anything. Hospitals have used checklists to massively lower infection rates, [although the results depend on how effectively they are used](https://www.nature.com/news/hospital-checklists-are-meant-to-save-lives-so-why-do-they-often-fail-1.18057). 
* Terraform is great for developing a deep understanding of all the services involved in your infrastructure. By describing your infrastructure in code you develop a mental model of all the components involved, and have a written record of the tiny fiddly settings that might normally be hidden away behind some unintuitive UI icons.
* Migrating production went pretty smooth, that's worth its weight in gold.

####James:
Pain Points:
* The monolith structure of Terraform modules results in very large artifacts - it would have preferable to break it down into multiple git repositories, one repo for each module.

What went well:
* We have been able to upgrade Terraform without incident.
* We were able to develop 30-40% faster because of having a small agile team without overly restrictive bureaucracy.

####Paul:
Pain points
* We should have pinned the Terraform version in version control to prevent accidental upgrade and to keep everyone on the same version (we ended up doing this later on with tfenv and a `.terraform-version` file). 
* None of the solutions for handling multiple environments seemed ideal, but James ended up solving this with Terragrunt once he came on board. 

What went well?
* It was a strong relief to have Terraform handle the setup of the environment. You just had to click a button, walk away, and the infrastructure would be ready by the time you came back. 
* Using Terragrunt was a very clean way of configuring infrastructure across multiple environments. 
* It checked off a lot of boxes for compliance and disaster recovery.

What could we have done better?
* In retrospect I could have handed off more product work to other people to have more time to write Terraform and ensure best practices. 
* We could have had nightly test runs - spinning up and tearing down infrastructure in another region to be confident it would always work. Metrics such as timing and money spent in setup would have been useful. The more metrics the better.

Any other thoughts?
* I wish I knew Terraform wrappers like [Palumi](https://www.pulumi.com/why-pulumi/) were an option. However, it probably would have been overkill at the time.
* I have found that it saves more time to configure a cloud service with Terraform first and then checking UI. Doing it in the UI first and then translating that to Terraform is slower.

<!--
Maybe wait till spinning up a new VPC before posting article? Then I would know Terraform was worth it and it would be a good demo of its advantage. ✅ done
UI sometimes has better error messages.
 We accidentally nuked a dev's config on the preview server.

cloud5 checklist: https://15five.quip.com/WOF3Azyhq4Lw
https://github.com/15five/fifteen5-terraform-modules/commits/master?after=a48bad9c4724599e767798de2911699cddad3763+384&branch=master
https://15five.15five.com/report/archive/?page=5
https://github.com/15five/fifteen5-terraform/commits/master?before=14cc7922cb7b55e242e21c38b060d2e9c06d1905+280&branch=master
https://circleci.com/gh/15five/workflows/fifteen5-terraform?page=32

Test a region that we could fail over to. Metrics around spinning up and tearing down - knowing how much money is spent doing so. Recording timing would be useful as well. It would be great if we could spin up and put a full prod db in play and do some smoke tests, and tear it down. Metrics metrics metrics!
I'm on the fence as to whether it would be worth it at 15Five's scale
and we are glad we did it. 

We should've started with a checklist in the lower environments. We also should've thought of some services ahead of time like metabase / chance of duplicate notifications

In Devops there is a saying. "Treat your servers like cattle, not pets." Your pet dies? A tragedy. One cow out of a thousand dies? A inconvenience, you just replace it with another cow. 


James:
Other notes:
* good that we did Terraform first [about kuberentes]

8 months total with a 3-person team.
-->