Terraforming Your Data Infrastructure on AWS: A Hands on guide for Data Engineers
Published: 2023-09-07 09:54:21

In the constantly evolving world of Data Engineering, if you haven't come across Terraform by now, you might have been living under a rock :) 

Terraform was originally embraced by DevOps Engineers for deploying cloud infrastructure. However, as Data Engineers recognized the need to provision similar resources, it has gained widespread adoption in the field.

**What is Terraform?**
Terraform is an open-source tool developed by [HashiCorp](https://www.hashicorp.com/) used for defining and provisioning infrastructure using the Declarative Approach of HashiCorp Configuration Language (HCL).

Terraform is used for Infrastructure as code. What does this mean? 
- **Infrastructure** are services like virtual servers, storage buckets, lambda functions, RDS, IAM policies and so on, commonly found on cloud providers such as Azure, AWS, GCP and Oracle. For this tutorial we will be focusing on AWS.
- **Infrastructure as code** refers to the use of code - in this case HCL, to create and manage the above resources as opposed to manually configuring them.

**Why do Data Enginners use Terraform?**

1. **IAC**: Terraform allows data engineers to define their infrastructure, including servers, networks, and storage, in  code. This enables them to version control their infrastructure configurations, track changes over time, and easily reproduce environments.

2. **Cloud Agnostic**: Terraform supports multiple cloud providers (e.g., AWS, Azure, Google Cloud) and even on-premises environments. Data engineers can use a consistent toolset regardless of their cloud provider.

3. **Collaboration**: Terraform configurations can be shared and collaboratively developed by data engineering teams. Changes can be reviewed, and infrastructure can be modified with the approval of team members.

4. **Change Management**: Data engineers can implement changes to infrastructure in a controlled manner. Terraform's plan and apply workflow allows them to preview changes before applying them, reducing the risk of errors.

5. **State Management**: Terraform maintains a state file that keeps track of the current state of the infrastructure. This state file helps Terraform understand which resources are already provisioned and how they are configured, preventing unnecessary modifications.

6. **Integration**: Terraform can be integrated into continuous integration/continuous deployment (CI/CD) pipelines, allowing data engineers to automate the testing and deployment of infrastructure changes.

7. **Cost Control**: By defining infrastructure in code, data engineers can implement cost-saving strategies, such as automatically shutting down non-essential resources when not in use or optimizing resource sizes based on actual usage.

**Terraform Lifecycle**
![terraform lifecycle](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hgoizdxvw5cq7k13398m.jpeg)

Terraform's lifecycle consists of four stages.

- **Terraform init** to initialize Terraform environment and cloud provider plugins
- **Terraform plan** to create and display execution plans of resources to be configured
- **Terraform apply** to execute the plan and create the resources
- **Terraform destroy** to delete the resources created in a specific terraform environment

In this tutorial you will learn how to provision and configure an EC2 instance, and an S3 bucket using terraform. Make sure you have a fully operational AWS account to proceed.
You can access the tutorial's code by visiting the [Github repository](https://github.com/priye-1/terraforming-data-infrastructure) provided for reference. 

**Disclaimer:**
This is a lengthy tutorial, particularly for those new to Terraform. Therefore, I recommend taking it step by step, pausing when needed, and avoiding any sense of being overwhelmed.

Now that you get the tea lets dive in and get hands-on!

**1. Install Terraform**
To use terraform locally you need to have it installed.

With the code editor of your choice, Terraform can be installed on Mac or Linux using a tool called [Terraswitch](https://warrensbox.github.io/terraform-switcher/). This enables you to switch between different terraform versions, just like pyenv.
Otherwise, you can use Package managers; Homebrew for mac - 
```
brew install terraform
```
wget for linux :
```
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install terraform
```
chocolatey for windows

```
choco install terraform
```
You can go through the official [documentation ](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli) for any step you need further clarifications on.

After installation, run `terraform -version` on your terminal to confirm installation. This should return the version of terraform installed.

![Terraform version](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tz9tpdmdacnac9kmkuvj.png)

**2. AWS Authentication - Generate new access**
In order to authorise terraform on AWS, you need to achieve the following steps: 
**a.**  Create a new IAM user with administrator access to get the AWS access key and AWS secret key.
On the AWS console, search for `IAM` services and then navigate to `Users` ---> `Create user`

![Create new user](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pxna25gey3kpjiek0ils.png)

Next step is to set permission, for this you will click on `Attach policies directly` then `Administrator access` and then `next` to create user

![set permissions](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1vw77cz2oz2glp3wqon7.png)
Now that you have created a user, you can then get access keys for that user by navigating back to `users` --> `security credentials` --> `create access key`. 

choose the `command line interface` option, write a description and finally `create access key`. You can download as CSV for future reference or just take note of the generated keys.

**b.** Configure AWS cli with new creds for AWS Authentication.
To download the AWS cli on Mac, Linux or Windows kindly follow [this guide](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).

After successful installation run `aws --version` for confirmation and then `aws configure`. This will prompt you to provide the Access code and Secret code copied from step (a).  

Alternatively, you can also authenticate AWS using environment variables:

```bash
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
```


After this is done, you are all set to use terraform on your aws account!

**3. Terraform Core Concepts**

**<u>File Structure</u>**
Terraform treats every configuration as a module and executes all configuration files that ends in the extension _.tf_

A typical file structure for a new module is:

├─ README.md
├─ main.tf
├─ variables.tf
├─ outputs.tf

- **main.tf** will contain the main set of configurations for your module. 

- **variables.tf** will contain the variable definitions for your module. When your module is used by others, the variables will be configured as arguments in the module block.  

- **outputs.tf** will contain the output definitions for your module. Module outputs are often used to pass information about the parts of your infrastructure defined by the module to other parts of your configuration.

Executing a terraform configuration file creates other files to be aware of, and ensure that you don't distribute them as part of your module:

- **terraform.tfstate** and **terraform.tfstate.backup**: These files contain your Terraform state, and is how Terraform keeps track of the relationship between your configuration and the infrastructure provisioned by it.

- **.terraform**: This directory contains the modules and plugins used to provision your infrastructure. These files are specific to a specific instance of Terraform when provisioning infrastructure.

- **.terraform.lock.hcl**: This file maintains specific provider versions, ensuring consistency in Terraform configurations and preventing unintended updates. It enhances control and stability, critical for production environments.

**<u>Terraform Syntax</u>**
These are the most essential parts of terraform

**Providers**: To configure any infrastructure you need to first declare the provider so that Terraform can install and use them. Terraform relies on plugins called providers to interact with cloud providers, and other APIs.

Terraform has a list of providers, such as AWS, Azure, GCP, etc. visit this [documentation](https://registry.terraform.io/browse/providers) to know how other providers are used.

An AWS provider is declared in a provider block by the following:

```terraform
  provider "aws" {
  region = "us-north-1"
}
```

**Resources**: Terraform uses resource blocks to define components of your infrastructure, such as virtual networks, compute instances, etc. 
Most Terraform providers have a number of different resources as documented [here](https://registry.terraform.io/providers/hashicorp/aws/latest/docs)
<table>
    <th>AWS Resource Type</th>
    <th>AWS Infrastructure</th>
<tr>
    <td>aws_instance</td>
    <td>EC2 Instance</td>
</tr>
<tr>
      <td>aws_security_group</td>
      <td>Security Group</td>
</tr>
</table>

To create a resource:

```terraform
resource "resource_type" "custom_resource_name" {
    sample_attribute = ""
}
```
Resource blocks declare a resource type and name. Together, the type and name forms a unique resource identifier (ID) in the format `resource_type.resource_name`
To display information about a resource as output, you have to reference the resource ID.

Resource types always start with the provider name followed by an underscore, for example; `aws_instance`

**Variables:** The general idea of variables in programming is also applied in Terraform. This allows you to parameterize your configurations and make them more flexible and reusable.

Terraform variables can be set in different ways;
- Using the variable.tf file
- Individually, with the -var command line option.
- In variable definitions (.tfvars) files, either specified on the command line or automatically loaded.
- As environment variables.

We will be using the first option, by declaring a variable block with the syntax:

```terraform
variable "var_name" {
  description = "description of var"
  type        = string
  default     = "value"
}
```
Note that Terraform variables can be of data types - string, number, boolean, list, object, map, tuples, etc. 

The value can then be accessed from within expressions as `var.<NAME>`, where `<NAME>` matches the label given in the declaration block. For example;
In `variables.tf` we create a new variable called `user_information` 
```terraform
variable "user_information" {
  type = object({
    name    = string
    address = string
  })
  sensitive = true
}
```
In `main.tf` we access the variable using `var.user_information`

```terraform
resource "some_resource" "a" {
  name    = var.user_information.name
  address = var.user_information.address
}
```


**Outputs:** Output values make information about your infrastructure available on the command line, and can expose information for other Terraform configurations to use. Output values are similar to return values in programming languages.

Each output value exported by a module must be declared using an output block:

```terraform
output "instance_ip_addr" {
  value = resource_type.custom_resource_name.attribute
}
```
The label immediately after the `output` keyword is the output name, which must be a valid identifier. 

Now that we have understood the syntax, let's provision our first AWS resource

**4. Creating an Ec2 instance**
For our first example we want to create an EC2 instance that we can access on the web. This means that we need to know the public IP to which the server is hosted, and will therefore be creating outputs in the outputs.tf file

To create an EC2 instance we need to declare a resource block that calls a resource named - [aws_instance](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/instance) 

From the documentation we can see the _argument references_  of `aws_instance` which points out different arguments needed to configure an instance. 
For this example we will define the following;
**a.** `user_data` file. which serves as a method to perform some configurations on the EC2 instance during creation, such as setting the hostname, or installing a software package.
**b.** `ami` ID which is a unique identifier for a specific Amazon Machine Image.
**c.** `security_group` to create a virtual firewall that controls inbound and outbound network traffic to Amazon EC2 instances.
**d.** `instance_type` to define the hardware of the virtual machine, specifying its CPU, memory, storage capacity, and so on.
**e.** `key_name` to specify the SSH keys required for secure access to an instance via SSH.

Let's go ahead to create some of these as variables in `variables.tf`

```terraform
variable "instance_name" {
    description = "Name of the EC2 instance"
    type        = string
    default     = "test_instance"
}
variable "instance_type" {
    description = "EC2 instance type"
    type        = string
    default     = "t3.micro"
}
variable "instance_ami" {
    description = "id of AMI"
    type        = string
    default     = "ami-065681da47fb4e433"
}
variable "key_name" {
    description = "key name"
    type        = string
    default     = "ssh-keypair"
}
```
You can obtain the `instance_type` and `ami` required to create the AWS instance by referencing the AWS console when manually setting up an EC2 instance.

On your AWS portal search for `EC2`, click on `instances`, and then `launch instances`
On this page you can view the AMI id and instance type of your choice. Since we are using a free tier, _t3.micro_ is good enough.

![Instance type](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/k918yxlj4ser01gxvex0.png)

Next we will use these variables in `main.tf` by declaring the provider and resource block. 

```terraform
provider "aws" {
    region = "eu-north-1"
}

resource "aws_security_group" "web_traffic" {
    name = "allow_tls"
    
    ingress {
        from_port        = 443
        to_port          = 443
        protocol         = "tcp"
        cidr_blocks      = ["0.0.0.0/0"]
    }
    ingress {
        from_port   = 80
        to_port     = 80
        protocol    = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
    }
    egress {
        from_port        = 443
        to_port          = 443
        protocol         = "tcp"
        cidr_blocks      = ["0.0.0.0/0"]
    }
    egress {
        from_port   = 80
        to_port     = 80
        protocol    = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
    }
}

resource "aws_key_pair" "ssh_key_pair" {
  key_name   = var.key_name
  public_key = file(".ssh/rsa.pub")
}

resource "aws_instance" "my_ec2" {
    ami = var.instance_ami
    instance_type = var.instance_type
    user_data = file("startup.sh")
    key_name = aws_key_pair.ssh_key_pair.key_name
    security_groups = [aws_security_group.web_traffic.name]

    tags = {
        Name = var.instance_name
    }
}
```
Here we specified three resources; `aws_instance`, `aws_key_pair`, and `aws_security_group`. 
Attributes of aws_security_group and aws_key_pair were parsed as arguments into aws_instance as `key_name` and `security_groups`. Note how we used the format `resource_type.resource_name.attribute` to access them.

Creating key pairs using `aws_key_pair` allows you to securely connect to your EC2 instance using SSH while `aws_security_group` allows you to configure inbound and outbound traffic using rules based on IP addresses, ports, and protocols. In this case we allowed traffic from any IP on ports 80 and 443.

keep in view that to utilize `aws_key_pair`, we must first generate an SSH key, which we can then provide to the AWS instance. To generate an SSH key with a length of 4096 bits,  run the command and follow the prompts. I created a `.ssh` folder and an `rsa` filename to store my keypair.

```bash
ssh-keygen -t rsa -b 4096
```
The ssh-keygen command will generate two files: a private key file and a public key file. The private key file should be kept secure and never shared with anyone. The public key file can be shared with Amazon EC2 instances to allow SSH access.

Thirdly, we declare the output block in `outputs.tf` so that we can view the IP of the hosted server

```terraform
output "instance_id" {
    value = aws_instance.my_ec2.public_ip
}
```
The _attribute_reference_ of the [documentation](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/instance) shows all the attributes that can be returned after provisioning a resource.

Lastly, create a `startup.sh` with the following content:

```bash
#!/bin/bash
sudo yum update
sudo yum install -y httpd
sudo systemctl start httpd
sudo systemctl enable httpd
echo "<h1>Hello from Terraform</h1>" | sudo tee /var/www/html/index.html]
```
This is the bash script that will be executed through `user_data`

Now that all requirements are in place:

- Initialize the project  - `terraform init`

- Next to preview the changes that Terraform will make to your 
  infrastructure, run `terraform plan`
![tf apply](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5ti5roz0vh73buhknkvh.png)
  This will show you all the configurations to be deployed.
- Finally to apply the configurations, run `terraform apply`
This will output the public IP of the instance

![tf-apply](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/iobc16ip3z1b4hsg0aax.png)

Typing the IP address into your web browser with the format: `http://<IP_address>` should yield the following page:

![website](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0b7gp2y43cl7xknyfctr.png)

and on your AWS console you will see the server up and running

![Instance](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/x907uhfl6f4thdm28pp2.png)

Congratulations! you have provisioned your first ec2 instance using terraform and you can ssh into the instance using the generated keypair with the command - `ssh -i "ssh-keypair.pem" ec2-user@ec2-<public_ip>.eu-north-1.compute.amazonaws.com.
`
**5. Creating an S3 bucket**
Now that we are familiar with the methods and syntax of terraform, this should be easy to grasp.

To create a simple s3 bucket we will be using the [aws_s3_bucket](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket) resource.

From the documentation linked above we can browse through  the _argument references_ to see what arguments we need to pass to this resource. 
Let's go ahead to define the `bucket_name ` in `variables.tf`. Make sure to use a unique name.

```terraform
variable "bucket_name" {
    description = "Name of s3 bucke"
    type        = string
    default     = "my-tf-test-bucket-priye"
}
```
Now we will add the aws_s3_bucket resource to `main.tf`

```terraform
resource "aws_s3_bucket" "my_s3_bucket" {
    bucket = var.bucket_name

    tags = {
        Name        = "My bucket"
        Environment = "Dev"
    }
}
```
now run terraform `plan` and `apply`. This will add an additional resource

![tf s3 apply](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2ttc7bircy5n6gpwnk4m.png)

You can also confirm the s3 bucket on your AWS console.

![bucket](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vec613p1syq6ft5o042n.png)

**6. Destroy all resources**
To remove all provisioned resources and reduce operational costs, execute the following command: 

```bash
terraform destroy
```

Congratulations! We have come to the end of this tutorial and you have taken your first steps into the world of Terraform!🎉

In this tutorial, you've learned the fundamentals of Terraform, from setting up your environment to creating your first resources. But remember, Terraform is a vast and flexible tool with many advanced features waiting for you to explore. 
As you embark on your data engineering journey with Terraform, delve into advanced topics such as:

**Modules:** Learn how to encapsulate and reuse your infrastructure code for better organization and maintainability.

**Dynamic Blocks:** Discover how to create dynamic and flexible configurations for resources using dynamic blocks.

**For Each:** Explore how to loop through lists and maps to create multiple similar resources with ease.

**Provider Configuration:** Extend your skills by configuring different providers for hybrid and multi-cloud setups.

**State Management:** Master the art of state files, remote backends, and locking mechanisms for safe and efficient collaboration.

If you enjoyed learning a thing or two from this tutorial, please consider sharing it. Additionally, if you'd be interested in a follow-up tutorial covering these advanced concepts, don't hesitate to express your interest in the comments below. Your feedback is highly appreciated!

Happy Terraforming! 🌍💻