Deploy Postgres with replications Database in a minutes, with Terraform (Infrastructure as Code)
Tags: aws, terraform, postgres, cloudskills
Published: 2021-01-22 21:23:27

Hi everyone ! 

So many thanks for your supports in my last blog post:
https://dev.to/simerca/use-native-gps-with-vuejs-for-ios-and-android-1kk7

*Today ! It's a great time for you !*
I'will show you how to deploy Postgres database in a minutes with Terraform and AWS, with Primary / Replica and Hourly backup !

*So first, whats is Terraform ?*
Terraform it's a tool for Infrastructure as Code, it's very simple to use it, for exemple, you just write the configuration of you need and execute it with one command like :
`terraform apply`

First install Terraform:
https://learn.hashicorp.com/tutorials/terraform/install-cli

*Next, start the configuration files*

please use my public repo in gitlab:
`git clone https://gitlab.com/Simerca/terraform-postgres`

Next, see the first file and edit with your own configuration
`variables.tf`

``` js
// Default AWS Access Credentials
variable "access_key" {
	default = "****"
}
variable "secret_key" {
	default = "*****"
}
variable "region" {
    default = "eu-west-3"
}
variable "bucket" {
    default = "bucket"
}
variable "aws_ami" {
    default = "ami-00798d7180f25aac2"
}
variable "aws_type" {
    default = "t2.micro"
}
// Other Configuration
variable "workspace" {
    default = "user"
}
variable "password" {
    default = "*****"
}
```

So next see the main file
`app-instances.tf`

``` js
/* Setup our aws provider */
provider "aws" {
  access_key  = "${var.access_key}"
  secret_key  = "${var.secret_key}"
  region      = "${var.region}"
}

resource "aws_instance" "primary_1" {
  ami           = "${var.aws_ami}"
  instance_type = "${var.aws_type}"
  security_groups = ["${aws_security_group.swarm.name}"]
  key_name = "${aws_key_pair.deployer.key_name}"
  connection {
    host = self.public_ip
    user = "ec2-user"
    private_key = "${file("${path.module}/id_rsa.pem")}"
  }
  provisioner "remote-exec" {
    inline = [
      "sudo yum update -y",
      "sudo yum install git -y",
      "sudo amazon-linux-extras install docker -y",
      "sudo service docker start",
      "sudo curl -L https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose",
      "sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose",
      "sudo chmod +x /usr/local/bin/docker-compose;",
      "sudo docker network create web",
      "sudo docker run --restart=unless-stopped --name=postgres -d -p 5432:5432 -e POSTGRES_DB=${var.workspace} -e POSTGRES_USER=${var.workspace} -e POSTGRES_PASSWORD=${var.password} -v $(pwd)/data:/var/lib/postgresql/data postgres",
      "sudo docker run -d --restart=unless-stopped -p 3000:3000 -e PW2_ADHOC_CONN_STR=\"postgresql://${var.workspace}:${var.password}@${self.public_ip}:5432/${var.workspace}\" -e PW2_GRAFANAUSER=admin -e PW2_GRAFANAPASSWORD=admin -e PW2_ADHOC_CONFIG=exhaustive -e PW2_ADHOC_CREATE_HELPERS=true --name pw2 cybertec/pgwatch2-postgres",
      "sudo docker run -d --restart=unless-stopped --name=postgres_backup -e SCHEDULE='@hourly' -e S3_REGION=${var.region} -e S3_ACCESS_KEY_ID=${var.access_key} -e S3_SECRET_ACCESS_KEY=${var.secret_key} -e S3_BUCKET=${var.bucket} -e POSTGRES_DATABASE=${var.workspace} -e POSTGRES_USER=${var.workspace} -e POSTGRES_HOST=${self.public_ip} -e POSTGRES_PASSWORD=${var.password} -e S3_PREFIX=${var.workspace} -e POSTGRES_EXTRA_OPTS='--schema=public --blobs' schickling/postgres-backup-s3"
    ]
  }
  tags = { 
    Name = "${var.workspace}-primary"
  }
}
resource "aws_instance" "replica_1" {
  ami           = "ami-00798d7180f25aac2"
  instance_type = "${var.aws_type}"
  security_groups = ["${aws_security_group.swarm.name}"]
  key_name = "${aws_key_pair.deployer.key_name}"
  connection {
    host = self.public_ip
    user = "ec2-user"
    private_key = "${file("${path.module}/id_rsa.pem")}"
  }
  provisioner "remote-exec" {
    inline = [
      "sudo yum update -y",
      "sudo yum install git -y",
      "sudo amazon-linux-extras install docker -y",
      "sudo service docker start",
      "sudo curl -L https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose",
      "sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose",
      "sudo chmod +x /usr/local/bin/docker-compose;",
      "sudo docker network create web",
      "sudo docker run --restart=unless-stopped --name=postgres -d -p 5432:5432 -e POSTGRES_DB=${var.workspace} -e POSTGRES_USER=${var.workspace} -e POSTGRES_PASSWORD=${var.password} -v $(pwd)/data:/var/lib/postgresql/data postgres",
      "sudo docker run -d --restart=unless-stopped -p 3000:3000 -e PW2_ADHOC_CONN_STR=\"postgresql://${var.workspace}:${var.password}@${self.public_ip}:5432/${var.workspace}\" -e PW2_GRAFANAUSER=admin -e PW2_GRAFANAPASSWORD=admin -e PW2_ADHOC_CONFIG=exhaustive -e PW2_ADHOC_CREATE_HELPERS=true --name pw2 cybertec/pgwatch2-postgres",
      "sudo docker run -d --restart=unless-stopped --name=postgres_backup -e SCHEDULE='@hourly' -e S3_REGION=${var.region} -e S3_ACCESS_KEY_ID=${var.access_key} -e S3_SECRET_ACCESS_KEY=${var.secret_key} -e S3_BUCKET=${var.bucket} -e POSTGRES_DATABASE=${var.workspace} -e POSTGRES_USER=${var.workspace} -e POSTGRES_HOST=${self.public_ip} -e POSTGRES_PASSWORD=${var.password} -e S3_PREFIX=${var.workspace} -e POSTGRES_EXTRA_OPTS='--schema=public --blobs' schickling/postgres-restore-s3"
    ]
  }
  tags = { 
    Name = "${var.workspace}-replica"
  }
}
```

So ! Try to understand how it works

# Step 1 :

Use the variable to set the configuration, very simple to do this.

```js
provider "aws" {
  access_key  = "${var.access_key}"
  secret_key  = "${var.secret_key}"
  region      = "${var.region}"
}
```

You can use environment variable with the schema :

`TF_VAR_access_key=XXXXXX terraform apply`

# Step 2:
Launch instance and basic configuration with set of command lines

```js
resource "aws_instance" "primary_1" {
  ami           = "${var.aws_ami}"
  instance_type = "${var.aws_type}"
  security_groups = ["${aws_security_group.swarm.name}"]
  key_name = "${aws_key_pair.deployer.key_name}"
  connection {
    host = self.public_ip
    user = "ec2-user"
    private_key = "${file("${path.module}/id_rsa.pem")}"
  }
  provisioner "remote-exec" {
    inline = [
      "sudo yum update -y",
      "sudo yum install git -y",
      "sudo amazon-linux-extras install docker -y",
      "sudo service docker start",
      "sudo curl -L https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose",
      "sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose",
      "sudo chmod +x /usr/local/bin/docker-compose;",
      "sudo docker network create web",
      "sudo docker run --restart=unless-stopped --name=postgres -d -p 5432:5432 -e POSTGRES_DB=${var.workspace} -e POSTGRES_USER=${var.workspace} -e POSTGRES_PASSWORD=${var.password} -v $(pwd)/data:/var/lib/postgresql/data postgres",
      "sudo docker run -d --restart=unless-stopped -p 3000:3000 -e PW2_ADHOC_CONN_STR=\"postgresql://${var.workspace}:${var.password}@${self.public_ip}:5432/${var.workspace}\" -e PW2_GRAFANAUSER=admin -e PW2_GRAFANAPASSWORD=admin -e PW2_ADHOC_CONFIG=exhaustive -e PW2_ADHOC_CREATE_HELPERS=true --name pw2 cybertec/pgwatch2-postgres",
      "sudo docker run -d --restart=unless-stopped --name=postgres_backup -e SCHEDULE='@hourly' -e S3_REGION=${var.region} -e S3_ACCESS_KEY_ID=${var.access_key} -e S3_SECRET_ACCESS_KEY=${var.secret_key} -e S3_BUCKET=${var.bucket} -e POSTGRES_DATABASE=${var.workspace} -e POSTGRES_USER=${var.workspace} -e POSTGRES_HOST=${self.public_ip} -e POSTGRES_PASSWORD=${var.password} -e S3_PREFIX=${var.workspace} -e POSTGRES_EXTRA_OPTS='--schema=public --blobs' schickling/postgres-backup-s3"
    ]
  }
  tags = { 
    Name = "${var.workspace}-primary"
  }
}
```

Be carreful to this line 
``` js
private_key = "${file("${path.module}/id_rsa.pem")}"
```
Yoo need to create new SSL key for access to your instances

`ssh-keygen -t rsa -N "" -b 2048 -C "assignment"`
`ssh-keygen -y -f ~/.ssh/id_rsa.pem > ~/.ssh/id_rsa.pub`

After,
The command line execute command in order.
- Install dependencies
- Install docker
- Docker run some containers

We run this set twice, 1 primary, 1 replica (master/slave)

# Step 3
Open security group

In other file we have :
`security-group.tf`

```js
/* Default security group */
resource "aws_security_group" "swarm" {
  name = "swarm-group-${var.workspace}"
  description = "Default security group that allows inbound and outbound traffic from all instances in the VPC"

  ingress {
    from_port   = "0"
    to_port     = "0"
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
    self        = true
  }

  ingress {
    from_port = 22
    to_port   = 22
    protocol  = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port = 5432
    to_port   = 5432
    protocol  = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  ....
```

Here we open the needed port to access on our Postgres


# Lets Try !

First try your configuration (it's just a test)
`terraform plan`

Next, you can use this exemple command line to launch Postgres instance inside your AWS account :

`TF_VAR_access_key=XXXX TF_VAR_secret_key=XXXX TF_VAR_user=DBUser TF_VAR_password=SomePassword terraform apply`

This command launch:
- 2 Ec2 instances (Primary /Replica)
- 2 PostgresSQL Database on port 5432 and with the user : DBUser and password : SomePassword
- 2 Monitoring container with Grafana
- 2 Hourly backup on S3 Bucket(Primary to Replica)

and when you want to destroy all instances

`terraform destroy`

very simple no ?

# Some pub ?

If you don't want to use your mind, you can try my deployment tool, to deploy Postgres with replications and Monitoring in a minutes

https://cloudgres.com

Dont forget to talk with me !

Thanks for your time!