Exploring GCP With Terraform: Setting Up The Environment And Project
Tags: terraform, gcp, tutorial
Published: 2023-08-19 19:02:22

This topic is nothing new. There are many articles and tutorials on the internet about this. I'll be setting up a Terraform project to explore GCP. I cover the basics of Terraform and GCP.

{% details Read Before Proceed.. %}
I replaced the real project ID with `project-id` in the following examples. You need to replace it with your project ID.
{% enddetails %}

## Prerequisites

### Create GCP Project

First, you must create a GCP account, then create a project and set up billing on that project. If you already have a Gmail account, you can use it to make a GCP account. Head to the [Google Cloud Console](https://console.cloud.google.com/) and create a new project.

My plan is to pay for the services I use. I need to feel the pain and learn more about cost optimization. You can be more intelligent than me and use the free tier for the first year. After that, you will be charged for the services you use. You can find more information about the [free tier here](https://cloud.google.com/free/docs/gcp-free-tier).

{% embed https://rnemet.substack.com/embed %}

### Install And Set Up The gcloud CLI Tool

The `gcloud` CLI is a part of the Google Cloud SDK. You can find the installation instructions for [`gcloud` CLI here](https://cloud.google.com/sdk/docs/install). Next, you need to set up the `gcloud` CLI tool. Docs are [here](https://cloud.google.com/sdk/docs/initializing).

For future reference, you can find the list of all `gcloud` commands [here](https://cloud.google.com/sdk/gcloud/reference).

After initialization, you can always check what you set with the following command:

```bash
$ gcloud config list [SECTION/PROPERTY] [--all]
```

### Install Terraform CLI Tool

You can find the Terraform installation instructions [here](https://learn.hashicorp.com/tutorials/terraform/install-cli). But I'm not following the instructions.

I'm using the tool [tfenv](https://github.com/tfutils/tfenv) to manage Terraform versions. Other tools can do that. You can use [`asdf`](https://asdf-vm.com/), too. I saw that `asdf` can do more than manage Terraform versions.

Let me explain how I do it since I use `tfenv`. I assume you already installed it. I use `tfenv` to install the desired version of Terraform. And to set it as default:

```bash
$ tfenv install 1.5.5
$ tfenv use 1.5.5
```

You can check the version with the following command:

```bash
$ terraform --version
```

I'll explain later why I use `tfenv`.

### Other Tools I Use

Other tools I'll be using are:

* [task](https://taskfile.dev/) - a task runner and a replacement for `make`
* [git](https://git-scm.com/) - version control system
* [direnv](https://direnv.net/) - unconsciously set and unset environment variables

## Setting Up The Project

I'm using this layout for now:

```text
.                                       # project root                 
├── README.md                           # project README
├── LICENSE                             # project LICENSE
├── .gitignore                          # project .gitignore
├── Taskfile.yml                        # project taskfile
└── gcp                                 # GCP Terraform code
    └── base                            # base GCP Terraform code
        ├── README.md                   # base README
        ├── .terraform-version          # base terraform version
        ├── main.tf                     # base main.tf
        ├── outputs.tf                  # base outputs.tf
        ├── variables.tf                # base variables.tf
        ├── terraform.tfvars            # base terraform.tfvars
        └── provider.tf                 # base provider.tf
```

Let me quickly explain. The `gcp` directory is the root directory for all GCP Terraform code. The `base` directory is the root directory for all base GCP Terraform code. That base code would cover the following, for now:

* Creating a bucket for storing Terraform state
* Creating Key and KeyRing for encrypting Terraform state file
* Setting up Terraform's backend
* Setting up Terraform provider
* Setting up Terraform base variables

Just create the directories and files, and we are ready to go.

{% details How I use tfenv? %}
Oh, one small thing. The file `.terraform-version` is used by `tfenv` to set and pin the Terraform version. Even if I set with `tfenv` the default Terraform version, I use this file to say which Terraform version should be used for the base. If you are using `tfenv` do:

```bash
$ cd gcp/base
$ tfenv pin
Pinned version by writing "1.5.5" to /somepath/gcp/base/.terraform-version
```

With this, it does not matter what is the default Terraform version. The `tfenv` will use the version from the `.terraform-version` file. If, for some reason, you
do not have the desired version, `tfenv` will install it for you.
{% enddetails %}

### Setting Up Provider

The first thing we need to do is to set up the provider. I'm using the latest version of the provider. You can find the latest version [here](https://registry.terraform.io/providers/hashicorp/google/latest):

```hcl
provider "google" {
  project = var.project_id
  region  = var.region
  zone    = var.zone
}

terraform {
  required_version = ">=1.5.5"

  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "4.77.0"
    }
  }

  backend "local" {
      path = "local-state.tfstate"
  }
}
```

I'm using the `local` backend for now. I plan to switch to the `gcs` backend later. The `local` backend is used for storing the Terraform state locally. The state file is called `local-state.tfstate`. And it will contain the Terraform state for the base. It is not encrypted. My plan is to keep it safe and encrypted.

Block `terraform` sets the required Terraform provider version, provider, and backend. You can find more [here](https://registry.terraform.io/providers/hashicorp/google/latest).

The `provider` block configures the provider by setting the project ID, region, and zone. You probably noticed that I'm using variables.

### Setting Up Variables

I'm using variables for defining variables I'll be using in the base. I'm using the following variables:

```hcl
variable "project_id" {
  description = "project id"
  type        = string
}

variable "region" {
  description = "default region"
  type        = string
}

variable "zone" {
  description = "default zone"
  type        = string
}
```

Once you defined the variables, you need to set them. You can do that in the `terraform.tfvars` file:

```hcl
project_id = "my-project"
region     = "europe-west1"
zone       = "europe-west1-b"
```

You should avoid committing the `terraform.tfvars` file. It may contain sensitive information. I'll show you how to handle that later. But for now, you must create the `terraform.tfvars` file and set the variables. For the `project_id` variable, you need to set the project ID you created earlier. Choose the region and zone that is closest to you. You can find the list of regions and zones [here](https://cloud.google.com/compute/docs/regions-zones).

{% details Regions&Zones Important? %}
Is this important? Yes, it is.

You need to choose the region and zone carefully, considering reliability, availability, and cost into account. For your users to have the best experience, your servers must be close to them and deliver the content quickly. It would be best if you considered the cost, too.

To learn more about choosing the correct region and zone, you can find more [here in Google documentation](https://cloud.google.com/solutions/best-practices-compute-engine-region-selection).
{% enddetails %}

### Init, Plan, and Apply

At this moment, I can initialize the Terraform workflow:

```bash
$ terraform init


Initializing the backend...

Successfully configured the backend "local"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Finding hashicorp/google versions matching "4.77.0"...
- Installing hashicorp/google v4.77.0...
- Installed hashicorp/google v4.77.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
```

This will create a directory `.terraform` and file `.terraform.lock.hcl` file. You should include `.terraform.lock.hcl` in your commit. It is used to lock the provider version. If you run `terraform init` again, it will use the same provider version. The folder `.terraform` stores the provider plugins and other files required for Terraform to work.

Next, I can run `terraform plan`:

```bash
$ terraform plan

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.
```

This will show me what Terraform will do. Since I did not create any resources, it will not do anything. But it is good to run `terraform plan` before `terraform apply`.

Next, I can run `terraform apply`:

```bash
$ terraform apply

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
```

Well, this is expected. I did not create any resources. If there are any changes, Terraform will show you what it will do and ask you to confirm them. If you want to apply the changes, type `yes` 
and hit enter.

### Setting Up The Bucket

From the `terraform init` output, I can see that the backend is configured. It is configured to use the `local` backend. My goal is to use the `gcs` backend. That means I want to store the Terraform state file in the GCP bucket. Let's look at how to do it:

```hcl
data "google_storage_project_service_account" "gcs_account" {
}

resource "google_kms_key_ring" "tf_states" {
  name     = "terraform-state-key-ring"
  location = "us"

  lifecycle {
    prevent_destroy = true
  }
}

resource "google_kms_crypto_key" "tf_states" {
  name            = "terraform-state-key"
  key_ring        = google_kms_key_ring.tf_states.id
  rotation_period = "100000s"

  lifecycle {
    prevent_destroy = true
  }
}

resource "google_kms_crypto_key_iam_binding" "binding" {
  crypto_key_id = google_kms_crypto_key.tf_states.id
  role          = "roles/cloudkms.cryptoKeyEncrypterDecrypter"

  members = ["serviceAccount:${data.google_storage_project_service_account.gcs_account.email_address}"]
}

resource "google_storage_bucket" "tf_states_bucket" {
  name          = "terraform-states-${var.project_id}"
  force_destroy = false
  location      = "us"
  storage_class = "STANDARD"
  versioning {
    enabled = true
  }
  encryption {
    default_kms_key_name = "your-crypto-key-id"
  }

  depends_on = [google_kms_crypto_key_iam_binding.binding]
  lifecycle {
    prevent_destroy = true
  }
}
```

When working with Terraform, you are calling the API of the provider. With each call, you are pushing configuration to the provider. The provider will then create, update, or delete your resources.

That is a lot of code. Let me explain.

First, I'm creating a Key(`google_kms_crypto_key`) and Key Ring(`google_kms_key_ring`) for encrypting the Terraform state file. These resources are coming from [Google KMS](https://cloud.google.com/kms/docs). The service allows you to work with cryptographic keys and execute cryptographic operations.

Next is adding the Role `roles/cloudkms.cryptoKeyEncrypterDecrypter` to the Service Account. Which is needed to encrypt/decrypt the state file. In the end, I'm creating a bucket for storing the Terraform state file, and I'm using the Key for encrypting the Terraform state file. There is a small gotcha. I need a Service Account for encrypting the Terraform state file. But it does not exist
yet. So, I'll use [an automatic Google Cloud Storage service account](https://cloud.google.com/storage/docs/projects#service-accounts) to get it when the resource is created. I'm using a `data` resource for that.

I'm using `lifecycle.prevent_destroy` to ensure the resources are not accidentally deleted.

If I now run the `terraform plan` I'll see the following:

```bash
$ terraform plan
data.google_storage_project_service_account.gcs_account: Reading...
data.google_storage_project_service_account.gcs_account: Read complete after 1s [id=service-964026605440@gs-project-accounts.iam.gserviceaccount.com]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # google_kms_crypto_key.tf_states will be created
  + resource "google_kms_crypto_key" "tf_states" {
      + destroy_scheduled_duration = (known after apply)
      + id                         = (known after apply)
      + import_only                = (known after apply)
      + key_ring                   = (known after apply)
      + name                       = "terraform-state-key"
      + purpose                    = "ENCRYPT_DECRYPT"
      + rotation_period            = "100000s"
    }

  # google_kms_crypto_key_iam_binding.binding will be created
  + resource "google_kms_crypto_key_iam_binding" "binding" {
      + crypto_key_id = (known after apply)
      + etag          = (known after apply)
      + id            = (known after apply)
      + members       = [
          + "serviceAccount:service-964026605440@gs-project-accounts.iam.gserviceaccount.com",
        ]
      + role          = "roles/cloudkms.cryptoKeyEncrypterDecrypter"
    }

  # google_kms_key_ring.tf_states will be created
  + resource "google_kms_key_ring" "tf_states" {
      + id       = (known after apply)
      + location = "us"
      + name     = "terraform-state-key-ring"
      + project  = (known after apply)
    }

  # google_storage_bucket.tf_states_bucket will be created
  + resource "google_storage_bucket" "tf_states_bucket" {
      + force_destroy               = false
      + id                          = (known after apply)
      + labels                      = (known after apply)
      + location                    = "US"
      + name                        = "terraform-states-project-id"
      + project                     = (known after apply)
      + public_access_prevention    = (known after apply)
      + self_link                   = (known after apply)
      + storage_class               = "STANDARD"
      + uniform_bucket_level_access = (known after apply)
      + url                         = (known after apply)

      + encryption {
          + default_kms_key_name = "your-crypto-key-id"
        }

      + versioning {
          + enabled = true
        }
    }

Plan: 4 to add, 0 to change, 0 to destroy.
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.  
```

Ok, let's try to apply the changes:

```bash
$ terraform apply

...

Plan: 4 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

google_kms_key_ring.tf_states: Creating...

Error: Error creating KeyRing: googleapi: Error 403: Google Cloud KMS API has not been used in project 3534534535 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudkms.googleapis.com/overview?project=3534534535 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.

  with google_kms_key_ring.tf_states,
  on main.tf line 4, in resource "google_kms_key_ring" "tf_states":
   4: resource "google_kms_key_ring" "tf_states" {
```

I had to cut output. But you can see the error. I need to enable the Cloud KMS API. This will be a standard message if you try to use the API, which is not enabled. You can enable it by following the link in the error message. Or you can enable it with the following command:

```bash
$ gcloud services enable cloudkms.googleapis.com
```

{% details gcloud CLI useful commands  %}
You can see all the APIs you enabled with the following command:

```bash
$ gcloud services list --enabled
```

And list all available APIs with the following command:

```bash
$ gcloud services list --available
```
{% enddetails %}

I already have enabled Google Storage API. But if you did not, you need to enable it, too. Try to figure out how. Ok, let's try again:

```bash
$ terraform apply
...

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

google_kms_key_ring.tf_states: Creating...
╷
│ Error: Error creating KeyRing: googleapi: Error 409: KeyRing projects/project-id/locations/us/keyRings/terraform-state-key-ring already exists.
│
│   with google_kms_key_ring.tf_states,
│   on main.tf line 4, in resource "google_kms_key_ring" "tf_states":
│    4: resource "google_kms_key_ring" "tf_states" {
```

I was impatient and ran `terraform apply` again. Google API responded that it was not enabled, but actually, it was. And not only that, it created the Key and KeyRing. This situation is rare. But it can happen. We can fix it by importing created resources. Let's do this:

```bash
$ terraform import google_kms_key_ring.tf_states project-id/us/terraform-state-key-ring
```

How do I know that? Well, I read the error message and the provider's documentation. You'll have a section **Import** in Terraform documentation for each resource.
There, you'll be able to see how to import it.

Now, let's try again:

```bash
$ terraform apply

...
Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

data.google_storage_project_service_account.gcs_account: Reading...
google_kms_key_ring.tf_states: Refreshing state... [id=projects/project-id/locations/us/keyRings/terraform-state-key-ring]
data.google_storage_project_service_account.gcs_account: Read complete after 0s [id=service-964026605440@gs-project-accounts.iam.gserviceaccount.com]
google_kms_crypto_key.tf_states_key: Refreshing state... [id=projects/project-id/locations/us/keyRings/terraform-state-key-ring/cryptoKeys/terraform-states-key]
google_kms_crypto_key_iam_binding.binding: Refreshing state... [id=projects/project-id/locations/us/keyRings/terraform-state-key-ring/cryptoKeys/terraform-states-key/roles/cloudkms.cryptoKeyEncrypterDecrypter]
google_storage_bucket.tf_states_bucket: Creating...
google_storage_bucket.tf_states_bucket: Creation complete after 2s [id=terraform-states-project-id]

```

The bucket is created. You'll see it if you go to [GCP console](https://console.cloud.google.com/storage/). You can check bucket properties and configuration there in the console or with the following command:

```bash
$ gsutil ls -L -b gs://terraform-states-project-id
```

You can see that the bucket is encrypted with the Key we created. And to check the Key properties and configuration in the console or with the following command:

```bash
$ gcloud kms keys describe terraform-states-key --location us --keyring terraform-state-key-ring
```

You can list the content of the bucket with the command:

```bash
$ gsutil ls gs://terraform-states-project-id
```

But you'll get nothing because it is empty. I still need to move Terraform's state file to the bucket.

### Setting Up The Backend

If you remember, I'm using the `local` backend. If you check the local directory, you'll see a file called `local-state.tfstate`. That is the Terraform state file. And that file I want to move to 
the bucket and encrypt it there. I can do that with the following code:

```hcl
  backend "gcs" {
    bucket  = "terraform-states-project-id"
    prefix  = "terraform/state"
  }

```

And removing this one in `provider.tf`:

```hcl
  backend "local" {
    path = "local-state.tfstate"
  }
```

Now, if I run `terraform init`, I'll see the following:

```bash
terraform init

Initializing the backend...
╷
│ Error: Backend configuration changed
│
│ A change in the backend configuration has been detected, which may require migrating existing state.
│
│ If you wish to attempt automatic migration of the state, use "terraform init -migrate-state".
│ If you wish to store the current configuration with no changes to the state, use "terraform init -reconfigure".
╵
```

Well, let's do as it says:

```bash
$ terraform init -migrate-state
```

When asked, type `yes` and wait to finish. Let's check if we have anything in the bucket:

```bash
$ gsutil ls gs://terraform-states-project-id/terraform/state
```

You'll see that the bucket has a file called `default.state`. That is the Terraform state file. It is encrypted. You can check it with the following command:

```bash
$ gsutil cat gs://terraform-states-project-id/terraform/state/default.tfstate | jq .
```

{% embed https://rnemet.substack.com/embed %}

### Conclusion

With I just set up a project. With it, I can start exploring GCP. We touched basic Terraform CLI commands(`init`, `plan`, `apply`, and `import`), Terraform variables, provider, backend, and resources. I hope you learned something new. I know I did. I'll continue exploring GCP with Terraform in the next article.

## References

* [Google Cloud Platform](https://cloud.google.com/)
* [Google Cloud Console](https://console.cloud.google.com/)
* [Google Cloud SDK](https://cloud.google.com/sdk/docs)
* [Terrafom](https://www.terraform.io/)
* [GCP Terraform Provider](https://registry.terraform.io/providers/hashicorp/google/latest/docs)