Exploring GCP With Terraform: VPC Firewall Rules, part 2
Tags: terraform, tutorial, gcp, devops
Published: 2023-09-02 09:05:12

This post would be 3rd part of the series about exploring GCP with Terraform. In the [previous part](https://rnemet.dev/posts/gcp/gcp_tf_vpc/), I created VPC networks, subnets, and a few firewall rules. In this part, I will explore more firewall rules and their parameters.

More precisely, I'll set up three VPCs: _back-office_, _services_ and _storage_. In VPC _back_office_, I'll have two subnets; in others, I'll have one subnet. For the sake of conversation, imagine that VMs in the _back-office_ have to call VMs in _services_ and _storage_. Also, direct access to  VMs from outside should not be allowed, except for one that will serve for maintenance.

## What I have done so far

I have two VPCs: _back-office_ and _services_. The VPC _back-office_ has a subnet _back-office_ and the VPC _services_ a subnet named _services_. In each VPC, I added firewall rules to allow SSH access from outside through IAP. I also added a firewall rule to allow ICMP traffic from anywhere. Each VPC has a VM instance.

The Terraform project is growing. If you look at how I set a `base` workflow in [the first post of this series](https://rnemet.dev/posts/gcp/start_with_gcp/#setting-up-the-project), now I have two more similar workflows for `networks` and `vms`. It is not the best practice to have it like this, but this will be addressed later.

## VPC Firewall rules

So, what is a firewall rule? A firewall rule is a set of conditions that define what traffic is allowed to enter or leave a VPC network. A collection of parameters defines each firewall rule. Those parameters are:

* direction - ingress or egress (default: ingress)
* priority - the priority of the rule. The lower the number, the higher the priority. The default value is 1000.
* action - allow or deny
* enforced - if the rule is enforced. The default value is true.
* target - the target of the rule. The target can be a tag, a service account, or a network.
* source - the source of the traffic. The source can be a tag, a service account, or a network.
* protocol - the protocol of the traffic. The protocols like TCP, UDP, ICMP, etc.
* logs - boolean. Make logs to see if the rule is matched or not.

### Setting up the stage

VPC _back_office_ has two subnets: _back-office_ and _back-office-private_. So far, I have firewall rules: `back-office-iap`, `back-office-icmp`, and `back-office-ssh`, allowing ingress traffic from Google IAP, ICMP from anywhere, and SSH from anywhere. So, adding a new subnet _back-office-private_ to the VPC _back-office_:

```hcl
# subnet for back office: private 
resource "google_compute_subnetwork" "back_office_private" {
  name          = "back-office-private"
  ip_cidr_range = "10.2.0.0/24"
  network       = google_compute_network.back_office.self_link
  region        = var.region
}
```

To create a VM, you can use this code as a template inside the `vms` workflow:

```hcl
resource "google_compute_instance" "services_vm_test" {
  name         = "services-vm-test"
  machine_type = "f1-micro"
  zone         = var.zone

  scheduling {
    preemptible        = true
    automatic_restart  = false
    provisioning_model = "SPOT"
  }

  boot_disk {
    initialize_params {
      image = "debian-cloud/debian-11"
    }
  }
  network_interface {
    network    = data.terraform_remote_state.network.outputs.vpc_services.id
    subnetwork = data.terraform_remote_state.network.outputs.vpc_services_subnetwork.id
  }
}
```

You may wonder what is this part:

```hcl
network_interface {
    network    = data.terraform_remote_state.network.outputs.vpc_services.id
    subnetwork = data.terraform_remote_state.network.outputs.vpc_services_subnetwork.id
  }
```

Where is that defined? Since I have a network creation in the different workflow `networks`, I need to get the network ID and subnet ID from that workflow. I can do that by using the `terraform_remote_state` data source. But first, I need to export it in the `networks` workflow in the `outputs.tf` file:

```hcl
output "vpc_services" {
  value = google_compute_network.services
}

output "vpc_services_subnetwork" {
  value = google_compute_subnetwork.services
}
```

Using the template above for VM and exposing all VPCs and subnets from `networks`, I can add two VMs in each subnet. The VMs in the _back-office_ subnet will be named _back-office-vm1_ and _back-office-vm2_. The VMs in the _back-office-private_ subnet will be named _back-office-private-vm1_ and _back-office-private-vm2_. First, I must change the `networks` and then the `vms`. Then, I can look for created instances:

```bash
$ gcloud compute instances list
NAME                     ZONE           MACHINE_TYPE  PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP  STATUS
back-office-private-vm1  us-central1-c  f1-micro      true         10.2.0.3                  RUNNING
back-office-private-vm2  us-central1-c  f1-micro      true         10.2.0.2                  RUNNING
back-office-vm1          us-central1-c  f1-micro      true         10.1.0.9                  RUNNING
back-office-vm2          us-central1-c  f1-micro      true         10.1.0.10                 RUNNING
services-vm-test         us-central1-c  f1-micro      true         10.1.0.5                  RUNNING
```

Then, let's try to connect to any VM from the local machine:

```bash
$ gcloud compute ssh back-office-vm1 --zone us-central1-c --tunnel-through-iap
```

Then, from it, let's try to connect to any other VM, ping it:

```bash
$ ping -c 5 10.2.0.3
PING 10.2.0.3 (10.2.0.3) 56(84) bytes of data.
64 bytes from 10.2.0.3: icmp_seq=1 ttl=64 time=0.884 ms
64 bytes from 10.2.0.3: icmp_seq=2 ttl=64 time=0.165 ms
64 bytes from 10.2.0.3: icmp_seq=3 ttl=64 time=0.233 ms
64 bytes from 10.2.0.3: icmp_seq=4 ttl=64 time=0.159 ms
64 bytes from 10.2.0.3: icmp_seq=5 ttl=64 time=0.238 ms

--- 10.2.0.3 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4103ms
rtt min/avg/max/mdev = 0.159/0.335/0.884/0.276 ms
```

Yes, you can connect to any VM inside VPC. Why? Because we have a firewall rule that allows ICMP traffic from anywhere. Let's delete that rule and try again(from the
local machine):

```bash
$ gcloud compute firewall-rules delete back-office-icmp
The following firewalls will be deleted:
 - [back-office-icmp]

Do you want to continue (Y/n)?  y

Deleted [https://www.googleapis.com/compute/v1/projects/network-playground-382512/global/firewalls/back-office-icmp].
```

Repeat the ping command from the connected VM to any other VM:


```bash
ping -c 5 10.2.0.3
PING 10.2.0.3 (10.2.0.3) 56(84) bytes of data.

--- 10.2.0.3 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4074ms
```

Now, at this moment VMs, can not communicate with each other.

#### Tightening access: Only one VM for SSH (bastion)

Let's say we want to allow SSH access only to one VM from outside. We can do that by adding a firewall rule that allows SSH access only to one VM. Let's remove the rule _back-office-ssh_ as it is not practical. Why? I'm accessing VMs with the tunnel through IAP, so I do not need it. It would be helpful only if I could access them directly. That means that the target VM has an external IP. But I do not want that. 

And let's modify the rule _back-office-iap_ to allow SSH access only to one VM. Let's add the tag _bastion_ to the VM:

```hcl
resource "google_compute_firewall" "back_office_iap" {
  name    = "back-office-iap"
  network = google_compute_network.back_office.self_link

  source_ranges = ["35.235.240.0/20"]
  target_tags   = ["bastion"]
  direction     = "INGRESS"
  allow {
    protocol = "tcp"
  }
}
```

I am using the tag _bastion_ to allow SSH access by applying this firewall rule only to VMs with particular tags. If I try to access the VM _back-office-vm1_ from the local machine, I will get an error:

```bash
$ gcloud compute ssh back-office-vm1 --zone us-central1-c --tunnel-through-iap
ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4003: 'failed to connect to backend']. (Failed to connect to port 22)
kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535

Recommendation: To check for possible causes of SSH connectivity issues and get
recommendations, rerun the ssh command with the --troubleshoot option.

gcloud compute ssh back-office-vm1 --project=network-playground-382512 --zone=us-central1-c --troubleshoot

Or, to investigate an IAP tunneling issue:

gcloud compute ssh back-office-vm1 --project=network-playground-382512 --zone=us-central1-c --troubleshoot --tunnel-through-iap

ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].
```

Let's add the tag _bastion_ to the VM _back-office-vm1_:

```shell
$ gcloud compute instances add-tags back-office-vm1 --zone=us-central1-c --tags=bastion
Updated [https://www.googleapis.com/compute/v1/projects/network-playground-382512/zones/us-central1-c/instances/back-office-vm1].
```

Now, connecting to the VM _back-office-vm1_ works. But connecting to any other VM in the VPC _back-office_ doesn't work. Why? Because we have a firewall rule that allows SSH access only to the VM _back-office-vm1_. The firewall matches the target with the tag value. If I tag any other VM in the VPC _back-office_ with the tag _bastion_, I can connect to it.

Well, I'm not happy that adding/removing tags is so easy. I want a more secure way to allow SSH access to VMs. I could use the service account to enable SSH access:

```hcl
resource "google_service_account" "back_office_fw_sa" {
  account_id   = "back-office"
  display_name = "back-office"
}

resource "google_compute_firewall" "back_office_iap" {
  name    = "back-office-iap"
  network = google_compute_network.back_office.self_link

  source_ranges           = ["35.235.240.0/20"]
  target_service_accounts = [google_service_account.back_office_fw_sa.email]
  direction               = "INGRESS"
  allow {
    protocol = "tcp"
  }
  depends_on = [google_service_account.back_office_fw_sa]
}
```

Connecting to the VM _back-office-vm1_ from the local machine, I will get the same error. Why? Because I'm not using the
service account to connect to the VM. I'm using the tunnel through IAP. Let's add the service account to the VM _back-office-vm1_:

```hcl
resource "google_compute_instance" "back_office_vm1" {
  name         = "back-office-vm1"
...

  allow_stopping_for_update = true

  service_account {
    email  = data.terraform_remote_state.network.outputs.back_office_fw_sa
    scopes = ["https://www.googleapis.com/auth/cloud-platform"]
  }

...
}
```

I added the service account to the VM _back-office-vm1_. Now, I can connect to it from the local machine. IAP can connect to it. Anyone who passes the IAP authentication can connect to it. But I can't connect to any other VM in the VPC _back-office_. Why? Because I'm not using the service account to connect to the VMs. I'm using the tunnel through IAP, and the firewall rule allows access only to the _back-office-vm1_ VM.

The scopes are required to allow the service account to access the resources. When adding service accounts to VMs, you must set `allow_stopping_for_update = true` and service account scopes. Changing the service account on the VM requires stopping the VM.  

This requirement makes a difference. You can add and remove tags on the fly but can't do that with service accounts. You must stop the VM to change the service account. That is why using service accounts is more secure than using tags.

Notice that I exported the service account email from the `networks` workflow. I did that because I needed to use it in the `vms` workflow.

OK, now I can connect to the VM _back-office-vm1_, in the VPC _back-office_, from the local machine via the IAP tunnel. But I can connect to any other VM in the VPC _back-office_. But I still can connect to all other VMs in the VPC _services_ and _storages_. I'll remove the IAP firewall rule for those VPCs to fix this. But then I'll need access from the VPC's _back_office_ VMs to the VPC's _services_ and _storages_ VMs. Keep the ICMP rule for VPC _services_ and _storages_. 

You can do it now if you did not add VPC _storages_, a subnet, and VMs. Just remember to add VPC and subnet to the `networks` workflow and VMs to the `vms` workflow. For the _storage_ subnet in the VPC _storage_, I used CIDR `10.120.0.0/24`. And just for a reminder, CIDR for subnet _services_ in VPC _services_ is `10.1.0.0/24`, and subnet _back-office_ in VPC _back_office_ is `10.1.0.0/24` and subnet _back-office-private_ in VPC _back_office_ is `10.2.0.0/24`.

#### Tightening access: Accessing other VPCs

OK, now what? I can't connect to any VM in the VPC _services_ and _datastorages_ from the local machine because I just removed IAP firewall rules for those VPCs. I can connect only to _back-office_vm1_ from the local machine. But from it, I can connect to any other VM in the VPC _back-office_, but not in other VPCs. To be able to connect from the local machine to _back-office-vm1_ and then to any VMs in VPC _services_ and _storages_, I need first to connect the VPC _back-office_
to VPC _services_ and _storages_. I can do that by setting up VPC peering. I'll set up VPC peering between VPC _back-office_ and _services_ and between VPC _back-office_ and _storages_:

```hcl
resource "google_compute_network_peering" "back_office_service_peering" {
  name         = "back-office-services-peering"
  network      = google_compute_network.back_office.self_link
  peer_network = google_compute_network.services.self_link
}

resource "google_compute_network_peering" "service_back_office_peering" {
  name         = "services-back-office-peering"
  network      = google_compute_network.services.self_link
  peer_network = google_compute_network.back_office.self_link
}

resource "google_compute_network_peering" "back_office_datastorage_peering" {
  name         = "back-office-storage-peering"
  network      = google_compute_network.back_office.self_link
  peer_network = google_compute_network.storage.self_link
}

resource "google_compute_network_peering" "datastorage_back_office_peering" {
  name         = "storage-back-office-peering"
  network      = google_compute_network.storage.self_link
  peer_network = google_compute_network.back_office.self_link
}
```

But we'll get an error when applying the changes:

```text
 Error: Error waiting for Adding Network Peering: An IP range in the peer network (10.1.0.0/24) overlaps with an IP range in the local network (10.1.0.0/24) allocated by 
 resource (projects/network-playground-382512/regions/us-central1/subnetworks/back-office).
```

Why? Because the VPC _back-office_ and _services_ have subnets with the same IP range. When setting up VPC peering, it is essential to know that the IP ranges of the subnets must not overlap. Changing the IP ranges of subnets in VPC _services_ is OK. But this means we need to plan the IP ranges of subnets. If you are using IPv6 in a VPC, then you don't need to worry about this. But, if you are using IPv4, you need to plan the IP ranges of subnets. The reason is that if you want to change CIDR for a subnet, you need to free all resources that use this subnet first, like deleting all VMs.

Changing the IP ranges of subnets in VPC _services_ is easy. I'll change the subnet _services_ IP range to _10.3.0.0/24_. But to do that, I need to delete the VMs in the VPC _services_ and recreate peering between VPCs. So, I'll remove VMs from the VPC _services__. Then, fixing the subnet range in VPC _services_, recreate peering between VPCs. And finally, adding back deleted VMs to the VPC _services_.

While I'm doing this, I'll revert SSH access to the VMs in all VPCs, but this time, I'll use CIDR ranges to allow access:

```hcl
resource "google_compute_firewall" "services_ssh" {
  name    = "services-ssh"
  network = google_compute_network.services.self_link

  source_ranges = ["10.1.0.0/24", "10.2.0.0/24"]
  direction     = "INGRESS"
  allow {
    protocol = "tcp"
    ports    = ["22"]
  }
}
```

This limits access to the VMs in the VPC _services_ from the VPC _back-office_ and _back-office-private_ subnets. I'll do the same for the VPC _storage_.

You can add one for the VPC _storage_ as well. On top of all this, you'll need to add public SSH keys to the VMs. You can do it per VM or project. I'll do it on 
the project level for now:

```hcl
resource "google_compute_project_metadata" "default" {
  metadata = {
    ssh-keys = <<EOF
      user:ssh-rsa public_key_goes_here user
    EOF
  }
}
```

When adding SSH public keys to the project level, you are allowing SSH access to all VMs in the project. For now, this is good enough. To generate SSH keys, you can use the command `ssh-keygen`:

```bash
$ ssh-keygen -f private-key -C rnemet -b 2048
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in private-key
Your public key has been saved in private-key.pub
...
```

The code above will generate two files: _private-key_ and _private-key.pub_. The file _private-key_ is the private key, and the file _private-key.pub_ is the public key. Now, you
can use the public key to add it to the project metadata. You can also use the private key to connect to the VMs. That means you must keep the private key safe and be on machines you use to connect to the VMs. After adding the public key to the project metadata, you can connect to the VMs in the VPC _services_ and _storages_ from VPC _back-office_ VMs. All you need when connecting to the VMs is the private key on bastion VMs:

```bash
$ gcloud compute scp ./private-key rnemet@back-office-vm1:.ssh/private-key --zone=us-central1-c
External IP address was not found; defaulting to using IAP tunneling.
WARNING:

To increase the performance of the tunnel, consider installing NumPy. For instructions,
please see https://cloud.google.com/iap/docs/using-tcp-forwarding#increasing_the_tcp_upload_bandwidth

private-key
```

When connecting to the VMs:

```bash
$ ssh rnemet@10.120.0.3 -i .ssh/private-key
```

With this setup, I can connect to the VMs in the VPC _services_ and _storages_ from the VM in the VPC _back-office_. Direct access to other VMs is not possible because
of IAP and firewall rules.

## Conclusion

So far, I have created VPCs, subnets, and VMs. I have also created firewall rules to allow SSH access to the VMs from one VM inside my networks. I have also created VPC peering between VPCs. Also, I started to share resources between Terraform state files. However, my TF project is growing and needs a better structure. I need to invest some time to improve it. But at the same time, I would like to put some services that are reachable from the outside.

What do you think? What would you do next? Let me know...

## References

* [GCP Firewall](https://cloud.google.com/firewall/docs/firewalls)
* [GCP TF firewall](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_firewall)
* [GCP TF instances](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_instance)
* [GCP scopes](https://cloud.google.com/sdk/gcloud/reference/alpha/compute/instances/set-scopes)
* [GCP VPC peering](https://cloud.google.com/vpc/docs/vpc-peering)