GCP Serverless deployments with Terraform: Cloud Run vs App Engine
Tags: gcp, serverless, terraform, containers
Published: 2023-09-04 10:46:10

As part of my job, I have been recently tasked to build the infrastructure hosting a NodeJs application on Google App Engine Flexible platform. 
One of the security guardrails on the project forbade the use of external IP addresses, therefore there was a specific requirement to deploy **App Engine** instances **with an internal IP address** only, and route traffic to the app through an external HTTP **Cloud Load Balancer**, with App Engine serving as its **serverless NEGs backend**. 
This solution allows to terminate **SSL encryption** on the Load Balancer and to have **multi-region load balancing**—App Engine is indeed a regional service, so it doesn’t come with global load balancing by default.

While building the infrastructure with **Terraform**, I came across a bunch of limitations and issues, which made me start wondering if it was the case to opt for another Google serverless product, namely Cloud Run. 

GCP **Cloud Run** and **App Engine** are fully managed serverless services that let you deploy your app without worrying too much about the underlying infrastructure or such requirements as capacity provisioning, load balancing, or scaling.

While being both serverless, scalable services, they present some major differences:

| App Engine | Cloud Run |
| ----------- | ----------- |
| -it cannot be deployed in different regions within the same project | -services can be deployed in different regions within a project |
| -it comes with a default service that must be deployed | -no top-level app resource or default service | 
| -it works with *versions* | -it works with *revisions* |
| -only App Engine standard can scale to zero | -it can scale to zero |
| -no lower costs for minimum idle instances | -it offers lower costs for idle minimum instances |
| -static content can be either served from Cloud Storage or by configuring handlers  | -static content can be served from Cloud Storage—it hasn't the handlers option  |
| -access can be controlled through ingress settings, IAP and firewalls | -it offers the ability to control access with IAM and Invoker Role, and Ingress rules |

**Terraform** is an open-source, cloud-agnostic tool that allows to achieve reliable, repeatable deployments. Terraform deploys your infrastructure taking care of all its dependencies and sorting them out. 

In this article I will **compare Cloud Run and App Engine**, going through the limitations of deploying their infrastructure with **Terraform**, and I will show you why Terraform works fine with Cloud Run but it **is not suited for** building the **App Engine** service as Infrastructure as code.  

**Use case**: we want to deploy a simple nodejs app on GCP. We do not want to expose the app publicly to the internet, but we want to access it through an external load balancer.
Cloud Run and App Engine will serve both as serverless network endpoint groups (NEGs) backend. 

**Prerequisites**
- Terraform installed locally. You can download it [here](https://www.terraform.io/downloads.html).
- Google Cloud [SDK](https://cloud.google.com/sdk/docs/install). 
- A Google Cloud project.

###Deploying on App Engine 
First thing to know is that App Engine is a regional service and **you cannot have more than one App Engine app** in your project.
Once created, **App Engine cannot be deleted** and you **cannot change the deployment region**, therefore, choose carefully where you want to deploy it.
The **App Engine Flexible** service provides a bunch of additional features that are not included in the Standard environment, among these, the possibility to deploy your app in a private subnet with an internal IP only. 

In order to deploy your app on App Engine, you need to provide a configuration file, called **app.yaml**, that must reside in the root of your application folder. This configuration file will be used by Cloud Build, which—under the hood—builds and deploys your app.
When deploying with Terraform, we zip our application code folder, upload it to a Cloud Storage bucket, and give Cloud Build Service Account permissions to read from that bucket and build the application. You can access the full Terraform code at this [link](https://github.com/elvias9/hello-world-app-engine).

Therefore, when we run `terraform apply`, two processes happen simultaneously:
- Terraform makes a call to google API and build the resources according to its state file
- Cloud Build retrieves the Cloud Storage object, unzips it and builds the app

This can lead to conflicts and mismatching during the deployment, which most of the times cause our `terraform apply` to fail. Furthermore, although Terraform gives you the option of specifying the network field in your  `google_app_engine_flexible_app_version` resource, unfortunately it hasn’t integrated the `internal_ip` argument in the network block yet.

```
# app-engine.tf

resource "google_app_engine_flexible_app_version" "myapp_v1" {
...

 network  {
   name             = google_compute_network.vpc.name
   subnetwork       = google_compute_subnetwork.private.name
   forwarded_ports  = null
   instance_tag     = "app-engine-vm"
   session_affinity = true
 }
...
}
``` 

Hence, running your Terraform code will most likely result in an error probably because of the mismatch between Terraform resource and the app.yaml file. This means that you will not be able to build your infrastructure exposed through the external load balancer.
At the moment of writing this article, the only way to avoid having an ephemeral external IP address assigned to your App Engine instances is to deploy the service through the gcloud CLI.  
App Engine services work with *versions*. Every time you change something in your application code, you can deploy a new version of your service. 
Therefore, the ideal way of managing App Engine deployments would be through a CI/CD pipeline, while Terraform wouldn’t be suited for this service, because it would require manual changes into the `google_app_engine_flexible_app_version` resource, every time a change in the source code is made. 
###Deploying on Cloud Run
Cloud Run is a managed serverless platform that lets you run request-serving containers, directly on Google's scalable infrastructure. It scales in or out automatically depending on requests, and **can scale to zero** if there are no incoming requests, allowing you to **completely cut your costs**. 

Terraform supports Cloud Run starting from Google provider *version 3.3*. In this example we built our Cloud Run service to be able to receive requests only from the internal Google network and the load balancer. This means that our container will be accessible from the internet only through the external load balancer. 
```
# cloud-run.tf

resource "google_cloud_run_service" "run_service" {
 provider = google-beta
 project  = var.project_id
 name     = var.service_name
 location = var.project_region
 template {
   spec {
     containers {
       image = local.app_image
     }
   }
 }
 metadata {
   annotations = {
     "run.googleapis.com/ingress" = "internal-and-cloud-load-balancing"
   }
 }
 autogenerate_revision_name = true
 traffic {
   percent         = 100
   latest_revision = true
 }
}
```
By default, Cloud Run services are secured by IAM, and to access them you need the Cloud Run Invoker permission set. For simplicity, in this example we allow unauthenticated access to our service, by granting the permission set to *allUsers*.
```
# cloud-run.tf

resource "google_cloud_run_service_iam_member" "allUsers" {
 service  = google_cloud_run_service.run_service.name
 location = google_cloud_run_service.run_service.location
 role     = "roles/run.invoker"
 member   = "allUsers"
}
```
Here we built our image directly with Terraform, using a `null_resource` with a `local-exec provisioner`, but you can automate the process with Cloud Build or any other CI/CD platform.
Once your *apply* is done, navigate to your Google Load Balancing console and click on the load balancer address: you’ll be able to see your Hello World app running. Full code [here](https://github.com/elvias9/hello-world-cloud-run).

Cloud Run works with *revisions*. Each revision corresponds to a change made in the configuration or a new image uploaded. Although Terraform is ideal to launch your initial infrastructure to Cloud Run, it is not the best choice to update it, because this would mean manually updating the template block in your resource every time you change something in your application code.
Hence, it is advisable to use Terraform to provision the infrastructure, and another CI step to build and deploy the container. A workaround to separate the two workflows is to add a lifecycle block in your `google_cloud_run_service` resource to ignore the changes in the deployed image, so that Terraform will not complain that the image is different from the one it manages.
```
# cloud-run.tf

lifecycle {
   ignore_changes = [template[0].spec[0].containers[0].image]
 }
```
###Conclusions
As Google state in their [documentation](https://cloud.google.com/appengine/migration-center/run/compare-gae-with-run):

*Cloud Run is designed to improve upon the App Engine experience [...]. Cloud Run services can handle the same workloads as App Engine services, but Cloud Run offers customers much more flexibility [...]. This flexibility, along with improved integrations with both Google Cloud and third-party services, also enables Cloud Run to handle workloads that cannot run on App Engine.*

From a developer perspective, we can add to this statement that, when dealing with the deployment of these two resources as *Infrastructure as Code*, due to the nature of both platforms, Terraform can be a good choice to build the initial infrastructure, while for the actual deployments of the code hosted on them, it is better to opt for a CI/CD tool (e.g. Google Cloud Build).
On top of that, considering that the current version of Terraform has not yet fully integrated all App Engine API features and its API calls often result in errors and deployment failures, when it comes to building the infrastructure with a IaC tool, Cloud Run is definitely the service that can be better implemented with Terraform.
With that said, let's keep an eye open for better integrations of App Engine in future versions of Terraform and Google provider.