TerraForm IaC on Google Cloud Platform provisioning CloudRun and CloudEndpoints 
Tags: googlecloud, terraform, openapi
Published: 2020-09-28 06:06:40

Side note: I'm an intermediate Sys admin and have been building my skills to be proficient. If there's a mistake, please let me know. 

### Warning: Using Terraform v0.13 

What is cloud run? Cloud run is a fully self-service managed infrastructure that is capable of automated scaling. 
[cloudrun](https://cloud.google.com/run/docs)
What is TerraForm? Terraform is a multi-cloud infrastructure as code that is open-source. This eliminates human error and reduces time. 
[TerraForm](https://www.terraform.io/intro/index.html)
In this how-to article, I'll show you how to deploy your own API on Cloud Endpoints and use the cloud run service for both Application and API gateway ESPv2 image with Terraform and cloud build. 
I followed this tutorial and replicated it through CI/CD pipeline GettingStarted [CloudEndpoints](https://cloud.google.com/endpoints/docs/openapi/get-started-cloud-run#deploy_configuration)

Prerequisites
* Create Google Cloud Project and account [GCPProject] (Fresh project will need API's and billing enabled) 
* Create Github Account and new repository on Github
* Enable Cloud Build app in Github Repo.  and  create cloud build triggers [CloudbuildTr]
(https://cloud.google.com/cloud-build/docs/automating-builds/create-github-app-triggers)

* Create Cloud run GO application, simple hello world CloudrunApp
* Enable Google container registry GCR

Let's start with CI/CD pipeline through, my environment has TerraForm in version control (GitHub) with Cloud build triggers app on Github and on Google Cloud Platform. 

I have another write-up. For senior or advance practitioners move to [here](#let's-start) 

Configuring terraform and google cloud platform will be up to you to decide what is best for your environment but I would highly recommend best practices for both TerraForm and GCP with security in mind. Securing Cloud Run services [HERE](https://cloud.google.com/run/docs/tutorials/secure-services#run_secure_invoker-python) Also, permissions are important and necessary in order to deploy TerraForm configuration for use of least privilege permissions. I like to keep TerraForm DRY with reusable infrastructure as code so that I can reuse modules. Another key part in this is the template version which I have set to version = "2.1.2".  
I have Cloud Build Triggers app within GitHub repo for dev, staging, and prod. This will create changes that I have made using "VScode" and commit to Github. Cloud Build triggers will be notified by branch name of the commit and running either dockerfile, cloudbuild.yaml. 
Cloudbuild.yaml configs, I've set the build time to 7200s just to make sure cloud build gives ample time to finish deploying TerraForm configuration. I grab these simple and customizable steps from this tutorial [HERE](https://cloud.google.com/solutions/managing-infrastructure-as-code) Basic fundamentals of using TerraForm and GCP Cloud Build.
Versions used in this tutorial 
~Google provider 3.35.0
~TerraForm v.0.13.0
Once everything is configured, let's start deploying some TerraForm GCP resources 

###Let's Start
Start by creating a dev folder structure. Here's what's going on, backend.tf is saving remote state to cloud storage bucket. Steps to setup remote state is in the above tutorial, Managing Infrastructure as code. Main.tf is Terraform root dir config, which holds the modules and google provider. Outputs.tf file will pass the output URL from the cloud run modules. Versions.tf is important to keep every TerraForm in sync. 

├── README.md
├── cloudbuild.yaml
├── gcloud_build_image
├── environments
│   └── dev
│       ├── backend.tf
│       ├── main.tf
│       ├── outputs.tf
│       └── versions.tf
│   ├── staging
│   ├── prod
├── modules
│   └── services
│       ├── CloudRun
│       │   ├── cloudrun.tf
│       │   ├── outputs.tf
│       │   └── variables.tf
│       ├── cloudEndpoints
│       │   ├── endpoints.tf
│       │   ├── openapi_spec.yml
│       │   └── variables.tf


In the Root dir. we have cloudbuild.yaml and gcloud_build_image, also do a readme.md to explain what the repo and how-to so that other team members can get started. 
We have modules/services/ TerraForm resources configurations of cloud run and cloud endpoints. For the sake of this tutorial, I have used hello world with a simple cloud endpoint that are used for the hello GO app. You may need to check configuration for security and if needed use CORS and set IAM permission to the cloud run services.  

Moving on to the cloud run module configs I will start with cloud run services. Here we will use the default of cloud run with IAM policy "no-auth" and created variables for the name, location, and a docker image. 
```tf
# ------------------------------------------------------------------------------
# GCP cloud run application 
# ------------------------------------------------------------------------------

resource "google_cloud_run_service" "default" {
  name     = var.name
  location = var.location
  template {
    spec {
      containers {
        image = var.dockerimg 
      }
    }

  }
  
  traffic {
    percent         = 100
    latest_revision = true
  }
  autogenerate_revision_name = true
}
data "google_iam_policy" "noauth" {
  binding {
    role = "roles/run.invoker"
    members = [ 
      "allUsers",
  ]
 }
}
resource "google_cloud_run_service_iam_policy" "noauth" {
  location = google_cloud_run_service.default.location
  project  = google_cloud_run_service.default.project
  service  = google_cloud_run_service.default.name
  policy_data = data.google_iam_policy.noauth.policy_data
}
```
```tf
# ------------------------------------------------------------------------------
#  variables for Cloud Run                                                              
# ------------------------------------------------------------------------------
variable "name" {
    description = "variable name for cloud run" 
    type = string 
}
variable "location" {
    description = "setting location of service"
    default = "us-central1"
}
variable "project" {
    description = " setting project name"
    type = string 
}
variable "dockerimg" {
  description = "docker img to be used"
  type = string
}
```
```tf
# ------------------------------------------------------------------------------
#  outputs for Cloud Run                                                              
# ------------------------------------------------------------------------------
output url {
  value = google_cloud_run_service.default.status[0].url
}
output urlesp {
  value = "${trimprefix(google_cloud_run_service.default.status[0].url, "https://")}"
}
```

Next, we have cloud endpoints module configs. I've used the data template file to import openapi_spec.yaml into the cloud endpoint config, I also have it in the cloud endpoint module dir. A couple of things going on here, I've created variables for data openapi.yaml config. This will pass in variables from the TerraForm root config into the cloud endpoint module. 
```tf
# # ------------------------------------------------------------------------------
# # Cloud endpoints
# # ------------------------------------------------------------------------------

data "template_file" "openapi_spec" {
 template = "${file("${path.module}/openapi_spec.yml")}"
 vars = {
   CloudRunES = var.CloudRunESurl ,
   HelloAPI = var.ClRnSrvapp
 }
}

resource "google_endpoints_service" "api-service" {
  service_name   = var.CloudRunES2url
  project        = var.project
  openapi_config    = data.template_file.openapi_spec.rendered
}
```
```yaml
 swagger: '2.0'
  info:
    title: Cloud Endpoints + Cloud Run
    description: Sample API on Cloud Endpoints with a Cloud Run backend
    version: 1.0.0
  host: ${CloudRunES}
  schemes:
    - https
  produces:
    - application/json
  x-google-backend:
    address:${HelloAPI}
    protocol: h2
  paths:
    /hello:
      get:
        summary: Greet a user
        operationId: hello
        responses:
          '200':
            description: A successful response
            schema:
              type: string
```
```tf
variable "project" {
  description = "name of project"
  type        = string
}
variable "CloudRunESurl" {
   type = string
}
variable "ClRnSrvapp" {
   type = string 
}
```
Ok so now that we have the module services configured we can jump back to the staging folder and configure the Terraform root config (main.tf) 
Cloud Run ESPv2 service will be created at the same time as Cloud Run GO application.
Further guide is at this [doc](https://cloud.google.com/endpoints/docs/openapi/get-started-cloud-run#deploy_configuration)
```tf
# # ------------------------------------------------------------------------------
# # Terraform provider
# # --------------------------------------------------------------------------------
provider google {
    project = "Project_ID"
    region = "var.region" 
}
provider "template" {
  version = "2.1.2"
}
# ------------------------------------------------------------------------------
#  cloud run and cloud endpoints
# ------------------------------------------------------------------------------
module "HelloAPI" {
   source = "../../modules/services/CloudRun"
   name = "hellotest"
   location = "us-central1"
   project = "Project_ID"
   dockerimg = "gcr.io/${Project_ID}/cloud-run-hello:v2"
}
module "CloudApiESP" {
  source = "../../modules/services/CloudRun"
  name = "cloudrunesp"
  location = "us-central1"
  project = "Project_ID"
  dockerimg = "gcr.io/${Project_ID}/endpoints-runtime-serverless:cloudapiesp-qutnc7nuq-uc.a.run.app-2020-08-017r0"
}
# ------------------------------------------------------------------------------
#  variables for Cloud endpoints                                                       
# ------------------------------------------------------------------------------
module "cloudEndpoints" {
  
    source = "../../modules/services/cloudEndpoints"
    project = "Project_ID"
    
    CloudRunESurl = "${module.CloudApiESP.urlesp}"
    ClRnSrvapp    = "${module.HelloAPI.url}"
}
```
The above config is calling module services cloud run and in that module we are creating name, location, project, and inputting docker image, which is from Google Container Registry. Cloud Run only runs images from GCR.  Next, in cloud endpoints module we are creating cloud endpoint and the only configuration we are doing is passing in env variables. What this means is passing the output urls from cloud run module to main config as an output and then to cloud endpoints module env. vars. This is the only way to pass environment variables to main config and then to other module service. 
Finally, we're going to create the cloud build yaml config for Continuous deployment. What this does is that cloud build will run steps to create a alpine busy box per say and install Terraform and allows for cloud build to cd into the root folder and modules configurations and be provisioned and deployed. I've also typed in TF_LOG=TRACE which will display TerraForm execution in the background through the cloud build, build log. 
Cloudbuild.yaml

```yaml
# Copyright 2019 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
timeout: 7200s
steps:

# - name: gcr.io/cloud-builders/gcloud
#   entrypoint: 'bash'
#   args: 
#     - '-c'
#     - |-
#       chmod +x gcloud_build_image 
#       ./gcloud_build_image -s ${cloudrun-esp}-${cloudrun-hash}-uc.a.run.app -c ${config-id} -p ${project-id}
 
  
- id: 'branch name'
  name: 'alpine'
  entrypoint: 'sh'  
  args: 
  - '-c'
  - | 
      echo "***********************"
      echo "$BRANCH_NAME"
      echo "***********************"
  #[start tf-init]
- id: 'tf init'
  name: 'hashicorp/terraform:0.13.0'
  entrypoint: 'sh'
  args: 
  - '-c'
  - |
      if [ -d "environments/$BRANCH_NAME/" ]; then
        cd environments/$BRANCH_NAME
        terraform init
      else
        for dir in environments/*/
        do 
          cd ${dir}   
          env=${dir%*/}
          env=${env*/}
          echo ""
          echo "*************** TERRAFORM INIT ******************"
          echo "******* At environment: ${env} ********"
          echo "*************************************************"
          terraform init || exit 1
          cd ../../
        done
      fi 
  
# [START tf-plan]
- id: 'tf plan'
  name: 'hashicorp/terraform:0.13.0'
  entrypoint: 'sh'
  args: 
  - '-c'
  - | 
      if [ -d "environments/$BRANCH_NAME/" ]; then
        cd environments/$BRANCH_NAME
        terraform plan
        
      else
        for dir in environments/*/
        do 
          cd ${dir}   
          env=${dir%*/}
          env=${env*/}  
          echo ""
          echo "*************** TERRAFOM PLAN ******************"
          echo "******* At environment: ${env} ********"
          echo "*************************************************"
          terraform plan  || exit 1
          cd ../../
          cat crash.log 
        done
      fi 
  
 # [END tf-plan]
#[START tf-apply]
- id: 'tf apply'
  name: 'hashicorp/terraform:0.13.0'
  entrypoint: 'sh'
  args: 
  - '-c'
  - | 
      if [ -d "environments/$BRANCH_NAME/" ]; then
        cd environments/$BRANCH_NAME      
        export TF_LOG=TRACE
        terraform apply -auto-approve 
      else
        echo "***************************** SKIPPING APPLYING *******************************"
        echo "Branch '$BRANCH_NAME' does not represent an oficial environment."
        echo "*******************************************************************************"
      fi

 
 
#[START tf-destroy]
# - id: 'tf destroy'
#   name: 'hashicorp/terraform:0.13.0'
#   entrypoint: 'sh'
#   args: 
#   - '-c'
#   - | 
#       if [ -d "environments/$BRANCH_NAME/" ]; then
#         cd environments/$BRANCH_NAME      
#         export TF_LOG=TRACE
#         terraform destroy -auto-approve 
#       else
#         echo "***************************** SKIPPING APPLYING *******************************"
#         echo "Branch '$BRANCH_NAME' does not represent an oficial environment."
#         echo "*******************************************************************************"
#       fi
  #[end tf-destroy]
```


Once all has been configured, we can then commit to GitHub branch to deploy. We will have two Cloud Run services and one Cloud Endpoints. 
Next, we will need to redeploy Cloud Run ESPv2 by rebuilding it, using the gcloud_build_image script provided at the bottom of this tutorial [GcloudBuildImage](https://cloud.google.com/endpoints/docs/openapi/get-started-cloud-run)



```yaml
# - name: gcr.io/cloud-builders/gcloud
#   entrypoint: 'bash'
#   args: 
#     - '-c'
#     - |-
#       chmod +x gcloud_build_image 
#       ./gcloud_build_image -s ${cloudrun-esp}-${cloudrun-hash}-uc.a.run.app -c ${config-id} -p ${project-id}
```

We will need to copy Google Container Registry image name so that we can enter that in the Cloud Run ESPv2 Cloud Run service to be redeployed. All we need to do is copy the ESPv2 image name and paste it in the dockimg= ESPv2 image name in the Cloud Run ESPv2 module of TerrForm main.tf. 
We should be able to go to the Cloud Endpoint URL is a form of Cloud Run ESPv2 URL /hello.

 i.e https://${cloudapiesp-url}-CloudRun-hash.a.run.app/hello


Next, we will change some code on the Cloud Run GO application for continuous integration while rebuilding Cloud Run service. I have a separate GitHub repo. for my Cloud Run application. 
Inside of that separate Cloud Run application GitHub Repo. I have a cloudbuild.yaml that is configured with Cloud Build Triggers to that GitHub repo. So everytime I make a commit, I can then run the Continous integration everytime I commit. 

```yaml
steps:
 -name: 'gcr.io/cloud-builders/docker'
 args: [ 'build', '-t', 'gcr.io/project_id/cloudrun-hello', '.']
-name: gcr.io/cloudbuilders/docker'
args: [ 'push', 'gcr.io/project_id/cloudrun-hello']
-name: 'gcr.io/cloud-builders/gcloud'
args:
[
 "run", 
"deploy",
"hellotest",
" - image",
"gcr.io/project_id/cloudrun-hello",
" - region",
"us-central1",
" - platform",
"managed",
" - allow-unauthenticated",
]
```

You could also build the Cloud Run hello application with a new tag then, update the cloud run on the TerraForm config to deploy with the new tag. Cloud Run is not aware of any application update and doesn't automatically pull the latest image from GCR. In the above gcloud cloud build config, you can run a build of docker container image with the new tag
i.e
```yaml 
args: [ 'build', '-t', 'gcr.io/project_id/cloudrun-hello:v2', '.'] 
```
Instead of running the docker push command and gcloud deploy command. 
Eliminating overhead and complexity. 

Then, head back to the module HelloAPI and change the docker image to the version that you set above in the cloudbuild step config.


Let me know of feedback