Things I did to run Kubernetes Pods with Terraform
Tags: kubernetes, terraform
Published: 2019-11-17 14:10:25

---
title: Things I did to run Kubernetes Pods with Terraform
published: true
description: Things I did to run Kubernetes Pods with Terraform
tags: Kubernetes, Terraform
---

# Terraform

I created an infrastructure in my project and configured the Kubernetes config for Google Cloud Kubernetes Engine.

```
gcloud container clusters list
gcloud config set project <your-project-name>
gcloud config list
gcloud container clusters list
gcloud container clusters get-credentials dev-cluster
kubectl config current-context
kubectl config view
kubectl config use-context gke_<your-project-name>_asia-northeast1-a_dev-cluster
kubectl apply -f ./manifests/generated/dev -R
```

We need to create _credentials.json_ for Cloud SQL Proxy and then apply manifests using it.

What I did for applying manifest


1. Create a service account for Cloud SQL Proxy's credential and encode using base64.
2. Create a secret in the Kubernetes
3. Create dev.env

```
$ kubectl create secret generic env-config --from-env-file envs/dev.env
```

```
kubectl logs dev-delivery-74b5975c64-cr2bx -c delivery
kubectl logs dev-delivery-74b5975c64-cr2bx -c cloudsql-proxy
```

Change Cloud SQL Proxy’s instance option to “-instances=<your-project-name>:asia-northeast1:.Values.env-master01-4eacaf8e22e3aa79=tcp:3306”

Use the Instance connection name.

Migration

1. lunch Cloud SQL Proxy
2. change the data source in dbconfig.yml
2. create database rdn
3. apply migration files

Create a docker image and push it.

1. `CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -installsuffix cgo -o dist/app ./microservices/device/main.go`
2. `docker build -t asia.gcr.io/<your-project-name>/device . --build-arg MICROSERVICE_NAME=device`
3. `docker push asia.gcr.io/<your-project-name>/device`

It uses for determining an image tag pulling from GCR.

I don't know how to determine the value of CPU and Memory resources.

Set enough resource value for CPU and Memory. If you be cheap, you will be stuck in some problems.

```
occurred upload error when CreateResource call repository: Post https://www.googleapis.com/upload/storage/v1/b/dev-delivery-resources/o?alt=json&prettyPrint=false&projection=full&uploadType=multipart: oauth2: cannot fetch token: Post https://oauth2.googleapis.com/token: net/http: TLS handshake timeout

UploadGcs Error ; msg objectName: d55bddf8d62910879ed9f605522149a81569317440.mp4 errors: Post https://www.googleapis.com/upload/storage/v1/b/dev-delivery-resources/o?alt=json&prettyPrint=false&projection=full&uploadType=multipart: oauth2: cannot fetch token: Post https://oauth2.googleapis.com/token: net/http: TLS handshake timeout
```

I thought above errors are authentication error; however, lack of resource (CPU, Memory) occur the errors.

Build delivery, docker build and push

```
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -installsuffix cgo -o dist/app ./microservices/device/main.go
sh docker.sh register delivery
```

```
An unknown error occurred, please check the HTTP result code and inner exception for server response.
```

If this error showed up, you should integrate a bucket between Firebase Storage and GCS.

# Tips

If we upgrade the version, we will need to change a template. So we should not update any version usually.

We use Ingress when to publish an IP outside a cluster.
Either Service type Node port or Load balancer can use.

Helm can use a conditional template such as

```
{- if eq .Values.env "dev" }
                  "-instances=.Values.project_name:asia-northeast1:.Values.env-master01=tcp:3306,,<your-project-name>:asia-northeast1:.Values.env-slave01=tcp:3307",
                  {- else }
                  "-instances=<your-project-name>:asia-northeast1:.Values.env-master01=tcp:3306,<your-project-name>:asia-northeast1:.Values.env-slave01=tcp:3307,<your-project-name>:asia-northeast1:.Values.env-slave02=tcp:3308",
                  {- end }
                  "-credential_file=/secrets/credentials.json"]
```

I used random ID as a suffix to Cloud SQL. However, we usually don't apply manifest repeatedly. So I removed the random id. If we need it soon, I will add it again.

Q: I want to install a specific version of Terraform.

It solved using by _tfswitch_

https://medium.com/@warrensbox/how-to-manage-different-terraform-versions-for-each-project-51cca80ccece

```
Error: Error, failed to create instance dev-master01: googleapi: Error 400: The incoming request contained invalid data., invalidRequest

  on modules/google-cloud-sql/main.tf line 1, in resource "google_sql_database_instance" "master":
   1: resource "google_sql_database_instance" "master" {
```

Cloud SQL and Terraform documents don't have information on the minimum disk size of Cloud SQL Second-generation. This information appears in the Cloud SQL API Document that the value is 10 GB. If we specify the disk size less than 10 GB, it occurs error.

```
"dataDiskSizeGb": "A String", # The size of data disk, in GB. The data disk size minimum is 10GB. Not used for First Generation instances.
```

https://developers.google.com/resources/api-libraries/documentation/sqladmin/v1beta4/python/latest/sqladmin_v1beta4.instances.html

I often run into the error "The incoming request contained invalid data." This error was caused by invalid parameters when I requested. At first, We should check the below list with GCP and GCP API document.

- Is the value in a valid range?
- Correct format?