How to use secured wildcard custom domains on AppEngine
Tags: googlecloud, ssl, appengine, https
Published: 2021-02-10 17:48:01

Today, we will learn to enable HTTPS and wildcard custom domain mapping on Google App Engine.

# But why ?

With my client, we wanted to use AppEngine for our frontend applications.

[Google AppEngine](https://cloud.google.com/appengine) deploys static web applications easily with performant and fully managed servers. It also offers the possibility to have different versions of the same app running and being available at the same time.

## Access specific application's version

For this to work, each version of each service has a dedicated URI in this form: `https://version-dot-service-dot-project.region.r.appspot.com`. This is very powerful and handy to manage multiple environments. In our project, we have a production version, a development version and we plan to have one version for each pull request opened on Github.

The only drawback is the readability and the difficulty to remember these URIs. Google enables you to configure your own custom domain and let you dispatch this domain to a given service. But a more flexible approach is to define a wildcard custom domain for the project. With a wildcard custom domain, you can enjoy flexible routing, juste like with default generated URI, but with your own custom domain: http://version.service.example.com.

## Here comes trouble

Fine! We have a clean URI for each version of each service. But here comes the biggest issue with this system: You can't enable HTTPS for your wild card custom domain since Google's managed SSL certificates are not available for wildcard domains.

"But why?" you will probably ask yourself. To issue a wildcard SSL certificate with LetsEncrypt, you need to achieve a [DNS-01](https://letsencrypt.org/docs/challenge-types/#dns-01-challenge) challenge instead of HTTP-01 challenge. To automate certificate renewal (which is required given their TTL), you need to modify your DNS zone via API, which is different for each DNS provider. That is why Google does not (for now) implement managed wildcard SSL certificates.

# Enable wildcard SSL certificate with automatic renewal

After this far too long introduction, we will finally learn how to add HTTPS for our beautiful URI with our beautiful custom domain.

At the end, we will have the following workflow:

![ssl certificate automation workflow](https://dev-to-uploads.s3.amazonaws.com/i/g0rbgul48rnzkf8sk991.png)

## Prerequisite

The main prerequisite is to use a compatible DNS provider. Let's encrypt automatic renewal of SSL certificates is only compatible with these providers:
  - google
  - cloudflare
  - digitalocean
  - dnsimple
  - linode
  - ovh
  - rfc2136
  - route53

To follow this guide, you will need a running App Engine project with at least 1 service. Of course it's more impressive if you have more than one !

If you don't have one (or prefer not to mess with your real app), you can deploy [this example](https://github.com/Zenika/gcloud-certbot/tree/main/example/app-engine) with a beautiful cat and dog !

## Step 1: Setup a service account

For our Cloud Run service to manage SSL certificates and access Cloud Storage bucket, it will need a service account with all the authorizations needed.

> You can also edit the default Cloud Run service account, but it is not recommended since it opens a potential security breach for existing or future Cloud Run services.

[Create a new dedicated service account](https://console.cloud.google.com/iam-admin/serviceaccounts) named `certbot` (`certbot@<project-id>.iam.gserviceaccount.com`) with the following role `App Engine Administrator` (needed to upload SSL certificates) and `Cloud Run Invoker` (needed by Cloud Scheduler to call the Cloud Run service).

> If your DNS zone is managed by google, you should also give the permissions to manage DNS records (`DNS Administrator`). This is needed by `certbot` to resolve the DNS challenge.

Add yourself as a user and administrator of this service account.

## Step 2: Add custom wildcard domain

Go to [App Engine custom domain setting page](https://console.cloud.google.com/appengine/settings/domains) to create and add the custom domain mappings.

### Add a custom domain

To add a custom domain, you have to first register it and give to the service account the permissions to manage it's certificates.

Go to the [domain registration page](https://www.google.com/webmasters/verification/verification).

> If you haven't registered any domain yet, you will have to follow the instructions and resolve a challenge to ensure that you are the owner of the domain.
> During this step, make sure to enter your domain without the wildcard. For example, if you want to use `*.my.domain.com`, make sure to input `my.domain.com`, without `*` nor `http://` (the placeholder is misleading here).
>
> **WARNING**: Not following this rule will cause issues later in this guide when we will try to upload an SSL certificate, the domain is recognized only if it matches exactly the one specified in the SSL certificate.

Open the details page for your domain, and add the previously created service account's email (`certbot@<project-id>.iam.gserviceaccount.com`) as an owner.

### Create the mappings

You should now be prompted for domain mappings. Add your wildcard domain (`*.my.domain.com` for example).

You should have 3 domain mapping :
 - `my.domain.com`
 - `www.my.domain.com`
 - `*.my.domain.com`

> With this configuration, the default service will be served on both `https://my.domain.com` and `https://www.my.domain.com`. If this is not desired, you can just remove them and only keep the wildcard mapping.

![domain mapping example](https://dev-to-uploads.s3.amazonaws.com/i/f7zo8ov6tlyj2x4s3qu1.png)

### Edit your DNS zone

You should now see a list of DNS records. You have to edit your DNS zone with these new records.

> If, in the previous step, you only kept the wildcard mapping, you can just add the CNAME entry

## Step 3: Create a Cloud Storage bucket

To manage automatic certificate renewal, `certbot` needs to keep a state on the filesystem. Since Fully Managed Cloud Run doesn't provide any persistent FS, we will have to backup and restore it from a Cloud Storage bucket.

Go to the [bucket creation page](https://console.cloud.google.com/storage/create-bucket) and give it a unique name (`certbot-sate_my-domain-com` for example).

You can choose whatever region you want (but try to put it at least in the same region as your Cloud Run and App Engine services).

Choose a storage class. Nearline is fine, or you can even choose Codline if you want to save the planet 🌳.

You can use Uniform access control since the entire bucket will only be used by our renewal service.

When created, go to `AUTHORIZATION` tab and add a new member. There, put your service account email (`certbot@<project-id>.iam.gserviceaccount.com`) and give it the `Old Storage Bucket Owner`) role.

## Step 4: Create the Cloud Run service

Go to [Cloud Run service creation page](https://console.cloud.google.com/run/create), choose a region and a name (`certbot` for example).

Use the `eu.gcr.io/gcloud-certbot/gcloud-certbot` image.

> You can also [build and publish the image yourself](https://github.com/Zenika/gcloud-certbot) in your project. This is better for reliability and security.

Under `Advanced Settings`, change this configurations:
 - Service Account: use the service account created during step 2 (`certbot@<project-id>.iam.gserviceaccount.com`).
 - Delay before request timeout: 600 (an SSL renewal is not a high speed process... so 10 minutes should be ok)
 - Max instances number: 1
 - Max number of requests by container: 1 (we don't want to be able to renew multiple time in parallel)
 
Go to `Variables` tab and add the following variables:
 - LETSENCRYPT_BUCKET: The bucket you have created (example: `gs://certbot-sate_my-domain-com`)
 - CERTIFICATE_NAME: The display name of the certificate. This will be visible in the App Engine Console, it should be unique for your application.
 - CUSTOM_DOMAIN: Your naked custom domain (example: `my.domain.com` if your wildcard is `*.my.domain.com`)
 - DNS_PROVIDER: The name of your DNS provider. It should be one of:
  - google
  - cloudflare
  - digitalocean
  - dnsimple
  - linode
  - ovh
  - rfc2136
  - route53
 - DNS_PROVIDER_CREDENTIALS: The credential file content. It should contain the access token for your DNS provider. You can find [documentation depending on your provider here](https://certbot.eff.org/docs/using.html#dns-plugins)
  > If you are using Google Domains for your DNS, omit this variable. You should instead add authorization to your service account to modify your DNS record.
 - LETSENCRYPT_CONTACT_EMAIL: Your contact email. This will be sent to Let's Encrypt for record and will be publicly visible, so be sure to create an email for this specific use.

Next, let the default publicly opened traffic (Cloud Scheduler uses external HTTP requests), but enable `Require Authentication` to avoid some malicious people from being able to trigger SSL certificate renewal.

Validate, you should be redirected to the service administration page. There, go to the `Authorization` tab and add a member. Add the service account (`certbot@<project-id>.iam.gserviceaccount.com`)


### Enable App Engine Admin API

If your project is new or you have never automated anything related to App Engine, you probably need to [enable App Engine Admin API](https://console.developers.google.com/apis/api/appengine.googleapis.com/overview).

This will be required by the Cloud Run service to upload the generated SSL certificate.

## Create a Cloud Scheduler task

To renew our certificate every 2 months, we will create a Cloud Scheduler task.

Go to the [creation page](https://console.cloud.google.com/cloudscheduler/jobs/new) and use this configuration:
 - Name: appengine_widlcard_certificate_renewal
 - Frequency: 0 1 1 */2 *
 - Time Zone: Your time zone :-)
 - Target: HTTP
 - URL: <your cloud run service URL>/renew
 - HTTP Method: GET
 - Authentication Header: OIDC token (click on `Show More` button)
 - Service Account: the create service account (`certbot@gcloud-certbot.iam.gserviceaccount.com`)
 - Target: <your cloud run service URL>/renew

Validate, and you should now see your task.

You try it by clicking on `Execute`. Click on `Display` in the `Logs` column to see it working !

If the task has finished without errors, you should see your custom SSL certificate in [the App Engine console](https://console.cloud.google.com/appengine/settings/certificates).

> **Troubleshooting**
>
>If there is a problem, and there will probably be at least one, you can disable the required authentication from your Cloud Run service. This will allow you to call the renewal endpoint manually and see the output logs.

> ```bash
gcloud run deploy certbot --image gcloud run deploy certbot --image eu.gcr.io/gcloud-certbot/gcloud-certbot --allow-unauthenticated
```

> Don't forget to enable it when you have fixed all issues to avoid malicious calls:

> ```bash
gcloud run deploy certbot --image eu.gcr.io/gcloud-certbot/gcloud-certbot:latest --no-allow-unauthenticated
```

Congratulations ! You can now access every version of every service with your beautiful custom domain `https://version.service.my.domain.com` \o/

# Conclusion

This guide has been made during my work at Zenika for my client. The final goal of all this was to being able to publish our features branches with ease. And I think this goal is now achieved, we can now deploy a new version an app for a given PR with a simple command:

```bash
gcloud app deploy . my-service --version pr-1234 --no-stop-previous-version --no-promote
```

I hope this will help you do the same for your own app !

That being said, since Google Cloud decided to put less efforts on GAE, it can be a good idea for new project to look for a different solution. Firebase is currently working on this and we have made another guide (longer of course) [guide to do the same thing on GKE](https://dev.to/zenika/google-kubernetes-engine-circleci-and-traefik-for-a-full-fledged-gitops-platform-in-the-cloud-part-1-2bai)