Beep boop: Scheduling a twitter bot with Cloud Functions and Cloud KMS encrypted secrets
Published: 2019-12-10 21:01:16

**2023 Update**: Instead of using Cloud KMS, you may have better success using Secret Manager, or [built-in secrets](https://dev.to/googlecloud/migrating-from-secret-manager-api-to-built-in-secrets-4k06)

---


![BeepBoop](https://thepracticaldev.s3.amazonaws.com/i/tzfz7rsll9f2vudsl759.png)


Sometimes, you just need a scheduled task to make the machine go 'beep boop'. With Google Cloud Scheduler, you can make parts of Google Cloud go 'beep boop' in a timely, but secure, manner. 

---

In this example, we'll be creating a Twitter bot that tweets daily. This is based on the real-world example [@python2sunset](https://twitter.com/python2sunset).

We'll be using the following components:

*   **Cloud Scheduler**, which targets a
*   **Cloud Pub/Sub** topic with a specific payload, which
*   **Cloud Functions** listens to, using
*   **Cloud KMS** encrypted environment variables.  

In our example, we'll be posting to a Twitter account. You'll need a set of API keys to use with Twitter, which you can get by [applying for a developer account](https://developer.twitter.com/en/apply-for-access) and [registering your app. ](https://developer.twitter.com/apps)


---

Table of contents: 

* [Example code: beepboop](#example-code-beepboop)
* [Configuring our actuators](#configuring-our-actuators)
* [Creating encrypted secrets](#creating-encrypted-secrets)
* [Deploying our function](#deploying-our-function)


---

### Example code: beepboop

Let's work backwards through this list and start with the function: 

{% gist https://gist.github.com/glasnt/d2d20a9c5ce34f4832823da3c09dc30a file=beepboop.py %}

This function accepts an _event_ and _context_, which our Pub/Sub message payload will supply. We haven't created this payload yet, but we can make it anything we want. In this case, we're going to use a dictionary: 

`{"tweet": "🤖"}`

Our _event data_ will be a base64 encoded representation of this string, which we can decode and then [safely parse](https://docs.python.org/3.8/library/ast.html#ast.literal_eval) to extract our data. We're only continuing on with our processing if we see a specific payload (a dictionary with a "tweet" key). Using this method we can tell our function to tweet any string, functionality we will mention later.

This function also has a third-party package dependency ([`python-twitter`](https://pypi.org/project/python-twitter/)) which we'll need to ensure we include in our provisioning. 

---

### Configuring our actuators

Now we have our function code, we need the parts that make our function run (the 'actuators', "a device that causes a machine or other device to operate"). For that, we'll need to create a Pub/Sub _topic_ and Scheduler _job_ that will trigger our _function_. We'll name all of these literally, prepending our function name.

_For each of these components, we'll link to the related section of the Google Cloud Platform Console with an an example screenshot, and the equivalent [`gcloud`](https://cloud.google.com/sdk/gcloud/) command. You can choose to use either or._

For the _topic_, we'll need a new [Pub/Sub](https://console.cloud.google.com/cloudpubsub/topic/list) topic called beepboop-topic: 


![Creating a topic](https://thepracticaldev.s3.amazonaws.com/i/iljxh824pg5m0x6fksbw.png)

```shell
gcloud pubsub topics create beepboop-topic 
```

For the _job_, we'll need a new [Cloud Scheduler job](https://console.cloud.google.com/cloudscheduler/jobs/new): 

*   called `beepboop-job`, 
*   on a daily frequency ([following cron standards](https://cloud.google.com/scheduler/docs/configuring/cron-job-schedules?&_ga=2.173512739.-109878897.1555001195#defining_the_job_schedule)),
*   in Greenwich Mean Time (GMT),
*   targeting Pub/Sub and the `beepboop-topic` topic we just created,
*   with our aforementioned payload `{"tweet": "🤖"}`

![Creating a Cloud Scheduler Job](https://thepracticaldev.s3.amazonaws.com/i/xkv2we9l3b5e26zdf9oa.png)

```shell
gcloud scheduler jobs create pubsub beepboop-job \
    --schedule "0 0 * * *"  \
    --topic beepboop-topic \
    --message-body "{'tweet': '🤖'}"
```

The schedule "`0 0 * * *`" means ["daily at midnight"](https://crontab.guru/#0_0_*_*_*) (minute 0, hour 0, every day of the month, every month of the year, and every day of the week).

Given our function passes through any value of "tweet", we could setup: 

*   Tweeting "🤖" at midnight, and 
*   tweeting "👾" at midday

by creating a second scheduled job, targeting the same topic, with these different payload values. This allows us to keep the one function code that performs multiple tasks ✨

---

### Creating encrypted secrets 

Before we get to the function, we need to do something about those four environment variables. They are super secret values that if exposed can have nasty consequences. In our case, people can tweet as us, access our account, and our direct messages. 

So, we need to encrypt them. For this, we'll use [Cloud KMS](https://cloud.google.com/kms/) for encryption and decryption. There are other ways to do secrets in Google Cloud. If you're using another part of Google Cloud such as App Engine or Cloud Run, [`berglas`](https://github.com/GoogleCloudPlatform/berglas#integrations) (before v0.5.0) simplifies this process by wrapping around Cloud KMS. Newer versions of `berglas` use Secret Manager (a topic for another post).
 

We're going to create a _key_ on a _keyring_, then _encrypt_ our secrets, add these as _environment variables_ for our function via the console, and _decrypt_ them at run time.

First, we create a [keyring](https://console.cloud.google.com/security/kms/keyring/create) to store our key on. 


![Creating a KMS Key Ring](https://thepracticaldev.s3.amazonaws.com/i/ngqq8dq277jwtzo6ts4n.png)

```shell
gcloud kms keyrings create beepboop-keyring --location global
```

Then a [key](https://console.cloud.google.com/security/kms) on our keyring: 

![Creating a KMS Key](https://thepracticaldev.s3.amazonaws.com/i/4oll6jnr35w8chf7h5nv.png)

```shell
gcloud kms keys create beepboop-key \
    --purpose encryption \
    --keyring beepboop-keyring \
    --location global
```

From here, we can encrypt our secrets. For this, we'll need the combination keyring/key identifier known as the _resource_. For our key, this resource will be `projects/beepboop-project/locations/global/keyRings/beepboop-keyring/cryptoKeys/beepboop-key`

---

For any project, location, keyring and key, this will be: 

`projects/PROJECT/locations/LOCATION/keyRings/KEYRING/cryptoKeys/KEY`

You can generate the specific resource identifier for your key by: 

 * going to the Key on the Keyring in the Google Cloud Platform Console, clicking the "⋮" icon, and selecting "Copy Resource ID"
 * using the gcloud CLI
  * `gcloud kms keys list --location global --keyring beepboop-keyring --format="value(name)"`

---

Now we have our key, we can encrypt our values. Since we have four values to encrypt, and later decrypt, we need to consider the most efficient way to do this. 

To make our application [twelve factor](https://cloud.google.com/blog/products/serverless/a-dozen-reasons-why-cloud-run-complies-with-the-twelve-factor-app-methodology), we can use four separate environment variables. For each of our secret values, we can encrypt them using the same key, then decrypt them in our function. For the command-line deployment method, we can put all these values into a yaml file, which will then be processed for us. For the console method, we will have to copy the values in separately.

For ease of ensuring we don't get any encoding issues, since we're using strings, we'll use a [method similar](https://dev.to/googlecloud/using-secrets-in-google-cloud-functions-5aem#encrypting-your-secret) to our reference article, using base64 encoding -- but processing all the keys in bulk. 

Using this small code snippet, we can take a yaml file of key-value pairs, and encrypt only the value, and returning a new set of key-encryptedvalue pairs: 

{% gist https://gist.github.com/glasnt/d2d20a9c5ce34f4832823da3c09dc30a file=encrypt.py %}

Given a yaml file `secrets.yaml`, which contains one `KEY_RESOURCE_NAME` of your resource identifier, and any other number of key-value pairs, it will encrypt all of them (apart from the `KEY_RESOURCE_NAME` itself), and output the result into a `secrets.yaml.enc` file (the `.enc` extension is just a flag to us that this file is the encrypted version)


```shell
$ cat secrets.yaml
KEY_RESOURCE_NAME: "projects/beepboop-project/locations/global/keyRings/beepboop-keyring/cryptoKeys/beepboop-key"
CONSUMER_TOKEN: AAAAAAAAAAAAAAAAAAAAAAAA
CONSUMER_SECRET: YIJJJjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj
ACCESS_TOKEN: 1111111111111111111-0oaaaaaaaaaaaaaaaaaaaaaaaa
ACCESS_SECRET: z4hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh

$ pip install google-cloud-kms pyyaml
$ python encrypt.py

$ cat secrets.yaml.enc
ACCESS_SECRET: CiQAgOOBPVx...==
ACCESS_TOKEN: CiQAgOOBPeaDhy/NP3rYYP5hc9yJnZZvw+kiIHMdbXWWQhknBQwS...==
CONSUMER_SECRET: CiQAgOOBPQm49gQd+N34gd9NyM7GoY8y//3n5qEqsbbLwxEVEv4SWAb...==
CONSUMER_TOKEN: CiQAgOOBPSp0zXOkD8q1+1tlIA7kWua0CkzncZapHOxhjIsRCS4SQgAb...==
KEY_RESOURCE_NAME: projects/beepboop-project/locations/global/keyRings/beepboop-keyring/cryptoKeys/beepboop-key
```

Our keys may change order, but since we're referencing them by key name, the order doesn't matter.

--- 

Side note: if you want to save this setup for later in a git repo, **do not commit these files**. There is no point making these values secret only to share them on the public internet  😫. To ensure you can't accidentally add them, create a `.gitignore` file and add "`secrets.yaml*`" so you don't commit them. To ensure `glcoud` ignores these files too, add [`!include:.gitignore`](https://cloud.google.com/sdk/gcloud/reference/topic/gcloudignore#ADVANCED-TOPICS) to your `.gcloudignore` file. This will ignore the contents of `.gitignore`!

---

To decrypt these secrets, we'll need to adjust our original function:

{% gist https://gist.github.com/glasnt/d2d20a9c5ce34f4832823da3c09dc30a file=main.py %}

We'll also need to make sure we update our `requirements.txt`:

{% gist https://gist.github.com/glasnt/d2d20a9c5ce34f4832823da3c09dc30a file=requirements.txt %}

We'll **also** need to give our function permission to _decrypt_ using this key. (We as an admin user have rights to do this, but our function, by default, does not have permission).

```shell
gcloud kms keys add-iam-policy-binding beepboop-key \
    --location global \
    --keyring beepboop-keyring \
    --role roles/cloudkms.cryptoKeyDecrypter \
    --member serviceAccount:beepboop-project@appspot.gserviceaccount.com
```

The `serviceAccount` here is the default service account for Cloud Functions. In our local python scripts and in the `gcloud` command line we act as an admin user, so we have permission to do a lot of things. Our function doesn't (and really shouldn't) have all the permissions we the admin have, so we need to be explicit.

---

### Deploying our function

And finally for the _function_, we'll need a [new Function](https://console.cloud.google.com/functions/add): 

*   called `beepboop-function`,
*   triggered on the topic `beepboop-topic`,
*   using our `main.py` and `requirements.txt` files from earlier,
*   executing the function `beepboop`, and
*   specifying our environment variables

![Creating the function](https://thepracticaldev.s3.amazonaws.com/i/qarqyw4seizz79glnq7u.png)

![Environment Variables](https://thepracticaldev.s3.amazonaws.com/i/xppzhaqtervy7qgv8c8b.png)


_Important note: for the `--source` flag you specify a folder, and **everything** apart from the files mentioned in `.gcloudignore` are uploaded. Since we've been playing with secrets files we want to super make sure we don't upload anything we don't want to. So we should create a folder containing only our code and requirements. Put the `main.py` and `requirements.txt` files in their own `code/` folder and the `gcloud` command below will work._

```shell
gcloud functions deploy beepboop-function \
    --trigger-topic beepboop-topic \
    --runtime python37 \
    --entry-point beepboop \
    --source code/ \
    --env-vars-file secrets.yaml.enc \
    --no-allow-unauthenticated
```

Both these methods will result in the same five environment variables. 

---

To test this function, you can go back to the [Cloud Scheduler](https://console.cloud.google.com/cloudscheduler), and click "Run Now": 


![Cloud Scheduler Job Listing](https://thepracticaldev.s3.amazonaws.com/i/8v6ml8yl30kvzh36ycgh.png)

```shell
gcloud scheduler jobs run beepboop-job
```

If everything went to plan, we should have a response on twitter: 

![Beep boop tweet](https://thepracticaldev.s3.amazonaws.com/i/ga7h8jico70kra0voznl.png)

Beep boop!