Migrating from Secret Manager API to built-in secrets
Tags: googlecloud, secretmanager, tutorial
Published: 2022-06-19 23:17:12

Secret Manager is directly integrated into a number of products, including Cloud Run and Cloud Build. 

Many samples I've authored use the Secret Manager API by calling the API from within the code. This can be [more secure](https://cloud.google.com/secret-manager/docs/best-practices#coding_practices), but variable leakage is something that needs to be considered in any deployment. Removing the direct API calls can help with portability, and reduce the amount of dependencies in the deployment, and the complexity of the code itself. 

So let's look at how we can migrate to built-in secrets. 

---

## Integrating Secret Manager API

![Secret value: Input your secret value or import it directly from a file.", showing a file upload dialog, or a text area input field.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5i6owi4brzc6hfkm1ub8.png)

_Screenshot: "Secret value: Input your secret value or import it directly from a file.", showing a file upload dialog, or a text area input field._

When you create secrets, you can create them from a file or from direct input. When you retrieve these secrets, you may presume they're a single value, or a file of content you have to process. 

Once you create the secret, you can [access it with the Secret Manager API](https://cloud.google.com/secret-manager/docs/creating-and-accessing-secrets#secretmanager-access-secret-version-python) using a client library for Secret Manager available in many different languages. 

This post will use Python examples and packages, but the same patterns can be applied to other languages. 


---
For example, this code snippet will retrieve the latest version of the secret `mySecret` in the current project:


```python
import google.auth
from google.cloud import secretmanager as sm

# trick to detect current project
_, project = google.auth.default() 
client = sm.SecretManagerServiceClient()

secret = "mySecret"

name = f"projects/{project}/secrets/{secret}/versions/latest"
secret_value = client.access_secret_version(path).payload.data.decode("UTF-8")
```


The `secret_value` can then be used as required. This presumes that `mySecret` is a single value, for example an API key.

One way developers can include multiple secrets in one format is with `.env` files. These files contain one or more key/value pairs which mean you can store multiple secrets in one file, then use helper packages to parse these values into your code. This prevents you from having to define multiple separate secret values. 

If `mySecret` was a `.env` file, I could load it using `python-dotenv: `


```python
## pip install python-dotenv
import io
from dotenv import load_dotenv

load_dotenv(stream=io.StringIO(secret_value))
```


Or I could to similar with `django-environ`, for example: 


```python
## pip install django_environ
import io
import environ
env = environ.Env()
env.read_env(io.StringIO(secret_value))
```


In both of these examples, these packages presume that the values are in a `.env` file within the same directory as the running process. Using `StringIO` means that a string variable can be sent to these methods in a file-like object, so the values can be read as though they were a file.

When using the Secret Manager API, you can run the application in any place where Google Cloud authentication works. That is, any Google Cloud service, or your local machine if you have set up `gcloud`. 


## Consuming built-in secrets

If you have your secret configured as an environment variable, then you only have to retrieve it as an environment variable: 


```python
import os
secret_value = os.environ.get("MYSECRET", None)
```


_This method prevents KeyErrors if the key is not found._

For products with built-in secrets, they may be made available in additional ways. 

In Cloud Build you can connect secrets [directly to environment variables](https://cloud.google.com/build/docs/securing-builds/use-secrets). But Cloud Run additionally allows [mounting the secret as a volume](https://cloud.google.com/run/docs/configuring/secrets), which means it can be read as a file. Any time the file is read, if you specify `latest`, the current `latest` will be received! However, you must mount the file in a new volume so you can't mount it directly in the default `.env`. 

If you want to adapt your code to handle a local `.env` file, or a separately mounted file, or an environment variable, you will have to ensure all methods are possible in your code. 

You also need to consider which configuration takes priority, as by default both `python-dotenv` and `django-environ` accept the first declared value as the value they use. You can override this by using the <code>[--override](https://github.com/theskumar/python-dotenv#variable-expansion)</code> or <code>[--overwrite](https://github.com/theskumar/python-dotenv#variable-expansion)</code> respectively.

When developing applications, I might choose to say my local `.env` file takes priority, then any mounted secrets, and then any declared variables. 

Another point to mention is that using `python-dotenv`, if a file is not found or a value is empty, it silently continues. This means you can include the method calls without having to explicitly handle errors:


```python
import io
import os

from dotenv import load_dotenv

load_dotenv()
load_dotenv("/secrets/.env")
load_dotenv(stream=io.StringIO(os.environ.get("MYSECRET", None)))
```


The same code works for `django-environ`, where you can just import in the order of priority without having to worry about missing files.


```python
import io
import os

import environ
env = environ.Env()

env.read_env()
env.read_env("/secrets/.env")
env.read_env(io.StringIO(os.environ.get("MYSECRET", None)))
```


Note that in these examples, I'm choosing `/secrets/`as my volume, and keeping the path the same name as the original file. You can choose any volume and path, as long as the volume is not already used by the application (for example, if you choose `/app/` as your working directory, you cannot mount secrets there.)


## Deploying built-in secrets

To run this code locally, you'd create a `.env` file with the contents `MYSECRET=serkitValue`. 

If you're committing this code to git, ensure you're not committing the secret file! Make sure you add `.env` to your `.gitignore` file!

You can also choose to ignore any contents of your `.gitignore` file in your Google Cloud commands by adding `.gitignore`'s contents to <code>[.gcloudignore](https://cloud.google.com/sdk/gcloud/reference/topic/gcloudignore)</code>: 


```git
.git
#!include:.gitignore
```


You can then create the secret from this file with `gcloud`: 


```shell
gcloud secrets create mySecret --data-file .env
```


For Cloud Build, you will need to ensure the secret is available in the environment ([which your script can then use](https://cloud.google.com/build/docs/securing-builds/use-secrets#example_accessing_secrets_from_scripts_and_processes)): 


```yaml
steps: 
 - name: python:slim
   entrypoint: pip
   args: ['install', '-r', 'requirements.txt', '--user']

 - name: python:slim
   secretEnv: ['MYSECRET']
   entrypoint: python
   args: ['main.py']

availableSecrets:
  secretManager:
  - versionName: projects/$PROJECT_ID/secrets/mySecret/versions/latest
    env: 'MYSECRET'
```


You can also reference secrets in the args call itself, [using bash variables](https://cloud.google.com/build/docs/securing-builds/use-secrets#access-utf8-secrets).

For Cloud Run, you'll have to deploy the service specifying either a mount or an environment variable: 


```shell
# for mounted volume
gcloud run deploy myservice --source .  \
  --update-secrets /secrets/.env=mySecret:latest

# for environment variable
gcloud run deploy myservice --source .  \
  --update-secrets MYSECRET=mySecret:latest
```

## And never forget IAM!

Don't forget: you'll also need to make sure the service account you're using has [permissions](https://cloud.google.com/secret-manager/docs/creating-and-accessing-secrets#access) to access your secret!


*Katie is a Developer Advocate for Google Cloud, focusing on Python and Serverless. She tweets [@glasnt](https://twitter.com/glasnt) about clouds, crafts, and cats.*