How to store secrets in Azure Databricks
Tags: azure, database, datascience, cloud
Published: 2022-09-27 01:18:31

## Background
In Azure Databricks, we can write code to perform data transformation on data stored in various Azure Services, e.g. Azure Blob Storage, Azure Synapse. However, as other programs, sometimes, you want to protect credentials used in Azure Databricks, Azure Databricks provides a solid secret management approach to help you achieve that.

## Steps

### Prepare Databricks command-line interface (CLI) in Azure Cloud Shell

#### Configure your cloud shell environment

Open Cloud Shell & make sure you select “Bash” for the Cloud Shell Environment.



![Launch bash cloud shell](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/oncwddz9dnu67zdp25q3.jpg)

#### Set up Virtual Environment
Create Virtual Environment with below command.

```bash
# Bash
virtualenv -p /usr/bin/python2.7 databrickscli
```

![Create virtual environment](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hu07fhe1shhob0m7vgmp.jpg)

#### Activate your virtual environment

Activate your virtual environment with below command.

```bash
# Bash
source databrickscli/bin/activate
```

![Activate virtual environment](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/09ug5xi6uibh2gscxvbc.jpg)

#### Install Databricks CLI
Install Databricks CLI with below command.

```bash
# Bash
pip install databricks-cli
```

![Install Databricks CLI](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/00uc56wsykq6e0cbkb0m.jpg)

### Create secret in Azure Databricks

#### Set up authentication
Before you can create a secret, you need to authenticate as a user of the Azure Databricks, which requires your Azure Databrics workspace’s URL and a token

##### Get your Azure Databricks workspace’s URL
You can navigate to your Azure Databricks workspace and copy its URL.

![Get Databricks URL](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kve809wme6dgtq2mwiuf.jpg)

##### Generate Access Token for your Azure Databricks workspace
You can follow below steps to retrieve access token

1. Launch Databricks workspace
    ![Launch Databricks Workspace](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/m1ytt0y9k3dr0g0iuawf.jpg)
2. Click 'User Settings'
    ![Click 'User Settings'](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5do8aha7z1bj7fxcb8c3.jpg)
3. Click 'Generate New Token'
    ![Generate access token](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kautbynrv2tpsbpabrm1.jpg)
4. Configure access token & click 'Generate'
    ![Configure access token](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gan07dwuq1l1pu8ebope.jpg)
5. Copy access token
    ![Copy access token](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7w83zooacj4pzt1lny3n.jpg)


#### Create Secret Scope
After authentication, you need to first create a secret scope which you may group several secrets. 

If your databricks is in Standard plan, you can only create secret scope which will be shared with other users in the same workspace.

![Check databricks plan](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gzqg3736si1suzazctwx.jpg)

```bash
# Bash
databricks secrets create-scope --scope <<scope>>

# Example
databricks secrets create-scope --scope storage --initial-manage-principal users # Standard Plan
databricks secrets create-scope --scope storage # Premium plan
```

![Create secret scope](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3jfp5asp2gax2wiiixmb.jpg)

#### Create Secret
You can use below command to create secret under the specified scope.
```bash
# Bash
databricks secrets put --scope <<scope>> --key <<key name>>

databricks secrets put --scope storage --key blob #Example
```

![Type command to launch secret editor](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ebyf7d6kb6vtptgjgemg.jpg)

![Type your secret and save](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zw9a9i05haf3727lugnd.jpg)

#### Use Secret in Notebook
You can use secret by below command in notebook.

```python
# Python
dbutils.secrets.get(scope=<<scope>>,key=<<key>>)

dbutils.secrets.get(scope=storage,key=blob) #Example
```

![Use stored secret in notebook](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/s8590ycthanyhng0nbr8.jpg)

Original Post: [https://joeho.xyz/blog-posts/how-to-store-secrets-in-azure-databricks/](https://joeho.xyz/blog-posts/how-to-store-secrets-in-azure-databricks/)