RaspberryPi cloud backup Part 3
Tags: bash, azure, rclone, raspberrypi
Published: 2022-09-11 20:14:37

## Introduction

This is the third article in a series of articles that explain how you can backup an inexpensive NAS solution (built with raspberry pi) to Azure.
In the previous posts we've seen the cloud storage solutions available as well as their pricing. We've decided upon what service we would use (Azure Storage) and built the secure infrastructure using Terraform.
In this final article we will focus on the solution used to backup our files to Azure Storage using the Archive tier for reduced costs.

## Backup Solution

Enter Rclone. Rclone is a command-line program to manage files on cloud storage. It is a feature-rich alternative to cloud vendors' web storage interfaces (as it is described in it's web page).

And that is what we need to automate our backups. We don't want to push the button, we want our files to backed up for several occasions during the day.

We also want to focus on security, and for that we will have to open and close the firewall on our storage account when the backup occurs.

So let's start.

## The credentials

First we will need to generate a credentials file.
Open your text editor and paste the following json text:

```json
{
    "appId": "replaceme",
    "displayName": "replaceme",
    "password": "replaceme",
    "tenant": "replaceme"
}
```
if you followed my previous post you remember we deployed the infrastructure using terraform, and that generated some outputs. If you stored them then you can replace the values in the text file:

```terraform
Changes to Outputs:
  + appId       = (known after apply)
  + displayName = "backupapplication"
  + password    = (known after apply)
  + tenant      = "xxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxx"
```
If you didn't store the values you can still obtain them either by going to the azure portal or via command line. If you don't remember the password then a new password for the service principal may have to be generated.

Now that we have the file in place let's leave it aside and start with the script.

## The script

Let's use our friend bash in order to put our solution together.

Starting with the security part. We need to be able to open and close the storage firewall in order to backup our files. For that we need to authenticate against azure (with our credentials file) and add our public ip address to the firewall.

Getting the public ip. Let's use aws! You read it correctly. AWS provides a free endpoint to discover your public ip address.
So our function will look like this:

```bash
getPublicIp() {
    curl checkip.amazonaws.com
}
```

Now that we have a way for retrieving the public ip we need to authenticate. Using the service principle credentials we can easily retrieve an access token that will allow us to interact with the azure storage via rest api. So let's do it:

```bash
getAzureToken() {
    client_id=$1
    client_secret=$2
    tenant=$3
    scope="https%3A%2F%2Fmanagement.core.windows.net%2F.default"
    headers="Content-Type:application/x-www-form-urlencoded"
    data1="client_id=${client_id}&scope=${scope}&client_secret=${client_secret}&grant_type=client_credentials"
    data2="https://login.microsoftonline.com/${tenant}/oauth2/v2.0/token"
    curl -X POST -H $headers -d $data1 $data2
}
```
As you are seeing the parameters we need are available in the file we've generated. We just have to retrieve them from the file and feed them to the function.

So now that we have the token and the public ip we can start opening and closing the communication in the azure firewall.
Let's work on the open function. We will need to call the api that allows us to interact with our storage account (represented by the resource id) and pass a json payload that specifies that the storage denies all traffic by default but allows our public ip to communicate:

```bash
addNetworkException() {
    token=$1
    subscriptionId=$2
    resourceGroupName=$3
    accountName=$4
    publicIp=$5

    headers1="Content-Type:application/json"
    headers2="Authorization:Bearer ${token}"
    url="https://management.azure.com/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.Storage/storageAccounts/${accountName}?api-version=2021-09-01"
    data="{\"properties\":{\"networkAcls\":{\"defaultAction\":\"deny\",\"ipRules\":[{\"action\":\"allow\",\"value\":\"${publicIp}\"}]}}}"
    curl -X PATCH -H "$headers1" -H "$headers2" -d "$data" "$url"
}
```

Closing the firewall is similar. We just have to remove the ip address from the json and pass an empty array:

```bash
removeNetworkException() {
    token=$1
    subscriptionId=$2
    resourceGroupName=$3
    accountName=$4
    publicIp=$5

    headers1="Content-Type:application/json"
    headers2="Authorization:Bearer ${token}"
    url="https://management.azure.com/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.Storage/storageAccounts/${accountName}?api-version=2021-09-01"
    data="{\"properties\":{\"networkAcls\":{\"defaultAction\":\"deny\",\"ipRules\":[]}}}"
    curl -X PATCH -H "$headers1" -H "$headers2" -d "$data" "$url"
}
```

Now that we have taken care of the security functions let's work on the backup itself.

Let's use our credentials file to add the azure information to our rclone configuration (This will run only once):

```bash
rclone_config() {
    echo 'Adding configuration...'
    rclone config create "$name" azureblob account="$storageaccountname" access_tier="$accesstier" service_principal_file="$credential_file"
    echo 'Config entry added!'
}
```
What are the function parameters?

- $name - The configuration name
- $storageaccountname - The name of the azure account
- $accesstier - The Archive access tier will be used to store our files in a cheap way
- $credential_file - The filesystem path for our manually created credentials file

Now we need the function that executes the backup:

```bash
rclone_backup() {
    echo 'Starting backup...'
    rclone sync "$sourcepath" "$name":"$container" "--azureblob-archive-tier-delete" "-v"
    if [ $? -eq 0 ]; then
        echo 'Backup completed!'
    else
        echo 'Something went wrong!'
        exit 1
    fi
}
```
The parameters once again.

- $sourcepath - The filesystem path to your folder
- $name - The rclone configuration name we used in the previous function
- $container - The azure storage container name

And now let's put this all together.

```bash
getPublicIp() {
    curl checkip.amazonaws.com
}
getAzureToken() {
    client_id=$1
    client_secret=$2
    tenant=$3
    scope="https%3A%2F%2Fmanagement.core.windows.net%2F.default"
    headers="Content-Type:application/x-www-form-urlencoded"
    data1="client_id=${client_id}&scope=${scope}&client_secret=${client_secret}&grant_type=client_credentials"
    data2="https://login.microsoftonline.com/${tenant}/oauth2/v2.0/token"
    curl -X POST -H $headers -d $data1 $data2
}

addNetworkException() {
    token=$1
    subscriptionId=$2
    resourceGroupName=$3
    accountName=$4
    publicIp=$5

    headers1="Content-Type:application/json"
    headers2="Authorization:Bearer ${token}"
    url="https://management.azure.com/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.Storage/storageAccounts/${accountName}?api-version=2021-09-01"
    data="{\"properties\":{\"networkAcls\":{\"defaultAction\":\"deny\",\"ipRules\":[{\"action\":\"allow\",\"value\":\"${publicIp}\"}]}}}"
    curl -X PATCH -H "$headers1" -H "$headers2" -d "$data" "$url"
}

removeNetworkException() {
    token=$1
    subscriptionId=$2
    resourceGroupName=$3
    accountName=$4
    publicIp=$5

    headers1="Content-Type:application/json"
    headers2="Authorization:Bearer ${token}"
    url="https://management.azure.com/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.Storage/storageAccounts/${accountName}?api-version=2021-09-01"
    data="{\"properties\":{\"networkAcls\":{\"defaultAction\":\"deny\",\"ipRules\":[]}}}"
    curl -X PATCH -H "$headers1" -H "$headers2" -d "$data" "$url"
}

rclone_config() {
    echo 'Adding configuration...'
    rclone config create "$name" azureblob account="$storageaccountname" access_tier="$accesstier" service_principal_file="$credential_file"
    echo 'Config entry added!'
}

rclone_backup() {
    echo 'Starting backup...'
    rclone sync "$sourcepath" "$name":"$container" "--azureblob-archive-tier-delete" "-v"
    if [ $? -eq 0 ]; then
        echo 'Backup completed!'
    else
        echo 'Something went wrong!'
        exit 1
    fi
}

publicIp=$(getPublicIp)

credentials=$(cat "$credential_file")

azureToken=$(jq -j '.access_token' <<<"$(getAzureToken "$client_id" "$client_secret" "$tenant")")

addNetworkException "$azureToken" "$subscriptionid" "$resourcegroupname" "$storageaccountname" "$publicIp"

config=$(rclone config dump)
if [ -z "$config" ]; then
    rclone_config
    rclone_backup
else
    rclone_backup
fi
```

To make it easier I've compiled it into a bash script available at [](https://github.com/mr-scripting/AzureToolkit/blob/main/Solution%20Deployment/Linux%20FS%20Cloud%20Backup/scripts/linuxFSAzureBackup.sh)

You can go over the description of each block of code and also read the script parameter description.

Using the downloaded script we can just make it executable and pass the parameters:

```bash
chmod +x linuxFSAzureBackup.sh
```
And then run it with the code bellow (this is a dummy example. Please replace with your own values).

```bash
./linuxFSAzureBackup.sh -n "raspberry2azure" -k "/home/pi/azurecredentials.json" -s "myspecialstorage" -r "specialresourcegroup" -i "497b54be-ccfb-47be-994d-b4d549a191cb" -a "Archive" -p "/mnt" -c "myspecialcontainer"
```
The parameter order:

-     -n The name that we will assign to the rclone configuration"
-     -k Path for the azure Cli generated credentials file"
-     -s The Azure Storage Account Name"
-     -r The resource Group Name where the Azure Storage Account resides"
-     -i The Azure subscription id"
-     -a The desired storage account accesstier for the backed up files"
-     -p The path where the items to be backed up reside"
-     -c The Azure Storage account container name"

These are the results:

Adding the public ip address

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/otwoki5cu4g0erctdljj.png)

Copying the files

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zsc2cravi5jjm1bkbr9d.png)

Removed the ip rule

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cr0xlptdi50lrq8wipyh.png)

In the end I had to create a filter file for rclone because some files were not getting copied. So that will be my next contribution for the script. But please feel free to modify it as you need.

And that's it. You can now add the script to [crontab](https://www.redhat.com/sysadmin/automate-linux-tasks-cron) and have a cloud backup solution on your raspberry pi.

I hope you enjoyed this series of articles.

See you soon.