Use Azure Functions to Periodically Download Your Tweet Stats
Tags: python, twitter, azure, tutorial
Published: 2022-02-01 15:34:44

In the [last part of this tutorial](https://dev.to/bascodes/using-twitters-api-to-gather-tweet-stats-and-follower-data-in-python-kkk) we learned how to access the Twitter API to download Tweet Stats and Follower Data. Today, we will use these scripts to run them periodically in Azure and store the results to Azure Blobs.

Here are the steps for today:

1. Creating an Azure Function that runs periodically
2. Gather the analytics data from the Twitter API
3. Storing the data to Azure Blobs

In further articles in this series, we will also learn how to evaluate this collected data later, using Python of course. So if this is something that interests you, make sure you [follow me on Twitter](https://twitter.com/bascodes) so you don’t miss the upcoming posts!

## [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#prerequisites)Prerequisites

In order to proceed, you will need an Azure account. Don’t worry, you can start for free with Azure. If you do not already have an account for Azure, [please make sure you sign up and install the required software](https://bas.surf/start-azure-dev)

## [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#creating-an-azure-function)Creating An Azure Function

To create an Azure Function app, open your VSCode in a new working dir. You should have at least the [Azure Functions Extension](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-azurefunctions) installed.

With that, click on the Azure Icon on the left side of your VSCode and click on “Create Function…“.

[![azure 01](https://bas.codes/static/aaef7176bda5e5b504c1b0aff1a642e6/6a068/azure-01.jpg "azure 01")](/static/aaef7176bda5e5b504c1b0aff1a642e6/6a068/azure-01.jpg)

VSCode will start a Wizard for you.

[![azure 02](https://bas.codes/static/23efb0e4a1e0e3b1070e4c540e64fe33/508ef/azure-02.png "azure 02")](/static/23efb0e4a1e0e3b1070e4c540e64fe33/508ef/azure-02.png)

First, accept that VSCode will create a basic Azure Functions skeleton for you.

[![azure 03](https://bas.codes/static/d751761fb119d0e8d8481c95d51824ab/d9199/azure-03.png "azure 03")](/static/d751761fb119d0e8d8481c95d51824ab/1134b/azure-03.png)

Of course, we choose Python as our programming language.

[![azure 04](https://bas.codes/static/709d3a3fc70541e0a00caa727a333e2d/d9199/azure-04.png "azure 04")](/static/709d3a3fc70541e0a00caa727a333e2d/6f464/azure-04.png)

Since we want to run something on a regular schedule, the right trigger for our case is a “Time Trigger”.

[![azure 05](https://bas.codes/static/87a6b167e17d1fef620aafef6e8d1074/d9199/azure-05.png "azure 05")](/static/87a6b167e17d1fef620aafef6e8d1074/e4ba2/azure-05.png)

Provide a name for your project.

[![azure 06](https://bas.codes/static/b1c9f8ae2d4d3b5977edafe06f30575a/d9199/azure-06.png "azure 06")](/static/b1c9f8ae2d4d3b5977edafe06f30575a/95e59/azure-06.png)

The time trigger in Azure is similar to the crontab notation. There are 6 space separated values. We choose `0 0 * * * *`.

This means that we get the following values 

| time unit   | value |
|-------------|-------|
| second      | `0`   |
| minute      | `0`   |
| hour        | `*`   |
| day         | `*`   |
| month       | `*`   |
| day of week | `*`   |

What that means is, at _every_ (`*`) hour, on _every_ day in _every_ month, on _any_ day of the week, we execute the function on minute `0` at second `0`, i.e. on every full hour.

[![azure 07](https://bas.codes/static/704a77ecb249e927f8f5b0416541e88f/e9beb/azure-07.png "azure 07")](/static/704a77ecb249e927f8f5b0416541e88f/e9beb/azure-07.png)

Now, you should see a local project in your VSCode.

## [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#time-to-code)Time to Code

[![azure explorer](https://bas.codes/static/4d4a6367ebd5450acbe40f5c53c698cc/94829/azure-explorer.png "azure explorer")](/static/4d4a6367ebd5450acbe40f5c53c698cc/94829/azure-explorer.png)

Back to the Explorer View of your VSCode you should see that the Azure extension created a project structure for you. Head over to the `requirements.txt` file and change it to include the `twitterstats` package.

It should look like this:

```
# DO NOT include azure-functions-worker in this file
# The Python Worker is managed by Azure Functions platform
# Manually managing azure-functions-worker may cause unexpected issues

azure-functions
twitterstats
```

Now, open the ` __init__.py` file.

```python
import datetime
import logging
import os
import json
import gzip
from twitterstats import Fetcher
from azure.storage.blob import BlobServiceClient
import azure.functions as func

def main(mytimer: func.TimerRequest) -> None:
    utc_timestamp = datetime.datetime.utcnow().replace(
        tzinfo=datetime.timezone.utc).isoformat()

    # Getting follower data
    fetcher = Fetcher()
    followers = fetcher.get_followers()
    
    # Connecting to Azure Storage
    AZURE_CONNECTION_STRING = os.getenv('AzureWebJobsStorage')
    container_name = str("output")
    blob_name = f"FOLLOWERS_{utc_timestamp}.json.gz"
    blob_service_client = BlobServiceClient.from_connection_string(AZURE_CONNECTION_STRING)
    blob_client = blob_service_client.get_blob_client(container=container_name, blob=blob_name)

    # Upload compressed data to Azure Storage
    json_data = json.dumps(followers)
    encoded = json_data.encode('utf-8')
    compressed = gzip.compress(encoded)
    blob_client.upload_blob(compressed)
```

**What are we doing here?**

First, we use the `Fetcher` class of the [`twitterstats` package](https://pypi.org/project/TwitterStats/). In case you missed it: This downloads the Twitter followers from the Twitter API. We learned how that works in the [last post of this mini series](https://dev.to/bascodes/using-twitters-api-to-gather-tweet-stats-and-follower-data-in-python-kkk).

_Note_ that we do not include any credentials here. Bear with me for a second, I will explain why we do not need the Twitter API credentials here in a few seconds.

Next, we create a connection to an Azure Storage. This storage has been automatically created by the Azure Function extension. We can get it’s connection string by reading the _environment variable_ called ”`AzureWebJobsStorage`“.

The last step is just converting the follower data to JSON, compress it with `gzip` and upload it to Azure storage.

That’ all. There is no more code needed to periodically download your Twitter Follower data.

However, before we continue, we need to head to the [Azure Portal](https://portal.azure.com/) to add a little bit of configuration.

**Create an Azure Function App**

In order to create the Azure Function App, switch back to the “Azure” view in VSCode. Here select your “Local Project” and click on the “Deploy to Function App” icon.

Again, a wizard will start:

[![azure 08](https://bas.codes/static/0d621024f5c72705d9eae54a34f334f1/d9199/azure-08.png "azure 08")](/static/0d621024f5c72705d9eae54a34f334f1/e4ba2/azure-08.png)

First, select the Azure Subscription you want the function to be deployed to.

[![azure 09](https://bas.codes/static/3d55581f9e2e85126b6f92816718092f/d9199/azure-09.png "azure 09")](/static/3d55581f9e2e85126b6f92816718092f/6f464/azure-09.png)

Next, choose “Create new Function App in Azure”.

[![azure 10](https://bas.codes/static/181446e1dac4aca45f890876d07819f4/d9199/azure-10.png "azure 10")](/static/181446e1dac4aca45f890876d07819f4/6f464/azure-10.png)

Name your function. _Note_ that this name has to be globally unique (among all Azure customers). So, prefix it with your username or domain name should get you there. I chose `bas-codes-twitter-stats`.

[![azure 11](https://bas.codes/static/1221cf66c6e2efb5bb0c3397a58e505f/d9199/azure-11.png "azure 11")](/static/1221cf66c6e2efb5bb0c3397a58e505f/e4ba2/azure-11.png)

Select the `Python 3.9` runtime.

[![azure 12](https://bas.codes/static/080a29019402300ce4a32660f23f280b/d9199/azure-12.png "azure 12")](/static/080a29019402300ce4a32660f23f280b/6f464/azure-12.png)

Select any location where you want your Azure function deployed.

_After a minute or two, your Azure function should be deployed._

**Time for configuration**

**Adding a storage container**

[![azure 13](https://bas.codes/static/ad41ade1b0ce1653e6eb0ce65015ac78/d9199/azure-13.png "azure 13")](/static/ad41ade1b0ce1653e6eb0ce65015ac78/1d7f7/azure-13.png)

For our output files, we create a storage container in the [“Storage accounts” section of the Azure Portal](https://portal.azure.com/#blade/HubsExtension/BrowseResource/resourceType/Microsoft.Storage%2FStorageAccounts).

Choose your storage account, which is named after the name you gave your Azure Function App. In my case it’s `bascodestwitterstats` (without the dashes).

Click on “Container” and on the `+`-sign to add a new container. Give that new container the name `output`.

**Adding Twitter Credentials**

[![azure 14](https://bas.codes/static/fa0947a7d7c741422730d264e31585e4/d9199/azure-14.png "azure 14")](/static/fa0947a7d7c741422730d264e31585e4/90eea/azure-14.png)

Now it’s time to add the Twitter credentials. Remember that we did not include any credentials in our source code? The reason for this is that the `twitterstats` package will automatically get the credentials from the _environment variables_. To make use of these environment variables, head to the [“Function App Section of the Azure Portal”](https://portal.azure.com/#blade/HubsExtension/BrowseResource/resourceType/Microsoft.Web%2Fsites/kind/functionapp).

Select your function and click on “Configuration”.

Click on “New Application Setting” and add one setting called `TWITTER_ACCESS_TOKEN`. The value for that setting is, of course, your API key from Twitter. Repeat that process for the three other settings we need: `TWITTER_ACCESS_TOKEN_SECRET`, `TWITTER_CONSUMER_KEY` and `TWITTER_CONSUMER_SECRET`.

Don’t forget to hit the “Save” button.

If you don’t know how to get the values for these settings, go one step back to the [first post of this series](https://dev.to/bascodes/using-twitters-api-to-gather-tweet-stats-and-follower-data-in-python-kkk).

## [<svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg>](#validating-the-results)Validating the Results

Now, your Azure Function App should run every hour and produce a result in the `output` bucket of your Storage account.

And, that’s it! We are ready now. Your Twitter followers are downloaded once per hour and we did that in about 20 lines of code!

Let’s head over to the [“Storage accounts” section of the Azure Portal](https://portal.azure.com/#blade/HubsExtension/BrowseResource/resourceType/Microsoft.Storage%2FStorageAccounts) again and see if we find our “FOLLOWERS” files there.

Wait for the function to trigger. After a while, your `output` bucket will start to fill like this:

[![azure check](https://bas.codes/static/4fe007d5c83d596ad9d47bf5ea70099a/3d4b6/azure-check.png "azure check")](/static/4fe007d5c83d596ad9d47bf5ea70099a/3d4b6/azure-check.png)

In the next part of this series we will analyze the Follower Data and extract new followers and lost followers (unfollowers) from it. [Follow me on Twitter](https://twitter.com/bascodes) if you don’t want to miss that!