Make a Dashboard for Your Twitter Stats With Python And Anvil
Tags: python, azure, twitter, anvil
Published: 2022-02-10 07:33:37


Welcome to the last part of our tutorial on building your own Twitter Analytics dashboard with Python, Azure Functions and Anvil.

## A Quick Recap

Let us summarise what we have done so far.

1. In the first part of our series, we got to know the Twitter API and learned how to retrieve Tweet stats and Follower/Unfollower data.

2. In the second part, we used Azure Functions to periodically download and store these data to Azure Blobs

3. In the previous blog post, we used Azure Blog Storage triggers to compare our point-in-time snapshots to their predecessors automatically. Thus, we compiled an aggregated result of our Twitter Followers over time and a consolidated data structure for our Tweet stats.

## What we will learn today

There is little value in any statistic unless it can be made visible. We still need a way to plot graphs of our Twitter performance over time. That's why we will build a simple dashboard today. To make this article short and easy to follow, we will just focus on new and lost follower data over time. However, you will see how easy adding more graphs and insights into this dashboard will be.

## How it will work

To make a modern dashboard, we will build a web-based application. This means that you will be able to access your dashboard from within your browser. You have seen these kinds of applications quite frequently. Twitter, Gmail, Google Analytics,… – all of those run in your browser interactively. 

Web-based applications consist of multiple layers: One part of the dashboard runs in the browser. Browsers can understand HTML, CSS and JavaScript. This is just intended to control how the dashboard will look and feel. Then, there is a server-side responsible for all the heavy lifting like data gathering, preparation, authorisation, etc. 

The good news is that we will be able to create our dashboard in pure Python. No HTML, no CSS, no JavaScript. We will use a tool called Anvil which will *bridge* our Python code to modern web technologies. To put it in simple terms, Anvil will *translate* our Python code so that the browser can understand it.


## Getting started with Anvil

You have two options to start programming with Anvil: 

1. You can download the open-source version and run everything on your own machine, or
2. You can sign up for an Anvil account and use their service for your development.

I have outlined both variants in the [Getting Started with Anvil Guide](https://bas.surf/start-anvil)


## Part I: Creating A Dashboard with Anvil's Open Source version


### Create a `hello-world` app

Before we start, we create a skeletton app with Anvil. To do this, we create a blank Anvil app to start with

```
create-anvil-app hello-world MyTwitterDashboard
```

This will create a new Anvil app in a directory called `MyTwitterDashboard`. Of course, you can name it whatever you want.

We can check whether the creation of the app has been successful and whether anvil starts up correctly by using the `anvil-app-server` command:

```
anvil-app-server --app MyTwitterDashboard
```

If you now head over to you browser at [localhost:3030](http://localhost:3030) and see an empty page, everything worked fine.

### Code for the Server Side

Now, we create a server side function in anvil which is responsible for connecting to Azure and download our data. Head over to the `ServerModule1.py` file in the `server_code` directory. Here we can re-use the `get_blob_service_client` function which we coded in the previous part of this series and expose two functions, `get_follower_data` and `get_follower_data` to Anvil's server.

The decorator `@anvil.server.callable` will instruct Anvil to expose this function over HTTP, so that we can call it from the front end.

*Note* that we rely on Anvil's *secret storage*. In this storage we place a variable called `AZURE_CONNECTION_STRING` to access our Azure Connection String. This way you don't have to hard-code your sensitive information into the code.


Here is the full source code of `ServerModule1.py`:

```python
import json
import os
import gzip
import anvil.server
import anvil.secrets
from azure.storage.blob import BlobServiceClient


def get_blob_service_client():
    AZURE_CONNECTION_STRING = anvil.secrets.get_secret("AZURE_CONNECTION_STRING")
    blob_service_client = BlobServiceClient.from_connection_string(AZURE_CONNECTION_STRING)
    return blob_service_client


def get_data_from_azure(blob_service_client, blob_name, container_name="output", default=None):
    try:
        blob_client = blob_service_client.get_blob_client(container=container_name, blob=blob_name)
        data = blob_client.download_blob().readall()
        uncompressed = gzip.decompress(data)
        return json.loads(uncompressed)
    except:
        return default

@anvil.server.callable
def get_follower_data():
    data = get_data_from_azure(get_blob_service_client(), "NEW_FOLLOWERS.json.gz")
    return [dict(dt=k, n=len(v)) for k,v in data.items()]

@anvil.server.callable
def get_unfollower_data():
    data = get_data_from_azure(get_blob_service_client(), "LOST_FOLLOWERS.json.gz")
    return [dict(dt=k, n=len(v)) for k,v in data.items()]
```


### Create the Front End

The *front end* is the visible part of your Anvil application, i.e. the part that will appear in your browser. To start, we just add two `Plot`s, one for our followers and one for our unfollowers over time to our `Form`. Open the `_template.py` file in the `client_code/Form` directory and add two plot components to the end of the `init_components` method like so:

```python
def init_components(self, **properties):
    ...
    self.followers_plot = Plot()
    self.add_component(self.followers_plot)
    self.unfollowers_plot = Plot()
    self.add_component(self.unfollowers_plot)
```

The full `Form/_template.py` would look like this:

```python
from anvil import *

class Form1Template(HtmlPanel):

  def init_components(self, **properties):
  	# Initialise HtmlPanel
    super().__init__() 
    # Set the html template for the app
    self.html = '@theme:standard-page.html'

    # Add a GridPanel to the Form
    self.content_panel = GridPanel()
    self.add_component(self.content_panel)

    # Add a FlowPanel to accept NavBar links
    self.nav_links = FlowPanel()
    self.add_component(self.nav_links, slot="nav-right")

    # Add a title to the app
    self.title_label = Label(text="Bas.codes – Twitter Analytics")
    self.add_component(self.title_label, slot="title")

    # Add a sidebar to the app. Comment out the following two rows if you don't want a sidebar in your app.
    self.left_nav = ColumnPanel()
    self.add_component(self.left_nav, slot="left-nav")

    self.followers_plot = Plot()
    self.add_component(self.followers_plot)
    self.unfollowers_plot = Plot()
    self.add_component(self.unfollowers_plot)
```

Nothing interesting has happened so far. We just created and initialized to member variables, `self.followers_plot` and `self.unfollowers_plot` and set both of them to an empty plot. 

Now, it's time to populate these plots with data. Luckily, these plots are not specific to Anvil, but are good old [Plotly](https://plotly.com/) graphs.
So, we can simply use the `go` functions from the `plotly.graph_objects` module.

We can build the followers graph very simple like so:

```python
def build_followers_graph(self):
  followers = anvil.server.call('get_follower_data')
  scatter = go.Scatter(x = [x['dt'] for x in followers],
                        y = [x['n'] for x in followers],
                        fill = 'tozeroy',
                        line=dict(color='#2196f3'))
  self.followers_plot.data = scatter
  self.followers_plot.layout.title = "FOLLOWERS"
```

And here, something interesting happens: The first line (`followers = anvil.server.call('get_follower_data')`) will instruct Anvil to call the server side function `get_follower_data` we just created over the network. This way, your browser does not directly talk to Azure, but to your Anvil server which in turn fetches the follower data from Azure. This is important as you might want to share your Anvil app but not the logic behind it. You can thus easily hide the storage part in the server. 

The next lines are pretty straight forward: A scatter plot is created with the data we got from Azure and this scatter plot is assigned to the still empty `Plot` object of our `_template.py` file.

Now, we just have to ensure that our graphs are build when the application is started. We do this by calling our `build_followers_graph` method in the constructor of our `Form`.

For the unfollowers, we repeat the steps above.

The whole code of the `Form/__init__.py` file now looks like this:

```python
from _template import Form1Template
from anvil import *
import anvil.server
import plotly.graph_objects as go
from datetime import datetime

class Form1(Form1Template):

  def __init__(self, **properties):
    # Set Form properties and Data Bindings.
    self.init_components(**properties)

    # Any code you write here will run when the form opens.
    self.build_followers_graph()
    self.build_unfollowers_graph()


  def build_followers_graph(self):
    followers = anvil.server.call('get_follower_data')
    scatter = go.Scatter(x = [x['dt'] for x in followers],
                         y = [x['n'] for x in followers],
                         fill = 'tozeroy',
                         line=dict(color='#2196f3'))
    self.followers_plot.data = scatter
    self.followers_plot.layout.title = "FOLLOWERS"

  def build_unfollowers_graph(self):
    unfollowers = anvil.server.call('get_unfollower_data')
    scatter = go.Scatter(x = [x['dt'] for x in unfollowers],
                         y = [x['n'] for x in unfollowers],
                         fill = 'tozeroy',
                         line=dict(color='#2196f3'))
    self.unfollowers_plot.data = scatter
    self.unfollowers_plot.layout.title = "UNFOLLOWERS"
```




If you now run


```
anvil-app-server --app MyTwitterDashboard --secret AZURE_CONNECTION_STRING=<Your Azure Connection String>
```

again, refresh your browser at [localhost:3030](http://localhost:3030), you will see your follower/unfollower dashboard which should look like this:


![Screenshot](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/csgf1gpwb8s56fyizinb.png)

*Note* that you have to export the `AZURE_CONNECTION_STRING` environment variable, so that Anvil can pick it up and use it in your server code.


## Part II: Using the Anvil Hosted Service

Let's recap what we have done so far. With only very few lines of code we created a full-fledged web application with a responsive layout (yes, your application will work on your smartphone, too!). On the way, we connected to Azure, processed the results and drew graphs. As if this would not be impressive enough, we can even simplify the whole process even further.

By using the hosted *Anvil.works* environment, we can interactively design our app. Let's do that!

If you haven't already, sign up at [anvil.works](https://bas.surf/get-anvil) and log into your account.

First, click on *New Blank app* and choose *Material Design*


![Screenshot](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n072qbpox8jz26v7wvld.png)

![Screenshot](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bs1paqec7fjv0vnmkckb.png)

Next, you should see your Anvil Development Environment which should look like this:


![Screenshot](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uojd23lj9pplfo9w75ax.png)

From the Toolbox Menu on the right, drag and drop the *Plot* component right into the application canvas in the middle


![Screenshot](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/eueg617fcehb63c75v0b.png)

Make sure, that you re-name `self.plot_1` and `self.plot_2` to `followers_plot` and `unfollowers_plot` respectively.


![Screenshot](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bki4dx9bwshg7a588hig.png)

Now, head over to the *Server Code* section in the left toolbox. Click *Add new Server Module* and paste the server code of the previous section here:


![Screenshot](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0j2v9bx3ky1oxmllog8c.png)

It should look like this:


![Screenshot](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/grwfcmiy0bbx4evpaf83.png)

The last step is to connect our Plot components with our server code just the same way we did it in the Open Source edition of the previous section.

Click on *Form1* in the left menu. Switch to the *Code* view in the middle of your editor. Then, you can paste in the missing parts of your code like so:



![Screenshot](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rum7g7ge0q3suzfexwes.png)


Now, there is only one thing left: Our server code does not know about your `AZURE_CONNECTION_STRING`. In Anvil you do have *Application Secrets* instead of environment variables which are specifically designed to store access secrets like our connection string.


![Screenshot](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mq6y0p3j64m06vq8z8dd.png)

![Screenshot](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pcnrodivurw8dlk7funl.png)

You can create an App secret by clicking the `+` Sign on the *SERVICES* headline in the left toolbox. Now, click on App Secret and create the `AZURE_CONNECTION_STRING` secret.

Inside your server code, you have to add this line to the top:

```python
import anvil.secrets
```

As a last step, change the line
```python
AZURE_CONNECTION_STRING = os.getenv('AZURE_CONNECTION_STRING')
```
in your server code to this line:
```python
AZURE_CONNECTION_STRING = anvil.secrets.get_secret("AZURE_CONNECTION_STRING")
```

If you now hit the *Run* button, you should see the exact same app - but this time it's hosted on Anvil.works.


### Limitations of the free edition

Running other peoples' Python code is a very challenging task to do. Anvil has to work out security and performance considerations to provide a secure and responsive platform for everyone. Unfortunately, this means that the free plan of hosted Anvil is restricted to avoid overloading of the platform.

In our sample code we use the `azure` module as well as the `gzip` module. **These modules are not available in the free plan.** Luckily, Anvil.works provides a free trial of their professional version. Also, the Open Source version is unrestricted. Should you decide to proceed with your Twitter Dashboard project, you will can always use the Open Source version on your own machine for free.


## Conclusion and Outlook

We have built Azure Functions to Download Data from our Twitter Account, have them processed on the Azure Platform and stored into Azure Blob Storage. Then, we went on and built a frontend for it in Anvil. 

All of this happened with nothing else than Python and it did not require us to buy or rent a server. Of course, all the components we discussed are way more powerful than our small example app might make you think. Of course, you can think of extending your Twitter stats dashboard with more `plotly` graphs or other frontend elements provided by Anvil.

I hope you've enjoyed building a Twitter Follower and Unfollower Dashboard with me! Let me know how this did work out for you and how you want to extend this app. If you like stuff like this, make sure to [follow me on Twitter](https://twitter.com/bascodes) so that you don't miss future projects we can build together!