Using Twitter's API to Gather Tweet Stats (and Follower Data) in Python
Tags: python, twitter, beginners
Published: 2022-01-26 11:36:40

In this little tutorial, we will learn how to use Twitter's API to download statistics about your tweets, like number of impressions, profile clicks etc. Also, we download a list of your followers, which would also allow you to track your unfollowers.

## Getting Started

Before we begin, we need an API key for the Twitter API. You can get one by signing up [here](https://developer.twitter.com/en/portal/petition/essential/basic-info).

Once you have your Twitter Developer Account, head over to the [Dashboard](https://developer.twitter.com/en/portal/dashboard) to create a new App.

Click on your newly created app and head to the "Keys and tokens" tab.

![Twitter Dashboard - Keys and Tokens](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/py7le2gdahvq0ipon3t2.png)

Ensure that you generate Authentication tokens with "Read, Write and Direct Messages Permissions" for your Twitter user.

In general, we need two pairs of key and secret to make use of our app:

1. API Key and Secret: This tells Twitter that it's your App which makes a request.
2. Access Token and Secret: Since apps can be used by multiple users, the access token and secret authenticates your user to Twitter.

Please store these secrets somewhere as you have no chance to display them again on the Twitter Dashboard. If you lose them, you have to regenerate them.

## Installing the TwitterAPI PyPI package

Since the Twitter API is a [well-documented](https://developer.twitter.com/en/docs) [REST-API](https://en.wikipedia.org/wiki/Representational_state_transfer), we could build all our queries to it ourselves. However, there are Python packages that allow us to access the Twitter API in a much more comfortable way. The most prominent packages for that purpose are [`tweepy`](https://www.tweepy.org/) and [`TwitterAPI`](https://github.com/geduldig/TwitterAPI). For this guide, we use the latter.

We install it as always by using `pip`:

```bash
pip install TwitterAPI
```

## Accessing the API

### Getting Tweet Stats

Now, we have everything to get started. Let's create a file `stats.py` and first import the `TwitterAPI` classes. We also create a client object by providing our credentials.

```python
from TwitterAPI import TwitterAPI, TwitterPager

consumer_key        = "<YOUR API KEY>"
consumer_secret     = "<YOUR API KEY SECRET>"
access_token        = "<YOUR ACCESS TOKEN>"
access_token_secret = "<YOUR ACCESS TOKEN SECRET>"

client = TwitterAPI(consumer_key, consumer_secret, access_token, access_token_secret, api_version="2")
```

The first request to the API is for retrieving our own `user_id`, which is a numeric value rather than your self-chosen username. We can get that by calling the `/users/me` endpoint like so:

```python
my_user = client.request(f'users/:me')
USER_ID = my_user.json()['data']['id']
```

Now, we can get stats for our tweets:

```python
params = {
        "max_results": 100,
        "tweet.fields": "created_at,public_metrics,non_public_metrics,in_reply_to_user_id",
    }
r = client.request(f"users/:{USER_ID}/tweets", params)
```

There are two things to note here:

1. the `params` dictionary contains a value for `max_results` set to `100`. This is the maximum Twitter allows for one API request.
2. To access the statistic fields, we have to instruct Twitter to attach these values to the API response. We do that by requesting a comma-separated list of fields for the tweet (`tweet.fields`). If you have paid ads running on Twitter's platform, you can also include the values `organic_metrics,promoted_metrics` to the `tweet.fields` params. However, these fields are available on promoted tweets only so that non-promoted tweets won't appear in your result when you request `organic_metrics,promoted_metrics`. So, you should make two API requests, one for promoted and one for non-promoted tweets.


### Getting Follower Data

Just like other social networks, Twitter doesn't really seem to want you to get information about your followers other than their sheer number. In particular, Twitter is reluctant to give out information about who has unfollowed you. The following API accesses are therefore more strictly limited, and you should take care not to make too many requests.

That being said, let's get our list of followers:

```python
params = {
        "max_results": 1000,
        "user.fields": "id,name,username,created_at,description,profile_image_url,public_metrics,url,verified",
    }
followers = client.request(f"users/:{USER_ID}/followers", params)
```

Even though Twitter is a bit reluctant to access the follower endpoint, we can still increase the number of users returned to 1000. That is good news.

#### Paging results

However, what do we do if you're lucky enough to have more than 1000 followers on Twitter? The `TwitterAPI` package gets us covered and provides a `TwitterPager` object for that purpose. If a request produces more results than fit into one response, the further results are hidden in further responses that can be accessed with a `next_token`. This means that the response is "accompanied" by a link to be followed for the next results. Basically, it works like pages in your favourite online shop: "Page 1 of 453", you get the idea. That's why the tool we need is called a `Pager`. Here is how to use it:

```python
params = {
        "max_results": 200,
        "user.fields": "id,name,username,created_at,description,profile_image_url,public_metrics,url,verified",
    }
pager = TwitterPager(client, f"users/:{USER_ID}/followers", params)
followers = list(pager.get_iterator())
```

#### Getting Unfollowers

How to get the users who unfollowed you on Twitter now?

There is no particular API endpoint for getting unfollowers. To circumvent this limitation, we store the result of our follower query on our disk. When we make a subsequent request, we can compare the old and new results. User IDs not present in the old result but present in the new result are *new followers*, and User IDs present in the old result but not present in the new result anymore are therefore *unfollowers*.

If you query the followers, let's say, every two hours, you will know who unfollowed you in the meantime.

The logic looks like this:

```python
follower_ids = [follower['id'] for follower in followers]

old_followers = json.load(open('last_run.json'))

old_follower_ids = [follower['id'] for follower in old_followers]

new_followers = [follower_id for follower_id in follower_ids if follower_id not in old_follower_ids]
unfollowers = [follower_id for follower_id in old_follower_ids if follower_id not in follower_ids]
```

## The Whole Package

The complete source code for our [`stats.py` file is on GitHub](https://gist.github.com/codewithbas/73ee2b8536ff4479e11e464d90763cdb).


## The `TwitterStats` PyPI package

For added convenience, I created a [package on PyPI](https://pypi.org/project/TwitterStats/), which combines all the ideas of this article on one class for fetching all the stats. You can install it as always with `pip`:

```bash
pip install TwitterStats
```

and use it in this simple way:

```python
from twitterstats import Fetcher

consumer_key        = "<YOUR API KEY>"
consumer_secret     = "<YOUR API KEY SECRET>"
access_token        = "<YOUR ACCESS TOKEN>"
access_token_secret = "<YOUR ACCESS TOKEN SECRET>"

fetcher = Fetcher(consumer_key, consumer_secret, access_token, access_token_secret)

promoted_tweets = fetcher.get_tweets(promoted=True)
unpromoted_tweets = fetcher.get_tweets(promoted=False)
followers = fetcher.get_followers()
```


## tl;dr

The `TwitterAPI` package makes it easy to download your tweet and follower/unfollower stats from Twitter. The `TwitterStats` package makes it even more convenient to access these data.

If you enjoyed this article, consider [following me on Twitter](https://twitter.com/bascodes) where I regularly share tips on Python, SQL and Cloud Computing.

<!-- Twitter single-event website tag code -->
<script src="//static.ads-twitter.com/oct.js" type="text/javascript"></script>
<script type="text/javascript">twttr.conversion.trackPid('q5o0o', { tw_sale_amount: 0, tw_order_quantity: 0 });</script>
<noscript>
<img height="1" width="1" style="display:none;" alt="" src="https://analytics.twitter.com/i/adsct?txn_id=q5o0o&p_id=Twitter&tw_sale_amount=0&tw_order_quantity=0" />
<img height="1" width="1" style="display:none;" alt="" src="//t.co/i/adsct?txn_id=q5o0o&p_id=Twitter&tw_sale_amount=0&tw_order_quantity=0" />
</noscript>
<!-- End Twitter single-event website tag code -->