Twitter API: Searching tweets, replies
Tags: api, twitter, node, typescript
Published: 2020-05-09 14:49:01

_Get started with Twitter's API, searching tweets, loading replies_

This project is on [Github repo](https://github.com/riturajborpujari/twitter-app) as an API server. Clone this to use twitter API immediately(after you've created your twitter developer app), to search for tweets, load replies to tweets.

### What you need to get started?
You will need a twitter developer app to access the Twitter API. The App will provide you with two keys: API key, and API secret which we will use for requesting data from the API endpoints. Follow the following steps to get yourself the necessary keys.

1. Apply for a [twitter developer account](https://developer.twitter.com/en/apply)
2. Create a [twitter developer app](https://developer.twitter.com/en/apps)
3. Generate your app's keys on the app's details page under __Keys and tokens__ tab
4. Open terminal and run the following command to generate your app's Bearer token.
`curl -u '<API key>:<API secret key>' --data 'grant_type=client_credentials' 'https://api.twitter.com/oauth2/token'`

Replace `<API key>` and `<API  secret key>` with your app's key and secret.

__Note:__ With Bearer token, your app can only search for public data. It can not post tweet, read/write messages from your account. For such purposes, you'll need the Access token and secret too. You can generate them on the same page where you generated your keys. For our use case this will be sufficient.

### Project setup
[Nodejs Typescript boilerplate here](https://github.com/riturajborpujari/typescript-boilerplate).

You are now ready to use Twitter API. We are using a Node.js application and writing in Typescript, but you can use any language you prefer. But depending on your language of choice, you may find a twitter library or may have to build your own. For Node.js, we have a library called [twitter](https://www.npmjs.com/package/twitter), which we will use to make the requests

#### Add the dependencies
1. Twitter library `npm install twitter`
2. Types `npm install @types/twitter`

#### Modify environment variables
Open up the `src/.env` file and add your app's keys and token.
```
TWITTER_APP_API_KEY=<your-api-key>
TWITTER_APP_API_SECRET=<your-api-secret>
TWITTER_APP_BEARER_TOKEN=<your-bearer-token>
```

Add this to the config file `src/config.js` to load these env variables
```
export const twitter = {
    consumer_key: process.env.TWITTER_APP_API_KEY,
    consumer_secret: process.env.TWITTER_APP_API_SECRET,
    bearer_token: process.env.TWITTER_APP_BEARER_TOKEN
}

```

### Let's search for some tweets. Shall we?
In `src/app.ts` file write the following code.
```
import Twitter from 'twitter';
import {twitter} from './config';

let tw = new Twitter(twitter);

tw.get('search/tweets', {
    q: '#webdevelopment',
    count: 2
})
    .then(res => {
        console.log('Response: ',res);
    })
    .catch(err => {
        console.error(err);
    })
```
Run this code by `npm run start` and you should see Twitter respond you with two tweets as an array, and some metadata about the search itself

##### Search parameters
Here, we are only using two properties `q` and `count`. we are searching for tweets that contain the hashtag `#webdevelopment`, and we are retrieving only 2 tweets.
See the full list of properties available [here](https://developer.twitter.com/en/docs/tweets/search/api-reference/get-search-tweets).

##### Search operators
For _standard search_(free with 7 days of data), your search query `q` can use a list of [standard search operators](https://developer.twitter.com/en/docs/tweets/search/guides/standard-operators). In our example we are searching by hashtag `#webdevelopment`.

### Pagination
For obvious reasons, Twitter allow only a maximum of 100 tweets to load at a time. You can load the next tweets(older), or refresh the search(newer tweets) using two parameters `max_id` and `since_id`. When you get response from twitter, you get one `search_metadata` field which contains `next_results` and `refresh_url`. From these two query strings, you can filter out the *max_id* and *since_id* respectively. Then if you want to load the older tweets simply add the max_id to your parameters.(alongside `q`).
In my [repo](https://github.com/riturajborpujari/twitter-app), there is a `PaginationController` class which automatically filters out the max_id and since_id if you just supply next_results and refresh_url respectively.

### Schema for tweets
While twitter gives a whole lot of detail with tweets, we do not need all that. For example tweets contain both `id_str` and `id`, but we will only use `id_str` as id as it contains the original value. Also what's good is typescript, if we don't use a type for tweet. So, I have created a schema for handling tweets, and also a filter function to filter out tweets in our schema from twitter responses. Check them out at my [repo](https://github.com/riturajborpujari/twitter-app). They are in file `src/api/twitter/schema.ts` and `src/api/twitter/filter/tweetFilter.ts`

### Loading replies
Loading replies is a bit tricky. Because, twitter does not provide us API to load replies for a tweet directly. That's sad. But thankfully, twitter standard search operators can be used to fulfill our need. It provides us a `to:screen_name` operator which allows us to load replies to a twitter user. The screen name is the twitter handle like `@thisisrituraj`, without the `@` sign. So to search for replies to my account, I would search using `q: "to:thisisrituraj"`.
 
This will load replies to my account. But I wanted replies for a particular tweet of mine. Not all. That's what we have to filter out by looking a field in the responded tweets. The `in_reply_to_status_id_str` field which holds the id of the tweet for which this tweet is a reply.

We have to load all the replies to my account, using pagination, and then filter out replies to a particular tweet using its id. But do we have to load all replies to filter out replies to just one tweet. Unfortunately, twitter does not provide us with a tweet's reply count to know how many replies are there for a tweet. BUT..., we it provides us with the timestamp when the tweet was created(`created_at` field inside a tweet). And we know that, _replies to a tweet can not predate the tweet itself_.

So, the steps to load replies are
1. Get the id of the tweet(`id_str`) for which we wan't to load replies to.
2. Get the user's screen name who posted this tweet.
3. Load replies to this user using `to` operator
4. Filter out replies by comparing `in_reply_to_status_id_str` with `id_str`
5. Use pagination to keep on loading replies until one of the following occurs
    * We have run out of replies. This means no more replies are there, _or_ we have loaded replies for past 7 days
    * We find a reply whose timestamp predates the timestamp of the tweet

Here's the code for loading replies. (This is an excerpt from my [repo](https://github.com/riturajborpujari/twitter-app))
```
export default class TwitterClient {
    // code omitted for brevity

    private async loadRepliesForTweet(tweet: ITweet): Promise<ITweetsResponse> {
        let { user, id_str } = tweet;

        // save tweet's post datestamp
        let tweet_at = new Date(tweet.created_at);

        // load all replies to this user
        let allRepliesToTweet: ITweet[] = [];
        let replies: ITweetsResponse;
        let max_id = null;

        do {
            // load all replies to user
            replies = await this.searchTweets({
                q: `to:${user.screen_name}`,
                max_id,
                include_entities: false,
                count: 100
            });

            // select only replies done to THIS tweet
            replies.tweets.forEach(tweet => {
                if (tweet.in_reply_to.status_id_str === id_str) {
                    allRepliesToTweet.push(tweet);
                }
            })

            // Get max_id from next_results
            if (replies.search_metadata.next_results) {
                let next = <string>replies.search_metadata.next_results;

                // Use PaginationController to get max_id
                max_id = PaginationController.getParameterFromQueryString(next, 'max_id');
            }
            else {
                // BREAK loop if no more results exist
                break;
            }

            // get last reply tweet's post datestamp
            let last_reply_at = new Date(replies.tweets[replies.tweets.length - 1].created_at);

            // BREAK loop if last reply is earlier than tweet itself
            if (last_reply_at.valueOf() < tweet_at.valueOf()) {
                break;
            }

        } while (true);

        return { tweets: allRepliesToTweet, search_metadata: replies.search_metadata };
    }
};
```

### That's all folks!
That was all about using the Twitter API for searching tweets and loading their replies. We saw, that although Twitter does not allow us to load replies to a tweet directly, we can use its Standard Search Operators and some coding to solve that.

How do you like this post. Do you have any idea about building something with Twitter API. Let me know. I develop software applications and am very much into problem solving. Connect with [me on linkedin](https://in.linkedin.com/in/riturajborpujari)