Getting started with analyzing past conversations with the new Twitter API v2
Tags: beginners, tutorial, twitter
Published: 2020-09-30 16:38:11

# Introduction

Twitter data is used by developers, students, and researchers to study various topics. You may want to study the conversation around a topic using hashtags, set of keywords, etc. from the last few days. For example, maybe you want to study the sentiment of Tweets about a football game over the weekend. You can get data for such use cases using Twitter’s [recent search endpoint](https://developer.twitter.com/en/docs/twitter-api/tweets/search/introduction). This endpoint gives you conversations from Twitter for the last 7 days.

In this guide, you will learn how to:

- Identify past conversation you wish to study
- Connect and authenticate to the recent search endpoint to receive Tweets
- Analyze the data for past conversations


# Prerequisites
 
- You must have a [developer account](https://developer.twitter.com/en/portal/petition/essential/basic-info). Access is available with active keys and tokens for a developer App that is attached to a [Project](https://developer.twitter.com/en/docs/projects) created in the [developer portal](https://developer.twitter.com/en/docs/developer-portal).
- A [Bearer Token](https://developer.twitter.com/en/docs/authentication/oauth-2-0) from your App in the [Twitter developer portal](https://developer.twitter.com/en/docs/basics/developer-portal/overview)

# Steps to consider
 
## Step 1: Identify which past conversation you wish to study
 
As a first step, you need to identify the type of conversation you wish to study. For example, you may decide to study all Tweets about the climate for the last few days. In order to do this, you might be looking for Tweets with a certain hashtag or words. Once you have identified these words, you will need to convert these into a query (also referred to as rule or filter). You can define the rule to include/exclude keywords, images, etc. You can also specify certain account names to get Tweets from. Check out [this guide](https://developer.twitter.com/content/developer-twitter/en/docs/twitter-api/tweets/search/integrate/build-a-rule) that shows you how to use the operators that are supported by the recent search endpoint that lets you get the data you need.

## Step 2: Authenticate and connect to the recent search endpoint to receive Tweets
 
Once you have decided what data you want and what operators you will use, you can connect to the [recent search endpoint](https://developer.twitter.com/content/developer-twitter/en/docs/twitter-api/tweets/search/introduction). The example below shows how you can use cURL to do so. To authenticate, replace `$BEARER_TOKEN` with the [bearer token](https://developer.twitter.com/content/developer-twitter/en/docs/authentication/oauth-2-0) from your App in the [developer portal](https://developer.twitter.com/content/developer-twitter/en/docs/basics/developer-portal/overview) and `$QUERY` using your query.

```bash
curl -X GET -H "Authorization: Bearer $BEARER_TOKEN" "https://api.twitter.com/2/tweets/search?tweet.fields=created_at,author_id,lang&query=$QUERY"
```

So for example, if you wish to get all Tweets with images about the hashtag #Caturday, excluding retweets your query will look like:

`“#caturday has:images -is:retweet”`

The '-' before a operator indicates negation, so the rule above excludes Tweets that are retweets.

And thus, your cURL call will be as follows:

```bash
curl -X GET -H "Authorization: Bearer $BEARER_TOKEN" "https://api.twitter.com/labs/2/tweets/search?query=caturday%20has:images%20-is:retweet&tweet.fields=created_at,author_id,lang"
```

You can also connect to the [recent search endpoint](https://developer.twitter.com/content/developer-twitter/en/docs/twitter-api/tweets/search/introduction) using one of our code samples:

- [Python](https://github.com/twitterdev/Twitter-API-v2-sample-code/blob/master/Recent-Search/recent_search.py)
- [Ruby](https://github.com/twitterdev/Twitter-API-v2-sample-code/blob/master/Recent-Search/recent_search.rb)
- [Node.js](https://github.com/twitterdev/Twitter-API-v2-sample-code/blob/master/Recent-Search/recent_search.js)
- [Java](https://github.com/twitterdev/Twitter-API-v2-sample-code/blob/master/Recent-Search/RecentSearchDemo.java)
 
The cURL call will return [10 Tweets](https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet) by default. You can also specify a maximum number of results you want to receive per API call. To do so, you can use the `max_results` query parameter with your API call. In order to get additional Tweets, you need to handle [pagination](https://developer.twitter.com/content/developer-twitter/en/docs/pagination). To do so, you will need to access the `next_token` from the object called `meta` and connect to, get the additional Tweet objects back for the next 10 Tweets (by default, unless you specify a number with the max_results parameter, which can be up to 100) until there is no longer a `next_token` available. 

By default, you will get back the id and text of each Tweet. However, you can customize what additional fields you want to return as part of the Tweet payload. You can include these additional fields in the payload by adding additional [fields](https://developer.twitter.com/content/developer-twitter/en/docs/twitter-api/fields) and [expansions](https://developer.twitter.com/content/developer-twitter/en/docs/twitter-api/expansions) to your query. 

Instead of making curl calls from your terminal, you can use a REST client such as [Postman](https://www.postman.com) or [Insomnia](https://insomnia.rest) to get data from the recent search endpoint. Our [Postman collection](https://explore.postman.com/api/7013/twitter-api-v2) will help you get started quickly with calling the recent search endpoint.

## Step 3: Analyzing the data for past conversations
 
The Twitter API provides a good opportunity to study historical data. Examples of analysis that you might be interested in doing with Twitter data include:

- Mapping Tweets by location
- Sentiment analysis on Tweets about past events
- Identifying influencers on Twitter

There are various libraries available in programming languages such as Python, R, etc. that let you analyze the data obtained from Twitter API. 

In Python, you can use libraries such as [pandas](https://pandas.pydata.org/), [numpy](https://numpy.org/), etc. that let you do data analysis and wrangling. You can also use libraries such as [matplotlib](https://matplotlib.org/) to build visualizations from Twitter data.

In R, you can use something like [Tidyverse](https://www.tidyverse.org/) (which is a collection of R packages for data science) for data analysis. You can use [ggplot2](https://ggplot2.tidyverse.org/), which is part of Tidyverse to build data visualization using Twitter data.


There are a variety of visualizations that you can build using these libraries mentioned above. Some examples of such visualizations that can be built using Twitter data include:

- Histograms for displaying Tweet frequency
- Map for displaying Tweets by Geo-Location
- Time series analysis of Trends

Check out [this sample app](https://developer.twitter.com/content/developer-twitter/en/docs/tutorials/how-to-analyze-the-sentiment-of-your-own-tweets) that demonstrates how to use the recent search endpoint to analyze sentiments of your own Tweets.

# Resources
 
- Learn more about the [recent search endpoint](https://developer.twitter.com/content/developer-twitter/en/docs/twitter-api/tweets/search/introduction).
- Get started building with our [quick start guide](https://developer.twitter.com/content/developer-twitter/en/docs/twitter-api/tweets/search/quick-start).
- Check out our [API reference](https://developer.twitter.com/content/developer-twitter/en/docs/twitter-api/tweets/search/api-reference/get-tweets-search-recent) to learn more about what’s available.
- Learn more about [building search queries](https://developer.twitter.com/content/developer-twitter/en/docs/twitter-api/tweets/search/integrate/build-a-rule).
- Get inspired by reading our other [tutorials](https://developer.twitter.com/content/developer-twitter/en/docs/twitter-api/tutorials).