You can tell a man by his tweets
Tags: dataviz, dataanalysis, python, api
Published: 2018-05-14 22:51:55

---
title: You can tell a man by his tweets
published: true
description: 
tags: dataviz,dataanalysis,python,api
cover_image: https://www.jisc.ac.uk/sites/default/files/styles/1144px-wide/public/twitter-head.png
---

Some time ago I read a tweet of Kenneth Reitz, a very well known Python developer I follow on Twitter, asking:
{% twitter 952553176925958145 %}
Starting from this, I decided to analyze some tweets from pretty popular Python devs in order to understand a priori how they use Twitter, what they tweet about and what I can gather using data from Twitter APIs only.
Obviously you can apply the same analysis on a different list of Twitter accounts.

#Setting up the environment

For my analysis I set up a Python 3.6 virtual environment with the following main libraries:
- [Tweepy](https://github.com/tweepy/tweepy) for interaction with Twitter APIs
- [Pandas](https://pandas.pydata.org/) for data analysis and visualization
- [Beautiful Soup 4](https://pypi.org/project/beautifulsoup4/), [NLTK](https://www.nltk.org/) and [Gensim](https://radimrehurek.com/gensim/) for processing text data

Some extra libraries will be introduced later on, along with the explanation of the steps I did.

In order to access the Twitter APIs I [registered my app](https://apps.twitter.com/) and then I provided the tweepy library with the consumer_key, access_token, access_token_secret and consumer_secret.

We're now ready to get some real data from Twitter!

#Choosing a list of Twitter accounts

First of all, I chose a list of 8 popular Python devs, starting from [the top 360 most-downloaded packages on PyPI](https://pythonwheels.com/) and selecting some libraries I know or use daily.

Here's my final list, including links to the Twitter account and the libraries (from the above mentioned list) for which those guys are known for:
- [@kennethreitz](https://twitter.com/kennethreitz): requests
- [@mitsuhiko](https://twitter.com/mitsuhiko): jinja2/flask
- [@zzzeek](https://twitter.com/zzzeek): sqlalchemy
- [@teoliphant](https://twitter.com/teoliphant): numpy/scipy/numba
- [@benoitc](https://twitter.com/benoitc): gunicorn
- [@asksol](https://twitter.com/asksol): celery
- [@wesmckinn](https://twitter.com/wesmckinn): pandas
- [@cournape](https://twitter.com/cournape): scikit-learn

#Getting data from Twitter

I got all the data with two endpoints only:
- with a call to [lookup users](https://developer.twitter.com/en/docs/accounts-and-users/follow-search-get-users/api-reference/get-users-lookup) I could get all the information about the accounts (creation date, description, counts, location, etc.)
- with a call to [user timeline](https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-user_timeline.html) I could get the tweets about a single user and all the information related to every single tweet. I configured the call to get also retweets and replies.

I saved the results from the two calls in two Pandas dataframes in order to ease the data processing and then into CSV files to be used as starting point for the next steps without re-calling each time the Twitter API.

#Preprocessing tweets

The users dataframe contained all the information I needed; I just created three more columns:
- a followers/following ratio, a sort of "popularity" indicator
- a tweets per day ratio, dividing the total number of tweets by the number of days since the creation of the account
- the coordinates starting from the location, if available, using [Geopy](https://pypi.org/project/geopy/). @benoitc doesn't have a location, while @zzzeek has a generic "northeast", geolocated in Nebraska :-)

Here's the final users dataframe:

screen_name|name|verified|description|created_at|location|location_coo|time_zone|total_tweets|favourites_count|followers_count|following_count|listed_count|followers/following|tweets_per_day
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
kennethreitz|Kenneth Reitz|False|@DigitalOcean & @ThePSF. Creator of Requests: HTTP for Humans. I design art — with code, cameras, musical instruments, and the English language.|2009-06-24 23:28:06|Winchester, VA|[39.1852184, -78.1652404]|Eastern Time (US & Canada)|58195|26575|18213|480|822|37.94375|17.950339296730412
mitsuhiko|Armin Ronacher|True|Creator of Flask; Building stuff at @getsentry; prev @fireteamltd, @splashdamage — writing and talking about system architecture, API design and lots of Python|2008-02-01 23:12:59|Austria|[47.2000338, 13.199959]|Vienna|31774|2059|21801|593|941|36.76391231028668|8.470807784590775
zzzeek|mike bayer|False||2008-06-11 19:22:19|northeast|[41.7370229, -99.5873816]|Quito|14771|1106|3013|209|194|14.416267942583731|4.080386740331492
teoliphant|Travis Oliphant|False|Creator of SciPy, NumPy, and Numba; founder and Director of Anaconda, Inc. Founder of NumFOCUS. CEO of Quansight|2009-04-17 20:04:57|Austin, TX|[30.2711286, -97.7436995]|Central Time (US & Canada)|3875|1506|18052|483|746|37.374741200828154|1.1706948640483383
benoitc|benoît chesneau|False|web craftsman|2007-02-13 16:53:37|||Paris|27172|548|1971|704|247|2.799715909090909|6.620857699805068
asksol|Ask Solem|False|sound, noise, stream processing, distributed systems, data, open source python, etc. Works at @robinhoodapp 🌳🌳🌴🌵|2007-11-11 18:56:33|San Francisco, CA|[45.4423543, -73.4373087]|Pacific Time (US & Canada)|3249|191|2509|513|126|4.89083820662768|0.8476389251239238
wesmckinn|Wes McKinney|True|Data science toolmaker at https://t.co/YVn0VFqgj0. Creator of pandas, @IbisData. @ApacheArrow @ApacheParquet PMC. Wrote Python for Data Analysis. Views my own|2010-02-18 21:01:15|New York, NY|[40.7306458, -73.9866136]|Eastern Time (US & Canada)|7749|3021|33130|784|1277|42.25765306122449|2.5804195804195804
cournape|David Cournapeau|False|Python and Stats geek. Former NumPy/Scipy core contributor. Lead the ML engineering team @cogentlabs  Occasional musings on economics/music/Japanese culture|2010-06-14 03:17:21|Tokyo, Japan|[34.6968642, 139.4049033]|Amsterdam|15577|505|800|427|112|1.873536299765808|5.395566331832352

The tweets dataframe on the contrary needed some extra preprocessing.

First of all, I discovered an annoying limitation about the Twitter user timeline API: there's a maximum number of tweets that can be returned (more or less 3200 including retweets and replies). Therefore I decided to group the tweets by username and to get the oldest tweet date for each user:

username|count|min|max
---|---|---|---
asksol|2991|2009-07-19 19:24:33|2018-05-10 14:58:17
benoitc|3199|2017-02-21 14:55:37|2018-05-11 19:36:21
cournape|3179|2017-06-12 19:13:20|2018-05-11 16:55:39
kennethreitz|3200|2017-08-26 20:48:35|2018-05-11 21:07:46
mitsuhiko|3226|2017-06-14 13:23:57|2018-05-11 19:26:07
teoliphant|3201|2013-02-19 03:54:16|2018-05-11 16:48:39
wesmckinn|3205|2014-01-26 17:45:07|2018-05-03 14:51:29
zzzeek|3214|2015-05-05 13:35:38|2018-05-11 14:17:02


Then I filtered out all the tweets before the maximum value between the first dates (2017-08-26 20:48:35).
Starting from these data, @kennethreitz is influencing the cut date because he's tweeting a lot more than some other users; but in this way we can at least get the same timeframe for all the users and compare tweets from the same period.

After this filter I got 25418-14470=10948 tweets, split in this way:

username|count|min|max
---|---|---|---
asksol|52|2017-09-11 18:36:19|2018-05-10 14:58:17
benoitc|1849|2017-08-26 20:49:13|2018-05-11 19:36:21
cournape|1888|2017-08-27 01:02:11|2018-05-11 16:55:39
kennethreitz|3200|2017-08-26 20:48:35|2018-05-11 21:07:46
mitsuhiko|2328|2017-08-27 08:22:19|2018-05-11 19:26:07
teoliphant|443|2017-08-26 22:57:23|2018-05-11 16:48:39
wesmckinn|591|2017-08-28 18:07:08|2018-05-03 14:51:29
zzzeek|596|2017-08-26 22:14:11|2018-05-11 14:17:02

Other preprocessing steps:
- I parsed the source information using Beautiful Soup because it contained HTML entities
- I removed smart quotes from text
- I converted the text to lower case
- I removed URLs and numbers

I filtered out all the tweets with empty text after these steps (i.e. containing only urls, etc) and I got 10948-125=10823 tweets.

I finally created a new column containing the "tweet type" (standard, reply or retweet) and another column with the tweet length.

Here are some columns from the final tweets dataframe (first 5 rows):

username|created_at|full_text|text_clean|lang|favorite_count|retweet_count|source|tweet_type|tweet_len
---|---|---|---|---|---|---|---|---|---
kennethreitz|2018-05-11 21:07:46|RT @IAmAru: Trio is @kennethreitz-approved. #PyCon2018|rt trio is approved|en|0.0|1|Tweetbot for iΟS|retweet|54
kennethreitz|2018-05-10 21:55:35|If you want to say hi, swing by the DigitalOcean booth during the opening reception! #PyCon2018|if you want to say hi swing by the digitalocean booth during the opening reception|en|20.0|2|Tweetbot for iΟS|standard|95
kennethreitz|2018-05-10 20:34:20|@dbinoj 24x 1x dynos right now|x x dynos right now|en|0.0|0|Tweetbot for iΟS|reply|30
kennethreitz|2018-05-10 20:11:37|Swing by the @IndyPy booth for your chance to win a signed copy of The Hitchhiker's Guide to Python! ✨🍰✨ https://t.co/CZhd2If5s0 https://t.co/3kUaqu5TMX|swing by the booth for your chance to win a signed copy of the hitchhiker s guide to python|en|25.0|3|IFTTT|standard|152
kennethreitz|2018-05-10 13:53:31|Let's do this https://t.co/6xLCE4WCqA https://t.co/ERiMmffe8L|let s do this|en|22.0|1|IFTTT|standard|61

#Explorative Data Analysis

The users dataframe itself already shows some insights:
- There are only two accounts with the verified flag: @mitsuhiko and @wesmckinn
- @wesmckinn, @kennethreitz, @teoliphant and @mitsuhiko are the most popular accounts in the list (according to my "popularity" indicator): ![popindicator](https://thepracticaldev.s3.amazonaws.com/i/1pwqnv2m23tmnwxajg10.png)
- @kennethreitz wrote since the creation of his account at least twice the number of tweets per day compared to the other devs in the list: ![tweetsperday](https://thepracticaldev.s3.amazonaws.com/i/wjxpj95ew9r6js55hz3b.png)
- Most of the accounts in the list live in the US; I used [Folium](https://github.com/python-visualization/folium) to create a map showing the locations: ![map](https://thepracticaldev.s3.amazonaws.com/i/g8y6ghlaltkquc8hdk7v.png)

The tweets dataframe needs instead some manipulation before we can gather some good insights.

First of all let's check the tweet "style" of each account. From the following chart we can see for example that @cournape is retweeting a lot, while @mitsuhiko is replying a lot: ![tweettype](https://thepracticaldev.s3.amazonaws.com/i/nv1czyjrk6q9ipoxdxfj.png)

We can also group by username and tweet type, and show a chart with the mean tweet length. @kennethreitz for example writes replies shorter than standard tweets, while @teoliphant writes tweets longer than the other guys (exceeding the 140 chars limit): ![tweetlen](https://thepracticaldev.s3.amazonaws.com/i/4jm1m8gt31f8eg1o93mn.png)

Ok, now let's filter out the retweets and let's focus on the machine-detected language used in standard tweets and replies. The five most common languages are: English, German, French, undefined and a rather weird "tagalog" (ISO 639-1 code "tl", maybe an error in auto-detection?). Most of the tweets are in English; @mitsuhiko tweets a lot in German, while @benoitrc in French: ![lang](https://thepracticaldev.s3.amazonaws.com/i/x15a61kev9e9r9p1zn2q.png)

So, let's just select tweets in English or undefined: all the next charts are just considering tweets and replies in English (but obviously you can tune differently your analysis).
Let's group by username and get statistics about the number of favorites/retweets per user:

username|favorite_count count|favorite_count max|favorite_count mean|favorite_count std|retweet_count max|retweet_count mean|retweet_count std
---|---|---|---|---|---|---|---
asksol|46|41.0|1.608695652173913|6.111840097055933|3.0|0.10869565217391304|0.48204475908203187
benoitc|1009|30.0|0.6531219028741329|1.8313280878865186|17.0|0.13676907829534193|0.7644934696088941
cournape|214|60.0|1.2757009345794392|4.449367547428712|25.0|0.205607476635514|1.7481758044670788
kennethreitz|2637|3932.0|10.062571103526736|82.09998594317476|2573.0|2.620781190747061|50.79602602503255
mitsuhiko|1547|752.0|9.657401422107304|41.06463543974671|220.0|1.8526179702650292|9.932970595417615
teoliphant|186|808.0|26.080645161290324|69.54002504187612|134.0|7.806451612903226|17.085639972995896
wesmckinn|433|2081.0|45.750577367205544|142.2699008271913|695.0|12.270207852193995|48.083342617014644
zzzeek|439|85.0|2.173120728929385|6.417876507767421|28.0|0.44874715261959|1.896040581838119

From this table we can see that:
- @kennethreitz has the most retweeted and favorited tweet in the dataframe. Here's the tweet: {% twitter 981547972239417345 %}
- @wesmckinn has the second most retweeted and favorited tweet in the dataframe. Here's the tweet: {% twitter 986998077767716865 %}
- @wesmckinn has highest mean value for retweet count and favorite count

Since @wesmckinn has also the highest followers count, how these stats change if we normalize the dataframe using the followers count?
Obviously one tweet can get favorited/retweeted even from non-followers, but this normalization will probably produce more fair results because the higher the followers count, the most the tweet will probably be viewed.

username|favorite_count perc count|favorite_count perc max|favorite_count perc mean|favorite_count perc std|retweet_count perc max|retweet_count perc mean|retweet_count perc std
---|---|---|---|---|---|---|---
asksol|46|1.634117178158629|0.06411700486942658|0.243596655920922|0.11956954962136308|0.0043322300587450395|0.019212624913592345
benoitc|1009|1.5220700152207|0.0331365754882869|0.09291365235345102|0.8625063419583967|0.0069390704360904|0.038787086230791176
cournape|214|7.5|0.1594626168224299|0.556170943428589|3.125|0.02570093457943925|0.21852197555838485
kennethreitz|2637|21.588974908032725|0.055249388368344046|0.4507768404061633|14.127271728984791|0.014389618353632417|0.27889982992935036
mitsuhiko|1547|3.4493830558231275|0.04429797450624903|0.18836124691411713|1.009128021650383|0.008497857760034094|0.04556199530029643
teoliphant|186|4.475958342565921|0.14447510060541938|0.38522061290647097|0.7423000221582097|0.043244247800261586|0.09464679798911974
wesmckinn|433|6.281316027769393|0.138094106149126|0.42942922072801465|2.0977965590099608|0.037036546490172004|0.1451353535074393
zzzeek|439|2.8211085297046132|0.07212481675836008|0.21300619010180605|0.9293063391968137|0.01489369905806803|0.06292866185987782

After the normalization we can see that @cournape and @teoliphant are getting higher mean values, in terms of retweets and favorites.

We can also see how the monthly number of tweets changes over time, per user. From the following chart we can see for example that @kennethreitz tweeted a lot in september 2017 (more than 800 tweets): ![monthly](https://thepracticaldev.s3.amazonaws.com/i/bk9lx56uecyupdpwjcnr.png)

Or we can even see which tools are used the most to tweet, per user: ![sources](https://thepracticaldev.s3.amazonaws.com/i/fj4jd4618hbsithnj6j4.png)
I grouped a lot of less used tools under "Other" (Tweetbot for iΟS, Twitter for iPad, OS X, Instagram, Foursquare, Facebook, LinkedIn, Squarespace, Medium, Buffer).

Finally, we can build a kind of punchcard chart for each user, showing an aggregation of tweets dates by day of the week and hours of the day: ![punch](https://thepracticaldev.s3.amazonaws.com/i/tki4ig3kewzvtz21mev5.png)


#Topics

But what are the devs in the list talking about?

Let's start with a simple visualization, a word cloud.
After some basic preprocessing of the text from standard tweets only (tokenization, pos tagging, stopwords removal, bigrams, etc), I grouped the tweets by username and got the most common words for each one:

username|tweet count|most_common
---|---|---
@asksol|6|[('python', 3), ('enjoy', 1), ('seeing', 1), ('process', 1), ('handle', 1)]
@benoitc|488|[('like', 40), ('erlang', 33), ('use', 31), ('code', 30), ('people', 30)]
@cournape|43|[('japan', 8), ('japanese', 6), ('#pyconjp', 6), ('shibuya', 4), ('python', 4)]
@kennethreitz|1109|[('pipenv', 157), ('python', 84), ('new', 77), ('requests', 64), ('released', 53)]
@mitsuhiko|399|[('rust', 53), ('like', 36), ('people', 27), ('new', 25), ('way', 20)]
@teoliphant|113|[('#pydata', 39), ('#python', 36), ('@anacondainc', 18), ('great', 18), ('new', 15)]
@wesmckinn|129|[('@apachearrow', 32), ('data', 21), ('pandas', 16), ('python', 12), ('new', 10)]
@zzzeek|170|[('like', 14), ('years', 11), ('python', 10), ('time', 10), ('use', 9)]

Then I created a word cloud for each username using [word_cloud](https://github.com/amueller/word_cloud). All the guys are talking about Python or their libraries (like pipenv, pandas, sqlalchemy, etc); we can also spot some other programming languages like erlang and rust.
![cloud](https://thepracticaldev.s3.amazonaws.com/i/c8xc4mjy1pvkajpgowl5.png)

The next step is to identify real topics, using an LDAmodel from Gensim. I still used standard tweets from the two accounts with the higher number of tweets (@kennethreitz and @mitsuhiko) and I performed the same preprocessing used for wordclouds generation.
I run the model using two dynamic values:
- the number of topics (ranging between 2 and 14)
- the alpha value (with possible values 0.2, 0.3, 0.4).
Then I chose the best solution using the Gensim built-in Coherence Model, using c_v as a metric: the optimal model is the one with 9 topics and alpha=0.2 ![coherence](https://thepracticaldev.s3.amazonaws.com/i/unan0uxh9o8smessm88w.png)

Here are the topics:

topic number|top words
---|---
0|(0, '0.125*"way" + 0.094*"favorite" + 0.076*"feature" + 0.067*"oh" + 0.063*"think"')
1|(1, '0.140*"pipenv" + 0.124*"released" + 0.098*"pipenv_released" + 0.082*"want" + 0.073*"code"')
2|(2, '0.271*"python" + 0.132*"today" + 0.093*"people" + 0.039*"month" + 0.036*"kenneth"')
3|(3, '0.183*"requests" + 0.134*"love" + 0.081*"work" + 0.071*"html" + 0.057*"github"')
4|(4, '0.164*"like" + 0.100*"rust" + 0.098*"time" + 0.058*"day" + 0.047*"things"')
5|(5, '0.297*"pipenv" + 0.062*"support" + 0.058*"includes" + 0.045*"right" + 0.044*"making"')
6|(6, '0.271*"new" + 0.076*"getting" + 0.075*"better" + 0.058*"use" + 0.049*"photos"')
7|(7, '0.161*"good" + 0.097*"going" + 0.092*"got" + 0.067*"happy" + 0.058*"current"')
8|(8, '0.114*"great" + 0.091*"ipad" + 0.076*"finally" + 0.066*"heroku" + 0.057*"working"')

We can check the intertopic distance map and the most relevant terms for each topic using [pyLDAvis](https://github.com/bmabey/pyLDAvis): you can explore the interactive data in the jupyter notebook in my github account. ![ldavis](https://thepracticaldev.s3.amazonaws.com/i/adzpp304ongpmq6e4gxe.png)

# Conclusions and future steps

In this post I showed how to get data from Twitter APIs and how to perform some simple analysis in order to know in advance some features about an account (e.g. tweet style, statistics about tweets, topics).
Your mileage may vary depending on the initial account list and the configuration of the algorithms (especially in topics detection).

I uploaded a [jupyter notebook](https://github.com/eric-bonfadini/python-notebooks/blob/master/blog_posts/You_can_tell_a_man_by_his_tweets.ipynb) on my github, with some snippets I used in order to create this blog post.

Next steps:
- Improve preprocessing using lemmatization and stemming
- Try different algorithms for topics detection using Gensim (e.g. AuthorTopicModel or LDAMallet) or scikit-learn
- Add sentiment analysis