Exploratory data analysis of Instagram using instascrape and Python
Tags: python, datascience, opensource, webscraping
Published: 2020-10-22 15:35:14

## Table of Contents 

* [The Content](#the-content)
  * [Working with static content...](#working-with-static-content)
  * [...and dealing with the dynamic](#and-dealing-with-the-dynamic)
* [The Tools](#the-tools)
  * [Data processing: the before and after](#data-processing)
  * [Data visualization and interaction](#data-visualization)
* [The Exploration](#the-exploration)
  * [Takin' a peak at politicians](#politicians)
  * [The rise and fall of @chris_greening](#the-rise-and-fall)
  * [Determining best time of day/day of week to post](#best-time)
  * [Scraping a post in real time](#post-real-time)
* [The Conclusion](#the-conclusion)

My recent two posts have been light introduction's into [_`instascrape`_](https://github.com/chris-greening/instascrape): the lightweight, open source Instagram web scraper written for Python :snake:!

{% link https://dev.to/chrisgreening/scrape-data-from-instagram-with-instascrape-5e3e %}
{% link https://dev.to/chrisgreening/visualizing-instagram-engagement-with-instascrape-326h %}

In this post, I'm going to show some of the ways I've personally explored Instagram programatically using Python and _`instascrape`_ :wink:

{% embed https://www.christophergreening.com %}

---

# The Content <a name="the-content"></a>
## :walking: Working with static content... <a name="working-with-static-content"></a>

On its own, _`instascrape`_ is a purely static web scraper. That means it only scrapes the initial source HTML served back by Instagram and does not deal with dynamic content rendered by JavaScript.

## ...and dealing with the dynamic :runner: <a name="and-dealing-with-the-dynamic"></a>


Like many other modern websites, Instagram uses a combination of server-side and client-side technology's such as [AJAX](https://www.w3schools.com/xml/ajax_intro.asp) to dynamically load content as you scroll. This allows Instagram to respond to an HTTP request quickly and then load more content as it's needed. By doing this, the user is presented with a clean, seamless user-experience (UX) with infinite scrolls and fast page refresh times. 

While great for UX, this dynamically rendered content can become a bit of a pain for web scraping... but no worries :raised_hands:! There are ways we can get around this and take it in step. For the most part, I use [_`selenium`_](https://selenium-python.readthedocs.io/) which allows us to automate web browsers such as [Google Chrome](https://www.google.com/chrome/?brand=CHBD&gclid=Cj0KCQjwuL_8BRCXARIsAGiC51BEkQ8bcay9MXWXY2-YM3SXVc6wcQAOsKF5AqmnWfW1jKo-uDWeYV8aAkg7EALw_wcB&gclsrc=aw.ds) and [Firefox](https://www.mozilla.org/en-US/firefox/new/) using Python :computer:! With this tool at our disposal, we can render the JavaScript and grab the HTML as it's loaded, integrating it into _`instascrape`_ for scraping. 

---

# The Tools <a name="the-tools"></a>
## :floppy_disk: Data processing: the before and after <a name="data-processing"></a>

Regarding the data analysis, I use a combination of 
- [_`pandas`_](https://pandas.pydata.org/): powerful tools and data structures for analyzing and exploring data 
- [_`numpy`_](https://numpy.org/): support for multidimensional arrays 
- [_`scikit-learn`_](https://scikit-learn.org/stable/): machine learning library that we will use for preprocessing data and building regression models 

# :eye: Data visualization and interaction<a name="data-visualization"></a>

The library I use for data visualization is [matplotlib](https://matplotlib.org/api/pyplot_summary.html). I use [Jupyter Notebook](https://jupyter.org/) or the [IPython console](https://ipython.org/) for interactivity.

# The Exploration
## :older_man: Takin' a peak at politicians <a name="politicians"></a>
With one of the early iterations of _`instascrape`_ in early March 2020, I used it to take a look at how various politician's Instagram game's stacked up against one another, specifically [Bernie Sanders](https://www.instagram.com/berniesanders/?hl=en) and [Joe Biden](https://www.instagram.com/joebiden/?hl=en):

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/uoanzmaywoh0jckah8ql.png)

Fascinating! Let's take a look at Bernie first. It appears he's enjoyed very steady growth on his Instagram since 2016, nearly quintupling his likes per post. Additionally, we can see when he's on the campaign trail based on the frequency of posts. 

Now let's take a look at Joe. He has no posts prior to mid-2018 and it's clear he enjoys less likes than Bernie did at the time of this data collection. This certainly makes sense considering Bernie is so popular with younger voters who make up a larger portion of social media platforms! 

## :chart_with_downwards_trend: The rise and fall of @chris_greening
<a name="the-rise-and-fall"></a>

Yes that's a David Bowie reference; yes I am Chris Greening and [my insta](https://www.instagram.com/chris_greening/?hl=en) is in fact falling :cry:... but that's okay :shrug:! It made for a fun exercise to analyze. Let's check out the data: 

 ![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/dlb3y116ss5sqb9qq9wq.png)

Gasp! Shock! I know, it's tragic. But let's get down to it :smirk:. We can see that my growth was quite stagnant between 2016 and 2020 until mid-March of this year when my page suddenly blew up :open_mouth: (quarantine was beginning and I decided to learn Photoshop) Let's zoom in a bit to just 2020 :mag_right::

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/ztz05njt6ea38zjgim2d.png)

I went from averaging <100 likes per post to almost 400 likes in just a matter of months with some posts netting over 800! We can also see that I was pretty steady with my frequency of posting until June when I slipped up and missed an entire month! Whoops! And it's all been downhill from there :smile:. This type of data can be great though for seeing how a page is performing! 

Let's take a look at a popular Instagram page right now and see how they're doing:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/kwxg6tru55f5xck8qyhp.png)

Wow, honestly kind of incredible how linear [@dudewithsign](https://www.instagram.com/dudewithsign/?hl=en)'s growth has been since he started posting, nearly an exact straight line. 

## :watch: Determining best time of day/day of week to post <a name="best-time"></a>

In the same vein using the same data as the previous exploration we just did, we can also create a heatmap that will show us the best time of day/day of week for @chris_greening (me) to post to net the highest average engagement :fire:: 

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/btwhrukx2ead7b1thvcc.png)

It certainly seems that I get the most engagement when I post in the late morning/early afternoon but additionally, we can see some of my best engaged posts were posted in the middle of the week on Wednesday and Thursday. This is great information to keep in mind the next time I go to post something :thumbsup:. 

## :hourglass: Scraping a post in real time <a name="post-real-time"></a>

For the final bit of data exploration I'll show in this post, let's take a look at the output of a program I wrote that watches a post's engagement as it grows in real time. The program tracked a post by [@dacre_montgomery](https://www.instagram.com/dacremontgomery/?hl=en) and gathered how many likes/comments it got as a time-series across a 30 minute window:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/bqvk6nq8s1c1l0ccjtp5.png)

The red/left y-axis represents likes on the post while the blue/right y-axis represents comments on the post. Incredibly enough, Dacre was able to amass over _35,000_ likes and almost _400_ comments in that time period alone (and that was after the post had already been up for an hour or so). That's more likes/comments than I have probably ever gotten on all my posts combined :grimacing:

A future idea could be to write a script that watches a user's page continuously and as soon as a new post is detected, the real-time engagement tracker is triggered and we could watch the post grow across a longer period, say 8 hours! 

# The Conclusion <a name="the-conclusion"></a>

And there you have it! Leveraging _`instascrape`_ to gather data, I was able to perform some really great data exploration I wouldn't have been able to do otherwise. Not only was I able to explore my own profile but I was able to look at the profile's of some public figures as well. These are just drops in a great ocean of possibilities you can accomplish using _`instascrape`_ and the data is just out there waiting for you! 

Keep an eye out for a future post with more data exploration that will take a look at real-time hashtag growth, real-time post growth, and some more interesting examples to mess around with. 

Let me know what you think in the comments below or even better, contribute to the [official repo](https://github.com/chris-greening/instascrape) :smiley:

{% github chris-greening/instascrape %}
{% embed https://www.christophergreening.com %}