Creating a scatter matrix of Instagram data using Python
Tags: python, datascience, webscraping, opensource
Published: 2020-10-25 06:51:07

In this post, I'm going to show you how you can use Python to visualize the relationship between Instagram variables using a scatter matrix. This will allow you to optimize your engagement and work towards beating the [infamous algorithm](https://later.com/blog/how-instagram-algorithm-works/)!  

# :books: Libraries 
We'll be working in three distinct steps with four distinct libraries:
- [_`Selenium`_](https://selenium-python.readthedocs.io/): Automate a web browser to get the HTML
- [_`BeautifulSoup`_](https://www.crummy.com/software/BeautifulSoup/bs4/doc/): Parsing and scraping HTML
- [_`Instascrape`_](https://github.com/chris-greening/instascrape): Scrape the posts and load the data
- [_`Pandas`_](https://pandas.pydata.org/): Organize and visualize the data

# :open_file_folder: Gathering the posts

For the sake of brevity, I will mostly summarize the code for this part but a very similar full version can be found in this [repo](https://github.com/chris-greening/instascrape/blob/master/examples/profile_scrape/dynamic_profile.py).

Since Instagram dynamically loads content, we aren't able to make simple GET requests and call it a day. Instead, we have to be clever and use a tool that can render JavaScript, simulating a user interacting with a webpage. 

This is where _`selenium`_ is gonna come in handy; it's a library that allows us to automate web browsers such as [Google Chrome](https://www.google.com/chrome/) or [Firefox](https://www.mozilla.org/en-US/firefox/new/) programatically using Python :snake:. 

The script we use will:
1. go to an Instagram page
2. scroll it automatically
3. gather the HTML at each scroll 
4. compare the HTML and find differences

One of the handy features that _`selenium`_ provides us is the ability to inject our own JavaScript scripts in the browser that interact with the webpage. In this case, we will use this script to continuously scroll the page in a loop: 

```javascript
//JavaScript scroll script
window.scrollTo(0, document.body.scrollHeight);
var lenOfPage=document.body.scrollHeight;
return lenOfPage;
``` 

After each scroll, we use _`BeautifulSoup`_ to get the unique shortcode of every post on the profile that we just scrolled. This can be used by _`instascrape`_ to construct a _`instascrape.Post`_ object for scraping with `Post.from_shortcode`

# :wrench: Scraping the data

Assuming we have created a `list` of `Post` objects called `post_objects`, we are now ready to scrape the data we need. Leveraging `instascrape`, all we have to do to scrape each post is:

```python
for post in post_objects:
    post.scrape()
```
And that's it! Each scrape loads a ton of data points with everything ranging from the amount of likes, hashtags used, tagged users, upload datetime, etc. 

# :bar_chart: Analysis and visualization
 
To get our data all neat and tidy, we're going to instantiate a [pandas.DataFrame](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html) that will store our data:

```python
import pandas as pd 

dataframe = pd.DataFrame([post.to_dict() for post in post_objects])
```

Now that we have an expressive and powerful way of handling our data, we can create some more useful columns with 

```python
dataframe["upload_hour"] = dataframe['upload_date'].dt.hour 
dataframe["upload_weekday"] = dataframe['upload_date'].dt.weekday
dataframe["amt_tagged_users"] = dataframe['tagged_users'].str.len()
dataframe["amt_hashtags"] = dataframe['hashtags'].str.len()
```

Now, to visualize it we use [pandas.plotting.scatter_matrix](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.plotting.scatter_matrix.html) which will let us view a matrix of scatter plots that show the different interactions between variables! For this example, we'll compare the 
- hour of the upload
- day of the week 
- amount of comments 
- amount of likes 
- amount of tagged users
- amount of hashtags 

Using my own personal Instagram page ([@chris_greening](https://www.instagram.com/chris_greening)), we get: 

```python
pd.plotting.scatter_matrix(dataframe[['likes', 'comments', 'amt_tagged_users', 'upload_hour']], figsize=(8,8))
```

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/cwzxgefi30r6qfrga43w.png)

Analyzing this scatter matrix, we can now look at how different variables interact with one another and get an idea of what we can do to better boost our engagement :raised_hands: 

For example, looking at the scatter plot that compares `upload_hour` and `likes`, we see a peak sometime around noon. This indicates that on average, the best time for me to post to my Instagram is around noon. 

This is just one relationship and there are plenty more to be discovered! Let me know what other relationships you found interesting in the comments below :heart:

# :newspaper: Additional resources

If you want to learn more about exploratory data analysis using `instascrape`, check out my other blog posts 

{% link https://dev.to/chrisgreening/exploratory-data-analysis-of-instagram-using-python-1o5c %}
{% link https://dev.to/chrisgreening/visualizing-instagram-engagement-with-instascrape-326h %}

# :computer: The official repo

`instascrape` is always looking for more contributors, come join us at the official repo 
{% github chris-greening/instascrape %}