How to get a JSON file of your Medium Stats and create a Dashboard using Streamlit
Tags: python, javascript, programming, computerscience
Published: 2021-06-13 18:32:13

<!-- wp:separator -->
<hr class="wp-block-separator"/>
<!-- /wp:separator -->

<!-- wp:heading {"level":3} -->
<h3>How to get a JSON file of your Medium Stats and create a Dashboard using Streamlit</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>We will be doing the following&nbsp;</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>Getting Medium Stats in a JSON format</li><li>Use Pandas and Plotting libraries to plot charts</li><li>Display the charts on a UI created using Streamlit</li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>You can find the repo <a href="https://github.com/rahulbanerjee26/Medium-Stats-Analysis" rel="noreferrer noopener" target="_blank">here</a></p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Getting Medium Stats in a JSON&nbsp;format</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Unfortunately Medium doesn’t seem to have any easy-to-use API publicly available so we will have to use a slightly hacky solution. We will essentially be scraping the stats, we will have to do it using JavaScript since it will have to run in your browser’s console. Check <a href="https://balsamiq.com/support/faqs/browserconsole/" rel="noreferrer noopener" target="_blank">this article</a> to see how you can access your browser’s console.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>You don’t need to know JavaScript, you can run the script as-is. However, if you want more metrics/stats about your articles, you will have to tinker around with the script.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>We will store the following data</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>Title</li><li>Read time of Each Article</li><li>Publication the article was published in</li><li>Number of Views</li><li>Number of Reads</li><li>Read Percentage</li><li>Number of Fans</li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>The JSON file will basically be an array of objects having the above properties.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>You can find the script <a href="https://github.com/rahulbanerjee26/Medium-Stats-Analysis/blob/main/getData.js" rel="noreferrer noopener" target="_blank">here</a></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Basically, we will have to go to <a href="https://medium.com/me/stats" rel="noreferrer noopener" target="_blank">https://medium.com/me/stats</a> and scroll all the way to the bottom, i.e scroll till you see all your articles. Run the script in your console and it will download a JSON file with all your stats.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The following article talks about downloading your Medium Stories. We will need to use the <a href="https://github.com/rahulbanerjee26/Medium-Stats-Analysis/blob/main/getData.js" rel="noreferrer noopener" target="_blank">script</a> I provided earlier.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><a href="https://medium.com/the-mighty-programmer/how-to-use-the-browser-console-to-scrape-and-save-data-in-a-file-with-javascript-b40f4ded87ef">How to use the browser console to scrape and save data in a file with JavaScript<br><em>A while back I had to crawl a site for links, and further use those page links to crawl data using selenium or…</em>medium.com</a></p>
<!-- /wp:paragraph -->

<!-- wp:separator -->
<hr class="wp-block-separator"/>
<!-- /wp:separator -->

<!-- wp:heading {"level":3} -->
<h3>Setting Up Environment&nbsp;</h3>
<!-- /wp:heading -->

<!-- wp:list -->
<ul><li>Create a new folder to store your JSON data and the Python files</li></ul>
<!-- /wp:list -->

<!-- wp:code -->
<pre class="wp-block-code"><code lang="bash" class="language-bash">mkdir medium-analysis</code></pre>
<!-- /wp:code -->

<!-- wp:list -->
<ul><li>Create Virtual Environment</li></ul>
<!-- /wp:list -->

<!-- wp:code -->
<pre class="wp-block-code"><code lang="bash" class="language-bash">python3 -m venv venv</code></pre>
<!-- /wp:code -->

<!-- wp:list -->
<ul><li>Activate Virtual Environment</li></ul>
<!-- /wp:list -->

<!-- wp:code -->
<pre class="wp-block-code"><code lang="bash" class="language-bash">source venv/Bin/activate</code></pre>
<!-- /wp:code -->

<!-- wp:list -->
<ul><li>Create Python File</li></ul>
<!-- /wp:list -->

<!-- wp:code -->
<pre class="wp-block-code"><code lang="bash" class="language-bash">touch main.py</code></pre>
<!-- /wp:code -->

<!-- wp:list -->
<ul><li>Install Required Libraries</li></ul>
<!-- /wp:list -->

<!-- wp:code -->
<pre class="wp-block-code"><code lang="bash" class="language-bash">pip3 install streamlit,pandas,seaborn,matplotlib</code></pre>
<!-- /wp:code -->

<!-- wp:list -->
<ul><li>Import the Libraries</li></ul>
<!-- /wp:list -->

<!-- wp:code -->
<pre class="wp-block-code"><code lang="bash" class="language-bash">import pandas as pd
import matplotlib.pyplot as plt
import streamlit as st
import seaborn as sns</code></pre>
<!-- /wp:code -->

<!-- wp:list -->
<ul><li>Run the streamlit app to ensure everything is working fin</li></ul>
<!-- /wp:list -->

<!-- wp:code -->
<pre class="wp-block-code"><code lang="bash" class="language-bash">streamlit run main.py</code></pre>
<!-- /wp:code -->

<!-- wp:separator -->
<hr class="wp-block-separator"/>
<!-- /wp:separator -->

<!-- wp:heading {"level":3} -->
<h3>Displaying DataFrame of Medium&nbsp;Stats</h3>
<!-- /wp:heading -->

<!-- wp:image -->
<figure class="wp-block-image"><img src="https://cdn-images-1.medium.com/max/1600/1*42tZN1WgNQ3jGil0QDSVug.png" alt=""/><figcaption>Screenshot by&nbsp;Author</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Before we begin, we will set the default theme to dark mode and the layout to wide mode.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code lang="python" class="language-python">plt.style.use("dark_background")
st.set_page_config(layout = "wide")</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>We will use pandas’ <strong>read_json()</strong> function to convert the JSON file to a dataframe.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code lang="python" class="language-python">df = pd.read_json("story.json")</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Now, we will display a title and the dataframe in the UI</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code lang="python" class="language-python">st.title("Medium Stats")
st.dataframe(df)</code></pre>
<!-- /wp:code -->

<!-- wp:separator -->
<hr class="wp-block-separator"/>
<!-- /wp:separator -->

<!-- wp:heading {"level":3} -->
<h3>Summary Section</h3>
<!-- /wp:heading -->

<!-- wp:image -->
<figure class="wp-block-image"><img src="https://cdn-images-1.medium.com/max/1600/1*SoMSM0DWANISNxWi7yIPig.png" alt=""/><figcaption>Screenshot by&nbsp;Author</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Our Summary section will basically consist of the sum of the various features in our dataframe and the average.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>We can use Pandas’ <strong>sum()</strong> function to find the sum of a column and use the standard <strong>len()</strong> function to find the number of rows (articles) in our dataframe.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code lang="python" class="language-python">minReadSum = df['minRead'].sum()
minReadSumString = f'{minReadSum} minutes' if minReadSum &lt; 60 else f'{int(minReadSum/60)} hours and {minReadSum%60} minutes'
totalNumViews = df['views'].sum()
totalNumReads = df['reads'].sum()
totalNumFans = df['numFans'].sum()
numArticles = len(df)</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>To calculate the average, simply divide the sum by the length.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>We will use the subheader() function in Streamlit to display the text.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code lang="python" class="language-python">st.subheader(f'You have written a total of {numArticles} articles')
st.subheader(f'Your articles have a total read time of {minReadSumString}. In total, they have been viewed {totalNumViews} times and read {totalNumReads} times. Your articles have received {totalNumFans} upvotes')
st.subheader(f'On average, your article has a read time of {round(minReadSum/numArticles , 2)} minutes. They get {int(totalNumViews/numArticles) } views, {int(totalNumReads/numArticles)} reads and {int(totalNumFans/numArticles)} fans')</code></pre>
<!-- /wp:code -->

<!-- wp:separator -->
<hr class="wp-block-separator"/>
<!-- /wp:separator -->

<!-- wp:heading {"level":3} -->
<h3>Top 5 By Column&nbsp;Name</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Since we will be doing it for multiple columns, we will write a function that will accept the column name as a parameter and return a figure.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code lang="python" class="language-python">def plot_top_5(df, colName):
    top5 = df.sort_values(by=[colName],ascending = False)[:5]
    fig,ax = plt.subplots(figsize = (15,15)
    ax = sns.barplot(x = colName, y = "title", data = top5)
    return fig</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>The column is sorted the first 5 values are plotted on a bar graph</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>We will streamlit’s <strong>beta_columns() </strong>function to split the UI into two columns. Each column will contain one of the graphs</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code lang="python" class="language-python">st.title("Your Top 5")
colNames = ["views" , "reads" , "numFans" , "readPercentage"]
p1,p2 = st.beta_columns(2)
for idx,colName in enumerate(colNames):
    curr = p1 if idx%2==0 else p2
    curr.subheader(f'Top 5 by {colName}')
    curr.pyplot(plot_top_5(df, colName))</code></pre>
<!-- /wp:code -->

<!-- wp:separator -->
<hr class="wp-block-separator"/>
<!-- /wp:separator -->

<!-- wp:heading {"level":3} -->
<h3>Publication Distribution</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Before we use the publications data, we will need to clean it up a bit. For the stories which were not published in any publication, you will notice that the publication is “View Story”. We will change this to “No Publication.&nbsp;</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code lang="python" class="language-python">df['publication'] = df.apply( lambda row : row['publication'] if row['publication'] != 'View story' else "No Publication" , axis = 1)</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>The <strong>value_counts()</strong> function returns the occurrence of each type of publication. This can be plotted as a pie chart.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code lang="python" class="language-python">Publicationscount = df['publication'].value_counts()
fig = plt.figure(figsize=(20,20))
plt.pie(Publicationscount, labels = Publicationscount.index,autopct='%1.0f%%')
p1.pyplot(fig)</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>This chart will be plotted in the first column we created earlier.</p>
<!-- /wp:paragraph -->

<!-- wp:separator -->
<hr class="wp-block-separator"/>
<!-- /wp:separator -->

<!-- wp:heading {"level":3} -->
<h3>Article Read Time&nbsp;Count</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Now will plot a chart to showcase the number of articles in various read-time intervales, i.e Number of articles with read time of 1 -3 mins, Number of articles with read time of 4–7 mins, etc</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>First, we will need to classify the articles based on their read time. Let's write a function to do so.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code lang="python" class="language-python">def categorizeArticle(readTime):
    if 1&lt;= readTime &lt;=3:
       return '1-3 mins'
    elif 4 &lt;= readTime &lt;= 6:
       return '4-6 mins'
    elif 7&lt;= readTime &lt;= 9:
       return '7-9 mins'
    else:
       return '9+ mins'</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>The above function will be called on each row of the dataframe and the read time of the article will be passed in as an argument.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code lang="python" class="language-python">df['articleCategory'] = df.apply(lambda row: categorizeArticle(row['minRead']) , axis =1)</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>This will create a new categorical column.&nbsp;</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Finally, we will plot it as a countplot</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code lang="python" class="language-python">p2.title("Article Read Time")
fig,ax = plt.subplots(figsize=(25,25))
ax = sns.countplot(data=df,x='articleCategory',order=['1-3 mins','4-6 mins','7-9 mins','9+ mins'])
p2.pyplot(fig)</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>This will be plotted in the second column&nbsp;</p>
<!-- /wp:paragraph -->

<!-- wp:separator -->
<hr class="wp-block-separator"/>
<!-- /wp:separator -->

<!-- wp:heading {"level":3} -->
<h3>Conclusion</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>We can further build on top of this</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>Plot Co-relation graphs between the features</li><li>Update the Script to get the date the article was published and plot distribution of articles by months</li><li>Update the Script to open the stats of the article and scrape the earnings, avg read time and member read time.</li></ul>
<!-- /wp:list -->

<!-- wp:separator -->
<hr class="wp-block-separator"/>
<!-- /wp:separator -->

<!-- wp:quote -->
<blockquote class="wp-block-quote"><p><em>Connect with me on </em><a href="https://www.linkedin.com/in/rahulbanerjee2699/" rel="noreferrer noopener" target="_blank"><em>LinkedIn</em></a><em>, </em><a href="https://twitter.com/rahulbanerjee99" rel="noreferrer noopener" target="_blank"><em>Twitter</em></a></p></blockquote>
<!-- /wp:quote -->