Web Scraping Blogs Posts using Node.js
Tags: showdev, webdev, tutorial, beginners
Published: 2019-09-08 14:06:00

---
title: Web Scraping Blogs Posts using Node.js
published: true
description: Web Scraping Blogs Posts using Node.js
tags: showdev,webdev,tutorial,beginners
canonical_url: https://cloudnweb.dev/2019/09/web-scraping-blogs-posts-using-node-js/
---
<!-- wp:paragraph -->
<p>In this article, we will see how to scrape medium blogs using node.js. Web Scraping Blogs Posts using Node.js</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Recent Node.js article</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><a href="https://cloudnweb.dev/2019/09/building-p2p-video-chat-application-using-webrtc-and-node-js/">Building P2P Video Chat Application using webRTC and Node.js</a></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><a href="https://cloudnweb.dev/2019/08/apache-kafka-for-node-js-developers/">Apache Kafka for Node.js Developers</a></p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Set up</h3>
<!-- /wp:heading -->

<!-- wp:list -->
<ul><li><strong>Express </strong>- we will be using express to show the scrap results in the browser.</li><li><strong>Request</strong> - it is used to make API calls to medium blogs to get the data.</li><li><strong>Cheerio</strong> - it is used to manipulate the DOM in the response data from the URL. consider it just like JQuery.</li><li><strong>Handlebars</strong> - View engine to render the web pages in the express application.</li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>Let's set up the project to scrape medium blogs. Create a Project directory.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>$ mkdir nodescraper
$ cd nodescraper
$ npm init --yes</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>  Install all the dependencies mentioned above. </p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>$ npm install express request cheerio express-handlebars</code></pre>
<!-- /wp:code -->

<!-- wp:heading {"level":3} -->
<h3>Getting Blog posts from Medium</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>we will be scraping blog posts based on the tag of it. Medium provides a search bar where we can search for blogs based on tags.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>we are going to use it to scrape all the blog posts for a particular tag.</p>
<!-- /wp:paragraph -->

![](https://cloudnweb.dev/wp-content/uploads/2019/09/node-scraperdemo.gif)

<!-- wp:paragraph -->
<p>For Example, if you are going to scrape node.js blog post in the medium.you can search through the url  <a href="https://medium.com/search?q=node.js">https://medium.com/search?q=node.js</a> .</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>After that, open the Inspector in chrom dev tools and see the DOM elements of it.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":1327} -->
<figure class="wp-block-image"><img src="https://cloudnweb.dev/wp-content/uploads/2019/09/nodescraper-1024x532.png" alt="" class="wp-image-1327"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>If you see it carefully, it has a pattern. we can scrap it using the element class names. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Firstly, get the webpage elements using <strong>request</strong> package.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>request(`https://medium.com/search?q=${tag}`,(err,response,html) => {
    //returns all elements of the webpage
});</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Once you get the data, load the data to cheerio to scrap the data that you need.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>const $ = cheerio.load(html);</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>This loads the data to the dollar variable. if you have used JQuery before, you know the reason why we are using $ here(Just to follow some old school naming convention).</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Now, you can traverse through the DOM tree.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Since we need only the title and link for the blog posts on the page. we will get the elements in the HTML using either the class name of it or class name of the parent element.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Firstly, we need to get all the blogs DOM which has <strong>.js-block</strong> as a class name.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>$('.js-block').each((i,el) => {
 
   //This is the Class name for all blog posts DIV.
    
 }); </code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Most Importantly, each keyword loops through all the element which has the class name as <strong>js-block</strong>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Secondly, we scrap the title and link of  each blog posts.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>$('.js-block').each((i,el) => {

            const title = $(el).find('h3').text();
            const article = $(el).find('.postArticle-content').find('a').attr('href');
            
            let data = {
                title,
                article
            }
            
            console.log(data);
           
        })  </code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>This will scrap the blogs posts for a given tag. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p> Meanwhile, we will wrap this functionality with express application which takes a tag name as input and returns blogs for the particular tag.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Complete Source code can be found <a href="https://github.com/ganeshmani/nodescraper">here</a></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p></p>
<!-- /wp:paragraph -->