Web Scraping using axios and Cheerio
Tags: javascript, webdev, beginners, codenewbie
Published: 2021-10-09 08:02:34

Hello folks, today I will be sharing information on <mark>Web Scraping</mark>. Web scraping is simply the process of extracting content and data from a website. This post is only for <mark>Educational purpose</mark>❗

![scraping](https://media.giphy.com/media/3o6MbflbRNuRqwBhss/giphy.gif)

## Prerequisite
👨‍💻 Nodejs 
👨‍💻 Developer Tool knowledge (DevTools)
👨‍💻 Document Object Model knowledge


---
## Let's Start
🥦 Make a new <mark>directory</mark> in my case `nodescraping` and initiate a node js app
`npm init -y`

![pik1](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xnpfubowub92fpseptdp.PNG)

🎯 Result: Creates your `package.json` file ![pik2](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2db9ppzswth8xzpxcdjg.PNG)
 
 
🥦 Install dependencies
`npm i express axios cheerio`
![pik3](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ee1wsgpokp32xtz8thzw.PNG)
 

🎯 Result: 
![pik 4](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ul345pj7ra8b2g0ujhau.PNG)
 

🥦 Install Dev dependencies (for development purposes), nodemon restarts our node app automatically when files change..
`npm i nodemon -save-dev`

🎯 Result:
![pik6](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xh97z428ueq1d1ruotiq.PNG)
 

🥦 Edit your start script
```JavaScript
  "start": "node app.js ",
   "dev:": "nodemon app.js"
```
🎯 Result:
![pik7](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lqmpwh7j9mt6t1shh99h.PNG)
 

---

🥦 create  a file `app.js` and import the packages
```Javascript 
const axios = require('axios');
const cheerio = require('cheerio');
const express = require('express');

const port = process.env.PORT || 4000;

const app = express();
```

🥦 I will be using the `axios` package to fetch the website. I will be using a site called [Dev.to](https://https://dev.to/)😁. Be at liberty to use any website of your choice. We will be scraping and exporting our result into a plain text file `CSV`.

 ---

🥦 Right-click, to inspect the website to select elements (class, is) and their respective attributes (a, li).
![pik1](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7a2a0te1zn0e51bdeupj.PNG)

🎯 This gives us the ability to inspect the classes we want to select.

🥦 I want to target the following. Blog title, link, author, and read the time. 
![pik2](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/weh57r64go0z8n8coxzi.PNG)


---
## Side note:
Always use `.` before the class name you want to target.  

```Javascript 
axios.get('https://dev.to/')
    .then(res => {
        const $ = cheerio.load(res.data)
        $('.crayons-story').each((index, element) => {
            const blogTitle = $(element).find('.crayons-story__title').text()

    }).catch(err => console.error(err))
```
In the logic above, i am target the child element of the class <mark>crayons-story</mark>.

The `.text()` method is converting the result to text.

🥦 I repeated the whole process to select the Blog link, author, and read the time. 


---

🥦 Final logic is: 
```Javascript
const axios = require('axios');
const cheerio = require('cheerio');
const express = require('express');
require('dotenv').config();
const fs = require('fs');
const writeStream = fs.createWriteStream('devBlog.csv');

const port = process.env.PORT || 4000;

const app = express();

//write headers
writeStream.write(`author, BlogTitle, bloglink, readtime \n`);


axios.get('https://dev.to/')
    .then(res => {
        const $ = cheerio.load(res.data)
        $('.crayons-story').each((index, element) => {

            const author = $(element).find('.profile-preview-card__trigger').text().replace(/\s\s+/g, '')
            const blogTitle = $(element).find('.crayons-story__title').text().replace(/\s\s+/g, '')
            const blogLink = $(element).find('a').attr('href');
            const readTime = $(element).find('.crayons-story__tertiary').text()
            const dev = 'https://dev.to'
            const joinedBlogLink = `${dev}` + `${blogLink}`;
            writeStream.write(`Author: ${author}, \n Blog title is : ${blogTitle} ,\n Blog link: ${joinedBlogLink}, \n Blog read time : ${readTime} \n`);
        });


    }).catch(err => console.error(err))

//Listen to server
app.listen(port, () => {
    console.log(`Server Established and  running on Port ⚡${port}`)
})
```

View source code here: [here](https://github.com/drsimplegraffiti/web-scraping-nodejs)

---
## Notes and explanation
* fs module was used to write the final result into the devBlog.csv file
* \n equals to a new line
* .replace(/\s\s+/g, '') removes white spaces between the author's field.
* axios fetch markup data from the URL
* cheerio grabs the html data from the URL. Cheerio is a tool for parsing HTML and XML in Node.js.
* the cheerio.load method loads the website mark up and stores the value in the declared variable, in my case `$`
* `.each` method loops through the selected elements.

🥦 Run server `npm run dev`


🎯 Result: 
```Javascript
author, BlogTitle, bloglink, readtime 
Author: Gracie Gregory (she/her), 
 The blog title is : What was your win this week? ,
 Blog link: https://dev.to/devteam/what-was-your-win-this-week-5h25, 
 Blog read time :  for Oct 8
            1 min read
           
Author: Jeremy Friesen, 
 Blog title is : Trick or Treat, I've Joined the DEV Team ,
 Blog link: https://dev.to/jeremyf/trick-or-treat-i-ve-joined-the-dev-team-4283, 
 Blog read time : Oct 8
            5 min read
           
Author: Michael, 
 Blog title is : How To See Which Branch Your Teammate Is On In Android Studio ,
 Blog link: https://dev.to/gitlive/how-to-see-which-branch-your-teammate-is-on-in-android-studio-2n3i, 
 Blog read time :  for Oct 8
            1 min read
           
Author: Iain Freestone, 
 Blog title is : 🚀10 Trending projects on GitHub for web developers - 8th October 2021 ,
 Blog link: https://dev.to/iainfreestone/10-trending-projects-on-github-for-web-developers-8th-october-2021-102e, 
 Blog read time : Oct 8
            3 min read
           
Author: AM, 
 Blog title is : Django Cloud Task Queue ,
 Blog link: https://dev.to/txiocoder/django-cloud-task-queue-27g2, 
 Blog read time : Oct 8
            1 min read
           
Author: Ankit Anand ✨, 
 Blog title is : AWS X-Ray vs Jaeger - key features, differences and alternatives ,
 Blog link: https://dev.to/signoz/aws-x-ray-vs-jaeger-key-features-differences-and-alternatives-322, 
 Blog read time :  for Oct 8
            6 min read
           
Author: Raquel Román-Rodriguez, 
 Blog title is : Algo Logging: the Longest Substring of Unique Characters in JavaScript ,
 Blog link: https://dev.to/raquii/algo-logging-the-longest-substring-of-unique-characters-in-javascript-4i3, 
 Blog read time : Oct 8
            3 min read
           
Author: Shaher Shamroukh, 
 Blog title is : Working With Folders & Files In Ruby ,
 Blog link: https://dev.to/shahershamroukh/working-with-folders-files-in-ruby-2l97, 
 Blog read time : Oct 8
            3 min read
           
Author: Roberto Ruiz, 
 Blog title is : Untangling Your Logic Using State Machines ,
 Blog link: https://dev.to/robruizr/untangling-your-logic-using-state-machines-2epj, 
 Blog read time : Oct 8
            5 min read
           
Author: Cubite, 
 Blog title is : How To Manage Open edX® Environment Variables Using Doppler and Automating The Deployment ,
 Blog link: https://dev.to/corpcubite/how-to-manage-open-edx-environment-variables-using-doppler-and-automating-the-deployment-4c5e, 
 Blog read time : Oct 8
            5 min read
           
Author: OpenReplay Tech Blog, 
 Blog title is : Building an Astro Website with WordPress as a Headless CMS ,
 Blog link: https://dev.to/asayerio_techblog/building-an-astro-website-with-wordpress-as-a-headless-cms-47mo, 
 Blog read time : Oct 8
            9 min read
           
Author: Anamika, 
 Blog title is : How to setup Appwrite on Ubuntu ,
 Blog link: https://dev.to/noviicee/how-to-setup-appwrite-on-ubuntu-3j67, 
 Blog read time : Oct 8
            4 min read
           
Author: Bryan Robinson, 
 Blog title is : Building server-rendered search for static sites with 11ty Serverless, Netlify, and Algolia ,
 Blog link: https://dev.to/algolia/building-server-rendered-search-for-static-sites-with-11ty-serverless-netlify-and-algolia-13e2, 
 Blog read time :  for Oct 8
            8 min read
           
Author: bhupendra, 
 Blog title is : Understanding Redux without React ,
 Blog link: https://dev.to/bhupendra1011/understanding-redux-without-react-223n, 
 Blog read time : Oct 8
            4 min read
           
Author: Rizel Scarlett, 
 Blog title is : Add Fuzzy Search to Your Web App with this Open Source Tool ,
 Blog link: https://dev.to/github/add-fuzzy-search-to-your-web-app-with-this-open-source-tool-22d7, 
 Blog read time :  for Oct 8
            6 min read
           
Author: Marcelo Sousa, 
 Blog title is : Ship / Show / Ask With Reviewpad ,
 Blog link: https://dev.to/reviewpad/ship-show-ask-with-reviewpad-47jh, 
 Blog read time :  for Oct 8
            5 min read
           
Author: Shantanu Jana, 
 Blog title is : Random Gradient Generator using JavaScript & CSS ,
 Blog link: https://dev.to/shantanu_jana/random-gradient-generator-using-javascript-css-529c, 
 Blog read time : Oct 8
            6 min read
           
Author: Miles Watson, 
 Blog title is : URL Shortener with Rust, Svelte, & AWS (6/): Deploying to AWS ,
 Blog link: https://dev.to/mileswatson/url-shortener-with-rust-svelte-aws-6-deploying-to-aws-2gi0, 
 Blog read time : Oct 8
            4 min read
           
Author: Jon Deavers, 
 Blog title is : Publishing my first NPM package ,
 Blog link: https://dev.to/lucsedirae/publishing-my-first-npm-package-200g, 
 Blog read time : Oct 8
            3 min read
           
Author: Anjan Shomooder, 
 Blog title is : CSS positions: Everything you need to know ,
 Blog link: https://dev.to/thatanjan/css-positions-everything-you-need-to-know-2ng4, 
 Blog read time : Oct 8
            4 min read
           
Author: Alvaro Montoro, 
 Blog title is : Divtober Day 8: Growing ,
 Blog link: https://dev.to/alvaromontoro/divtober-day-8-growing-1182, 
 Blog read time : Oct 8
            1 min read
           
Author: Jambang J, 
 Blog title is : Deploying an discordjs bot to Qovery ,
 Blog link: https://dev.to/jambang067/deploying-an-discordjs-bot-to-qovery-51e, 
 Blog read time : Oct 8
            7 min read
           
Author: Sadee, 
 Blog title is : How to create responsive navbar {twitter clone} with HTML CSS ,
 Blog link: https://dev.to/codewithsadee/how-to-create-responsive-navbar-twitter-clone-with-html-css-6fa, 
 Blog read time : Oct 8
            1 min read
           
Author: Jeremy Grifski, 
 Blog title is : Support The Sample Programs Repo This Hacktoberfest ,
 Blog link: https://dev.to/renegadecoder94/support-the-sample-programs-repo-this-hacktoberfest-42ad, 
 Blog read time : Oct 8
            5 min read
           
Author: Sebastian Rindom, 
 Blog title is : Making your store more powerful with Contentful ,
 Blog link: https://dev.to/medusajs/making-your-store-more-powerful-with-contentful-3efk, 
 Blog read time :  for Oct 8
            7 min read
           
Author: Shalvah, 
 Blog title is : A practical tracing journey with OpenTelemetry on Node.js ,
 Blog link: https://dev.to/shalvah/a-practical-tracing-journey-with-opentelemetry-on-node-js-5706, 
 Blog read time : Oct 8
            16 min read
           
Author: Kingsley Ubah, 
 Blog title is : How to build an Accordion Menu using HTML, CSS and JavaScript ,
 Blog link: https://dev.to/ubahthebuilder/how-to-build-an-accordion-menu-using-html-css-and-javascript-3omb, 
 Blog read time : Oct 7
            6 min read
           
Author: mike1237, 
 Blog title is : Create Proxmox cloud-init templates for use with Packer ,
 Blog link: https://dev.to/mike1237/create-proxmox-cloud-init-templates-for-use-with-packer-193a, 
 Blog read time : Oct 8
            3 min read
           
Author: Prosper Yong, 
 Blog title is : Get Paid Writing ,
 Blog link: https://dev.to/yongdev/get-paid-writing-2i2j, 
 Blog read time : Oct 8
            1 min read
           
Author: Debbie O'Brien, 
 Blog title is : Understanding TypeScript ,
 Blog link: https://dev.to/debs_obrien/understanding-typescript-378g, 
 Blog read time : Oct 8
            5 min read
           
Author: Matias D, 
 Blog title is : Show me your portfolio ,
 Blog link: https://dev.to/matiasdandrea/show-me-your-portfolio-1l9h, 
 Blog read time : Oct 8
            1 min read
           
Author: Marcos Henrique, 
 Blog title is : You should use Buildpacks instead Dockerfile and I'll tell you why ,
 Blog link: https://dev.to/wakeupmh/you-should-use-buildpack-instead-dockerfile-and-i-ll-tell-you-why-2n6, 
 Blog read time : Oct 8
            2 min read
           
Author: Gaurav Gupta, 
 Blog title is : Smart Notes - A Build-in Public Product. BuildLog[1] ,
 Blog link: https://dev.to/gauravgupta/smart-notes-a-build-in-public-product-buildlog-1-kj6, 
 Blog read time : Oct 8
            4 min read
           
Author: Andrea Giammarchi, 
 Blog title is : About bitwise operations ,
 Blog link: https://dev.to/webreflection/about-bitwise-operations-29mm, 
 Blog read time : Oct 8
            10 min read
           
Author: AbcSxyZ, 
 Blog title is : Business models of Free and Open Source software ,
 Blog link: https://dev.to/abcsxyz/business-models-of-free-and-open-source-software-2cg8, 
 Blog read time : Oct 8
            4 min read
           
Author: Saharsh Laud, 
 Blog title is : Face Detection in just 15 lines of Code! (ft. Python and OpenCV) ,
 Blog link: https://dev.to/saharshlaud/face-detection-in-just-15-lines-of-code-ft-python-and-opencv-37ci, 
 Blog read time : Oct 8
            4 min read
           
Author: Kaustubh Joshi, 
 Blog title is : Hello, I'm HTTP and these are my request methods👋🏻 ,
 Blog link: https://dev.to/elpidaguy/hello-i-m-http-and-these-are-my-request-methods-co, 
 Blog read time : Oct 8
            3 min read
           
Author: SilvenLEAF, 
 Blog title is : Easiest way to create a ChatBOT from Level 0 ,
 Blog link: https://dev.to/silvenleaf/easiest-way-to-create-a-chatbot-from-level-0-31pf, 
 Blog read time : Oct 8
            6 min read
           
Author: whykay 👩🏻‍💻🐈🏳️‍🌈 (she/her), 
 Blog title is : 👏 New EuroPython Fellows ,
 Blog link: https://dev.to/europython/new-europython-fellows-2ob2, 
 Blog read time :  for Oct 8
            1 min read
           
Author: Zaw Zaw Win, 
 Blog title is : How to pass props object from child component to parent ,
 Blog link: https://dev.to/hareom284/how-to-pass-props-object-from-child-component-to-parent-2a8d, 
 Blog read time : Oct 8
            2 min read
           
Author: Zack DeRose, 
 Blog title is : The "DeRxJSViewModel Pattern": The E=mc^2 of State Management [Part 1] ,
 Blog link: https://dev.to/zackderose/the-derxjsviewmodel-pattern-the-e-mc-2-of-state-management-part-1-3dka, 
 Blog read time : Oct 8
            23 min read
           
Author: john methew, 
 Blog title is : Serverless Cloud Application Development with AWS Lambda ,
 Blog link: https://dev.to/johnmethew18/serverless-cloud-application-development-with-aws-lambda-3o7l, 
 Blog read time : Oct 8
            1 min read
           
Author: Antonio-Bennett, 
 Blog title is : Hacktoberfest Week 1 ,
 Blog link: https://dev.to/antoniobennett/hacktoberfest-week-1-4ebc, 
 Blog read time : Oct 8
            2 min read
           
Author: ZigRazor, 
 Blog title is : Hacktoberfest Beginners and Advanced Repos to Contribute to ,
 Blog link: https://dev.to/zigrazor/hacktoberfest-beginners-and-advanced-repos-to-contribute-to-p1, 
 Blog read time : Oct 8
            1 min read
           
Author: Rahul kumar, 
 Blog title is : Added option to share the blog on any social media | @dsabyte.com ,
 Blog link: https://dev.to/ats1999/added-option-to-share-the-blog-on-any-social-media-dsabyte-com-57oo, 
 Blog read time : Oct 8
            2 min read
           
Author: Kavindu Santhusa, 
 Blog title is : Top 10 trending github repos of the week💜. ,
 Blog link: https://dev.to/ksengine/top-10-trending-github-repos-of-the-week-k7, 
 Blog read time : Oct 8
            1 min read
           
Author: Andre Willomitzer, 
 Blog title is : OpenAQ - My first open source PR :) ,
 Blog link: https://dev.to/andrewillomitzer/openaq-my-first-open-source-pr-3k32, 
 Blog read time : Oct 8
            2 min read
           
Author: Kinanee Samson, 
 Blog title is : Observables Or Promises ,
 Blog link: https://dev.to/kalashin1/observables-or-promises-29a8, 
 Blog read time : Oct 8
            9 min read
           
Author: Amador Criado, 
 Blog title is : How to enable versioning in Amazon S3 ,
 Blog link: https://dev.to/aws-builders/how-to-enable-versioning-in-amazon-s3-17m8, 
 Blog read time :  for Oct 8
            2 min read
           
Author: Bartosz Zagrodzki, 
 Blog title is : React Context - jak efektywnie go używać? ,
 Blog link: https://dev.to/bartek532/react-context-jak-efektywnie-go-uzywac-41l, 
 Blog read time : Oct 8
            8 min read
                    
```



## Conclusion: 
This is a quick guide on how to scrape websites, there are other packages that can be used to perform the same function such as puppeteer, fetch, request and so on.

## Reference
[Web scraping by Thomas W.Smith](https://www.youtube.com/watch?v=XX8Q_39mue4&t=22s)
[Web scraping by Traversy Media](https://www.youtube.com/watch?v=LoziivfAAjE)
[Cheerio Docs](https://www.npmjs.com/package/cheerio)

Thanks for reading