Lighter Web Scraping Using NodeJS
Tags: node, webscraping, javascript
Published: 2021-05-07 06:31:35

---
title: Lighter Web Scraping Using NodeJS
published: true
date: 2021-05-04 15:29:14 UTC
tags: NodeJS,WebScraping,JavaScript
canonical_url: https://www.christiandimas.com/lighter-web-scraping-using-nodejs/
---

### An alternative way for doing web scraping using NodeJS
 ![Lighter Web Scraping Using NodeJS](https://images.unsplash.com/photo-1607446045657-959e0cccb4fc?crop=entropy&cs=tinysrgb&fit=max&fm=jpg&ixid=MnwxMTc3M3wwfDF8c2VhcmNofDJ8fHNjcmFwZXxlbnwwfHx8fDE2MjAxNDIwMTA&ixlib=rb-1.2.1&q=80&w=2000)

If you search for Web Scrapping using NodeJS, probably Puppeteer examples/articles will come up. It is an awesome library to use for complex web scraping because you are actually automating a browser when using Puppeteer. With that said, I think it’s an overkill library to use for a simpler web scrapping. So in this article, we’ll look into how we can scrape data from the web without using Puppeteer



> This article is originally published in my blog here: https://www.christiandimas.com/lighter-web-scraping-using-nodejs/

* * *

## Getting Started

To do this we need to solve two problems. The first one is, how we can get the website HTML code. After that’s solved, the second problem is how to get the actual data that we need from the HTML code.

Let’s start coding! First, scaffold a new Node project by running

```
yarn init -y
```

Now that we have a project ready to use, let’s install some dependencies

```
yarn install axios cheerio
```

### Axios

You might be familiar with this package because it’s quite a popular package to use for doing HTTP requests. Nowadays we usually use this to interact with API and get the result as JSON, but there’s a setting that we can tweak so the response will be an HTML instead of JSON.

### Cheerio

Taken from their NPM Package description, it’s a “_Fast, flexible & lean implementation of core jQuery designed specifically for the server_” I think that explains it really well. Basically, with this package, we can run jQuery commands on the server.

* * *

## Building The Scraper

We'll be using [https://books.toscrape.com/](https://books.toscrape.com/) website to test our scraper. First off, create a file called `index.js` in your project folder root, we’ll use this file to build our scraper.

From the list of books on the website we'll grab a couple of things including:

- Title
- Price
- Cover Image
- Rating
- Availability
- URL

Let's get coding!

{% gist https://gist.github.com/kenanchristian/0b2ab2622c360470f0963dd0a1da3387.js file=function.js %}

First, we import both `axios` and `cheerio` and then we create an async function called `scrape`.

Now let's grab the HTML code from the website using `axios` and load it to `cheerio` so we can query the data, to do this we'll do it like this

{% gist https://gist.github.com/kenanchristian/0b2ab2622c360470f0963dd0a1da3387.js file=load-html.js %}

After inspecting the website we can see that the book listing looks like this. This will help us get the data.

![Lighter Web Scraping Using NodeJS](https://www.christiandimas.com/content/images/2021/05/CleanShot-2021-05-04-at-9.14.44@2x.png)<figcaption>HTML Structure for the Book Item</figcaption>

With that information, let's grab the book elements first. We can do that by using cheerio like this

{% gist https://gist.github.com/kenanchristian/0b2ab2622c360470f0963dd0a1da3387.js file=get-book.js %}

Alright, we got the books. Now it's time to grab the simple data first, these are something that we can directly see in the element

{% gist https://gist.github.com/kenanchristian/0b2ab2622c360470f0963dd0a1da3387.js file=get-title-price-cover.js %}

After that's done, now we can also grab the data that's a bit more complicated like `rating`, `availability`, and `url`.

{% gist https://gist.github.com/kenanchristian/0b2ab2622c360470f0963dd0a1da3387.js file=get-rating-availability-url.js %}

First off, for `rating` we can grab the `p` element and check the class because it contains how many ratings the book has (e.g. Three). Next up, for the availability we can just check is there any div with a class of `.instock.availability`, we query for both classes to make sure that the `.instock` class is really for the availability, and the `.availability` has `.instock` class to show that it is available.

All done! This is what the complete code looks like

{% gist https://gist.github.com/kenanchristian/0b2ab2622c360470f0963dd0a1da3387.js file=complete.js %}
* * *

## Conclusion

I think this is the simplest way to do web scraping, and there are some pros and cons of doing it this way.

### Pros

- Simpler to build
- Fewer resources needed (library like Puppeteer needs to install Chromium to run)
- Smaller package size

### Cons

- Cannot scrape a website where navigation is needed (sign in, scroll, etc.)
- Cannot take a screenshot of the page

In the end, it depends on what website do you want to scrape and what data that you want to get. If you want to get something from a complex website then yes, use something like Puppeteer! It has a powerful API and you can interact with a complex website. But if you need something simple, then `axios` and `cheerio` might be a better choice

* * *

## Resources

Here are some resources for all the things that I've mentioned in this tutorial

- Axios:  [https://github.com/axios/axios](https://github.com/axios/axios)
- Cheerio: [https://github.com/cheeriojs/cheerio](https://github.com/cheeriojs/cheerio)
- Puppeteer: [https://github.com/puppeteer/puppeteer](https://github.com/puppeteer/puppeteer)