Scraping with puppeteer and export in JSON format
Tags: sraping, puppeteer, node, javascript
Published: 2023-11-04 08:27:53

App development is no walk in the park, figuring out the right themes to use, frameworks or libraries when building an app can be a tedious process, also sourcing for the right APIs to build on can be quite challenging especially when the one you need is not available.

Well in this case scraping the data can be all you need to provide the right data for your app.

In this blog post, we are going to be learning how to use Puppeteer to scrape data from the web and then export it in JSON format.

This will enable you to know how to scrape data with Puppeteer and implement it on any project of your choice.

# Prerequires

A basic knowledge of Node.js and JavaScript is required to be able to follow along.

# Setup

To get started, create a folder and name it as `puppeteer`, then open it up as a workspace on VS Code as indicated below 👇

![VS-Code Screenshot with Puppeteer](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pe3gvxvkqgb2by6byibq.png)

Click on `Ctrl` + `` ` `` backtick to open the terminal on VS Code. 👇

![VS Code Terminal and puppeteer](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/94vymotdt9mjwfo2p6ol.png)

You should already have `node.js` installed on your system, but if you are unsure if you have it installed. run the command below.

```bash
node -v
```

This will return the current version you have installed on your system

```bash
👉 // v21.1.0
```

Am currently running on version `v21.1.0` but depending on when you are reading this or what version you have installed, it will be different from mine.

Next step, generate a `package.json` file. Type the command below and hit enter.

```bash
npm init -y
```

This will create and set up a `package.json` file where all dependencies and devDependencies will be saved.

Now that we have that figured out, let's install Puppeteer.

```bash
npm i puppeteer
```

The above command will install Puppeteer on your system if your default package manager is `npm` but if you are making use of `yarn` or `pnpm` you can try the following command below.

```bash
# using yarn
yarn add puppeteer

# using pnpm
pnpm i puppeteer*
```

Directly on your terminal, the installed `node_modules` should indicate as follows. 👇

![VS Code Terminal with install puppeteer](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/weu8zy1m4yqcrx7ocem1.png)

By default, Puppeteer will install `node_modules` and `package-lock.json` and update the `package.json` file with the latest version of Puppeteer. As at the time of writing, I have `21.5.0` installed, yours should be similar or higher. Use `npm i puppeteer@12.5.0` to install the exact version used in this tutorial.

# Entry Point Setup

Create `index.js` file. This will serve as an entry point for our application.

```jsx
const puppeteer = require('puppeteer');
```

click to open the `index.js` file, type the code above. The syntax above is [commonJS](https://en.wikipedia.org/wiki/CommonJS), this is what we’ll be using in this tutorial.

```javascript

(async () => {

 👉 // your code should come write here

})();
```

The code above is [immediately-invoked function expression (IIFE)](https://developer.mozilla.org/en-US/docs/Glossary/IIFE), this allows the code specified inside the function to be executed immediately and it is associated with [async/await](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function) syntax for handling asynchronous operations.

Now that we have set the function boilerplate, let's create a headless browser instance with Puppeteer.

```javascript
(async () => {

 const browser = await puppeteer.launch({ headless: "new" });

 const page = await browser.newPage();

 await browser.close();

})();
```

Here, we are creating an instance of a `browser` and `puppeteer` is set to `launch`. This opens up the browser.

The key and value [{ headless: "new" }](https://developer.chrome.com/articles/new-headless/) object syntax indicates that the browser should operate in a headless mode, that is to run in the background without displaying its UI.

Next the `browser.newPage()` opens up a new tag and assigns it to a `page` variable.

Then the `await browser.close()` method closes the browser when the function stops running.

# Scraping web page data

To begin scraping data, you have to decide which platform data you want to scrap. While it is easy to scrap data, some platforms prevent access to scraping of it data and you will have to use a third-party tool like [ZenRows](https://www.zenrows.com/) or [Oxylabs](https://oxylabs.io/) just to mention a few.

In this example, I will be making use of [Techmeme.com](http://Techmeme.com) for scraping.

```javascript
(async () => {
  const browser = await puppeteer.launch({ headless: "new" });

  const page = await browser.newPage();

 👉 await page.goto("https://www.techmeme.com/");
 
  await browser.close();
}
  )();
```

Here, the `goto()` method is used to specify the web URL we want to scrap.

Once the URL has been specified. visit and `inspect` the website you want to scrap to have an overview of the HTML structure. In this case, we will visit [techmeme.com](http://techmeme.com) and inspect the page to know which element to grab and return.

![Techmeme.com screenshot](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/s0k2qo6mndc7rd5m0vv8.png)

Right-click on the page, this will open a dialog box, then click `inspect`. for short hit `F12` key. See the example below 👇

![Inspecting the Techmene website](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mcnmu8tibcnpy8ua4yzp.png)

Your DevTools should open up as indicated below.

![Inspect element on Techmeme with chrome DevTools](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0eo8r59eofvhn5gp2mvw.png)

From the above screenshot, we have identified the tag we want to scrap which is the `div` with the class of `clus`. The parent of `clue` has a class of `topcol1`. So we will be grabbing this with Puppeteer.

Try the code sample below. 👇

```javascript
const browser = await puppeteer.launch({ headless: "new" });

  const page = await browser.newPage();
  await page.goto("https://www.techmeme.com/");

 👉 const techNewsApis = await page.evaluate(() =>
    Array.from(document.querySelectorAll("#topcol1 .clus"), (e) => ({
     // Some code example here.
    }))
  );
```

Here, the [Array.from()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/from) method is used to grab all the elements with a class of `clus` that is a direct parent of `topcol1`, this will create a shallow-copied Array instance from the returned elements.

Also, notice we are using the `evaluate()` method, which takes a call-back function. It evaluates the response of a Promise and returns its values when the Promise is resolved. In this case, the resolved value is returned to the `techNewsApis` variable.

We have seen the basic example of how to grab elements, but this is not all, what we have are elements that Puppeteer does not know what to do with it.

What we want to do is return the headline on the webpage and the link associated with it. We also want to implement a unique `id` to the data.

![inspecting an anchor tag with name attribute on techmeme](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sa9gqkll1rb86il9mjg0.png)

From the above screenshot, I inspect the first `<a>` tag and return the name value as an id.

![inspecting an achor tag with title and link on Techmeme](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/931a4g8ry2v686gsmrt9.png)

Similarly, I inspect the `<a>` tag with a class of `ourh` which are children of the `clue` tag and then return both the headline and the link associated with it.

See the code below

```javascript
const techNewsApis = await page.evaluate(() =>
    Array.from(document.querySelectorAll("#topcol1 .clus"), (e) => ({
      
     👉   id: e.querySelector(".clus > a").name,
     👉   title: e.querySelector("a.ourh").innerText,
     👉   url: e.querySelector("a.ourh").href,

    }))
  );
```

Here, `querySelector()` method is used to access the elements and a property and two attribute is attached to return the respective value associated with it.

* `name` the attribute returns the number string value specified on the `<a>` tag
    
* `innerText` returns the text content in between the open and close `<a>` tag
    
* `href` returns the link specified on the `<a>` tag
    

Now that we have successfully scraped the data, let's return it and see what the data looks like.

```javascript
const techNewsApis = await page.evaluate(() =>
    Array.from(document.querySelectorAll("#topcol1 .clus"), (e) => ({
      id: e.querySelector(".clus > a").name,
      title: e.querySelector("a.ourh").innerText,
      url: e.querySelector("a.ourh").href,
    }))
  );

 👉 console.log(techNewsApis);

  await browser.close();
```

Parse the resolved value to the `console.log()` method.

```bash
node index.js
```

Go to the terminal and run the above command.

![Logged data from the Techmeme website](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lbczi6a2nrqv0l5gpodl.png)

The scraped data is now logged to the console as shown above. With the `id`, `title` and `url` specified respectively.

# Exporting scraped data in a JSON format

Logging data to the terminal is not a good use case since we can not access or use it directly from the terminal.

What to do next is to save it on the local directory of the project we are working on so that we can make use of it.

```jsx
    const puppeteer = require('puppeteer');
 👉 const fs = require('fs');
```

With the [fs](https://nodejs.org/api/fs.html#fswritefilefile-data-options-callback) (file system) module specified, we can write to save the data to something we can use in an application.

```javascript
const puppeteer = require('puppeteer');
const fs = require('fs');

(async () => {
  const browser = await puppeteer.launch({ headless: "new" });

  const page = await browser.newPage();
  await page.goto("https://www.techmeme.com/");

  const techNewsApis = await page.evaluate(() =>
    Array.from(document.querySelectorAll("#topcol1 .clus"), (e) => ({
      id: e.querySelector(".clus > a").name,
      title: e.querySelector("a.ourh").innerText,
      url: e.querySelector("a.ourh").href,
    }))
  );

  //Save data to JSON file
👉    fs.writeFile("techNewsApis.json", JSON.stringify(techNewsApis), (error) => {
👉    if (error) throw error;
👉       console.log(`techNewsApis is now saved on your project folder`);
		  });

  console.log(techNewsApis);
  await browser.close();
}
  )();
```

Here, the fs module is implemented as follows;

* `fs.writeFile("techNewsApis.json")` set the name of the file to be saved.
    
* `JSON.stringify(techNewsApis)` is used to receive and parse the value to a valid JSON file
    
* `if (error) throw error` throws an error if any occur
    

Next, go to the terminal and run the command as shown below.

```bash
node index.js
```

This will save the exported data as a valid JSON file for external use.

![Exported JSON data shown in VS-Code](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6i6wvi1isq2ttfaaa0qs.png)

From the above screenshot, the exported data is saved directly in the project folder.

# Conclusion

Scraping can be very useful in getting specific data in situations where the data we need is not readily available or can be found on another website, in this case, we can scrape such data and use it in our project.

However, this comes with some caveats as some web owners do not grant access for the scraping of their website.