Web Scraping With Puppeteer for Total Noobs: Part 1
Tags: javascript, beginners, tutorial, webdev
Published: 2023-01-31 02:28:46

Web scraping is something I never thought I'd do. I'm primarily a UI developer, although my career started as a backend developer. I've never been asked at a job to perform any scraping tasks, and I'd never had a personal project that required me to scrape some data until recently. I'll share what I've learned, which is honestly probably just scratching the surface of what you can do with a technology like [Puppeteer](https://pptr.dev/). In this post I'll walk you through installing and writing a Node.js script that will scrape a weather forecast website. The weather forecast site isn't my personal project that I mentioned earlier, it's more of a contrived example of how to get started with web scraping using Node.js and Puppeteer. 

This post assumes that you have some knowledge of Node.js, NPM, async/await, and the command line.

This is intended to be a multi-series post/tutorial. We'll start off slow, and eventually have a project that will net us a 10 day weather forecast in `json` format.

*Disclaimer: I'm no expert in Node.js and certainly not a Puppeteer expert, so, if you see anything that could be done in a more proficient manner, or that's just plain wrong, please let me know. Thanks.*

*Note: I'll be using `npm` in this post. Feel free to use your preferred package manager if you're comfortable doing so.*

## Project Setup

The website that we'll be scraping is [weather.com](https://weather.com/weather/tenday/l/Austin+TX). We'll work towards getting the 10-day forecast of Austin, Texas. You can most certainly swap out the city for your preferred city.

Let's go ahead and create a new directory for our project:
`mkdir weather-scraper`

Now navigate in to the directory:
`cd weather-scraper`

Let's initialize the project (I'll use the `-y` flag to skip all the questions):
`npm init -y`

Next, I'm going to go ahead and open my favorite editor and create a JavaScript file. I'll name mine `scraper.js`.

![project directory image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1e49twnxojmux4sitysh.png)


Before we get too far, let's add one line to the `package.json` file so that we can use the import declaration. Add this line:
`"type": "module",`

Your `package.json` should look something like this:

```json
{
  "name": "weather-scraper",
  "type": "module",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "puppeteer": "^19.6.2"
  }
}
```

Let's now install [Puppeteer](https://pptr.dev/).

`npm i puppeteer`

## First Few Lines of Code

First things first, let's import puppeteer on line 1:

`import puppeteer from 'puppeteer';`

We'll create an async function called `scrape` and write out the first bit of code. The reason that this function is an async function will become quite clear in just a bit (we basically have to await everything).

```javascript
async function scrape() {
  const browser = await puppeteer.launch({ dumpio: true });
  const page = await browser.newPage();
}
```

We've created two variables, `browser` and `page`. The `browser` variable is created using puppeteer's `launch` method, which has a return type of `Promise<Browser>` . See [this page](https://pptr.dev/api/puppeteer.browser) for more info on the Browser type.

The `page` variable is created using the [browser context's](https://pptr.dev/api/puppeteer.browsercontext) `newPage` method, which returns a type of `Promise<Page>`. See [this page](https://pptr.dev/api/puppeteer.page) for more info on the Page class.

The Page class has a method that we'll use to navigate to the weather website we're trying to scrape, the [`goto` method](https://pptr.dev/api/puppeteer.page.goto). This method takes in one parameter, the url, along with an optional options parameter which we won't use at this time. This method returns a type of `Promise<HttpResonse | null>`. 

We'll add that next:
`await page.goto('https://weather.com/weather/tenday/l/Austin+TX')`

Our `scraper.js` file should look like this at this point:

```javascript
import puppeteer from 'puppeteer';

async function scrape() {
    const browser = await puppeteer.launch({ dumpio: true });
    const page = await browser.newPage();

    await page.goto('https://weather.com/weather/tenday/l/Austin+TX')
}
```

## Let the Scraping Begin

After the `goto` method, we can use the `Page` class's [`evaluate` method](https://pptr.dev/api/puppeteer.page.evaluate) to work some magic. Inside of the `evaluate` method is where we can write out function inside the page's context and have the result returned. Essentially, we'll write some code inside this method to get the data from the page that we want to get. I'm going to put the code below, then discuss:

```javascript
async function scrape() {
  const browser = await puppeteer.launch({ dumpio: true });
  const page = await browser.newPage();

  await page.goto("https://weather.com/weather/tenday/l/Austin+TX");

  const weatherData = await page.evaluate(() =>
    Array.from(
      document.querySelectorAll(".DaypartDetails--DayPartDetail--2XOOV"),
      (e) => ({
        date: e.querySelector("h3").innerText,
      })
    )
  );

  await browser.close();
  return weatherData;
}
```

The `evaluate` method is a higher-order function, meaning we can pass in another function as a parameter, which is exactly what we're doing. Inside the anonymous function that we pass in, we have access to the `document` object. If you go inspect the weather.com url that I shared, you should be able to find the `DaypartDetails--DayPartDetail--2XOOV` class.

We are using the `Array.from` method and passing in `document.querySelectorAll(".DaypartDetails--DayPartDetail--2XOOV")`, which will return a `NodeList` of all elements in the document with that class. The `Array.from` method has a second, optional parameter, which is a mapping function. We are using that mapping function to select all `h3` elements inside of each element in the `NodeList` and assigning the value of it's `innerText` to a property that we call `date`;

## Viewing Our Data

After the `scrape` function, add in these two lines and save:

```javascript
const scrapedData = await scrape();
console.log(scrapedData);
```

Let's go to the terminal and run:
`node scraper.js`

We get results that look something like this:

```javascript
[
  { date: 'Tonight' },
  { date: 'Tue 31' },
  { date: 'Wed 01' },
  { date: 'Thu 02' },
  { date: 'Fri 03' },
  { date: 'Sat 04' },
  { date: 'Sun 05' },
  { date: 'Mon 06' },
  { date: 'Tue 07' },
  { date: 'Wed 08' },
  { date: 'Thu 09' },
  { date: 'Fri 10' },
  { date: 'Sat 11' },
  { date: 'Sun 12' },
  { date: 'Mon 13' }
]
```

Very cool.

The entire `scraper.js` file looks like this:

```javascript
import puppeteer from "puppeteer";

async function scrape() {
  const browser = await puppeteer.launch({ dumpio: true });
  const page = await browser.newPage();

  await page.goto("https://weather.com/weather/tenday/l/Austin+TX");

  const weatherData = await page.evaluate(() =>
    Array.from(
      document.querySelectorAll(".DaypartDetails--DayPartDetail--2XOOV"),
      (e) => ({
        date: e.querySelector("h3").innerText,
      })
    )
  );

  await browser.close();
  return weatherData;
}

const scrapedData = await scrape();
console.log(scrapedData);
```

## Wrapping Up
I implore you to play around with this. See if you're able to scrape other bits of data. In my next post I'll continue where I left off here. By the end of my web scraping posts I'll have shown you how to scrape, create a GitHub Action to automatically scrape, and have the GitHub Action save the data in to a `.json` file in the same repo.

I hope you found this post interesting. Thanks for sticking around. Until next time.