Web Scraping With Puppeteer for Total Noobs: Part 2
Tags: webdev, javascript, tutorial, beginners
Published: 2023-02-02 02:29:53

Hello and welcome to the second post in this series on web scraping with Puppeteer. If you missed the first post you can check it out [here](https://dev.to/juniordevforlife/web-scraping-with-puppeteer-for-total-noobs-38j4). In this post we'll pick up where we left off and scrape some weather data from [weather.com](https://weather.com/weather/tenday/l/Austin+TX). The current goal is to scrape the 10-day forecast of Austin, Texas. Feel free to swap out Austin for your favorite city.

## Picking up where we left off

Our `scrape` function that we created in the previous post looks like this:

```javascript
async function scrape() {
  const browser = await puppeteer.launch({ dumpio: true });
  const page = await browser.newPage();

  await page.goto("https://weather.com/weather/tenday/l/Austin+TX");

  const weatherData = await page.evaluate(() =>
    Array.from(
      document.querySelectorAll(".DaypartDetails--DayPartDetail--2XOOV"),
      (e) => ({
        date: e.querySelector("h3").innerText,
      })
    )
  );

  await browser.close();
  return weatherData;
}

const scrapedData = await scrape();
console.log(scrapedData);
```

Let's now add to the weatherData. In addition to the `innerText` of the `h3` , we'll get the high temperature, the low temperature, and the precipitation percentage for the day.

Let's have a look at how we can do that:

```javascript
const weatherData = await page.evaluate(() =>
    Array.from(
      document.querySelectorAll(".DaypartDetails--DayPartDetail--2XOOV"),
      (e) => ({
        date: e.querySelector("h3").innerText,
        highTemp: e.querySelector(".DetailsSummary--highTempValue--3PjlX")
          .innerText,
        lowTemp: e.querySelector(".DetailsSummary--lowTempValue--2tesQ")
          .innerText,
        precipitationPercentage: e.querySelector(
          ".DetailsSummary--precip--1a98O"
        ).innerText,
      })
    )
```

As you can see I am adding three new properties to the object that's returned in the `Array.from` mapping function. These properties are `highTemp`, `lowTemp`, and `precipitationPercentage`. I found the class names by inspecting the document in the browser. These values seem to work, but only time will tell if something will have to be updated.

Let's now run `node scraper.js` in the terminal and check out the results:

```javascript
[
  {
    date: 'Tonight',
    highTemp: '--',
    lowTemp: '31°',
    precipitationPercentage: '84%'
  },
  {
    date: 'Thu 02',
    highTemp: '41°',
    lowTemp: '32°',
    precipitationPercentage: '53%'
  },
  {
    date: 'Fri 03',
    highTemp: '55°',
    lowTemp: '30°',
    precipitationPercentage: '6%'
  },
  {
    date: 'Sat 04',
    highTemp: '57°',
    lowTemp: '40°',
    precipitationPercentage: '7%'
  },
  {
    date: 'Sun 05',
    highTemp: '64°',
    lowTemp: '47°',
    precipitationPercentage: '9%'
  },
  {
    date: 'Mon 06',
    highTemp: '71°',
    lowTemp: '58°',
    precipitationPercentage: '14%'
  },
  {
    date: 'Tue 07',
    highTemp: '68°',
    lowTemp: '50°',
    precipitationPercentage: '54%'
  },
  {
    date: 'Wed 08',
    highTemp: '60°',
    lowTemp: '47°',
    precipitationPercentage: '40%'
  },
  {
    date: 'Thu 09',
    highTemp: '60°',
    lowTemp: '42°',
    precipitationPercentage: '52%'
  },
  {
    date: 'Fri 10',
    highTemp: '62°',
    lowTemp: '38°',
    precipitationPercentage: '17%'
  },
  {
    date: 'Sat 11',
    highTemp: '59°',
    lowTemp: '42°',
    precipitationPercentage: '11%'
  },
  {
    date: 'Sun 12',
    highTemp: '64°',
    lowTemp: '48°',
    precipitationPercentage: '15%'
  },
  {
    date: 'Mon 13',
    highTemp: '67°',
    lowTemp: '51°',
    precipitationPercentage: '24%'
  },
  {
    date: 'Tue 14',
    highTemp: '71°',
    lowTemp: '51°',
    precipitationPercentage: '24%'
  },
  {
    date: 'Wed 15',
    highTemp: '70°',
    lowTemp: '50°',
    precipitationPercentage: '21%'
  }
]
```

Very cool. We're getting the values I'd expect to get.

## GitHub Repo

I've set up a GitHub repository for this project. You can find the link [here](https://github.com/JasonFritsche/weather-scraper). Feel free to fork/clone this repository and play around. If you're not too comfortable with using `git`, there's a plethora of resources out there. If you'd be interested in a tutorial for noobs, please let me know in the comment section.

## Wrapping up
In this post we were able to scrape a bit more weather forecast data and return it in our `scrape` function. In the next post I'll show you how to create a GitHub Action that will run the `scrape` function once a day and save the scraped weather data in a `.json` file in the same GitHub repository.