Scrape Google News Results
Tags: tutorial, javascript, node, scraping
Published: 2022-07-18 17:37:00

## Introduction

In this article, I will teach you to scrape Google News Results using Node JS.

## Requirements

Before we begin this tutorial, we need to install some NPM packages, which we will use further in this tutorial.

1. [Unirest JS](https://www.npmjs.com/package/unirest) 
2. [Cheerio JS](https://www.npmjs.com/package/cheerio)

We will use NPM libraries Unirest to extract our target HTML data and Cheerio JS for the parsing the extracted HTML data.

## Target

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zk00f3r8k1lx6vnxmdta.png)

We will scrape the Google News Results of query Football.

## Procedure

We have installed and set up all the things to prepare our scraper. Now we will use Unirest to make a get request to our target URL, which is: `https://www.google.com/search?q=football&gl=us`
, and Cheerio JS to parse the HTML. 


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z069fda9vkfpd7de79d6.png)

If you inspect the Google News page you will find that all the main articles are contained inside this `div.BGxR7d` container. By searching in this container we got the tag for title as `.MBeuO`, for the snippet as `.GI74Re`, for the thumbnail as `div.NUnG9d img`, for the date as `div.ZE0LJd`and for the link as `.WlydOe`.

This makes our code as: 

```
const unirest = require("unirest");
const cheerio = require("cheerio");

const getNewsData = () => {
  return unirest
    .get("https://www.google.com/search?q=football&gl=us&tbm=nws")
    .headers({
      "User-Agent":
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36",
    })
    .then((response) => {
      let $ = cheerio.load(response.body);

      let news_results = []; 

      $(".BGxR7d").each((i,el) => {
        news_results.push({
         link: $(el).find("a").attr('href'),
         title: $(el).find("div.mCBkyc").text(),
         snippet: $(el).find(".GI74Re").text(),
         date: $(el).find(".ZE0LJd span").text(),
         thumbnail: $(el).find(".NUnG9d img").attr("src")
        })
      })
      
    console.log(news_results)
    });
};

getNewsData();
```

## Results: 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/g8gbxfevnlbecv1vtf4t.png)

## By using [Google News API](https://docs.serpdog.io/google-news-api)

If you want to scrape Google News Results easily, without making a scraper, as scraping can take a lot of time sometimes, you can try this [Google Search API](https://serpdog.io). 

Serpdog also provides 100 free searches per month, and if you want to scale your requests quota, you can buy paid plans also.

Example request code: 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/x33perjbr6jhkh7838qs.png)

Results: 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/eyoeqgbx5liunkreaa8y.png)

## Conclusion 

In this tutorial we learned to scrape Google News Results. If you have any questions, feel free to ask me in the comments. Follow me on [Twitter](https://twitter.com/serpdogAPI). Thanks for reading! 

## Additional Resources

1. [Web Scraping Google With Node JS - A Complete Guide](https://serpdog.io/blog/web-scraping-google-with-node-js)
2. [Web Scraping Google Without Getting Blocked](https://serpdog.io/blog/web-scraping-google-without-getting-blocked)
3. [Google Maps Reviews Scraper](https://serpdog.io/blog/scrape-google-maps-reviews)
4. [Web Scraping Google Organic Search Results](https://serpdog.io/blog/scrape-google-organic-search-results)