How to scrape Google Shopping results?
Tags: javascript, node, webscraping, tutorial
Published: 2022-08-20 13:12:00

In this post, we will learn to scrape Google Shopping Results using Node JS.

## Requirements:
Before we begin, install these libraries so we can move forward and prepare our scraper.

1. [Unirest JS](https://www.npmjs.com/package/unirest)
2. [Cheerio JS](https://www.npmjs.com/package/cheerio)

To extract our HTML data we will use Unirest JS and for parsing the HTML data we will use Cheerio JS.

## Target:


![Google Shopping Nike shoe Results](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/27vcoyeqyejkdfj69lm7.png)
We will target to scrape the shopping results of Nike shoes.

## Process:

We have installed all the things which we will need for our scraper. Now we will hit our target URL using Unirest JS to get our HTML data and then we will parse our extracted HTML data with the help of Cheerio JS.
We will target this URL:  

```
https://www.google.com/search?q=nike shoes&tbm=shop&gl=us
```
                          
Look at the tbm parameter and its value(`shop`, here). This value `shop` will tell Google that we are looking for shopping results.

Open this URL in your browser. Inspect the code. You will see that every organic shopping result is inside this tag `.sh-dgr__gr-auto`.

![Nike shoes Google Shopping Results HTML](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3udkd1qhd786jcfklt7m.png)

Now, we will search the tags for title, product link, price, rating, reviews, delivery, and source.

![NIke Shoes Google Shopping HTML tags ](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lbhwp5luds37qkr8lyz4.png)

![Nike Shoes Google Shopping HTML tags](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9h3bvqh08t5ufjqtsgei.png) 

**The above images are in the pattern of two at the top and one at the bottom.**

We have completed our search for tags of organic shopping results. Now, we will search for the tags of ad results.

![Google Shopping ad results](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gt7jjotlfi5hpmkx14rp.png) 

If you inspect the ad results you will see that all the ad results are inside the tag `.sh-np__click-target`. This tag contains all the information about the title, link, price, and source.

All the above things make our code look like this:

```
const unirest = require("unirest");
const cheerio = require("cheerio");

const getShoppingData = () => {
  
  try
  {
  return unirest
    .get("https://www.google.com/search?q=nike shoes&tbm=shop&gl=us")
    .headers({
      "User-Agent":
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36",
    })
    .then((response) => {
      let $ = cheerio.load(response.body);


    let ads = [];
    
    $(".sh-np__click-target").each((i,el) => {
     ads.push({
        title: $(el).find(".sh-np__product-title").text(),
        link: "https://google.com" + $(el).attr("href"),
        source: $(el).find(".sh-np__seller-container").text(),
        price: $(el).find(".hn9kf").text(),
        delivery: $(el).find(".U6puSd").text(),
     })
     if($(el).find(".rz2LD").length)
     {
      let extensions = []
        extensions = $(el).find(".rz2LD").text()
        ads[i].extensions = extensions
     }
     })

    for (let i = 0; i < ads.length; i++) {
        Object.keys(ads[i]).forEach(key => ads[i][key] === "" ? delete ads[i][key] : {});  
    }

    let shopping_results = [];

    $(".sh-dgr__gr-auto").each((i,el) => {
        shopping_results.push({
            title: $(el).find(".Xjkr3b").text(),
            link: $(el).find(".zLPF4b .eaGTj a.shntl").attr("href").substring($(el).find("a.shntl").attr("href").indexOf("=")+1),
            source: $(el).find(".IuHnof").text(),
            price: $(el).find(".XrAfOe .a8Pemb").text(),
            rating: $(el).find(".Rsc7Yb").text(),
            reviews: $(el).find(".NzUzee div").attr("aria-label") ? $(el).find(".NzUzee div").attr("aria-label").substring(0,$(el).find(".NzUzee div").attr("aria-label").indexOf(" ")) : "",
            delivery: $(el).find(".vEjMR").text()
        })
        if($(el).find(".Ib8pOd").length)
        {
            let extensions = [];
            extensions = $(el).find(".Ib8pOd").text();
            shopping_results[i].extensions = extensions
        }
    })

    for (let i = 0; i < shopping_results.length; i++) {
        Object.keys(shopping_results[i]).forEach(key => shopping_results[i][key] === "" ? delete shopping_results[i][key] : {});  
     }
     
     console.log(ads)
     console.log(shopping_results)
    })
  }
 catch(e)
 {
    console.log(e)
 }
}


getShoppingData();

```

## Result:

![Google-Shopping-Results](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/p27nga73cnogj3aiq91y.png)

Our result should look like this 👆🏻.

## [With Google Shopping API](https://docs.serpdog.io/google-shopping-api)

If you don't want to code and maintain the scraper in the long run then you can definitely try [Serpdog | Google Search API](https://serpdog.io).
Serpdog also offer 100 free requests on the first sign-up. 

```
const axios = require('axios');

axios.get('https://api.serpdog.io/shopping?api_key=APIKEY&q=shoes&gl=us')
  .then(response => {
    console.log(response.data);
  })
  .catch(error => {
    console.log(error);
  });
```

## Result:

![Google Shopping API docs](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/763bwdujz2pqrqcv3neq.png)

## Conclusion:

In this tutorial, we learned to scrape Google Shopping Results using Node JS. Feel free to comment anything you need clarification on. Follow me on [Twitter](https://twitter.com/darshan0_1) Thanks for reading!

## Additional Resources

1. [How to scrape Google Organic Search Results using Node JS?](https://medium.com/@darshankhandelwal12/how-to-scrape-google-organic-search-results-with-node-js-d3abe0274f40)
2. [Scrape Google Images Results](https://hackernoon.com/scrape-google-images-with-node-js)
3. [Scrape Google News Results](https://dev.to/darshan0_1/scrape-google-news-results-1je4)
4. [Scrape Google Maps Reviews](https://serpdog.io/blog/scrape-google-maps-reviews)