Web scraping Google Reverse Images results with Nodejs
Tags: webscraping, node, serpapi, google
Published: 2022-09-22 14:49:37

<h2 id='how'>How reverse search happens</h2>

First of all, we need to paste image link to Google Image search:

![process](https://user-images.githubusercontent.com/64033139/189161123-c67df002-f29a-42ff-854d-e2aeb1e735a1.png)

Next, we need to click on the "Find image source":

![process](https://user-images.githubusercontent.com/64033139/189377350-b17fbf8e-504e-4347-bab9-b0addac3d5ac.png)

<h2 id='what'>What will be scraped</h2>

![what](https://user-images.githubusercontent.com/64033139/189154956-9767d598-cdf7-4f2a-a560-72dc183b7ef8.png)

<h2 id='full_code'>Full code</h2>

If you don't need an explanation, have a look at [the full code example in the online IDE](https://replit.com/@MikhailZub/Scrape-Google-Reverse-Image-with-NodeJS-SerpApi#withPuppeteer.js)

```javascript
const puppeteer = require("puppeteer-extra");
const StealthPlugin = require("puppeteer-extra-plugin-stealth");

puppeteer.use(StealthPlugin());

const imageUrl = "https://www.bugatti.com/fileadmin/_processed_/sei/p63/se-image-ce40627babaa7b180bc3dedd4354d61c.jpg"; // what we want to search

const URL = `https://images.google.com`;

async function setImage(page) {
  const isPopup = await page.evaluate(() => {
    return Array.from(document.querySelectorAll("iframe")).find((el) => el.style.visibility !== "hidden");
  });
  if (isPopup) {
    for (let i = 0; i < 14; i++) {
      await page.keyboard.press("Tab");
      await page.waitForTimeout(500);
    }
    await page.keyboard.press("Enter");
  }
  await page.waitForTimeout(1500);
  await page.click(".nDcEnd");
  await page.waitForTimeout(1500);
  await page.click(".PXT6cd input");
  await page.keyboard.type(imageUrl);
  await page.waitForTimeout(1500);
  await page.click(".PXT6cd div");
  await page.waitForTimeout(5000);
  await page.click(".QeWRZ .WpHeLc");
}

async function fillInfoFromPage(page) {
  return await page.evaluate(async () => {
    return Array.from(document.querySelectorAll("#search .Ww4FFb")).map((el) => ({
      title: el.querySelector(".yuRUbf > a > h3").textContent.trim(),
      link: el.querySelector(".yuRUbf > a").getAttribute("href"),
      snippet: el.querySelector(".VwiC3b").textContent.trim(),
    }));
  });
}

async function getReverseImageInfo() {
  const browser = await puppeteer.launch({
    headless: false,
    args: ["--no-sandbox", "--disable-setuid-sandbox"],
  });

  const page = await browser.newPage();

  await page.setDefaultNavigationTimeout(60000);
  await page.goto(URL);
  await page.waitForSelector(".nDcEnd");

  await setImage(page);

  await page.waitForTimeout(5000);
  const pages = await browser.pages();
  const page2 = pages[pages.length - 1];

  let imageOrganicResults = [];

  while (true) {
    await page2.waitForSelector(".Ww4FFb");
    imageOrganicResults.push(...(await fillInfoFromPage(page2)));
    const nextButton = await page2.$$(".d6cvqb");
    let isButtonActive;
    if (nextButton) {
      isButtonActive = await nextButton[1]?.$("a");
    } else {
      isButtonActive = await page2.$(".acRNod");
    }
    if (!isButtonActive) break;
    await isButtonActive.click();
  }

  await browser.close();

  return imageOrganicResults;
}

getReverseImageInfo().then((result) => console.dir(result, { depth: null }));
```

<h2 id='preparation'>Preparation</h2>

First, we need to create a Node.js\* project and add [`npm`](https://www.npmjs.com/) packages [`puppeteer`](https://www.npmjs.com/package/puppeteer), [`puppeteer-extra`](https://www.npmjs.com/package/puppeteer-extra) and [`puppeteer-extra-plugin-stealth`](https://www.npmjs.com/package/puppeteer-extra-plugin-stealth) to control Chromium (or Chrome, or Firefox, but now we work only with Chromium which is used by default) over the [DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/) in [headless](https://developers.google.com/web/updates/2017/04/headless-chrome) or non-headless mode.

To do this, in the directory with our project, open the command line and enter `npm init -y`, and then `npm i puppeteer puppeteer-extra puppeteer-extra-plugin-stealth`.

\*<span style="font-size: 15px;">If you don't have Node.js installed, you can [download it from nodejs.org](https://nodejs.org/en/) and follow the installation [documentation](https://nodejs.dev/learn/introduction-to-nodejs).</span>

📌Note: also, you can use `puppeteer` without any extensions, but I strongly recommended use it with `puppeteer-extra` with `puppeteer-extra-plugin-stealth` to prevent website detection that you are using headless Chromium or that you are using [web driver](https://www.w3.org/TR/webdriver/). You can check it on [Chrome headless tests website](https://intoli.com/blog/not-possible-to-block-chrome-headless/chrome-headless-test.html). The screenshot below shows you a difference.

![stealth](https://user-images.githubusercontent.com/64033139/173014238-eb8450d7-616c-42ae-8b2f-24eeb5fd5916.png)

<h2 id='process'>Process</h2>

The first step is to extract data from HTML elements, then change the page and repeat again. The process of getting the right CSS selectors is fairly easy via [SelectorGadget Chrome extension](https://selectorgadget.com/) which able us to grab CSS selectors by clicking on the desired element in the browser. However, it is not always working perfectly, especially when the website is heavily used by JavaScript.

We have a dedicated [web Scraping with CSS Selectors](https://serpapi.com/blog/web-scraping-with-css-selectors-using-python/#css_gadget) blog post at SerpApi if you want to know a little bit more about them.

The Gif below illustrates the approach of selecting different parts of the results.

![how](https://user-images.githubusercontent.com/64033139/189158082-852874ad-e619-47dc-944d-07b4ace9a9f0.gif)

<h3 id='code_explanation'>Code explanation</h3>

Declare [`puppeteer`](https://www.npmjs.com/package/puppeteer-extra) to control Chromium browser from `puppeteer-extra` library and [`StealthPlugin`](https://www.npmjs.com/package/puppeteer-extra-plugin-stealth) to prevent website detection that you are using [web driver](https://www.w3.org/TR/webdriver/) from `puppeteer-extra-plugin-stealth` library:

```javascript
const puppeteer = require("puppeteer-extra");
const StealthPlugin = require("puppeteer-extra-plugin-stealth");
```

Next, we "say" to `puppeteer` use `StealthPlugin`, write the image link that we want to search, and Google Image URL:

```javascript
puppeteer.use(StealthPlugin());

const imageUrl = "https://www.bugatti.com/fileadmin/_processed_/sei/p63/se-image-ce40627babaa7b180bc3dedd4354d61c.jpg"; // what we want to search

const URL = `https://images.google.com`;
```

Next, we write a function to paste image URL in the Google Search:

```javascript
async function setImage(page) {
  ...
}
```

In this function, first, we need to check if Google proposes you Sign in or register (using [`evaluate()`](https://pptr.dev/api/puppeteer.page.evaluate) and[`querySelectorAll()`](https://developer.mozilla.org/en-US/docs/Web/API/Document/querySelectorAll) methods to get access to right HTML selectors, and make the new array from got [NodeList](https://developer.mozilla.org/en-US/docs/Web/API/NodeList) with [`Array.from()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/from), and finally [`find()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/find) methods to get the necessary data from an array):

```javascript
const isPopup = await page.evaluate(() => {
  return Array.from(document.querySelectorAll("iframe")).find((el) => el.style.visibility !== "hidden");
});
```

If it's true, we need to close this popup. Because this popup is placed in `iframe` element, there's challenging to get control from the Puppeteer on it. So we use the simple way, just press the "Tab" key fourteen times (using [`keyboard.press()`](https://pptr.dev/api/puppeteer.keyboard.press/) method) with 0,5 sec timeout (using [`waitForTimeout`](https://pptr.dev/api/puppeteer.page.waitfortimeout) method) until the need button has been in focus and press the "Enter" button:

```javascript
if (isPopup) {
  for (let i = 0; i < 14; i++) { // 14 is the number of press the "Tab" key
    await page.keyboard.press("Tab");
    await page.waitForTimeout(500);
  }
  await page.keyboard.press("Enter");
}
```

Then we click on necessary buttons (using [`click()`](https://pptr.dev/api/puppeteer.page.click) method) and type the `imageUrl`(using [`keyboard.type()`](https://pptr.dev/api/puppeteer.keyboard.type) method). Before the last click we use [`waitForTimeout`](https://pptr.dev/api/puppeteer.page.waitfortimeout) method:

```javascript
await page.waitForTimeout(1500);
await page.click(".nDcEnd");
await page.waitForTimeout(1500);
await page.click(".PXT6cd input");
await page.keyboard.type(imageUrl);
await page.waitForTimeout(1500);
await page.click(".PXT6cd div");
await page.waitForTimeout(5000);
await page.click(".QeWRZ .WpHeLc");
```

Next, we write a function to get need information from HTML selectors. We can do this with [`textContent`](https://developer.mozilla.org/en-US/docs/Web/API/Node/textContent) and [`trim()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/trim) methods, which get the raw text and removes white space from both sides of the string. If we need to get links, we use [`getAttribute()`](https://developer.mozilla.org/en-US/docs/Web/API/Element/getAttribute) method to get `"href"` HTML element attribute:

```javascript
async function fillInfoFromPage(page) {
  return await page.evaluate(async () => {
    return Array.from(document.querySelectorAll("#search .Ww4FFb")).map((el) => ({
      title: el.querySelector(".yuRUbf > a > h3").textContent.trim(),
      link: el.querySelector(".yuRUbf > a").getAttribute("href"),
      snippet: el.querySelector(".VwiC3b").textContent.trim(),
    }));
  });
}
```

Next, write a function to control the browser, and get information:

```javascript
async function getReverseImageInfo() {
  ...
}
```

In this function first we need to define `browser` using `puppeteer.launch({options})` method with current `options`, such as `headless: false` and `args: ["--no-sandbox", "--disable-setuid-sandbox"]`.

These options mean that we use [headless](https://developers.google.com/web/updates/2017/04/headless-chrome) mode and array with [arguments](https://peter.sh/experiments/chromium-command-line-switches/) which we use to allow the launch of the browser process in the online IDE. And then we open a new `page`:

```javascript
const browser = await puppeteer.launch({
  headless: false,
  args: ["--no-sandbox", "--disable-setuid-sandbox"],
});

const page = await browser.newPage();
```

Next, we change default ([30 sec](https://github.com/puppeteer/puppeteer/blob/2a0eefb99f0ae00dacc9e768a253308c0d18a4c3/src/common/TimeoutSettings.ts#L17)) time for waiting for selectors to 60000 ms (1 min) for slow internet connection with [`.setDefaultNavigationTimeout()`](https://pptr.dev/api/puppeteer.page.setdefaultnavigationtimeout) method, go to `URL` with [`.goto()`](https://pptr.dev/api/puppeteer.page.goto) method and use [`.waitForSelector()`](https://pptr.dev/api/puppeteer.page.waitforselector) method to wait until the selector is load:

```javascript
await page.setDefaultNavigationTimeout(60000);
await page.goto(URL);
await page.waitForSelector(".nDcEnd");
```

Then, we wait until the `setImage` functions was finished and change the page context from the new tab (get an array with all opened pages with [`browser.pages()`](https://pptr.dev/api/puppeteer.browser.pages) method and pick the last one):

```javascript
await setImage(page);

await page.waitForTimeout(5000);
const pages = await browser.pages();
const page2 = pages[pages.length - 1];
```

Then we create the empty `imageOrganicResults` array, use `while` loop in which we wait for load results, and add results to the end of the `imageOrganicResults` array (using [`push()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/push) method and the [spread syntax(`[...]`)](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Spread_syntax)).

After that we need to go to the next page. We check if the next page button is present on the page, we click it, and repeat our loop, otherwise, we end the loop:

```javascript
let imageOrganicResults = [];

while (true) {
  await page2.waitForSelector(".Ww4FFb");
  imageOrganicResults.push(...(await fillInfoFromPage(page2)));
  const nextButton = await page2.$$(".d6cvqb");
  let isButtonActive;
  if (nextButton) {
    isButtonActive = await nextButton[1]?.$("a");
  } else {
    isButtonActive = await page2.$(".acRNod");
  }
  if (!isButtonActive) break;
  await isButtonActive.click();
}
```

And finally, we close the browser, and return the received data:

```javascript
await browser.close();

return imageOrganicResults;
```

Now we can launch our parser:

```bash
$ node YOUR_FILE_NAME # YOUR_FILE_NAME is the name of your .js file
```

<h2 id='output'>Output</h2>

```json
[
   {
      "title":"Super Sport - Bugatti Veyron 16.4",
      "link":"https://www.bugatti.com/models/veyron-models/veyron-164-super-sport/",
      "snippet":"In the year of its market launch the Veyron 16.4 already set up a speed record for street cars. Adhering to the the Guiness World Record restrictions an ..."
   },
   {
      "title":"Bugatti Veyron - Wikipedia",
      "link":"https://en.wikipedia.org/wiki/Bugatti_Veyron",
      "snippet":"The Super Sport version of the Veyron is one of the fastest street-legal production cars in the world, with a top speed of 431.072 km/h (267.856 mph)."
   },
   ... and other results
]
```

<h2 id='serp_api'>Using<a href="https://serpapi.com/google-reverse-image"> Google Reverse Image API </a>from SerpApi</h2>

This section is to show the comparison between the DIY solution and our solution.

The biggest difference is that you don't need to use browser automation to scrape results, create the parser from scratch and maintain it.

There's also a chance that the request might be blocked at some point from Google, we handle it on our backend so there's no need to figure out how to do it yourself or figure out which CAPTCHA, proxy provider to use.

First, we need to install [`google-search-results-nodejs`](https://www.npmjs.com/package/google-search-results-nodejs):

```bash
npm i google-search-results-nodejs
```

Here's the [full code example](https://replit.com/@MikhailZub/Scrape-Google-Reverse-Image-with-NodeJS-SerpApi#withSerpApi.js), if you don't need an explanation:

```javascript
const SerpApi = require("google-search-results-nodejs");
const search = new SerpApi.GoogleSearch(process.env.API_KEY);

const imageUrl = "https://www.bugatti.com/fileadmin/_processed_/sei/p63/se-image-ce40627babaa7b180bc3dedd4354d61c.jpg"; // what we want to search

const params = {
  engine: "google_reverse_image", // search engine
  image_url: imageUrl, // search image
};

const getJson = () => {
  return new Promise((resolve) => {
    search.json(params, resolve);
  });
};

const getResults = async () => {
  const organicResults = [];
  while (true) {
    const json = await getJson();
    if (json.search_information?.organic_results_state === "Fully empty") break;
    organicResults.push(...json.image_results);
    params.start ? (params.start += 10) : (params.start = 10);
  }
  return organicResults;
};

getResults().then((result) => console.dir(result, { depth: null }));
```

<h3 id='serp_api_code_explanation'>Code explanation</h3>

First, we need to declare `SerpApi` from [`google-search-results-nodejs`](https://www.npmjs.com/package/google-search-results-nodejs) library and define new `search` instance with your API key from [SerpApi](https://serpapi.com/manage-api-key):

```javascript
const SerpApi = require("google-search-results-nodejs");
const search = new SerpApi.GoogleSearch(API_KEY);
```

Next, we write an image URL and the necessary parameters for making a request:

```javascript
const imageUrl = "https://www.bugatti.com/fileadmin/_processed_/sei/p63/se-image-ce40627babaa7b180bc3dedd4354d61c.jpg"; // what we want to search

const params = {
  engine: "google_reverse_image", // search engine
  image_url: imageUrl, // search image
};
```

Next, we wrap the search method from the SerpApi library in a promise to further work with the search results:

```javascript
const getJson = () => {
  return new Promise((resolve) => {
    search.json(params, resolve);
  });
};
```

And finally, we declare the function `getResult` that gets data from the page and return it:

```javascript
const getResults = async () => {
  ...
};
```

In this function first, we declare an array `organicResults` with results data:

```javascript
const organicResults = [];
```

Next, we need to use `while` loop. In this loop we get `json` with results, check if results are present on the page (`organic_results_state` isn't `"Fully empty"`), push results to `organicResults` array, define the start number on the results page, and repeat the loop until results aren't present on the page:

```javascript
while (true) {
  const json = await getJson();
  if (json.search_information?.organic_results_state === "Fully empty") break;
  organicResults.push(...json.image_results);
  params.start ? (params.start += 10) : (params.start = 10);
}
return organicResults;
```

After, we run the `getResults` function and print all the received information in the console with the [`console.dir`](https://nodejs.org/api/console.html#consoledirobj-options) method, which allows you to use an object with the necessary parameters to change default output options:

```javascript
getResults().then((result) => console.dir(result, { depth: null }));
```

<h2 id='serp_api_output'>Output</h2>

```json
[
   {
      "position":1,
      "title":"Best Bugatti Cars in India - CARS24",
      "link":"https://www.cars24.com/blog/best-bugatti-cars-in-india/",
      "displayed_link":"https://www.cars24.com › blog › best-bugatti-cars-in-in...",
      "thumbnail":"https://serpapi.com/searches/6319fad65c560673de2b144a/images/7c3a215cbf2776de47a9c447d0b97c5290a72394aec05f099004cb62a9250eee.jpeg",
      "image_resolution":"1920 × 1080",
      "snippet":"The Bugatti Veyron was originally launched in 2005 and was then, the fastest car ... Although the Divo is a super luxurious vehicle with a hefty price tag, ...",
      "snippet_highlighted_words":[
         "Bugatti Veyron",
         "super"
      ],
      "cached_page_link":"https://webcache.googleusercontent.com/search?q=cache:0BQ-hPQGl9IJ:https://www.cars24.com/blog/best-bugatti-cars-in-india/&cd=91&hl=en&ct=clnk&gl=us"
   },
   {
      "position":2,
      "title":"1056304 car, vehicle, road, Super Car, sports car, motion blur ...",
      "link":"https://rare-gallery.com/1056304-car-vehicle-road-super-car-sports-car-motion-blur-bugatti-bugatti-chiron-bugatti-veyron-performance-.html",
      "displayed_link":"https://rare-gallery.com › Another wallpapers",
      "thumbnail":"https://serpapi.com/searches/6319fad65c560673de2b144a/images/7c3a215cbf2776de9f26993a213d2a5cf9506bec4885508e13e62c8036abbf11.jpeg",
      "image_resolution":"1920 × 1080",
      "snippet":"Wallpaper name: car, vehicle, road, Super Car, sports car, motion blur, Bugatti, Bugatti Chiron, Bugatti Veyron, performance car, wheel, supercar, ...",
      "snippet_highlighted_words":[
         "Super",
         "Bugatti",
         "Bugatti",
         "Bugatti Veyron"
      ],
      "cached_page_link":"https://webcache.googleusercontent.com/search?q=cache:lcVswIAM3eMJ:https://rare-gallery.com/1056304-car-vehicle-road-super-car-sports-car-motion-blur-bugatti-bugatti-chiron-bugatti-veyron-performance-.html&cd=92&hl=en&ct=clnk&gl=us"
   },
   ... and other results
]
```

<h2 id='links'>Links</h2>

- [Code in the online IDE](https://replit.com/@MikhailZub/Scrape-Google-Reverse-Image-with-NodeJS-SerpApi#index.js)
- [Google Reverse Image API](https://serpapi.com/google-reverse-image)

If you want to see some projects made with SerpApi, [write me a message](mailto:miha01012019@gmail.com).

---

<p style="text-align: center;">Join us on <a href="https://twitter.com/serp_api">Twitter</a> | <a href="https://www.youtube.com/channel/UCUgIHlYBOD3yA3yDIRhg_mg">YouTube</a></p>
<p style="text-align: center;">Add a  <a href="https://github.com/serpapi/public-roadmap/issues">Feature Request</a>💫 or a <a href="https://github.com/serpapi/public-roadmap/issues">Bug</a>🐞</p>