Web scraping Google Trends Realtime search with Nodejs
Tags: webscraping, node, google
Published: 2022-09-19 11:21:50

<h2 id='intro'>Intro</h2>

Currently, we don't have an API that supports extracting data from Google Trends Realtime Search page.

This blog post is to show you way how you can do it yourself with provided DIY solution below while we're working on releasing our proper API.

The solution can be used for personal use as it doesn't include the [Legal US Shield](https://serpapi.com/#features) that we offer for our paid [production and above plans](https://serpapi.com/pricing) and has its limitations such as the need to bypass blocks, for example, CAPTCHA.

You can check our public roadmap to track the progress for this API:

   🗺️  [[New API] Google Trends Realtime Search Trends](https://github.com/serpapi/public-roadmap/issues/320)

<h2 id='what'>What will be scraped</h2>

![what](https://user-images.githubusercontent.com/64033139/187644462-ffb2bb20-3f86-4eb5-89ba-65d39ce284be.png)

<h2 id='full_code'>Full code</h2>

If you don't need an explanation, have a look at [the full code example in the online IDE](https://replit.com/@MikhailZub/Scrape-Google-Trends-Realtime-with-NodeJS#index.js)

```javascript
const puppeteer = require("puppeteer-extra");
const StealthPlugin = require("puppeteer-extra-plugin-stealth");

puppeteer.use(StealthPlugin());

const baseURL = `https://trends.google.com`;
const countryCode = "US";
const category = "all";
/* allows next categories: 
b - business,
e - entertainment,
m - health,
t - sci/tech,
s - sports,
h - top stories
*/
async function fillTrendsDataFromPage(page) {
  while (true) {
    const isNextPage = await page.$(".feed-load-more-button");
    if (!isNextPage) break;
    await page.click(".feed-load-more-button");
    await page.waitForTimeout(2000);
  }
  const dataFromPage = await page.evaluate((baseURL) => {
    return Array.from(document.querySelectorAll(".feed-item")).map((el) => ({
      index: el.querySelector(".index")?.textContent.trim(),
      title: Array.from(el.querySelectorAll(".title a"))
        .map((el) => el.getAttribute("title"))
        .join(" • "),
      titleLinks: Array.from(el.querySelectorAll(".title a")).map((el) => ({
        [el.getAttribute("title")]: `${baseURL}${el.getAttribute("href")}`,
      })),
      subtitle: el.querySelector(".summary-text a")?.textContent.trim(),
      subtitleLink: el.querySelector(".summary-text a")?.getAttribute("href"),
      source: el.querySelector(".source-and-time span:first-child")?.textContent.trim(),
      published: el.querySelector(".source-and-time span:last-child")?.textContent.trim(),
      thumbnail: `https:${el.querySelector(".feed-item-image-wrapper img")?.getAttribute("src")}`,
    }));
  }, baseURL);
  return dataFromPage;
}

async function getGoogleTrendsRealtimeResults() {
  const browser = await puppeteer.launch({
    headless: false,
    args: ["--no-sandbox", "--disable-setuid-sandbox"],
  });

  const page = await browser.newPage();
  page.setViewport({ width: 1200, height: 700 });

  const URL = `${baseURL}/trends/trendingsearches/realtime?geo=${countryCode}&category=${category}&hl=en`;

  await page.setDefaultNavigationTimeout(60000);
  await page.goto(URL);

  await page.waitForSelector(".feed-item");

  const realtimeResults = await fillTrendsDataFromPage(page);

  await browser.close();

  return realtimeResults;
}

getGoogleTrendsRealtimeResults().then((result) => console.dir(result, { depth: null }));
```

<h2 id='preparation'>Preparation</h2>

First, we need to create a Node.js\* project and add [`npm`](https://www.npmjs.com/) packages [`puppeteer`](https://www.npmjs.com/package/puppeteer), [`puppeteer-extra`](https://www.npmjs.com/package/puppeteer-extra) and [`puppeteer-extra-plugin-stealth`](https://www.npmjs.com/package/puppeteer-extra-plugin-stealth) to control Chromium (or Chrome, or Firefox, but now we work only with Chromium which is used by default) over the [DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/) in [headless](https://developers.google.com/web/updates/2017/04/headless-chrome) or non-headless mode.

To do this, in the directory with our project, open the command line and enter `npm init -y`, and then `npm i puppeteer puppeteer-extra puppeteer-extra-plugin-stealth`.

\*<span style="font-size: 15px;">If you don't have Node.js installed, you can [download it from nodejs.org](https://nodejs.org/en/) and follow the installation [documentation](https://nodejs.dev/learn/introduction-to-nodejs).</span>

📌Note: also, you can use `puppeteer` without any extensions, but I strongly recommended use it with `puppeteer-extra` with `puppeteer-extra-plugin-stealth` to prevent website detection that you are using headless Chromium or that you are using [web driver](https://www.w3.org/TR/webdriver/). You can check it on [Chrome headless tests website](https://intoli.com/blog/not-possible-to-block-chrome-headless/chrome-headless-test.html). The screenshot below shows you a difference.

![stealth](https://user-images.githubusercontent.com/64033139/173014238-eb8450d7-616c-42ae-8b2f-24eeb5fd5916.png)

<h2 id='process'>Process</h2>

[SelectorGadget Chrome extension](https://chrome.google.com/webstore/detail/selectorgadget/mhjhnkcfbdhnjickkkdbjoemdmbfginb) was used to grab CSS selectors by clicking on the desired element in the browser. If you have any struggles understanding this, we have a dedicated [Web Scraping with CSS Selectors blog post](https://serpapi.com/blog/web-scraping-with-css-selectors-using-python/#css_gadget) at SerpApi.

The Gif below illustrates the approach of selecting different parts of the results.

![how](https://user-images.githubusercontent.com/64033139/187644823-29de7ad0-a817-4e2a-ae76-17ba1b6d789b.gif)

<h3 id='code_explanation'>Code explanation</h3>

Declare [`puppeteer`](https://www.npmjs.com/package/puppeteer-extra) to control Chromium browser from `puppeteer-extra` library and [`StealthPlugin`](https://www.npmjs.com/package/puppeteer-extra-plugin-stealth) to prevent website detection that you are using [web driver](https://www.w3.org/TR/webdriver/) from `puppeteer-extra-plugin-stealth` library:

```javascript
const puppeteer = require("puppeteer-extra");
const StealthPlugin = require("puppeteer-extra-plugin-stealth");
```

Next, we "say" to `puppeteer` use `StealthPlugin`, write Google Trends URL, country code (check the [full list of supported Google Trends Locations](https://serpapi.com/google-trends-locations)) and category:

```javascript
puppeteer.use(StealthPlugin());

const baseURL = `https://trends.google.com`;
const countryCode = "US";
const category = "all";
```

All awailable categories:

- `b` - business,
- `e` - entertainment,
- `m` - health,
- `t` - sci/tech,
- `s` - sports,
- `h` - top stories.

Next, write a function to load all data and get information from the page:

```javascript
async function fillTrendsDataFromPage() {
  ...
}
```

In this function, first, we need to load more data until it is available. To do this we use `while` loop in which we check if "Load More" button is present on the page ([`page.$()`](https://pptr.dev/api/puppeteer.page._) method), [`click`](https://pptr.dev/api/puppeteer.page.click/) on this button, wait 2 seconds (using [`waitForTimeout`](https://pptr.dev/api/puppeteer.page.waitfortimeout) method) and repeat again until the button is absent from the page:

```javascript
while (true) {
  const isNextPage = await page.$(".feed-load-more-button");
  if (!isNextPage) break;
  await page.click(".feed-load-more-button");
  await page.waitForTimeout(2000);
}
```

Next, we get information from the page context (using [`evaluate()`](https://pptr.dev/api/puppeteer.page.evaluate) method) and save it in the returned array. First, we need to get all the trends results available on the page ([`querySelectorAll()`](https://developer.mozilla.org/en-US/docs/Web/API/Document/querySelectorAll) method) and make the new array from got [NodeList](https://developer.mozilla.org/en-US/docs/Web/API/NodeList) ([`Array.from()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/from)):

```javascript
return Array.from(document.querySelectorAll(".feed-item")).map((el) => ({
```

Next, we assign the necessary data to each object's key. We can do this with [`textContent`](https://developer.mozilla.org/en-US/docs/Web/API/Node/textContent) and [`trim()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/trim) methods, which get the raw text and removes white space from both sides of the string. If we need to get links, we use [`getAttribute()`](https://developer.mozilla.org/en-US/docs/Web/API/Element/getAttribute) method to get `"href"` and `"src"` HTML element attributes. To make `title` string looks like on the page, we need to get an array with title links and using [`join()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/join) method unite array elements into a string with the `•` separator:

```javascript
    index: el.querySelector(".index")?.textContent.trim(),
    title: Array.from(el.querySelectorAll(".title a"))
        .map((el) => el.getAttribute("title"))
        .join(" • "),
    titleLinks: Array.from(el.querySelectorAll(".title a")).map((el) => ({
        [el.getAttribute("title")]: `${baseURL}${el.getAttribute("href")}`,
        })),
    subtitle: el.querySelector(".summary-text a")?.textContent.trim(),
    subtitleLink: el.querySelector(".summary-text a")?.getAttribute("href"),
    source: el.querySelector(".source-and-time span:first-child")?.textContent.trim(),
    published: el.querySelector(".source-and-time span:last-child")?.textContent.trim(),
    thumbnail: `https:${el.querySelector(".feed-item-image-wrapper img")?.getAttribute("src")}`,
```

Next, write a function to control the browser, and get information:

```javascript
async function getGoogleTrendsDailyResults() {
  ...
}
```

In this function first we need to define `browser` using `puppeteer.launch({options})` method with current `options`, such as `headless: false` and `args: ["--no-sandbox", "--disable-setuid-sandbox"]`.

These options mean that we use [headless](https://developers.google.com/web/updates/2017/04/headless-chrome) mode and array with [arguments](https://peter.sh/experiments/chromium-command-line-switches/) which we use to allow the launch of the browser process in the online IDE. And then we open a new `page`:

```javascript
const browser = await puppeteer.launch({
  headless: false,
  args: ["--no-sandbox", "--disable-setuid-sandbox"],
});

const page = await browser.newPage();
```

Next, we define the full request URL, change default ([30 sec](https://github.com/puppeteer/puppeteer/blob/2a0eefb99f0ae00dacc9e768a253308c0d18a4c3/src/common/TimeoutSettings.ts#L17)) time for waiting for selectors to 60000 ms (1 min) for slow internet connection with [`.setDefaultNavigationTimeout()`](https://pptr.dev/api/puppeteer.page.setdefaultnavigationtimeout) method, go to `URL` with [`.goto()`](https://pptr.dev/api/puppeteer.page.goto) method and use [`.waitForSelector()`](https://pptr.dev/api/puppeteer.page.waitforselector) method to wait until the selector is load:

```javascript
const URL = `${baseURL}/trends/trendingsearches/realtime?geo=${countryCode}&category=${category}&hl=en`;

await page.setDefaultNavigationTimeout(60000);
await page.goto(URL);

await page.waitForSelector(".feed-item");
```

And finally, we save trends data from the page in the `realtimeResults` constant, close the browser and return the received data:

```javascript
const realtimeResults = await fillTrendsDataFromPage(page);

await browser.close();

return realtimeResults;
```

Now we can launch our parser:

```bash
$ node YOUR_FILE_NAME # YOUR_FILE_NAME is the name of your .js file
```

<h2 id='output'>Output</h2>

```json
[
   {
      "index":"1",
      "title":"Explore Financial Conduct Authority • Explore Finance • Explore Robo-advisor • Explore Financial services • Explore Debt management plan • Explore Consumer • Explore Investment • Explore Debtor • Explore Financial adviser",
      "titleLinks":[
         {
            "Explore Financial Conduct Authority":"https://trends.google.com/trends/explore?q=/m/0cc7rp_&date=now+7-d&geo=US"
         },
         {
            "Explore Finance":"https://trends.google.com/trends/explore?q=/m/02_7t&date=now+7-d&geo=US"
         },
         {
            "Explore Robo-advisor":"https://trends.google.com/trends/explore?q=/m/010vqqqk&date=now+7-d&geo=US"
         },
         {
            "Explore Financial services":"https://trends.google.com/trends/explore?q=/m/02h400t&date=now+7-d&geo=US"
         },
         {
            "Explore Debt management plan":"https://trends.google.com/trends/explore?q=/m/0crs3y&date=now+7-d&geo=US"
         },
         {
            "Explore Consumer":"https://trends.google.com/trends/explore?q=/m/025_b&date=now+7-d&geo=US"
         },
         {
            "Explore Investment":"https://trends.google.com/trends/explore?q=/m/0g_fl&date=now+7-d&geo=US"
         },
         {
            "Explore Debtor":"https://trends.google.com/trends/explore?q=/m/03rd6r&date=now+7-d&geo=US"
         },
         {
            "Explore Financial adviser":"https://trends.google.com/trends/explore?q=/m/08p4gp&date=now+7-d&geo=US"
         }
      ],
      "subtitle":"Robo advice shines for borrowers: study",
      "subtitleLink":"https://www.investmentexecutive.com/news/research-and-markets/robo-advice-shines-for-borrowers/",
      "source":"Investment Executive",
      "published":"13 hours ago",
      "thumbnail":"https://t0.gstatic.com/images?q=tbn:ANd9GcSmxvumRvWQdIKhvir_gth7zv6N3zSIsoG1WsbnCB84b2rWqidrbyIVY04xbem0jYwTQ5Yd4osF1ns"
   },
   ... and other results
]
```

If you want to see some projects made with SerpApi, [please write me a message](mailto:miha01012019@gmail.com).

---

<p style="text-align: center;">Join us on <a href="https://twitter.com/serp_api">Twitter</a> | <a href="https://www.youtube.com/channel/UCUgIHlYBOD3yA3yDIRhg_mg">YouTube</a></p>
<p style="text-align: center;">Add a  <a href="https://github.com/serpapi/public-roadmap/issues">Feature Request</a>💫 or a <a href="https://github.com/serpapi/public-roadmap/issues">Bug</a>🐞</p>