Web scraping YouTube autocomplete with Nodejs
Tags: webscraping, node, serpapi, youtube
Published: 2022-08-31 07:44:48

<h2 id='what'>What will be scraped</h2>

![what](https://user-images.githubusercontent.com/64033139/184880245-be261f7f-747f-4cc6-8b49-3fcd82ab7710.png)

📌Note: For now, we don't have an API that supports extracting autocomplete data.

This blog post is to show you way how you can do it yourself while we're working on releasing our proper API in a meantime. We'll update you on our Twitter once this API will be released.

<h2 id='full_code'>Full code</h2>

If you don't need an explanation, have a look at [the full code example in the online IDE](https://replit.com/@MikhailZub/Scrape-YouTube-autocomplete-with-NodeJS#index.js) 

```javascript
const puppeteer = require("puppeteer-extra");
const StealthPlugin = require("puppeteer-extra-plugin-stealth");

puppeteer.use(StealthPlugin());

const queries = ["javascript", "node", "web scraping"];
const URL = "https://www.youtube.com";

async function getYoutubeAutocomplete() {
  const browser = await puppeteer.launch({
    headless: false,
    args: ["--no-sandbox", "--disable-setuid-sandbox"],
  });

  const page = await browser.newPage();

  await page.setDefaultNavigationTimeout(60000);
  await page.goto(URL);
  await page.waitForSelector("#contents");

  const autocompleteResults = [];
  for (query of queries) {
    await page.click("#search-input");
    await page.keyboard.type(query);
    await page.waitForTimeout(5000);
    const results = {
      query,
      autocompleteResults: await page.evaluate(() => {
        return Array.from(document.querySelectorAll(".sbdd_a li"))
          .map((el) => el.querySelector(".sbqs_c")?.textContent.trim())
          .filter((el) => el);
      }),
    };
    autocompleteResults.push(results);
    await page.click("#search-clear-button");
    await page.waitForTimeout(2000);
  }

  await browser.close();

  return autocompleteResults;
}

getYoutubeAutocomplete().then(console.log);
```

<h2 id='preparation'>Preparation</h2>

First, we need to create a Node.js* project and add [`npm`](https://www.npmjs.com/) packages [`puppeteer`](https://www.npmjs.com/package/puppeteer), [`puppeteer-extra`](https://www.npmjs.com/package/puppeteer-extra) and [`puppeteer-extra-plugin-stealth`](https://www.npmjs.com/package/puppeteer-extra-plugin-stealth) to control Chromium (or Chrome, or Firefox, but now we work only with Chromium which is used by default) over the [DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/) in [headless](https://developers.google.com/web/updates/2017/04/headless-chrome) or non-headless mode. 

To do this, in the directory with our project, open the command line and enter `npm init -y`, and then `npm i puppeteer puppeteer-extra puppeteer-extra-plugin-stealth`.

*<span style="font-size: 15px;">If you don't have Node.js installed, you can [download it from nodejs.org](https://nodejs.org/en/) and follow the installation [documentation](https://nodejs.dev/learn/introduction-to-nodejs).</span>

📌Note: also, you can use `puppeteer` without any extensions, but I strongly recommended use it with `puppeteer-extra` with `puppeteer-extra-plugin-stealth` to prevent website detection that you are using headless Chromium or that you are using [web driver](https://www.w3.org/TR/webdriver/). You can check it on [Chrome headless tests website](https://intoli.com/blog/not-possible-to-block-chrome-headless/chrome-headless-test.html). The screenshot below shows you a difference.

![stealth](https://user-images.githubusercontent.com/64033139/173014238-eb8450d7-616c-42ae-8b2f-24eeb5fd5916.png)

<h2 id='process'>Process</h2>

[SelectorGadget Chrome extension](https://chrome.google.com/webstore/detail/selectorgadget/mhjhnkcfbdhnjickkkdbjoemdmbfginb) was used to grab CSS selectors by clicking on the desired element in the browser. If you have any struggles understanding this, we have a dedicated [Web Scraping with CSS Selectors blog post](https://serpapi.com/blog/web-scraping-with-css-selectors-using-python/#css_gadget) at SerpApi.

The Gif below illustrates the approach of selecting different parts of the results.

![how](https://user-images.githubusercontent.com/64033139/184881249-87c37e24-5fc8-4cd0-84d9-f5543c274989.gif)

<h3 id='code_explanation'>Code explanation</h3>

Declare [`puppeteer`](https://www.npmjs.com/package/puppeteer-extra) to control Chromium browser from `puppeteer-extra` library and [`StealthPlugin`](https://www.npmjs.com/package/puppeteer-extra-plugin-stealth) to prevent website detection that you are using [web driver](https://www.w3.org/TR/webdriver/) from `puppeteer-extra-plugin-stealth` library:

```javascript
const puppeteer = require("puppeteer-extra");
const StealthPlugin = require("puppeteer-extra-plugin-stealth");
```

Next, we "say" to `puppeteer` use `StealthPlugin`, write search queries and YouTube URL:

```javascript
puppeteer.use(StealthPlugin());

const queries = ["javascript", "node", "web scraping"];
const URL = "https://www.youtube.com";
```
       
Next, write a function to control the browser, and get information:

```javascript
async function getYoutubeAutocomplete() {
  ...
}
```

In this function first we need to define `browser` using `puppeteer.launch({options})` method with current `options`, such as `headless: false` and `args: ["--no-sandbox", "--disable-setuid-sandbox"]`.

These options mean that we use [headless](https://developers.google.com/web/updates/2017/04/headless-chrome) mode and array with [arguments](https://peter.sh/experiments/chromium-command-line-switches/) which we use to allow the launch of the browser process in the online IDE. And then we open a new `page`:

```javascript
  const browser = await puppeteer.launch({
    headless: false,
    args: ["--no-sandbox", "--disable-setuid-sandbox"],
  });

  const page = await browser.newPage();
```

Next, we change default ([30 sec](https://github.com/puppeteer/puppeteer/blob/2a0eefb99f0ae00dacc9e768a253308c0d18a4c3/src/common/TimeoutSettings.ts#L17)) time for waiting for selectors to 60000 ms (1 min) for slow internet connection with [`.setDefaultNavigationTimeout()`](https://pptr.dev/api/puppeteer.page.setdefaultnavigationtimeout) method, go to `URL` with [`.goto()`](https://pptr.dev/api/puppeteer.page.goto) method and use [`.waitForSelector()`](https://pptr.dev/api/puppeteer.page.waitforselector) method to wait until `#contents` selector is creating on the page.:

```javascript
  await page.setDefaultNavigationTimeout(60000);
  await page.goto(URL);
  await page.waitForSelector("#contents");
```

Then, we define an array with the results, called `autocompleteResults` and starts [`for...of`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for...of) loop to iterate over all queries:

```javascript
  const autocompleteResults = [];
  for (query of queries) {
    ...
  }
```

Next, in the loop we cick on `#search-input` ([`.click()`](https://pptr.dev/api/puppeteer.page.click) method), type current `query` with [`page.keyboard.type(query)`](https://pptr.dev/api/puppeteer.keyboard.type) method and wait 5 seconds, using [`.waitForTimeout(5000)`](https://pptr.dev/api/puppeteer.page.waitfortimeout) method:

```javascript
    await page.click("#search-input");
    await page.keyboard.type(query);
    await page.waitForTimeout(5000);
```

Then, we make the results object that have `query` and `autocompleteResults` keys. We get `autocompleteResults` using [`page.evaluate()`](https://pptr.dev/api/puppeteer.page.evaluate) method to run code in the brackets in the browser context.

There we need to use [`.querySelectorAll()`](https://developer.mozilla.org/en-US/docs/Web/API/Document/querySelectorAll) method which returns a static [NodeList](https://developer.mozilla.org/en-US/docs/Web/API/NodeList) representing a list of the document's elements that match the css selectors in the brackets and convert result to an array with [`Array.from()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/from) method to iterate over that array.

After that we find element with class name `.sbqs_c` ([`.querySelector()`](https://developer.mozilla.org/en-US/docs/Web/API/Document/querySelector) method), get raw text ([`textContent`](https://developer.mozilla.org/en-US/docs/Web/API/Node/textContent) property) and remove whitespace from both ends of a string with [`.trim()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/trim) method from each of `.sbdd_a li` elements. Because sometimes we find empty nodes in the end we need to filter our array and leave true elements (`.filter((el) => el)`):

```javascript
    const results = {
      query,
      autocompleteResults: await page.evaluate(() => {
        return Array.from(document.querySelectorAll(".sbdd_a li"))
          .map((el) => el.querySelector(".sbqs_c")?.textContent.trim())
          .filter((el) => el);
      }),
    };
```

Next, we push `results` object from current itaration step to the `autocompleteResults` array, click `#search-clear-button` to clear search input and wait 2 seconds before next itaration:

```javascript
    autocompleteResults.push(results);
    await page.click("#search-clear-button");
    await page.waitForTimeout(2000);
```

And finally, we close the browser and return received data:

```javascript
  await browser.close();

  return autocompleteResults;
```

Now we can launch our parser:

```bash
$ node YOUR_FILE_NAME # YOUR_FILE_NAME is the name of your .js file
```

<h2 id='output'>Output</h2>

```json
[
   {
      "query":"javascript",
      "autocompleteResults":[
         "javascript",
         "javascript tutorial for beginners",
         "javascript full course",
         "javascript tutorial",
         "javascript dom",
         "javascript mastery",
         "javascript course",
         "javascript interview questions and answers",
         "javascript for beginners",
         "javascript с нуля",
         "javascript project",
         "javascript ninja",
         "javascript game",
         "javascript interview"
      ]
   },
   {
      "query":"node",
      "autocompleteResults":[
         "node js",
         "node js tutorial",
         "node",
         "node js project",
         "node js express",
         "node js interview",
         "node video tutorial",
         "node video",
         "node js interview questions",
         "node js event loop",
         "node js уроки",
         "nodemailer",
         "node red",
         "nodemcu"
      ]
   },
   {
      "query":"web scraping",
      "autocompleteResults":[
         "web scraping weather data python",
         "web scraping",
         "web scraping python",
         "web scraping javascript",
         "web scraping amazon product",
         "web scraping amazon price",
         "web scraping amazon",
         "web scraping amazon reviews",
         "web scraping amazon reviews python",
         "web scraping indeed",
         "web scraping flight prices",
         "web scraping using python",
         "web scraping tutorial"
      ]
   }
]
```

<h2 id='extract_from_url'>Extract suggestions from Google Autocomplete Client</h2>

Previous example was a "hard" way. Also you can parse data using following URL which will output a txt file: 

```javascript
"https://clients1.google.com/complete/search?client=youtube&hl=en&q=minecraft"
```

If you want to see some projects made with SerpApi, [please write me a message](mailto:miha01012019@gmail.com).

___
<p style="text-align: center;">Join us on <a href="https://twitter.com/serp_api">Twitter</a> | <a href="https://www.youtube.com/channel/UCUgIHlYBOD3yA3yDIRhg_mg">YouTube</a></p>
<p style="text-align: center;">Add a  <a href="https://github.com/serpapi/public-roadmap/issues">Feature Request</a>💫 or a <a href="https://github.com/serpapi/public-roadmap/issues">Bug</a>🐞</p>