Browser automation with Puppeteer
Tags: puppeteer, node
Published: 2023-08-26 17:33:53

---
title: Browser automation with Puppeteer
published: true
tags: ['puppeteer', 'node']
canonical_url: https://sevic.dev/notes/browser-automation-puppeteer/
---

Puppeteer is a headless browser for automating browser tasks. Here's the list of some of the features:

- Turn off headless mode

  ```js
  const browser = await puppeteer.launch({
    headless: false
    // ...
  });
  ```

- Resize the viewport to the window size

  ```js
  const browser = await puppeteer.launch({
    // ...
    defaultViewport: null
  });
  ```

- Emulate screen how it's shown to the user via the `emulateMediaType` method

  ```js
  await page.emulateMediaType('screen');
  ```

- Save the page as a PDF file with a specified path, format, scale factor, and page range

  ```js
  await page.pdf({
    path: 'path.pdf',
    format: 'A3',
    scale: 1,
    pageRanges: '1-2',
    printBackground: true
  });
  ```

- Use preexisting user's credentials to skip logging in to some websites. The user data directory is a parent of the `Profile Path` value from the `chrome://version` page.

  ```js
  const browser = await puppeteer.launch({
    userDataDir: 'C:\\Users\\<USERNAME>\\AppData\\Local\\Google\\Chrome\\User Data',
    args: [],
  });
  ```

- Use Chrome instance instead of Chromium by utilizing the `Executable Path` from the `chrome://version` URL. Close Chrome browser before running the script

  ```js
  const browser = await puppeteer.launch({
    executablePath: puppeteer.executablePath('chrome'),
    // ...
  });
  ```

- Get value based on evaluation in the browser page

  ```js
  const shouldPaginate = await page.evaluate(() => {
    // ...
  });
  ```

- Get HTML content from the specific element

  ```js
  const html = await page.evaluate(
    () => document.querySelector('.field--text').outerHTML,
  );
  ```

- Wait for a specific selector to be loaded. You can also provide a timeout in milliseconds

  ```js
  await page.waitForSelector('.success', { timeout: 5000 });
  ```

- Manipulate with a specific element and click on some of the elements

  ```js
  await page.$eval('#header', async (headerElement) => {
    // ...
    headerElement
      .querySelectorAll('svg')
      .item(13)
      .parentNode.click();
  });
  ```

- Extend execution of the `$eval` method

  ```js
  const browser = await puppeteer.launch({
    // ...
    protocolTimeout: 0,
  });
  ```

- Manipulate with multiple elements

  ```js
  await page.$$eval('.some-class', async (elements) => {
    // ...
  });
  ```

- Wait for navigation (e.g., form submitting) to be done

  ```js
  await page.waitForNavigation({ waitUntil: 'networkidle0', timeout: 0 });
  ```

- Trigger hover event on some of the elements

  ```js
  await page.$eval('#header', async (headerElement) => {
    const hoverEvent = new MouseEvent('mouseover', {
      view: window,
      bubbles: true,
      cancelable: true
    });

    headerElement.dispatchEvent(hoverEvent);
  });
  ```

- Expose a function in the browser and use it in `$eval` and `$$eval` callbacks (e.g., simulate typing using the `window.type` function)

  ```js
  await page.exposeFunction('type', async (selector, text, options) => {
    await page.type(selector, text, options);
  });

  await page.$$eval('.some-class', async (elements) => {
    // ...
    window.type(selector, text, { delay: 0 });
  });
  ```

- Press the `Enter` button after typing the input field value

  ```js
  await page.type(selector, `${text}${String.fromCharCode(13)}`, options);
  ```

- Remove the value from the input field before typing the new one

  ```js
  await page.click(selector, { clickCount: 3 });
  await page.type(selector, text, options);
  ```

- Expose a variable in the browser by passing it as the third argument for `$eval` and `$$eval` methods and use it in `$eval` and `$$eval` callbacks

  ```js
  await page.$eval(
    '#element',
    async (element, customVariable) => {
      // ...
    },
    customVariable
  );
  ```

- Mock response for the specific request

  ```js
  await page.setRequestInterception(true);
  page.on('request', async function (request) {
    const url = request.url();
    if (url !== REDIRECTION_URL) {
      return request.continue();
    }

    await request.respond({
      contentType: 'text/html',
      status: 304,
      body: '<body></body>',
    });
  });
  ```

- Intercept page redirections (via interceptor) and open them in new tabs rather than following them in the same tab

  ```js
  await page.setRequestInterception(true);
  page.on('request', async function (request) {
    const url = request.url();
    if (url !== REDIRECTION_URL) {
      return request.continue();
    }

    await request.respond({
      contentType: 'text/html',
      status: 304,
      body: '<body></body>',
    });
    const newPage = await browser.newPage();
    await newPage.goto(url, { waitUntil: 'domcontentloaded', timeout: 0 });
    // ...
    await newPage.close();
  });
  ```

- Intercept page response

  ```js
  page.on('response', async (response) => {
    if (response.url() === RESPONSE_URL) {
      if (response.status() === 200) {
        // ...
      }
      // ...
    }
  });
  ```

### Boilerplate

Here is the [link](https://sevic.dev/nestjs-starter) to the boilerplate I use for the development.