Web scraping using a headless browser in NodeJS
Tags: programming, node, webscraping, codenewbie
Published: 2023-07-06 01:19:46

Web scraping collects and extracts unstructured data from a website to a more readable structured format like JSON, CSV format, and more. Organizations set [guiding principles](https://hackernoon.com/how-to-build-a-python-web-scraper-scrape-data-from-any-website) on scraped endpoints that are permitted.
 

When scraping a website for personal use, it can be stressful to manually change the code every time, as most big brand websites want people to refrain from scraping their public data. The following restrictions or problems might arise, such as CAPTCHAs, user agent (allowed and disallowed endpoints) blocking, IP blocking, and proxy network setup are set.

A practical use case of web scraping is notifying users of price changes for an item on sites like Amazon, eBay, etc. 

In this article, you will learn how to use Bright Data’s [Scraping Browser](https://get.brightdata.com/scrapingbrowser) to unlock websites at scale without being blocked because of its built-in unlocking capabilities.


## Sandbox

Test and run the complete code in this [Codesandbox](https://codesandbox.io/p/sandbox/keen-montalcini-yw6fr5?file=%2F.gitignore%3A1%2C5). 


## Prerequisites

It would help if you had the following to complete this tutorial:

- Basic knowledge of JavaScript. 
- Have [Node](https://nodejs.org/) installed on your local machine. It is required to install dependencies
- A code editor - VS Code


## What is Bright Data?

Bright Data is a data collection or aggregation service with a massive network of internet protocols (IPs) and proxies to scrape information off a website, thereby having the resource to avoid detection by company bots that prevent data scraping. 

In essence, Bright Data does the heavy lifting in the background because of its large datasets available on the platform, which removes the worry of being blocked or gaining access to website data.

{% embed https://www.youtube.com/watch?v=AGaiVApKfmc& %}


## What is a headless browser?

A headless browser is a browser that operates without a graphical user interface (GUI). Modern web browsers like Google, Safari, Brave, Mozilla, and so on; all have a graphical interface for interactivity and displaying visual content. For headless browsers, it functions in the background with scripts or in the command line interface (CLI) written by developers.

Using a headless browser for web scraping is essential because it allows you to extract data from any public website by simulating user behavior.

Headless browsers are suitable for the following:

- Automated testing 
- Web scraping 


## Benefits of Puppeteer

Puppeteer is an example of a headless browser. The following are some of the benefits of using Puppeteer in web scraping:

- Crawl single-page application (SPA)
- Allows for automated testing of website code
- Clicking on pages elements
- Downloading data 
- Generate screenshots and PDFs of pages


## Installation

Create a new folder for this app, and run the command below to install a node server.

```bash
    npm init -y
```

The command will initialize this project and create a package.json file containing all the dependencies and project information. The `-y` flag accepts all the defaults upon initialization of the app.

With the initialization complete, let’s install the `nodemon` dependency with this command:

```bash
    npm install -D nodemon
```

**Nodemon** is a tool that will automatically restart the node application when the file changes.

In the `package.json`, update the scripts object with this code:

`package.json`
```js
    {
      ...
      "scripts": {
        "start": "node index.js",
        "start:dev": "nodemon index.js"
      },
      ...
    }
```

Next, create a file, `index.js`, in the directory's root, which will be the entry point for writing the script.

The other package to install is the `puppeteer-core`, the automation library without the browser used when connecting to a remote browser.

```bash
    npm install puppeteer-core
```

## Building with Bright Data’s Scraping Browser

Create an account on [Bright Data](https://brightdata.com/) to access all its services. But for this project, the focus would be on the Scraping Browser functionality.

On your admin dashboard, click on the **Proxies and Scraping Infra.** 


![proxies and scraping infra](https://paper-attachments.dropboxusercontent.com/s_4CBE51D7D2EAE0886B936E84C1BF1CF4AEE57FF9FCF3E0499D73B25BAC6F820E_1686225099949_Screenshot+2023-06-08+at+10.49.57.jpg)


Scroll to the bottom of the page and select the **Scraping Browser**. After that, click the **Get started** button from the proxy products listed.

![Scraping browser](https://paper-attachments.dropboxusercontent.com/s_4CBE51D7D2EAE0886B936E84C1BF1CF4AEE57FF9FCF3E0499D73B25BAC6F820E_1686228328438_Screenshot+2023-06-08+at+13.31.34.jpg)


On opening the tool, give the proxy a name and click the button, **Add Proxy,** and when prompted about creating a new zone, select **Yes**.


![naming the proxy](https://paper-attachments.dropboxusercontent.com/s_4CBE51D7D2EAE0886B936E84C1BF1CF4AEE57FF9FCF3E0499D73B25BAC6F820E_1686228901763_Screenshot+2023-06-08+at+13.28.50.jpg)


The next screen should be something like this, with the host, username, and password displayed.


![host, username, and password](https://paper-attachments.dropboxusercontent.com/s_4CBE51D7D2EAE0886B936E84C1BF1CF4AEE57FF9FCF3E0499D73B25BAC6F820E_1686229110311_Screenshot+2023-06-08+at+13.57.15.jpg)


Now, click on the button **</> Check out code and integration examples** and on the next screen, select Node.js as the language of choice for this app.


## Creating environment variables

Environment variables are secret keys and credentials that should not be shared, hosted, or pushed to GitHub to prevent unauthorized access.

Before creating the `.env` file in the root of the directory, let’s install this command:

```bash
    npm install dotenv
```

Copy-paste this code to the `.env` file, and replace the entire value in the quotation from your **Access parameters** tab:

`.env`
```bash
    UNAME="<user-name>"
    HOST="<host>"
```

## Creating a web scraper using Puppeteer

Back to the entry point file, `index.js`, copy-paste this code:

`index.js`
```javascript
    const puppeteer = require("puppeteer-core");
    require("dotenv").config();
    
    const auth = process.env.UNAME;
    const host = process.env.HOST;
    
    async function run() {
      let browser;
      try {
        browser = await puppeteer.connect({
          browserWSEndpoint: `wss://${auth}@${host}`,
        });
    
        const page = await browser.newPage();
        page.setDefaultNavigationTimeout(2 * 60 * 1000);
        
        await page.goto("http://lumtest.com/myip.json");
        const html = await page.content();
    
        console.log(html);
      } catch (e) {
        console.error("run failed", e);
      } finally {
        await browser?.close();
      }
    }
    
    if (require.main == module) run();
```

The code above does the following:

- Import the modules, the `puppeteer-core`, and `dotenv`
- Read the secret variables with the `host` and `auth` variables 
- Define the asynchronous `run` function
- In the **try** block, connect the endpoint with `puppeteer` in the object using the key `browserWSEndpoint`
- The browser page launches programmatically to access the different pages like elements and fire up events
- Since this is an asynchronous method, the `setDefaultNavigationTimeout` sets a navigation timeout for 2 minutes
- Navigate to the page using the `goto` function, and afterward, get the URL's content with the `page.content()` method
- It is compulsory that after scraping the web, you must close it in the `finally` block

If you want to expand this project, you can take screenshots of the web pages in **png** or **pdf** format.

Check out [the documentation](https://help.brightdata.com/hc/en-us/articles/14726500347409-How-To-Common-Browser-functions-and-Examples#heading-5) to learn more.

## Conclusion

Scraping the web with Bright Data infrastructure makes the process quicker for your use case without writing your scripts from scratch, as it is already taken care of for you. 

Try it today to [explore the benefits](https://get.brightdata.com/ac7857082y9n) of Bright Data over traditional web scraping tools, restricted by proxy networks and make it challenging to work with large datasets.

## Resources

[Scraping Browser documentation](https://help.brightdata.com/hc/en-us/articles/13362921219729#heading-3)
[Scrape at scale with Bright Data Scraping Browser](https://brightdata.com/products/scraping-browser?gspk=dGVyaWV5ZW5pa2U1OTIy&gsxid=mYHfqXzoF0NP&pscd=get.brightdata.com&utm_source=affiliates&utm_campaign=dGVyaWV5ZW5pa2U1OTIy)