Scraping the unscrapable in Python using Playwright
Tags: python, programming, webscraping, codenewbie
Published: 2023-07-06 01:37:51

Automating your workflow [with scripts](https://hackernoon.com/how-to-build-a-python-web-scraper-scrape-data-from-any-website) to get results efficiently is better than being painstakingly done manually. Scraping the web is all about extracting data in a clean and readable format that developers, data analysts, and scientists deploy to read and download an entire web page of its data ethically.

In this article, you will learn and explore the benefits of using Bright Data infrastructure that connects to large datasets with great proxy networks using the [Scraping Browser](https://brightdata.com/products/scraping-browser?gspk=dGVyaWV5ZW5pa2U1OTIy&gsxid=mYHfqXzoF0NP&pscd=get.brightdata.com&utm_source=affiliates&utm_campaign=dGVyaWV5ZW5pa2U1OTIy).

Let’s get started.


## What is Bright Data?

Bright Data is a web data platform that helps organizations, small businesses, and academic institutions retrieve crucial public web data efficiently, reliably, and flexibly. Bright Data comprises ready-to-use datasets that are GDPR and CCPA-compliant.


## What is Playwright?

Playwright is used to navigating target websites just like the function of Puppeteer interacting with the site’s HTML code to extract the data you need.

## Installation

Before writing a single script, check if you have Python installed on your system using this command in the command line interface (CLI) or terminal: 

```bash
    python --version
```

If the version is not present in the terminal after running the command, go to the [official website of Python](https://www.python.org/downloads/) to download it to your local machine.


## Connecting to Scraping Browser

Create a new account on [Bright Data](https://brightdata.com/) to gain access to the admin dashboard of the Scraping Browser for the proxy integration with your application.

On the left pane of the dashboard, click on the **Proxies and Scraping Infra** icon.


![proxies and scraping infra](https://paper-attachments.dropboxusercontent.com/s_4CBE51D7D2EAE0886B936E84C1BF1CF4AEE57FF9FCF3E0499D73B25BAC6F820E_1686225099949_Screenshot+2023-06-08+at+10.49.57.jpg)

Scrolling down the page, select the **Scraping Browser.** After that, click on the **Get started** button. If you don’t find it, click the **Add** button dropdown from the previous image and select the **Scraping Browser**.


![Scraping browser](https://paper-attachments.dropboxusercontent.com/s_4CBE51D7D2EAE0886B936E84C1BF1CF4AEE57FF9FCF3E0499D73B25BAC6F820E_1686228328438_Screenshot+2023-06-08+at+13.31.34.jpg)


The next screen allows you to rename the proxy name. Click the **Add proxy** button to pop up a prompt display message. Accept the default change by clicking the **Yes** button. 


![naming the proxy](https://paper-attachments.dropboxusercontent.com/s_4CBE51D7D2EAE0886B936E84C1BF1CF4AEE57FF9FCF3E0499D73B25BAC6F820E_1686228901763_Screenshot+2023-06-08+at+13.28.50.jpg)


![host, username, and password](https://paper-attachments.dropboxusercontent.com/s_4CBE51D7D2EAE0886B936E84C1BF1CF4AEE57FF9FCF3E0499D73B25BAC6F820E_1686229110311_Screenshot+2023-06-08+at+13.57.15.jpg)


Next, click the **</> Check out code and integration examples** button to configure the code in Python.


## Creating environment variables in Python

Environment variables are stored secret keys and credentials in the form of values configured to keep the app running during development and prevent unauthorized access.

Like in a **Node.js** app, create a new file called `.env` in the root directory. But first, you will need to install the Python package `python-dotenv`.

```bash
    pip3 install python-dotenv
```

The package reads the key-value pairs of the environment variables set.

To confirm the installation of the package `python-dotenv`, run this command that lists all installed packages present:

```bash
    pip3 list
```

Next, copy-paste this code into the `.env` file:

`.env`
```bash
    USERNAME="<user-name>"
    HOST="<host>"
```

Replace the values in the **quotation** with the values from Bright Data.


## Creating the web scraper with Playwright

In the project directory, create a new file called `app.py` to handle scraping the web. 

**Installing packages**
You will need to install these two libraries, `asyncio`, and playwright, with this command:

```bash
    pip3 install asyncio
    pip3 install playwright
```

- **Asyncio**: It is a library to write concurrent code using the async/await syntax
- **Playwright**: This module provides a method to launch a browser instance

Now, copy-paste this code:

`app.py`
```python
    import asyncio
    import os
    from playwright.async_api import async_playwright
    from dotenv import load_dotenv
    
    load_dotenv()
    
    auth = os.getenv("USERNAME")
    host = os.getenv("HOST")
    
    browser_url = f'wss://{auth}@{host}'
    
    async def main():
        async with async_playwright() as pw:
            print('connecting');
            browser = await pw.chromium.connect_over_cdp(browser_url)
            print('connected');
            page = await browser.new_page()
            print('goto')
            await page.goto('http://lumtest.com/myip.json', timeout=120000)
            print('done, evaluating')
            print(await page.evaluate('()=>document.documentElement.outerHTML'))
            await browser.close()
    
    asyncio.run(main())
```

The code above does the following:

- Import the necessary modules like asyncio, `async_playwright`, `load_dotenv`, and `os`
- The `load_dotenv()` is responsible for reading the variables from the `.env` file
- The `os.getenv()` method returns the values of the environment variable key
- The `main()` function is asynchronous, and within the function, the playwright module connects to the data zone
- The `new_page()` method gets the page HTML and, with the `goto` method, leads to the destination site with a timeout of 2 minutes
- While the `page.evaluate()` method will query the page and print out the result after accessing the page elements and firing up the events
- It is a must to close the browser with the `browser.close()` method

To test this application, run with the command:

```bash
    python app.py
```

## Conclusion
The prospects of evaluating and extracting meaningful data are the heart and operation of what Bright Data offers. 

This tutorial showed you how to use the Scraping Browser in Python with the Playwright package to read data from a website. 

Try [Bright Data](https://get.brightdata.com/scrapingbrowser) today!