Pyppeteer Tutorial: The Ultimate Guide to Using Puppeteer with Python
Tags: beginners, programming, tutorial, webdev
Published: 2024-02-05 07:56:26


Pyppeteer, a Python library used for web browser automation, is a valuable resource in the testing world. It serves as the unofficial counterpart to [Puppeteer](https://www.lambdatest.com/puppeteer?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=webpage), a renowned JavaScript library. By leveraging Pyppeteer, users can manipulate web browsers, automate tasks, and extract data from websites using Python.

This Pyppeteer tutorial will walk you through the installation process and will cover all the major scenarios that can be automated with code examples.

So, let’s get started!

## What is Pyppeteer?

Pyppeteer, a Python library, is an unofficial Python port of Puppeteer (a Node.js Chromium browser automation library). Pyppeteer APIs can be leveraged for automating interaction with web elements, [scraping content from dynamic websites](https://www.lambdatest.com/blog/unit-testing-frameworks/?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=blog), handling iFrames, and more.

Though the last released version of Pyppeteer was 1.0.2, on Jan 11, 2022, the QA community still uses the library for web automation on the Chromium browser on the local machine (or cloud grid). Pyppeteer can be used with pytest and PyUnit (Python’s default [unit testing framework)](https://www.lambdatest.com/blog/unit-testing-frameworks/?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=blog).

***Note: **Please replace pip3 with pip depending on the pip version installed on your machine. Also, function and method are used interchangeably throughout the blog.*
>  [**Remote Test device](https://www.lambdatest.com/remote-test-lab?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=webpage) Lab enables efficient software testing on various devices remotely. Click to explore and enhance your development process now!**

## How to install Pyppeteer?

As stated in the Pyppeteer installation guide, Pyppeteer requires Python 3.6 (or above) installed on the machine. As a good practice, we would install Pyppeteer in a virtual environment *(venv)* to isolate it from the packages in the base environment.

When writing this blog, the latest version of Pyppeteer is 1.0.2.

![](https://cdn-images-1.medium.com/max/2000/0*queIGso65KyTRysG.png)

Run the commands *virtualenv venv* and *source venv/bin/activate* on the terminal to create the virtual environment.

![](https://cdn-images-1.medium.com/max/3200/0*5T0rZqBglRd2TCSg.png)

Now that the virtual environment is ready, let’s install Pyppeteer using either of the two ways mentioned below:

1. Run the command *pip3 install pyppeteer* (or *pip install pyppeteer*) on the terminal

![](https://cdn-images-1.medium.com/max/3200/0*Vm0OwEJ92RrYqCKh.png)

As seen above, Pyppeteer (v 1.0.2) was successfully installed in the virtual environment (venv).

2. The latest version of Pyppeteer, i.e., 1.0.2, can also be installed by executing *pip3 install -U git+https://github.com/pyppeteer/pyppeteer@dev* on the terminal.

![](https://cdn-images-1.medium.com/max/3200/0*b0xeEzABAoniwEuF.png)

If Pyppeteer is run for the first time and no Chromium is in the machine, it downloads and installs the latest version.

![](https://cdn-images-1.medium.com/max/3200/0*EdIuWJNo6dMmMZUn.png)

Pyppeteer also lets you use a specified version of Chromium. The Default value can be checked using *pyppeteer.__chromium_revision__*, which is 588429 in our case.

![](https://cdn-images-1.medium.com/max/2208/0*DmIDhT1STcmxM64i.png)

![](https://cdn-images-1.medium.com/max/2000/0*F22VLTuFjuqrvB6b.png)

As seen below, it downloaded & installed Chromium (v 71.0) on the machine.

![](https://cdn-images-1.medium.com/max/3200/0*vonT3XHgBFAX9aGc.png)

In further sections of the blog, we will demonstrate using the *$PYPPETEER_CHROMIUM_REVISION* environment variable to use a certain Chromium version.

Now that the Pyppeteer setup is complete, let’s look at some prominent test scenarios (or use cases) that can be automated using Pyppeteer.
>  **This article explains the [emulator vs simulator](https://www.lambdatest.com/blog/emulator-vs-simulator-vs-real-device/?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=blog) vs real device differences, the learning of which can help you select the right mobile testing solution for your business.**

## Demonstration: Web automation with Pyppeteer

For demonstration of major test scenarios, we have used Pyppeteer in conjunction with the pytest framework. This choice is primarily due to support for [pytest fixtures](https://www.lambdatest.com/blog/end-to-end-tutorial-for-pytest-fixtures-with-examples/?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=blog), test parameterization, and more.

## Directory Structure

As seen in the project structure, the configuration and execution are driven via a *Makefile*. All the relevant test scenarios (or use cases) that demonstrate the usage of Pyppeteer APIs are separated into different folders.

![](https://cdn-images-1.medium.com/max/2000/0*vt_pCufJhMEzxSne.png)

Here is a closer look at the directory structure:

## Configuration (or setup) files

The conftest.py file in pytest contains the implementation related to fixtures, hooks, and other configuration options used in the test code. Based on the value of the environment variable *EXEC_PLATFORM*, Pyppeteer tests will be executed on the local Chromium browser or on Chrome (which is built on Chromium) on a cloud grid like LambdaTest.

LambdaTest is an AI-powered test orchestration and execution platform that allows you to perform [Puppeteer testing](https://www.lambdatest.com/puppeteer?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=webpage) using Python on an [online browser farm](https://www.lambdatest.com/online-browser-farm?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=webpage) spanning 50+ browser versions on the cloud-based testing infrastructure, ensuring efficient [automation testing](https://www.lambdatest.com/automation-testing?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=webpage) using the Pyppeteer library.

With LambdaTest, you can also perform parallel testing at scale and accelerate software release cycles by multiple folds.

![](https://cdn-images-1.medium.com/max/2000/0*6KA2rzidvU9epvTM.png)

### Fixture: Creation of a new browser instance

The scope of the pytest fixture (i.e., *@pytest.fixture*) is set to function. Hence, the fixture is set up and torn down before & after executing a test function (or method). If ***EXEC_PLATFORM*** is set to ***local***, the Chromium browser on the local machine is instantiated using the *launch()* method in Pyppeteer.

![](https://cdn-images-1.medium.com/max/3200/0*QRHtLuxbYVr0nDdN.png)

The *launch()* method is a part of the *pyppeteer.launcher* module that is used to launch an instance of Chrome browser in headless mode (default). Since we intend to run the tests in non-headless mode, we set the *headless* parameter to *False*. The *[‘–start-maximized’]* argument starts the browser in a maximized window.

![](https://cdn-images-1.medium.com/max/2916/0*7MnS4-hdCXfHlZrP.png)

If ***EXEC_PLATFORM*** is set to ***cloud***, the Chrome browser on the LambdaTest grid is instantiated using the *connect()* method in Pyppeteer. In the demonstration, we used the WebSocket endpoint (*browserWSEndpoint*) that points to *wss://cdp.lambdatest.com/puppeteer*. We have used the Endpoint from the [Puppeteer sample](https://github.com/LambdaTest/puppeteer-sample/blob/main/puppeteer-parallel.js#L7) on LambdaTest.

![](https://cdn-images-1.medium.com/max/3200/0*IFXz1_Gohsg43KIM.png)

The capabilities (in dictionary format) generated using [LambdaTest Capabilities Generator](https://www.lambdatest.com/capabilities-generator/?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=webpage) are passed as a query parameter to *browserWSEndpoint*. The *dumps() *function of the *json* module converts the capabilities into a JSON formatted string.

As seen below, the values of *username* and *access_key* are obtained from the environment variables *LT_USERNAME* and *LT_ACCESS_KEY* specified in the Makefile. Since we want to check out the execution, the headless capability is set to *False*.

![](https://cdn-images-1.medium.com/max/2784/0*o4ToiYuI2xX1y0gq.png)

Finally, yield is used for defining the *teardown* code (i.e., releasing browser resources) after the test execution is complete. In our case, we close the browser instance in the fixture responsible for opening a new Browser page.

![](https://cdn-images-1.medium.com/max/2916/0*rH-We9JinRT3Un8Y.png)
>  **Looking for an effective way to test on [Safari browsers online](https://www.lambdatest.com/test-on-safari-browsers?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=webpage)? Skip the hassle of emulators and simulators. Experience authentic testing with LambdaTest’s real online Safari browsers. Start now!**

### Fixture: Creation of a new page

Next up, we have a Fixture of *function* scope primarily responsible for creating (or opening) a new page. The fixture takes the newly created *browser* instance as the input parameter.

The *newPage()* method of the *BrowserContext* class is invoked to open up a new web page. Since the viewport sizes differ when the tests are run on the local machine (i.e., MacBook Pro 13 inch) & cloud grid, the same is set using the *setViewport()* method of the page class. The required viewport width & height are passed in a dictionary format to the *setViewport()* method.

![](https://cdn-images-1.medium.com/max/2612/0*8nt5VPWo_XaL8pwB.png)

Finally, the resources used by the *browser* (created in the *browser()* fixture) and *page(s)* that are created in the *page()* fixture are released as a part of the teardown code.

![](https://cdn-images-1.medium.com/max/2480/0*3PbpUyrpScwmBewq.png)

### pyproject.toml

    [tool.poetry]
    name = "Web automation with Pyppeteer"
    version = "0.1.0"
    description = ""
    authors = ["Himanshu Jagdish Sheth <himanshu.sheth@gmail.com>"]
    
    
    [tool.poetry.dependencies]
    python = "^3.7.9"
    flake8 = "^4.0.1"
    autopep8 = "^1.6.0"
    pytest-asyncio = "^0.21.0"
    
    
    [tool.pytest.ini_options]
    asyncio_mode = "auto"

All the project dependencies (i.e., *flake8*, autopep8, and *pytest-asyncio*) are under the *[tool.poetry.dependencies]* section. *pytest-asyncio* is a pytest plugin that provides support for coroutines as test functions. Awaits inside the test code are made possible with this plugin. The [pytest asyncio](https://www.lambdatest.com/blog/pytest-asyncio-to-reduce-test-execution-time/?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=blog) is also instrumental in greatly reducing the test execution time!

![](https://cdn-images-1.medium.com/max/2000/0*wYzsfP-cRJvHcCLv.png)

In the *[tool.pytest.ini_options]* section, the *asyncio* mode is set to *auto*. This means that the *asyncio* mode will be automatically detected and enabled by pytest, depending on the presence of async fixtures (or tests).

![](https://cdn-images-1.medium.com/max/2000/0*2iVY0W2bchV0Ie7q.png)

### requirements.txt

The requirements.txt contains all the packages and libraries required for execution.

    pytest
    pyppeteer
    pytest-asyncio
    nest_asyncio
    pytest-xdist
    pytest-order
    py

The *pytest-order* plugin is used for ordering the execution of the tests. [pytest parallel execution](https://www.lambdatest.com/blog/pytest-tutorial-parallel-testing-with-selenium-grid/?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=blog) is performed for a couple of test scenarios. Hence, the *pytest-xdist* plugin is installed to realize the same.

All the dependencies and packages are installed by executing *poetry install* and *pip3 install -r requirements.txt* on the terminal.

![](https://cdn-images-1.medium.com/max/3200/0*-7xjOxu_XrcMD-a5.png)

![](https://cdn-images-1.medium.com/max/3200/0*_6kPQonrwDuNAriK.png)
>  **Test native, hybrid, and web apps on any mobile OS with our free [online Android emulator](https://www.lambdatest.com/android-emulator-online?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=webpage). Sign up to optimize app performance.**

### Makefile

As stated earlier, the execution is all driven using the Makefile. Depending on the Python and pip versions installed on your machine, you may change *python3* & *pip3* to *python* & *pip*, respectively.

    # Define variables
    PYTHON := python3
    POETRY := poetry
    PYTEST := pytest
    PIP := pip3
    PROJECT_NAME := web automation with Pyppeteer
    
    
    .PHONY: install
    install:
       $(POETRY) install
       @echo "Dependency installation complete"
    
    
       $(PIP) install -r requirements.txt
       @echo "Set env vars LT_USERNAME & LT_ACCESS_KEY"
       # Procure Username and AccessKey from https://accounts.lambdatest.com/security
       export LT_USERNAME=himansh
       export LT_ACCESS_KEY=Ia1MiqNfci
    
    
    .PHONY: install
    poetry-install:
       poetry install
    
    
    .PHONY: test
    test:
       export NODE_ENV = test
    
    
    .PHONY: test
    pyunit-pyppeteer:
       - echo $(EXEC_PLATFORM)
       - $(PYTHON) tests/pyunit-pyppeteer/test_pyunit_pyppeteer.py
    
    
    .PHONY: test
    pytest-pyppeteer:
       - echo $(EXEC_PLATFORM)
       - $(PYTEST) --verbose --capture=no -s -n 2 tests/pytest-pyppeteer/test_pytest_pyppeteer_1.py \
       tests/pytest-pyppeteer/test_pytest_pyppeteer_2.py
    
    
    .PHONY: test
    pyunit-pyppeteer-browser-session:
       - echo $(EXEC_PLATFORM)
       - $(PYTHON) tests/starting-browser-session/pyunit/test_pyppeteer_browser_session.py
    
    
    .PHONY: test
    pytest-pyppeteer-browser-session:
       - echo $(EXEC_PLATFORM)
       - $(PYTEST) --verbose --capture=no -s \
       tests/starting-browser-session/pytest/test_pyppeteer_browser_session.py
    
    
    .PHONY: test
    asyncio-run-pyppeteer-browser-session:
       - echo $(EXEC_PLATFORM)
       - $(PYTHON) tests/starting-browser-session/asyncio_run/test_pyppeteer_browser_session.py
    
    
    .PHONY: test
    asyncio-run-complete-pyppeteer-browser-session:
       - echo $(EXEC_PLATFORM)
       - $(PYTHON) tests/starting-browser-session/\
       asyncio_run_until_complete/test_pyppeteer_browser_session.py
    
    
    .PHONY: test
    pyppeteer-button-click:
       - echo $(EXEC_PLATFORM)
       - $(PYTEST) --verbose --capture=no -s tests/button-click/test_page_class_click.py
    
    
    .PHONY: test
    pyppeteer-activate-tab:
       - echo $(EXEC_PLATFORM)
       - $(PYTEST) --verbose --capture=no -s tests/active-tab/test_page_class_bringtofront.py
    
    
    ###### Testing Custom Environment - https://miyakogi.github.io/pyppeteer/reference.html#environment-variables
    # Available versions: 113, 121, and default
    .PHONY: test
    pyppeteer-custom-chromium-version:
       - echo $(EXEC_PLATFORM)
       - echo 'Browser Version:' $(CHROMIUM_VERSION)
       - $(PYTEST) --verbose --capture=no -s tests/custom-configuration/test_launcher_exe_path.py
    
    
    ###### Testing Headless - https://miyakogi.github.io/pyppeteer/reference.html#launcher
    # Available values: headless and non-headless
    .PHONY: test
    pyppeteer-custom-browser-mode:
       - echo $(EXEC_PLATFORM)
       - echo $(BROWSER_MODE)
       - $(PYTEST) --verbose --capture=no -s tests/custom-configuration/test_launcher_headless.py
    
    
    .PHONY: test
    pyppeteer-generate-pdf:
       - echo $(EXEC_PLATFORM)
       - $(PYTEST) --verbose --capture=no -s tests/generate-pdf/test_page_class_pdf.py
    
    
    .PHONY: test
    pyppeteer-generate-screenshot:
       - echo $(EXEC_PLATFORM)
       - $(PYTEST) --verbose --capture=no -s tests/generate-screenshots/test_page_class_screenshot.py
    
    
    .PHONY: test
    pyppeteer-cookies:
       - echo $(EXEC_PLATFORM)
       - $(PYTEST) --verbose --capture=no -s tests/handling-cookies/test_page_class_cookies.py
    
    
    .PHONY: test
    pyppeteer-dialog-box:
       - echo $(EXEC_PLATFORM)
       - $(PYTEST) --verbose --capture=no -s tests/handling-dialog-box/test_handling_dialog_box.py
    
    
    .PHONY: test
    pyppeteer-iframe:
       - echo $(EXEC_PLATFORM)
       - $(PYTEST) --verbose --capture=no -s tests/handling-iframe/test_page_class_iframe.py
    
    
    # Like Puppeteer, Navigation operations mentioned below only work in Headless mode
    # goBack: https://miyakogi.github.io/pyppeteer/reference.html#pyppeteer.page.Page.goBack
    # goForward: https://miyakogi.github.io/pyppeteer/reference.html#pyppeteer.page.Page.goForward
    
    
    # Bug Link
    # https://github.com/puppeteer/puppeteer/issues/7739
    # https://stackoverflow.com/questions/65540674/how-to-error-check-pyppeteer-page-goback
    
    
    .PHONY: test
    pyppeteer-navigate-ops:
       - echo $(EXEC_PLATFORM)
       - $(PYTEST) --verbose --capture=no -s tests/navigate-operations/test_page_class_navigation_ops.py
    
    
    .PHONY: test
    pyppeteer-request-response:
       - echo $(EXEC_PLATFORM)
       - $(PYTEST) --verbose --capture=no -s tests/request-response/test_page_class_req_resp.py
    
    
    .PHONY: test
    pyppeteer-viewport:
       - echo $(EXEC_PLATFORM)
       - echo $(BROWSER_MODE)
       - $(PYTEST) --verbose --capture=no -s tests/setting-useragent-viewports/\
       test_page_class_useragent_viewport.py::test_mod_viewport
    
    
    .PHONY: test
    pyppeteer-non-headless-useragent:
       - echo $(EXEC_PLATFORM)
       - echo $(BROWSER_MODE)
       - $(PYTEST) --verbose --capture=no -s tests/setting-useragent-viewports/\
       test_page_class_useragent_viewport.py::test_get_nonheadless_user_agent
    
    
    .PHONY: test
    pyppeteer-headless-useragent:
       - echo $(EXEC_PLATFORM)
       - echo $(BROWSER_MODE)
       - $(PYTEST) --verbose --capture=no -s tests/setting-useragent-viewports/\
       test_page_class_useragent_viewport.py::test_get_headless_user_agent
    
    
    .PHONY: test
    pyppeteer-dynamic-content:
       - echo $(EXEC_PLATFORM)
       - echo $(BROWSER_MODE)
       - $(PYTEST) --verbose --capture=no -s -n 4 tests/handling-dynamic-content/\
       test_page_class_lazy_loaded_content.py
    
    
    .PHONY: test
    pyppeteer-web-scraping:
       - echo $(EXEC_PLATFORM)
       - $(PYTEST) --verbose --capture=no -s tests/web-scraping-content/\
       test_scraping_with_pyppeteer.py
    
    
    .PHONY: clean
    clean:
       # This helped: https://gist.github.com/hbsdev/a17deea814bc10197285
       find . | grep -E "(__pycache__|\.pyc$$)" | xargs rm -rf
       rm -rf .pytest_cache/
       @echo "Clean Succeeded"
    
    
    .PHONY: distclean
    distclean: clean
       rm -rf venv
    
    
    .PHONY: help
    help:
       @echo ""
       @echo "install : Install project dependencies"
       @echo "clean : Clean up temp files"
       @echo "pyunit-pyppeteer : Running Pyppeteer tests with Pyunit framework"
       @echo "pytest-pyppeteer : Running Pyppeteer tests with Pytest framework"
       @echo "pyunit-pyppeteer-browser-session : Browser session using Pyppeteer and Pyunit"
       @echo "pytest-pyppeteer-browser-session : Browser session using Pyppeteer and Pytest"
       @echo "asyncio-run-pyppeteer-browser-session : Browser session using Pyppeteer (Approach 1)"
       @echo "asyncio-run-complete-pyppeteer-browser-session : Browser session using Pyppeteer (Approach 2)"
       @echo "pyppeteer-button-click : Button click demo using Pyppeteer"
       @echo "pyppeteer-activate-tab : Switching browser tabs using Pyppeteer"
       @echo "pyppeteer-custom-chromium-version : Custom Chromium version with Pyppeteer"
       @echo "pyppeteer-custom-browser-mode : Headless and non-headless test execution with Pyppeteer"
       @echo "pyppeteer-generate-pdf : Generating pdf using Pyppeteer"
       @echo "pyppeteer-generate-screenshot : Generating page & element screenshots with Pyppeteer"
       @echo "pyppeteer-cookies : Customizing cookies with Pyppeteer"
       @echo "pyppeteer-dialog-box : Handling Dialog boxes with Pyppeteer"
       @echo "pyppeteer-iframe : Handling iFrames with Pyppeteer"
       @echo "pyppeteer-navigate-ops : Back & Forward browser operations with Pyppeteer"
       @echo "pyppeteer-request-response : Request and Response demonstration using Pyppeteer"
       @echo "pyppeteer-viewport : Customizing viewports using Pyppeteer"
       @echo "pyppeteer-non-headless-useragent : Customizing user-agent (with browser in headed mode) using Pyppeteer"
       @echo "pyppeteer-headless-useragent : Customizing user-agent (with browser in headless mode) using Pyppeteer"
       @echo "pyppeteer-dynamic-content : Handling dynamic web content using Pyppeteer"
       @echo "pyppeteer-web-scraping : Dynamic web scraping using Pyppeteer"

Since environment variables *LT_USERNAME* and *LT_ACCESS_KEY* are used for accessing the LambdaTest cloud grid, replace them with credentials from [LambdaTest Accounts Page](https://accounts.lambdatest.com/security?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=webpage). Also, *make install* sets up configurations & installs dependencies from the .toml & requirements.txt, respectively.

![](https://cdn-images-1.medium.com/max/3200/0*mfaZq96X0Bh9xz84.png)

Invoking make clean on the terminal removes all the generated files or artifacts (e.g., *.pyc) created during the build process.

![](https://cdn-images-1.medium.com/max/2000/0*IOCprGd7F4JxmEKJ.png)

Typing *make help* provide all the options available for execution.

![](https://cdn-images-1.medium.com/max/3200/0*EKua42KhPsRcwlnF.png)

Like earlier, we are not getting into the nuances of Makefile since it is pretty much self-explanatory! With the stage all set, let’s get our hands dirty by implementing some prominent scenarios (including web scraping) with Pyppeteer.

***Note: **The examples below work on local Chromium (**EXEC_PLATFORM = local**) and Chrome on LambdaTest cloud grid (**EXEC_PLATFORM = cloud**).*
>  **Test native, hybrid, and web apps on any mobile OS with our free [online emulator Android](https://www.lambdatest.com/android-emulator-online?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=webpage). Sign up to optimize app performance.**

## Setting up a browser session with Pyppeteer

Like any other library (or framework), the first step is to instantiate the browser and navigate to the respective URL. The *launch()* function of the *pyppeteer.launcher* module is invoked to start up a new instance of the Chrome (or Chromium) browser.

As stated in the Pyppeteer official documentation, the *headless* option is set to *False* so that the browser is invoked in non-headless (or headed) mode. Additional arg (or argument), i.e., *–start-maximized*, is also passed to the *launch()* function so that the browser is maximized.

![](https://cdn-images-1.medium.com/max/2716/0*JEXiuc3Qh9YxaCL_.png)

Now that the browser is instantiated, the *newPage()* function is invoked to create a new page within the browser. The method returns a *Page* object that represents the newly opened page.

![](https://cdn-images-1.medium.com/max/2000/0*eUI_cpXK6yhxCQ3j.png)

Finally, the *goto()* method of the *Page* class is used to navigate to the URL under test.

![](https://cdn-images-1.medium.com/max/2000/0*EdFh_D2XKEb0BkXN.png)

We compare the page title with the expected title in the current test. An assert is raised in case the titles do not match.

![](https://cdn-images-1.medium.com/max/2000/0*OuTzVhI4i7eAWPEF.png)

There are a couple of ways to invoke the test scenarios. The choice depends on the automation framework being used in conjunction with Pyppeteer.

### Approach 1: Using the run function of the asyncio module

The **first approach** is using the *run_until_complete(future)* method (of the asyncio module) until the *future* has been completed. It blocks the execution of the code following it.

    import asyncio
    import pytest
    import os
    import sys
    import json
    from os import environ
    from urllib.parse import quote
    from pyppeteer import connect, launch
    from pyppeteer.errors import PageError
    
    
    test_url = 'https://search.brave.com/'
    exec_platform = os.getenv('EXEC_PLATFORM')
    
    
    # Get username and access key of the LambdaTest Platform
    username = environ.get('LT_USERNAME', None)
    access_key = environ.get('LT_ACCESS_KEY', None)
    
    
    # Capabilities array with the respective configuration for parallel tests
    cloud_capabilities = {
           'browserName': 'Chrome',
           'browserVersion': 'latest',
           'LT:Options': {
               'platform': 'Windows 11',
               'build': '[Build] Launching browser session with Pyppeteer (with Pytest)',
               'name': 'Launching browser session with Pyppeteer (with Pytest)',
               'user': username,
               'accessKey': access_key,
               'resolution': '1920x1080',
               'network': True,
               'video': True,
               'console': True,
               'headless': False
           }
    }
    
    
    local_capabilities = {
           'browserName': 'Chrome'
    }
    
    
    async def test_browser_session():
       # Launch a new browser instance
       # browser = await launch()
       if exec_platform == 'cloud':
           capability = quote(json.dumps(cloud_capabilities))
           print('Initializing test:: ', cloud_capabilities['LT:Options']['name'])
    
    
           browser = await connect(
               browserWSEndpoint=f'wss://cdp.lambdatest.com/puppeteer?capabilities={capability}'
           )
       elif exec_platform == 'local':
           print('Initializing test:: ', local_capabilities['browserName'])
           browser = await launch(headless = False, args=['--start-maximized'])
    
    
       # Create a new page
       page = await browser.newPage()
    
    
       # Navigate to a website
       await page.goto('https://search.brave.com/')
       title = await page.title()
       print(title)
    
    
       try:
           assert title == 'Private Search Engine - Brave Search', 'Expected page title is incorrect!'
           await page.evaluate('_ => {}', f'lambdatest_action: {json.dumps({ "action": "setTestStatus", "arguments": { "status": "passed", "remark": "Title matched" } })}')
       except PageError as e:
           await page.evaluate('_ => {}', f'lambdatest_action: {json.dumps({ "action": "setTestStatus", "arguments": { "status": "failed", "remark": str(e) } })}')
    
    
       # Release the resources
       await page.close()
       await asyncio.sleep(1)
       await browser.close()
    
    
    # Run the event loop
    asyncio.get_event_loop().run_until_complete(test_browser_session())

The **second approach** is to execute the coroutine directly by passing the coroutine to the *run()* function of the *asyncio* module. As stated in the official documentation, the *run()* function cannot be called when another asyncio event loop is running in the same thread.

As shown below, the test method [i.e., *test_lambdatest_search()* ] is passed to the *run()* function. These test execution steps are repeated for all the browser capabilities passed to the test method (or coroutine).

![](https://cdn-images-1.medium.com/max/2344/0*8y_hNByBaD5mFcrg.png)

Here is the complete implementation of setting up a browser session with Pyppeteer when coroutines (or test methods) are run using the *run()* function.

    import asyncio
    import pytest
    import os
    import sys
    import json
    from os import environ
    from urllib.parse import quote
    from pyppeteer import connect, launch
    from pyppeteer.errors import PageError
    
    
    test_url = 'https://search.brave.com/'
    exec_platform = os.getenv('EXEC_PLATFORM')
    
    
    # Get username and access key of the LambdaTest Platform
    username = environ.get('LT_USERNAME', None)
    access_key = environ.get('LT_ACCESS_KEY', None)
    
    
    # Capabilities array with the respective configuration for parallel tests
    cloud_capabilities = {
           'browserName': 'Chrome',
           'browserVersion': 'latest',
           'LT:Options': {
               'platform': 'Windows 11',
               'build': '[Build] Launching browser session with Pyppeteer (with Pytest)',
               'name': 'Launching browser session with Pyppeteer (with Pytest)',
               'user': username,
               'accessKey': access_key,
               'resolution': '1920x1080',
               'network': True,
               'video': True,
               'console': True,
               'headless': False
           }
    }
    
    
    local_capabilities = {
           'browserName': 'Chrome'
    }
    
    
    # Pytest fixture for browser setup
    @pytest.fixture(scope='function')
    async def browser():
       if exec_platform == 'cloud':
           capability = quote(json.dumps(cloud_capabilities))
           print('Initializing test:: ', cloud_capabilities['LT:Options']['name'])
    
    
           browser = await connect(
               browserWSEndpoint=f'wss://cdp.lambdatest.com/puppeteer?capabilities={capability}'
           )
       elif exec_platform == 'local':
           print('Initializing test:: ', local_capabilities['browserName'])
           browser = await launch(headless = False, args=['--start-maximized'])
      
       yield browser
    
    
       await asyncio.sleep(1)
    
    
    # Pytest fixture for page setup
    @pytest.fixture(scope='function')
    async def page(browser):
       page = await browser.newPage()
    
    
       yield page
       await page.close()
       await asyncio.sleep(1)
       await browser.close()
    
    
    @pytest.mark.asyncio
    async def test_browser_session(page):
       await page.goto('https://search.brave.com/')
       title = await page.title()
       print(title)
    
    
       try:
           assert title == 'Private Search Engine - Brave Search', 'Expected page title is incorrect!'
           await page.evaluate('_ => {}', f'lambdatest_action: {json.dumps({ "action": "setTestStatus", "arguments": { "status": "passed", "remark": "Title matched" } })}')
       except PageError as e:
           await page.evaluate('_ => {}', f'lambdatest_action: {json.dumps({ "action": "setTestStatus", "arguments": { "status": "failed", "remark": str(e) } })}')

### Approach 2: Using Pyppeteer with pytest

As mentioned earlier, Pyppeteer can also be used with the pytest framework. The *yield* in conjunction with pytest fixtures is used for implementing the *setup* and *teardown* logic for the following:

* Instantiating the browser [using either *connect()* or *launch()* methods]

* Creating a new page in the browser context [using *newPage()* method]

* Closing & releasing the resources held by the instantiated browser and page

![](https://cdn-images-1.medium.com/max/3200/0*0_XL5ahkiBG0OZNM.png)

Here is the complete implementation of setting up a browser session with Pyppeteer in conjunction with the pytest framework:

    import asyncio
    import pytest
    import os
    import sys
    import json
    from os import environ
    from urllib.parse import quote
    from pyppeteer import connect, launch
    from pyppeteer.errors import PageError
    
    
    test_url = 'https://search.brave.com/'
    exec_platform = os.getenv('EXEC_PLATFORM')
    
    
    # Get username and access key of the LambdaTest Platform
    username = environ.get('LT_USERNAME', None)
    access_key = environ.get('LT_ACCESS_KEY', None)
    
    
    # Capabilities array with the respective configuration for parallel tests
    cloud_capabilities = {
           'browserName': 'Chrome',
           'browserVersion': 'latest',
           'LT:Options': {
               'platform': 'Windows 11',
               'build': '[Build] Launching browser session with Pyppeteer (with Pytest)',
               'name': 'Launching browser session with Pyppeteer (with Pytest)',
               'user': username,
               'accessKey': access_key,
               'resolution': '1920x1080',
               'network': True,
               'video': True,
               'console': True,
               'headless': False
           }
    }
    
    
    local_capabilities = {
           'browserName': 'Chrome'
    }
    
    
    # Pytest fixture for browser setup
    @pytest.fixture(scope='function')
    async def browser():
       if exec_platform == 'cloud':
           capability = quote(json.dumps(cloud_capabilities))
           print('Initializing test:: ', cloud_capabilities['LT:Options']['name'])
    
    
           browser = await connect(
               browserWSEndpoint=f'wss://cdp.lambdatest.com/puppeteer?capabilities={capability}'
           )
       elif exec_platform == 'local':
           print('Initializing test:: ', local_capabilities['browserName'])
           browser = await launch(headless = False, args=['--start-maximized'])
      
       yield browser
    
    
       await asyncio.sleep(1)
    
    
    # Pytest fixture for page setup
    @pytest.fixture(scope='function')
    async def page(browser):
       page = await browser.newPage()
    
    
       yield page
       await page.close()
       await asyncio.sleep(1)
       await browser.close()
    
    
    @pytest.mark.asyncio
    async def test_browser_session(page):
       await page.goto('https://search.brave.com/')
       title = await page.title()
       print(title)
    
    
       try:
           assert title == 'Private Search Engine - Brave Search', 'Expected page title is incorrect!'
           await page.evaluate('_ => {}', f'lambdatest_action: {json.dumps({ "action": "setTestStatus", "arguments": { "status": "passed", "remark": "Title matched" } })}')
       except PageError as e:
           await page.evaluate('_ => {}', f'lambdatest_action: {json.dumps({ "action": "setTestStatus", "arguments": { "status": "failed", "remark": str(e) } })}')
>  **In this article, we take a look at some aspects of simulation and discuss some ways through which we can use [ios emulator for pc](https://www.lambdatest.com/blog/iphone-simulators-for-windows/?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=webpage) Windows.**

### Approach 3: Using Pyppeteer with PyUnit

As stated earlier, Pyppeteer can also be used with the PyUnit (or unittest) framework. However, I would always prefer pytest (with Pyppeteer) over PyUnit (with Pyppeteer) since PyUnit (or unittest) methods are synchronous — a deal breaker when used with the Pyppeteer library!

Since the intent is to run synchronous and asynchronous code, we have used the *IsolatedAsyncioTestCase* test class in the *unittest* module. The *IsolatedAsyncioTestCase* class provides an API similar to *TestCase* and accepts coroutines as test functions. It also isolates the test case’s *asyncio* event loop, providing each test case with its fresh event loop.

The *IsolatedAsyncioTestCase* class provides the *asyncSetUp()* & *asyncTearDown()* methods that accept coroutines. These methods are called after *setUp()* and before *tearDown()*, respectively.

![](https://cdn-images-1.medium.com/max/3200/0*kY2InztOz9efh_7X.png)

Here is the complete implementation of setting up a browser session with Pyppeteer in conjunction with the PyUnit (or unittest) framework:

    import asyncio
    import unittest
    from pyppeteer import connect, launch
    from pyppeteer.errors import PageError
    from urllib.parse import quote
    import json
    import os
    from os import environ
    
    
    exec_platform = os.getenv('EXEC_PLATFORM')
    
    
    # Get username and access key of the LambdaTest Platform
    username = environ.get('LT_USERNAME', None)
    access_key = environ.get('LT_ACCESS_KEY', None)
    
    
    # Capabilities array with the respective configuration for parallel tests
    cloud_capabilities = {
           'browserName': 'Chrome',
           'browserVersion': 'latest',
           'LT:Options': {
               'platform': 'Windows 11',
               'build': '[Build] Launching browser session with Pyppeteer (with unittest)',
               'name': 'Launching browser session with Pyppeteer (with unittest)',
               'user': username,
               'accessKey': access_key,
               'resolution': '1920x1080',
               'network': True,
               'video': True,
               'console': True,
               'headless': False
           }
    }
    
    
    local_capabilities = {
           'browserName': 'Chrome'
    }
    
    
    class LambdaTestAsyncTest(unittest.IsolatedAsyncioTestCase):
       async def asyncSetUp(self):
           if exec_platform == 'cloud':
               capability = quote(json.dumps(cloud_capabilities))
               print('Initializing test:: ', cloud_capabilities['LT:Options']['name'])
    
    
               self.browser = await connect(
                   browserWSEndpoint=f'wss://cdp.lambdatest.com/puppeteer?capabilities={capability}'
               )
           elif exec_platform == 'local':
               print('Initializing test:: ', local_capabilities['browserName'])
               self.browser = await launch(headless = False, args=['--start-maximized'])
    
    
           await asyncio.sleep(1)
           self.page = await self.browser.newPage()
    
    
       async def asyncTearDown(self):
           await self.page.close()
           await asyncio.sleep(1)
           await self.browser.close()
    
    
       async def test_page_title(self):
           await self.page.goto('https://search.brave.com/')
           title = await self.page.title()
           print('Scenario 1: Page Title ' + title)
    
    
           try:
               assert title == 'Private Search Engine - Brave Search', 'Expected page title is incorrect!'
               await self.page.evaluate('_ => {}', f'lambdatest_action: {json.dumps({ "action": "setTestStatus", "arguments": { "status": "passed", "remark": "Title matched" } })}')
           except PageError as e:
               await self.page.evaluate('_ => {}', f'lambdatest_action: {json.dumps({ "action": "setTestStatus", "arguments": { "status": "failed", "remark": str(e) } })}')
    
    
       async def test_page_content(self):
           # Navigate to a website to see the effect
           await self.page.goto('https://www.duckduckgo.com')
           element = await self.page.querySelector('[name="q"]')
    
    
           await element.click()
           await element.type('LambdaTest')
           await asyncio.gather(
               self.page.keyboard.press('Enter'),
               self.page.waitForNavigation()
           )
    
    
           page_title = await self.page.title()
           print('Scenario 2: Page Title ' + page_title)
           return page_title
    
    
    if __name__ == '__main__':
       unittest.main()

**Execution**

To run the set up a browser session with Pyppeteer, simply trigger the relevant *make* command from the terminal:

![](https://cdn-images-1.medium.com/max/2000/1*zgl6dgKjhercueM4Br7frw.png)

As seen below, we have set the *EXEC_PLATFORM* to local and successfully started a browser session using PyUnit (or unittest) with Pyppeteer.

![](https://cdn-images-1.medium.com/max/3200/0*5mL9YOaoo2LN3vDq.png)
>  **Perform browser automation testing on the most powerful cloud infrastructure. Leverage LambdaTest [test automation cloud](https://www.lambdatest.com/automation-testing?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=webpage) for faster, reliable and scalable experience on cloud.**

## Clicking buttons with Pyppeteer

*Page* class in Pyppeteer provides the *click()* method for clicking on an element that matches the selector. For example, when c[licking buttons with Selenium](https://www.lambdatest.com/blog/selenium-click-button-with-examples/?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=blog), the first step is to locate the element using XPath, CSS Selector, or any other suitable locator.

Once the element is located, the *click()* method also scrolls it into view (if needed). Once inside the view, it uses the mouse to click in the element’s center. Like the [Selenium exception](https://www.lambdatest.com/blog/49-common-selenium-exceptions-automation-testing/?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=blog), the *click() *method raises *PageError* if it is impossible to interact with the element (or if there is any other error on the page).

**Syntax**

    coroutine click(selector: str, options: dict = None, **kwargs)

The available options are below:

* *button* (str) — left, right, or middle (default: left)

* *clickCount* (int) — 1 (default)

* *delay* (int|float) — Time to wait between mouse-down and mouse-up in milliseconds (default — 0)

**Demonstration**

For demonstrating button click with Pyppeteer, we use the following test scenario:

 1. Navigate to [https://ecommerce-playground.lambdatest.io/](https://ecommerce-playground.lambdatest.io/).

 2. Click on the menu item ‘Shop by category’.

 3. Add Macbook Pro to the cart.

 4. Assert if the checkout is not successful.

The complete implementation of realizing button clicks with Pyppeteer:

    import asyncio
    import pytest
    from pyppeteer.errors import PageError
    from urllib.parse import quote
    import json
    import os
    import sys
    from os import environ
    from pyppeteer import connect, launch
    
    
    exec_platform = os.getenv('EXEC_PLATFORM')
    
    
    test_url = 'https://ecommerce-playground.lambdatest.io/'
    product_url = 'https://ecommerce-playground.lambdatest.io/index.php?route=product/category&path=57'
    
    
    # XPath's of the elements that need to be located on the page
    
    
    shopcategory = "//a[contains(.,'Shop by Category')]"
    megamenu = "//a[contains(.,'Mega Menu')]"
    # We could have also used XPath for the same
    # phonecategory = "//span[contains(.,'Phone, Tablets & Ipod')]"
    
    
    phonecategorySelector = '#widget-navbar-217841 > ul > li:nth-child(3) > a > div.info > span'
    # Macbook Air product
    macbook_locator = '#mz-product-grid-image-44-212408'
    
    
    # Buy Now button
    button_buynow = '.btn-buynow'
    target_checkout_url = 'https://ecommerce-playground.lambdatest.io/index.php?route=checkout/checkout'
    target_page_str = 'Billing Address'
    
    
    menu_hover_xpath = "//*[@id='__docusaurus']/nav/div[1]/div[2]/div[1]"
    
    
    @pytest.mark.asyncio
    @pytest.mark.order(1)
    async def test_click_element(page):
       # await page.goto(test_url)
       await page.goto(test_url,
           {'waitUntil': 'load'})
    
    
       # Wait for the 'Shop by Category' menu to be available
       menu_element = await page.waitForXPath(shopcategory)
    
    
       # Click on the 'Shop by Category' menu
       await menu_element.click()
    
    
       # Can be changed with non-blocking sleep
       await asyncio.sleep(2)
    
    
       shop_element = await page.waitForSelector(phonecategorySelector, {'visible': True})
    
    
       # Click on the 'Shop by Category' menu
       if exec_platform == 'local':
           await shop_element.click()
       elif exec_platform == 'cloud':
           await asyncio.gather(
               shop_element.click(),
               page.waitForNavigation({'waitUntil': 'networkidle2', 'timeout': 60000}),
           )
    
    
       # Can be changed with non-blocking sleep
       await asyncio.sleep(2)
    
    
       page_title = await page.title()
    
    
       try:
           assert page_title == "Tablets"
           print("Test Success: Reached the target URL")
           await page.evaluate('_ => {}', f'lambdatest_action: {json.dumps({ "action": "setTestStatus", "arguments": { "status": "passed", "remark": "Title matched" } })}')
       except PageError as e:
           print("Test Failure: Recheck the URL")
           await page.evaluate('_ => {}', f'lambdatest_action: {json.dumps({ "action": "setTestStatus", "arguments": { "status": "failed", "remark": str(e) } })}')
    
    
       await asyncio.sleep(2)
    
    
       elem_macbook = await page.waitForSelector(macbook_locator, {'visible': True})
    
    
       # Click on the 'Shop by Category' menu
       await elem_macbook.click()
    
    
       await asyncio.sleep(2)
    
    
       # Click on the Buy Now Button
       elem_buynow = await page.waitForSelector(button_buynow, {'visible': True})
       if exec_platform == 'local':
           await elem_buynow.click()
       elif exec_platform == 'cloud':
           await asyncio.gather(
               elem_buynow.click(),
               page.waitForNavigation({'waitUntil': 'networkidle2'}),
           )
    
    
       await asyncio.sleep(2)
    
    
       current_url = page.url
       print('Current URL is: ' + current_url)
    
    
       try:
           assert current_url == target_checkout_url
           print("Test Success: Product checkout successful")
           await page.evaluate('_ => {}', f'lambdatest_action: {json.dumps({ "action": "setTestStatus", "arguments": { "status": "passed", "remark": "Title matched" } })}')
       except PageError as e:
           print("Test Failure: Could not checkout Product")
           await page.evaluate('_ => {}', f'lambdatest_action: {json.dumps({ "action": "setTestStatus", "arguments": { "status": "failed", "remark": str(e) } })}')
    
    
       await asyncio.sleep(2)
    
    
    @pytest.mark.asyncio
    @pytest.mark.order(2)
    async def test_get_content(page):
       await page.goto('https://ecommerce-playground.lambdatest.io/')
    
    
       await asyncio.sleep(2)
    
    
       html_content = await page.content()
    
    
       await asyncio.sleep(2)
    
    
       assert 'Upto 50% Off on Fully Automatic' in html_content, "Expected string not found in page content"

**Code Walkthrough**

Since we are using the pytest framework with Pyppeteer library, we first import *pytest*, *pyppeteer*, and *json* modules.

![](https://cdn-images-1.medium.com/max/2000/0*i7LPH79VVJzOdryE.png)

The tests are marked with the *@pytest.mark.asyncio* decorator, making them coroutines instead of tests (or test methods). This lets us use await code inside the tests.

Next, the *@pytest.mark.order* marker of the *pytest-order* plugin is used to order the execution of the tests.

![](https://cdn-images-1.medium.com/max/2000/0*WA8ih6mup_ZUXjr3.png)

The *goto()* method of the *Page* class is invoked to navigate to the URL under test. The *waitUntil* option is set to *load* (i.e., default — navigation succeeds when the load event is fired).

The *waitUntil* option can also be set to *domcontentloaded* (i.e., *DOMContentLoaded* is fired), *networkidle0* (i.e., no more than 0 network connections for at least 500 ms), or *networkidle2* (i.e., no more than 2 network connections for at least 500 ms).

![](https://cdn-images-1.medium.com/max/2000/0*D4R7Y-2y_4tnIp4q.png)

Now that the page is loaded, we use the *waitForXPath()* method until the element matching the XPath appears on the page. The default wait is 3000 ms.

![](https://cdn-images-1.medium.com/max/3200/0*A9fUgam03G-fwBMC.png)

![](https://cdn-images-1.medium.com/max/3084/0*f-hdSOBSRNrfwv6D.png)

We locate the menu item using the [CSS Selector](https://www.lambdatest.com/blog/css-selectors/?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=blog), passed to the *waitForSelector() *method of the *Page *class. The method returns immediately in case the selector already exists on the page. On the other hand, it waits until the selector appears on the page for the duration specified by the *timeout* value (default — 3000 ms).

![](https://cdn-images-1.medium.com/max/3200/0*hQiC7dzBvdp35Pb6.png)

As seen below, the *visible* option is set to *True*; hence, it waits for the element to be visible when it is present in the DOM.

![](https://cdn-images-1.medium.com/max/3200/0*y53jNv8McFkb3vVL.png)

Once the element is located, the *click()* method is invoked for clicking on the element. In order to ensure that there is zero [test flakiness](https://www.lambdatest.com/learning-hub/flaky-test?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=learning_hub) (when using the cloud grid), we have also used the *waitForNavigation()* method with the timeout set to 60000 ms.

The *waitUntil* option in the method is set to *networkidle2*. Hence, the execution moves to the next line only if there is a *timeout* or when there are no more than 2 network connections for at least 500 ms.
>  [**What is Xcode](https://www.lambdatest.com/learning-hub/what-is-xcode?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=learning_hub): It is an IDE developed by Apple for developing your applications for macOS, iOS, iPadOS, watchOS, tvOS, and visionOS with powerful, integrated tools.**

![](https://cdn-images-1.medium.com/max/3188/0*QgYBk-G1Otn9BNxQ.png)

Then, we check the page title using the *title()* method of the *Page* class.

![](https://cdn-images-1.medium.com/max/2000/0*oQaPguUnp6mfU_Yd.png)

Once we have the page title, the *page.evaluate()* method in Pyppeteer is used to execute JavaScript code within the context of a page. An assert is raised in case the title (of the current page) does not match with the expected page title.

![](https://cdn-images-1.medium.com/max/3200/0*4QYQL4fHbmmYgE9r.png)

Now that we are on the Product Page, we locate the respective WebElement using the CSS Selector. The *visible* option in the *waitForSelector()* method is set to *True;* hence, it waits for the located element to be visible in the DOM. By default, the *timeout* is 30000 ms (or 30 seconds).

![](https://cdn-images-1.medium.com/max/3200/0*6IEPrKylmrjfJ3bP.png)

![](https://cdn-images-1.medium.com/max/3084/0*tJZTabPzyN6vNkYe.png)

Once the button is clicked, we pause the execution (for 2 seconds) by invoking the *asyncio.sleep()*. It is used for introducing asynchronous delays in an event loop.

After an asynchronous sleep of 2 seconds, the **Buy Now** button is located using the CSS Selector (i.e., *.btn-buynow*). Like earlier, the visibility of the located element is checked; otherwise, an exception is raised if the element is not visible within a default timeout of 30000 ms.

![](https://cdn-images-1.medium.com/max/3200/0*gdxHhfQicsrZdVs0.png)

![](https://cdn-images-1.medium.com/max/2984/0*JFza6jg_w5-A1pzw.png)

Here, we have used *asyncio.gather()*, a function provided by the *asyncio* library that allows multiple coroutines to be executed concurrently and waits for them to complete before moving further with the execution.

In our case, the click to the located element (or button) and [waiting for the page to load](https://www.lambdatest.com/blog/selenium-wait-for-page-to-load/?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=blog) with the *waitUntil *option set to *networkidle2 *are executed as a part of the *gather()* function.

![](https://cdn-images-1.medium.com/max/2580/0*jqUy9g9j3yJ0MVl7.png)

Finally, the current page’s URL is fetched by invoking the *url()* method of the Page class. An assert is raised if the current URL does not match the expected destination URL.

![](https://cdn-images-1.medium.com/max/3200/0*FBIMq8-s6jAQ-RYx.png)

**Execution**

To automate interactions with a button element with Pyppeteer, invoke the command *make pyppeteer-button-click *on the terminal. In our case, we have set the execution to the LambdaTest cloud grid.

![](https://cdn-images-1.medium.com/max/3200/0*eVSiEg_iEw_wOKBx.png)

Below is the snapshot from the [LambdaTest Automation Dashboard](https://automation.lambdatest.com/?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=webpage) indicating that the test execution was successful.

![](https://cdn-images-1.medium.com/max/3200/0*QO-x4IyyB3FDOhbn.png)
>  **Explore ‘What is TDD’ in Agile with our comprehensive tutorial. Learn [Test-Driven Development](https://www.lambdatest.com/learning-hub/test-driven-development?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=learning_hub) meaning, examples, and best practices for effective software development.**

## Printing PDF files with Pyppeteer

[Testing PDF files](https://www.lambdatest.com/blog/selenium-testing-pdf-files/?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=blog) is a common test scenario with any automation framework, and the same is the case with the Pyppeteer library. The *pdf() *method (or coroutine) of the *Page *class lets you generate a pdf of the entire web page.

As stated in official Pyppeteer documentation, generating PDFs is only supported in headless mode. The *pdf()* method provides many options, some mentioned below:

![](https://cdn-images-1.medium.com/max/2800/0*ta97APp9nQqKpYWO.png)

[*Image Source](https://miyakogi.github.io/pyppeteer/_modules/pyppeteer/page.html#Page.pdf)*

By default, the *pdf()* method generates a pdf of the page with *print* CSS media. For generating a PDF with *screen* media, *page.emulateMedia(‘screen’)* needs to be invoked before invoking the *pdf()* method.

    import asyncio
    import pytest
    from pyppeteer.errors import PageError
    from urllib.parse import quote
    import os
    import sys
    from os import environ
    from pyppeteer import connect, launch
    
    
    exec_platform = os.getenv('EXEC_PLATFORM')
    
    
    test_url = 'https://lambdatest.com/'
    
    
    # Selectors of the page
    # Pytest fixture for browser setup
    @pytest.fixture(scope='function')
    async def browser():
       if exec_platform == 'local':
           browser = await launch()
       yield browser
       await asyncio.sleep(1)   
       await browser.close()
    
    
    # Pytest fixture for page setup
    @pytest.fixture(scope='function')
    async def page(browser):
       page = await browser.newPage()
       yield page
       await page.close()
    
    
    @pytest.mark.asyncio
    @pytest.mark.order(1)
    async def test_print_pdf(page):
       await page.goto(test_url, {'waitUntil' : 'networkidle2'})
    
    
       asyncio.sleep(1)
    
    
       page.emulateMedia('screen')
    
    
       # Further details
       # https://miyakogi.github.io/pyppeteer/reference.html#pyppeteer.page.Page.pdf
       await page.pdf({'path': 'lambdatest.pdf', 'format': 'A4'})

**Code Walkthrough**

The *launch()* method is invoked for instantiating [headless Chrome](https://www.lambdatest.com/blog/headless-chrome/?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=blog). Since we are using pytest with Pyppeteer, *browser *instance and *new pages *on the browser are created under the pytest fixture with *function *scope.

![](https://cdn-images-1.medium.com/max/2000/0*V900oTLyyKX6QILk.png)

Before printing the page, the *emulateMedia()* method is called to simulate the appearance of the page when rendered for *screen *media.

![](https://cdn-images-1.medium.com/max/2000/0*yNU242gI55Hb6UJ4.png)

Finally, the *pdf()* method of the *Page *class is invoked with the path set to *current-working-directory/lambdatest.pdf* and format set to A4.

![](https://cdn-images-1.medium.com/max/2444/0*4vL2GqgfHNIvs1dT.png)

**Execution**

In order to generate a PDF of the current web page with Pyppeteer, invoke the command *make pyppeteer-generate-pdf *on the terminal. We have instantiated the Chromium browser on the local machine for this scenario.

![](https://cdn-images-1.medium.com/max/3200/0*j3qDgJ71k2ofuGTx.png)

Below is a screenshot of the generated PDF document:

![](https://cdn-images-1.medium.com/max/3200/0*18ic8jSU2wKUvoeE.png)
>  [**Dynamic testing](https://www.lambdatest.com/learning-hub/dynamic-testing?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=learning_hub) evaluates software by executing code to identify defects, ensuring functionality, and verifying system performance. Learn what is dynamic testing, its types, phases, and more.**

## Opening specified version of Chromium browser with Pyppeteer

As stated in the earlier sections, Pyppeteer also provides the flexibility to use a specified version of Chromium. The version can be specified in the *PYPPETEER_CHROMIUM_REVISION* environment variable.

The other option is downloading the required version of the Chromium browser on the machine. After that, we have to pass the path to *launch()* method of the *Launcher *class. The *executablePath* option in the *launch()* method lets you run the specified Chromium version instead of the default bundled Chromium.

![](https://cdn-images-1.medium.com/max/3152/0*5u0yjHvtSF_XGLbf.png)

To run a particular version of Chromium on the machine (e.g., macOS in my case), we downloaded a couple of Chromium versions from the following locations:

![](https://cdn-images-1.medium.com/max/2000/1*frP9UtxxYI4ZOsKlC42vBQ.png)

I have uploaded the downloaded different Chromium browser versions in the [mac-chrome](https://drive.google.com/drive/folders/1CDBGc2PmbiOKZtH66BSqwmyt3JzMnfF3?usp=sharing) folder.

![](https://cdn-images-1.medium.com/max/3200/0*1CSk_j7wEEDQlGqK.png)

Once downloaded, you need to copy the different versions of Chromium in the *mac-chrome* folder in the root of the working directory.

![](https://cdn-images-1.medium.com/max/3200/0*vv6Y6O2huY6AyRZA.png)

With this, we are all set to run Pyppeteer with the different versions of the Chromium browser!

**Demonstration**

In the sample shown below, choosing Chromium versions (*default*, *121*, and *113*) is flexible depending on the value of the *CHROMIUM_VERSION* environment variable. For this example, you must ensure that EXEC_PLATFORM is set to *local*.

    import asyncio
    import pytest
    from pyppeteer.errors import PageError
    from urllib.parse import quote
    import json
    import os
    import sys
    from os import environ
    from pyppeteer import connect, launch
    
    
    exec_platform = os.getenv('EXEC_PLATFORM')
    
    
    # Can take values - headless and non-headless
    chromium_version = os.getenv('CHROMIUM_VERSION')
    
    
    # Pytest fixture for browser setup
    @pytest.fixture(scope='function')
    async def browser():
       if exec_platform == 'local':
           if chromium_version == '121':
               custom_chrome_path = "mac-chrome/Chromium_121.app/Contents/MacOS/Chromium"
           elif chromium_version == '113':
               custom_chrome_path = "mac-chrome/Chromium_113.app/Contents/MacOS/Chromium"
           else:
               custom_chrome_path = "mac-chrome/Chromium.app/Contents/MacOS/Chromium"
      
       browser = await launch(headless = False,
                   executablePath = custom_chrome_path, args=['--start-maximized'])
       yield browser
       await asyncio.sleep(1)   
       await browser.close()
    
    
    # Pytest fixture for page setup
    @pytest.fixture(scope='function')
    async def page(browser):
       page = await browser.newPage()
       yield page
       await page.close()
    
    
    # Ported code from https://github.com/LambdaTest/puppeteer-sample/blob/main/puppeteer-parallel.js
    
    
    @pytest.mark.asyncio
    async def test_exe_path(page):
       await page.goto('https://www.duckduckgo.com')
       await page.setViewport({'width': 1920, 'height': 1080})
    
    
       element = await page.querySelector('[name="q"]')
       await element.click()
       await element.type('LambdaTest')
       await asyncio.gather(
           page.keyboard.press('Enter'),
           page.waitForNavigation()
       )
    
    
       page_title = await page.title()
    
    
       try:
           assert page_title == 'LambdaTest at DuckDuckGo', 'Expected page title is incorrect!'
           await page.evaluate('_ => {}', f'lambdatest_action: {json.dumps({ "action": "setTestStatus", "arguments": { "status": "passed", "remark": "Title matched" } })}')
       except PageError as e:
           await page.evaluate('_ => {}', f'lambdatest_action: {json.dumps({ "action": "setTestStatus", "arguments": { "status": "failed", "remark": str(e) } })}')

**Code Walkthrough**

To get started, we read the value of the environment variable *CHROMIUM_VERSION*.

![](https://cdn-images-1.medium.com/max/2748/0*aRMwTM07UzrC_5o7.png)

Once the value of the environment variable is read, the custom path is assigned to *executablePath *[of the *launch() *method].

![](https://cdn-images-1.medium.com/max/3200/0*dKjfyo2ccP3xcGbp.png)

As seen in the modified pytest fixture implementation, the custom Chromium browser is launched in non-headless mode and maximized state.

![](https://cdn-images-1.medium.com/max/3084/0*uCB72Of-2BxXSdYq.png)

As stated in the official documentation, Pyppeteer works best with the version of Chromium it is bundled with. However, we did not encounter any issues when testing with the custom versions of the Chromium!

In the test method [i.e. *test_exe_path()*], we first set the viewport size to (1920* 1080) using the *setViewport()* method.

![](https://cdn-images-1.medium.com/max/2480/0*Hbcthd8GJG9LJVFz.png)

After that, we locate the text-box element in the [DuckDuckGo](https://www.duckduckgo.com/) search page with the CSS Selector.

![](https://cdn-images-1.medium.com/max/3200/0*6FkdmMtuJlR4eltV.png)

![](https://cdn-images-1.medium.com/max/2108/0*buuZ_vtHAsymL48f.png)

Next, we enter the search term LambdaTest in the search box. As a part of *asyncio.gather()*, we first trigger the *Enter* key via the *press()* method of the *keyboard *class. The *waitForNavigation()* method waits (for a maximum of 30000 ms) for the navigation event to occur before it proceeds to the next step in execution.

![](https://cdn-images-1.medium.com/max/2000/0*1ZdPSOHSttQ7AiUu.png)

**Execution**

We first set the *EXEC_PLATFORM* variable to *local*. For execution on Chromium version 121, set the *CHROMIUM_VERSION* to 121 by triggering the *export CHROMIUM_VERSION=121* command on the terminal. Run the *make pyppeteer-custom-chromium-version* command to run the automated test on the specified Chromium version.

![](https://cdn-images-1.medium.com/max/3200/0*P4LC2K0DUOJrZZOx.png)

To verify the Chromium version, we temporarily disabled closing (or releasing resources) of the Page & Browser.

![](https://cdn-images-1.medium.com/max/2000/0*LCZtU5r2kntap-bx.png)

As seen below, the tests are rightly triggered on Chromium v121!

![](https://cdn-images-1.medium.com/max/3200/0*LBqJ7XbY0Lud3XqE.png)

For execution on Chromium version 113, set the *CHROMIUM_VERSION* to 113 by triggering the *export CHROMIUM_VERSION=113* command on the terminal. Run the *make pyppeteer-custom-chromium-version* command to run the automated test on the specified Chromium version.

![](https://cdn-images-1.medium.com/max/3200/0*MGnovhSjx084f8mV.png)

![](https://cdn-images-1.medium.com/max/3200/0*DgQzdBnSu5cLaA9h.png)
>  **Check out [top Java Frameworks](https://www.lambdatest.com/blog/top-java-frameworks/?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=blog) to use in 2023. Elevate your web development with top Java frameworks such as JUnit, Spring, Hibernate, Play, Apache Struts, Blade, Grails, and more.**

## Switching tabs with Pyppeteer

[Switching tabs in a browser](https://www.lambdatest.com/blog/switch-tabs-in-browser-using-selenium-python/?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=blog) is very commonly used when automating browser-based test scenarios. The same principle also applies to Pyppeteer as well. Every invocation of the *newPage() *method in Pyppeteer returns a Page Object. This becomes the identifier of the page, similar to window handles in Selenium. To learn more about it you can go through this blog on [handling windows in Selenium](https://www.lambdatest.com/blog/selenium-c-tutorial-handling-multiple-browser-windows/?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=blog).

In case more than one browser page is open, you use the *targets()* method that returns a list consisting of all active targets inside the browser. Switching to a certain tab (or activating a tab) in Pyppeteer is done via the *bringToFront()* method of the *Page* class.

**Demonstration**

For demonstration, let’s open up two browser tabs and switch to the first tab using the *bringToFront()* coroutine.

    import asyncio
    import pytest
    from pyppeteer.errors import PageError
    from urllib.parse import quote
    import json
    import os
    import sys
    from os import environ
    from pyppeteer import connect, launch
    
    
    exec_platform = os.getenv('EXEC_PLATFORM')
    
    
    @pytest.mark.asyncio
    async def test_bring_to_front(browser, page):
       await page.goto('https://www.duckduckgo.com', {'waitUntil' : 'networkidle2'})
       # Maximize the page
       await page.setViewport({'width': 1920, 'height': 1080})
    
    
       element = await page.querySelector('[name="q"]')
       await element.click()
       await element.type('LambdaTest')
       await asyncio.gather(
           page.keyboard.press('Enter'),
           page.waitForNavigation()
       )
    
    
       # Use asyncio.sleep within the async test function
       # await asyncio.sleep(1)
    
    
       page_title = await page.title()
    
    
       try:
           assert page_title == 'LambdaTest at DuckDuckGo', 'Expected page title is incorrect!'
           await page.evaluate('_ => {}', f'lambdatest_action: {json.dumps({ "action": "setTestStatus", "arguments": { "status": "passed", "remark": "Title matched" } })}')
       except PageError as e:
           await page.evaluate('_ => {}', f'lambdatest_action: {json.dumps({ "action": "setTestStatus", "arguments": { "status": "failed", "remark": str(e) } })}')
    
    
      
       page_1 = await browser.newPage()
       await page_1.goto('https://www.lambdatest.com', {'waitUntil' : 'networkidle2'})
    
    
       await page.bringToFront()
       await asyncio.sleep(2)

**Code Walkthrough**

Since the test scenario remains unchanged, we would only focus on how tabs are switched in the example. First, we open a new page with the URL under test set to DuckDuckGo.

![](https://cdn-images-1.medium.com/max/3084/0*DTW1uIVo6OeDZmsj.png)

Next, we open a new page with the URL under test set to [LambdaTest](https://automation.lambdatest.com/?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=webpage). As seen below, the page identifier is set to *page_1*.

![](https://cdn-images-1.medium.com/max/3152/0*fwAUX9sg7CzZuXca.png)

For switching to the earlier tab, we invoke the *bringToFront()* method on the page object — *page*.

![](https://cdn-images-1.medium.com/max/2000/0*-rGeE39RZpW8hwvv.png)

**Execution**

Invoke the command *make pyppeteer-activate-tab* on the terminal with the Chromium browser instantiated on the local machine.

![](https://cdn-images-1.medium.com/max/3200/0*0WGPUT4Ia1d09g4u.png)

As seen in the execution screenshot, the older tab (where the DuckDuckGo scenario is run) is brought back to focus.

## Taking screenshots with Pyppeteer

[Capturing screenshots with Selenium](https://www.lambdatest.com/blog/how-to-capture-screenshots-in-selenium-guide-with-examples/?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=blog), Pyppeteer, Cypress, etc., is a common practice for isolating and debugging visual issues in the website/app. Screenshots are also essential for [visual regression testing](https://www.lambdatest.com/learning-hub/visual-regression-testing?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=learning_hub), as it helps in comparing captured screenshots with the baseline screenshots.

Apart from identifying visual inconsistencies, screenshots (full-page or element-level) provide a visual confirmation of the page at a particular point in time. The *screenshot()* method of the *Page *class helps capture a screenshot of the entire page or a particular element that is currently under focus.

The *screenshot()* method throws an *ElementHandleError* in case the target element is detached from the [DOM](https://www.lambdatest.com/blog/document-object-model/?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=blog).

For capturing element screenshot, we first capture the element using an appropriate locator. Once located, the *boundingBox()* method is used to calculate the bounding box of the element.

![](https://cdn-images-1.medium.com/max/2932/0*Gk20bi20f8FwVds0.png)

[*BoundingBox](https://developer.mozilla.org/en-US/docs/Web/API/Element/getBoundingClientRect/element-box-diagram.png)*

The *boundingBox()* method returns a dictionary of items mentioned below:

* *x* (integer): X coordinate value in pixels

* *y* (integer): Y coordinate value in pixels

* *width *(integer): Element width in pixels

* *height *(integer): Element height in pixels

**Demonstration**

For demonstrating capturing screenshots with Pyppeteer, we automate the following test scenario on Chromium installed on the local machine (*EXEC_PLATFORM* = *local*):

 1. Navigate to the [Product Page](https://ecommerce-playground.lambdatest.io/index.php?route=product/category&path=30)

 2. Locate the required element using the most-suited selector.

 3. Click on the located element.

 4. Capture full-page screenshots and element screenshots.

    # Further details of screenshot API
    # https://miyakogi.github.io/pyppeteer/reference.html#pyppeteer.page.Page.screenshot
    
    
    import asyncio
    import pytest
    from pyppeteer.errors import PageError
    from urllib.parse import quote
    import os
    import sys
    from os import environ
    from pyppeteer import connect, launch
    
    
    exec_platform = os.getenv('EXEC_PLATFORM')
    
    
    timeOut = 60000
    
    
    test_url = 'https://ecommerce-playground.lambdatest.io/index.php?route=product/category&path=30'
    loc_product_1 = "#mz-product-grid-image-43-212408 > div > div.carousel-item.active > img"
    loc_final_product = "#image-gallery-216811 > div.image-thumb.d-flex > a > img"
    target_url = "https://ecommerce-playground.lambdatest.io/index.php?route=product/product&path=25_30&product_id=43"
    
    
    @pytest.mark.asyncio
    @pytest.mark.order(1)
    async def test_screenshot(page):
       # The time out can be set using the setDefaultNavigationTimeout
       # It is primarily used for overriding the default page timeout of 30 seconds
       page.setDefaultNavigationTimeout(timeOut)
       await page.goto(test_url,
           {'waitUntil': 'networkidle2', 'timeout': timeOut})
    
    
       await asyncio.sleep(1)
    
    
       # Wait for the element to be present in the DOM
       elem_prod_link = await page.waitForSelector(loc_product_1, {'visible': True})
       # elem_prod_link = await page.querySelector(loc_product_1)
    
    
       await asyncio.sleep(2)
    
    
       await asyncio.gather(
           elem_prod_link.click(),
           page.waitForNavigation()
       )
    
    
       # Assert if required, since the test is a simple one; we leave as is :D
       current_url = page.url
       print('Current URL is: ' + current_url)
    
    
       try:
           assert current_url == target_url
           print("Test Success: Product checkout successful")
       except PageError as e:
           print("Test Failure: Could not checkout Product")
           print("Error Code" + str(e))
    
    
       await asyncio.sleep(2)
    
    
       # Wait for the element to be present in the DOM
       await page.waitForSelector(loc_final_product)
       elem_prod_img = await page.querySelector(loc_final_product)
    
    
       # Further information on boundingBox
       # https://miyakogi.github.io/pyppeteer/reference.html#pyppeteer.element_handle.ElementHandle.boundingBox
       bounding_box = await elem_prod_img.boundingBox()
    
    
       # Take a screenshot of the element
       if bounding_box:
           await elem_prod_img.screenshot({'path': 'product-screenshot.png', 'clip': bounding_box})
    
    
       await asyncio.sleep(1)
    
    
       # Take a screenshot of the entire page
       await page.screenshot({'path': 'page-screenshot.png'})

**Code Walkthrough**

We first navigate to the [LambdaTest eCommerce Playground](https://ecommerce-playground.lambdatest.io/index.php?route=product/category&path=30) using the *goto() *method of the *Page *class.

![](https://cdn-images-1.medium.com/max/3200/0*vYZ6upflExjSsoYS.png)

Next, we locate the required element using the CSS Selector property. Instead of CSS Selector, you can also use [XPath](https://www.lambdatest.com/blog/complete-guide-for-using-xpath-in-selenium-with-examples/?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=blog), in which case you would need to use *waitForXPath() *instead of the *waitForSelector()* method.

![](https://cdn-images-1.medium.com/max/3200/0*Zrd7N7n5QbyNPFQo.png)

![](https://cdn-images-1.medium.com/max/3200/0*EGBkreG4ZGhMQSXg.png)

Click operation is performed on the located element using the *click()* method. Like before, the *waitForNavigation()* method after the click helps wait for the completion of the navigation operation before the browser proceeds with further actions.

![](https://cdn-images-1.medium.com/max/2000/0*VG1aEFALHSVPRyvk.png)

The product image is located via the CSS Selector. The *querySelector()* method of the *Page *class returns the *ElementHandle* of the element that matches the selector.

![](https://cdn-images-1.medium.com/max/3200/0*1j7uM0ISlYVBb-_z.png)

![](https://cdn-images-1.medium.com/max/3120/0*fkfjC40XhkMPpA0D.png)

Next, the *boundingBox()* method on the located element returns a dictionary of the bounding box.

![](https://cdn-images-1.medium.com/max/2108/0*rqQEZOPaXRn2uVN4.png)

To capture a screenshot of the element, the *screenshot()* method is used with the *clip* option that specifies an object (of type dict), which specifies the clipping region of the page. The screenshot is saved as *product-screenshot.png*.

![](https://cdn-images-1.medium.com/max/2540/0*NMhh5fHiY-xhIMQa.png)

![](https://cdn-images-1.medium.com/max/3200/0*Vah8AT2Fm7skxrbM.png)

The page screenshot (saved as *page-screenshot.png*) is captured by invoking the *screenshot()* method. The *fullPage* option is set to *False* (default) and takes a screenshot of the page currently under display.

![](https://cdn-images-1.medium.com/max/2952/0*Vna8KwNQatAey1eu.png)

Full-page (or scrollable) screenshot is captured by setting the *fullPage* option to *True*.

![](https://cdn-images-1.medium.com/max/3200/0*JmYYLaBIQxCB8Ofm.png)

**Execution**

Invoke the command *make pyppeteer-generate-screenshot* on the terminal with the Chromium browser instantiated on the local machine. As seen below, the test execution is successful.

![](https://cdn-images-1.medium.com/max/3200/0*tegLOTPsqy1oDKpi.png)

The full-page screenshot is below:

![](https://cdn-images-1.medium.com/max/3200/0*e4DngmW398JMmm3z.png)

The scrollable-page screenshot is below:

![](https://cdn-images-1.medium.com/max/2244/0*uN0D9KNJJzQb0EUk.png)

The element screenshot is below:

![](https://cdn-images-1.medium.com/max/2000/0*XQHStqvBU3EPIijF.png)

In summary, the *screenshot()* method of Pyppeteer has all the options to realize [visual regression testing with Python](https://www.lambdatest.com/learning-hub/python-visual-regression-testing?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=learning_hub)!

## Managing cookies with Pyppeteer

Website cookies plays an integral role in automated testing. Modifying cookies can be useful for test scenarios related to session management, [localization testing](https://www.lambdatest.com/blog/how-to-perform-localization-testing/?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=blog), handling cookie consent, and [cross browser testing](https://www.lambdatest.com/online-browser-testing?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=webpage).

I recommend reading this detailed blog on [handling cookies with Selenium](https://www.lambdatest.com/blog/handling-cookies-in-selenium-webdriver/?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=blog), as overall cookie handling principles remain unchanged irrespective of the framework (or library) used for [automated testing](https://www.lambdatest.com/learning-hub/automation-testing?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=learning_hub). The Pyppeteer library provides the following methods (or coroutines) for cookie management:

![](https://cdn-images-1.medium.com/max/2000/1*vh_W_p4m-nUBiR6eLEUpLQ.png)

**Demonstration**

For demonstrating cookie management with Pyppeteer, we automate the following test scenario on Chromium installed on the local machine (*EXEC_PLATFORM* = *local*):

 1. Navigate to [LambdaTest eCommerce Playground](https://ecommerce-playground.lambdatest.io/).

 2. Print cookies.

 3. Set a new cookie.

 4. Delete a cookie and print the cookies.

    import asyncio
    import pytest
    from pyppeteer.errors import PageError
    from urllib.parse import quote
    import json
    import os
    import sys
    from os import environ
    from pyppeteer import connect, launch
    
    
    exec_platform = os.getenv('EXEC_PLATFORM')
    test_url = 'https://ecommerce-playground.lambdatest.io/'
    
    
    cookie_dict = {}
    
    
    @pytest.mark.asyncio
    @pytest.mark.order(1)
    async def test_get_cookie_info(page):
       await page.goto(test_url)
    
    
       await asyncio.sleep(1)
    
    
       # Output is a list of dictionaries
       # https://miyakogi.github.io/pyppeteer/reference.html#pyppeteer.page.Page.cookies
       cookie_dict = await page.cookies()
    
    
       await asyncio.sleep(1)
    
    
       # Let's print values of all the cookies
       for cookie in cookie_dict:
           print("Cookie Name:")
           for key, val in cookie.items():
               print(f'  {key}: {val}')
           print()
    
    
    @pytest.mark.asyncio
    @pytest.mark.order(2)
    async def test_delete_cookies(page):
       before_del_cookies = {}
       after_del_cookies = {}
    
    
       await page.goto(test_url)
    
    
       await asyncio.sleep(12)
    
    
       # https://miyakogi.github.io/pyppeteer/reference.html#pyppeteer.page.Page.cookies
       # Create/Add a new cookie
       await page.setCookie({'name': 'pyppeteer', 'value': 'v1.0.2'})
    
    
       before_del_cookies = await page.cookies()
       print("\nBefore deletion\n")
    
    
       print_cookie(before_del_cookies)
    
    
       # Delete cookies
       await page.deleteCookie({'name': 'pyppeteer'})
    
    
       after_del_cookies = await page.cookies()
       print("After deletion\n")
    
    
       print_cookie(after_del_cookies)
    
    
    def print_cookie(cookie_info):
       # Let's print values of all the cookies
       for cookie in cookie_info:
           print("Cookie Name:")
           for key, val in cookie.items():
               print(f'  {key}: {val}')
           print()

**Code Walkthrough**

After navigating to the URL under test, get the cookies by invoking the *cookies()* method of the *Page *class.

![](https://cdn-images-1.medium.com/max/3200/0*OjeLwuWxAgEqL95x.png)

The *cookies()* method returns a dictionary of *key: value* pairs. An iterative *For* loop is used for printing the value of each cookie.

![](https://cdn-images-1.medium.com/max/2000/0*hsbFxSMD9aumkcr4.png)

Next up, we set a new cookie *pyppeteer *to *v1.0.2* value with the *setCookie()* method of the *Page *class.

![](https://cdn-images-1.medium.com/max/2580/0*qYiKzC2SPcZDkBao.png)

With the new cookie set, we delete the cookie *pyppeteer* by invoking the *deleteCookie()* method, where the cookie name is passed as a parameter to the method.

![](https://cdn-images-1.medium.com/max/2040/0*mBjuoAoNzqLBpxV8.png)

At each step, cookies are printed before & after the deletion of cookies.

**Execution**

Invoke the command *make pyppeteer-cookies* on the terminal with the Chromium browser instantiated on the local machine. As seen below, the newly-added cookie *pyppeteer* is successfully set to *v1.0.2*, and the cookie added by the *setCookie()* method was successfully deleted by the *deleteCookie()* method.

![](https://cdn-images-1.medium.com/max/3200/0*folqu3zBTphg7jwT.png)

## Handling iFrames with Pyppeteer

iFrames (or inline frames) are commonly used in websites to insert a document from an entirely different domain. [Handling iFrames with Selenium](https://www.lambdatest.com/blog/handling-frames-and-iframes-selenium-c-sharp/?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=blog), Cypress, Pyppeteer, and other automation frameworks is a common test scenario for automating interactions with modern websites.

The *< iframe >* tag in HTML specifies an inline frame. Every HTML page (or document) has a top-level frame called a parent frame. The parent (or main) frame is the container of the child frames or iFrames. Shown below is an example of an iFrame from [LambdaTest Selenium Playground](https://www.lambdatest.com/selenium-playground/iframe-demo/?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=webpage).

![](https://cdn-images-1.medium.com/max/3200/0*_iEqz4IhhwuFlj-U.png)

*iFrame Example*

The methods (or coroutines) for accessing iFrames with Pyppeteer are available in the [*Frame](https://miyakogi.github.io/pyppeteer/reference.html#frame-class)* class. Here are some prominent methods for handling iFrames with Pyppeteer:

![](https://cdn-images-1.medium.com/max/2000/1*pELAkkoBd6S5XrLagnDq7A.png)

**Demonstration**

For demonstrating iFrame handling with Pyppeteer, we automate the following test scenario on Chromium installed on the local machine (*EXEC_PLATFORM* = *local*):

 1. Navigate to [Selenium Playground iFrame Demo](https://www.lambdatest.com/selenium-playground/iframe-demo/?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=webpage).

 2. Locate the *first iFrame* using the best-suited selector.

 3. Enter the message — *LambdaTest is an awesome platform!* in the textbox inside the iFrame.

 4. Capture a screenshot of the Page for verification.

 5. Locate the second *iFrame* using the best-suited selector.

 6. Scroll to the element of the search string *Playwright Testing*.

 7. Capture a screenshot of the Page for verification.

    import asyncio
    import pytest
    from pyppeteer.errors import PageError
    from urllib.parse import quote
    import os
    import sys
    from os import environ
    from pyppeteer import connect, launch
    
    
    exec_platform = os.getenv('EXEC_PLATFORM')
    
    
    test_url = 'https://www.lambdatest.com/selenium-playground/iframe-demo/'
    
    
    # Selectors of the page
    
    
    # Simple iFrame containing Editor
    loc_iframe_1 = "#iFrame1"
    # Use class name to select an element
    loc_iframe_inside_con = ".rsw-ce"
    test_message = "LambdaTest is an awesome platform!"
    loc_bold_button = "#__next > div > div.rsw-toolbar > button:nth-child(1)"
    loc_underline_button = "//span[.='𝐔']"
    
    
    # Simple iFrame containing webpage
    loc_iframe_2 = "//*[@id='iFrame2']"
    loc_playwright_testing = "//a[.='Playwright Testing']"
    
    
    async def scroll_to_element(page, element):
       # Scroll until the element is detected
       await page.evaluateHandle(
           '''async (element) => {
               if (element) {
                   element.scrollIntoView();
               }
           }'''
       )
    
    
    @pytest.mark.asyncio
    @pytest.mark.order(1)
    async def test_handling_iframe_1(page):
       await page.goto(test_url)
      
       # Set the viewport - Apple MacBook Air 13-inch
       # Reference - https://codekbyte.com/devices-viewport-sizes/
       await page.setViewport({'width': 1440, 'height': 770})
    
    
       asyncio.sleep(2)
    
    
       # Get the iframe element handle
       iframe_handle = await page.querySelector(loc_iframe_1)
    
    
       # Switch to the iframe
       iframe = await iframe_handle.contentFrame()
      
       # Locate the Search button in the iFrame
       # Perform actions inside the iframe
       elem_text_box = await iframe.querySelector(loc_iframe_inside_con)
    
    
       # Get the element inside the view
       await elem_text_box.click()
    
    
       await asyncio.sleep(1)
    
    
       await elem_text_box.click(clickCount=3)
       await page.keyboard.press('Backspace')
    
    
       # Wait for 2000 ms
       await iframe.waitFor(2000)
    
    
       await elem_text_box.type(test_message)
       await asyncio.sleep(2)
       await elem_text_box.click(clickCount=3)
    
    
       elem_underline_button = await iframe.waitForXPath(loc_underline_button)
       elem_bold_button = await iframe.querySelector(loc_bold_button)
       await asyncio.sleep(1)
       await elem_underline_button.click()
       await asyncio.sleep(1)
       await elem_bold_button.click()
       await asyncio.sleep(1)
    
    
       await elem_text_box.click()
    
    
       # Switch back to the main frame if needed
       await page.bringToFront()
    
    
       await asyncio.sleep(2)
    
    
       # Take a screenshot
       await page.screenshot({'path': 'iFrame1-screenshot.png'})
    
    
    @pytest.mark.asyncio
    @pytest.mark.order(2)
    async def test_handling_iframe_2(page):
       await page.goto(test_url)
      
       # Set the viewport - Apple MacBook Air 13-inch
       # Reference - https://codekbyte.com/devices-viewport-sizes/
       await page.setViewport({'width': 1440, 'height': 770})
    
    
       asyncio.sleep(2)
    
    
       # Get the iframe element handle
       iframe_handle = await page.waitForXPath(loc_iframe_2)
    
    
       # Switch to the iframe
       iframe = await iframe_handle.contentFrame()
      
       # Locate the Search button in the iFrame
       # Perform actions inside the iframe
       elem_search = await iframe.waitForXPath(loc_playwright_testing)
    
    
       # Get the element inside the view
       await scroll_to_element(iframe, elem_search)
       await elem_search.click()
    
    
       await asyncio.sleep(2)
    
    
       # Take a screenshot
       await page.screenshot({'path': 'iFrame2-screenshot.png'})

**Code Walkthrough**

After navigating to the i[Frame Demo](https://www.lambdatest.com/selenium-playground/iframe-demo/?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=webpage) page, we first locate the iFrame using the ID locator (i.e., *iFrame1*). The *querySelector() *method is used to check the presence of the *iFrame *selector.

![](https://cdn-images-1.medium.com/max/3200/0*klKvlvQxbbt9OoZn.png)

![](https://cdn-images-1.medium.com/max/2312/0*SdB9Rbxr_QgbgIJr.png)

Next, we use the *contentFrame()* method on the located element to get the content frame for the element handle.

![](https://cdn-images-1.medium.com/max/2000/0*NacmAahD3EtvOKZy.png)

Now that we have switched to the iFrame, we locate the text-area element using the CSS Selector (i.e., *.rsw-ce*). To simplify the process of locating elements, I have used the [POM Builder plugin](https://pombuilder.com/) on Arc browser.

![](https://cdn-images-1.medium.com/max/3200/0*O_InkMvIteG_l5Yr.png)

![](https://cdn-images-1.medium.com/max/2680/0*BzSQSAwqlMVT5VrN.png)

Since the text-area contains the default text, we first select the text by setting the *clickCount* to 3 in *click()* — a method used for stimulating mouse click on a DOM element.

![](https://cdn-images-1.medium.com/max/3200/0*V6r81XxasPRZMP7e.png)

![](https://cdn-images-1.medium.com/max/2000/0*aRLF_k2UbZUYmkwA.png)

The content in the text-area is deleted by pressing the Backspace key using the [*press()](https://miyakogi.github.io/pyppeteer/reference.html#keyboard-class)* method of the *Keyboard* class.

![](https://cdn-images-1.medium.com/max/2000/0*9LkvtMsAMqOHKSEX.png)

Enter the required text *LambdaTest is an awesome platform!* in the text area by invoking the *type()* method on the text-area element. Since we also want to Bold & Italicise the text, we again select the entire text by setting the *clickcount* to 3.

![](https://cdn-images-1.medium.com/max/3200/0*JD6GuG3mFohgv6t8.png)

![](https://cdn-images-1.medium.com/max/2000/0*7KQkyi5z-QnwoYq4.png)

Since we want to underline the text, we locate the element using the XPath locator. Once the element (i.e., *loc_underline_button*) is located, a click operation is performed to underline the text in the text area.

![](https://cdn-images-1.medium.com/max/3200/0*zzEmGc_L52uRnnNl.png)

![](https://cdn-images-1.medium.com/max/2884/0*axVMZBA1Xf4ZKOhE.png)

Similarly, we locate the element (to bold the text) using the CSS Selector. Once the element (i.e., *loc_bold_button*) is located, a click operation is performed to bold the text in the text area.

![](https://cdn-images-1.medium.com/max/3200/0*BUlZbu1OJQOG-VCL.png)

![](https://cdn-images-1.medium.com/max/2952/0*3l_CVhZccQFQTQDw.png)

We switch back to the main window (or tab) by invoking the *bringToFront()* method. Post this, we take a full-page screenshot (*iFrame1-screenshot.png*) using the *screenshot()* method that was discussed in the earlier section.

![](https://cdn-images-1.medium.com/max/2412/0*UL2tTkYTxc4CHBPN.png)

For the **second test scenario**, we first locate the iFrame element using the XPath locator (i.e., *iFrame2*).

![](https://cdn-images-1.medium.com/max/3200/0*IR0r-hpIlQKpH5M6.png)

![](https://cdn-images-1.medium.com/max/2276/0*mdgA5vNvX2B_7uYK.png)

Switch to the iFrame by invoking the *contentFrame()* method on the element located in the previous step.

![](https://cdn-images-1.medium.com/max/2000/0*r-jbAc8BjZkWfJMH.png)

As seen below, two occurrences of the XPath *//a[.=’Playwright Testing’]* exists.

![](https://cdn-images-1.medium.com/max/3200/0*I5Xil8NU5oxPEKqT.png)

However, the element from the iFrame is selected since the *waitForXPath* is invoked on the iFrame element.

![](https://cdn-images-1.medium.com/max/3200/0*_Ajq-TuTRKpEKOyF.png)

![](https://cdn-images-1.medium.com/max/2612/0*CJW4nJNd4RybkxJs.png)

Now that the element within the iFrame is located, we scroll to the element by invoking the *scroll_to_element()* user-defined method.

![](https://cdn-images-1.medium.com/max/2000/0*5xMlMYrH3dffpbAW.png)

A *click()* operation is performed so that it navigates to the [Playwright testing](https://www.lambdatest.com/playwright-testing) documentation page.

![](https://cdn-images-1.medium.com/max/2000/0*LEyj6hZBrBJzFjEU.png)

Finally, we capture a screenshot of the entire page via the *screenshot()* method.

![](https://cdn-images-1.medium.com/max/2412/0*2QKcVobYUYVsRElw.png)

**Execution**

Invoke the command make *pyppeteer-iframe* on the terminal with the Chromium browser instantiated on the local machine. As seen below, the test execution is successful.

![](https://cdn-images-1.medium.com/max/3200/0*C72UURcR-GCx-6bK.png)

The iFrame screenshots are also successfully generated at the end of the test execution.

![](https://cdn-images-1.medium.com/max/2000/0*_XCsozPBWxfqb9PG.png)

![](https://cdn-images-1.medium.com/max/2868/0*sohoIzUfTYPy1dK2.png)

*iFrame grab of test scenario — 1*

![](https://cdn-images-1.medium.com/max/3200/0*n1HKSrEKFec28ny_.png)

*iFrame grab of test scenario — 2*

With this, you are also set to tackle iFrame-related challenges with the Pyppeteer library!

## Handling alerts and pop-ups with Pyppeteer

Alerts and Pop-ups (or pop-ups) are used extensively in websites for gathering user information, seeking confirmation from end-users, and more. Simple alerts, information alerts, and prompt alerts are some of the most common types of alerts used across websites. You can learn more about handling alerts through this blog on [handling alerts in windows using Selenium](https://www.lambdatest.com/blog/selenium-c-tutorial-handling-alert-windows/#Alerts?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=blog).

![](https://cdn-images-1.medium.com/max/3200/0*rSBQk8UjqA9i7a0x.png)

*Examples of Alerts*

All websites show a ‘Cookie Consent’ popup when you visit the same for the first time or from an incognito window. Below is an Exit popup on the LambdaTest website that prompts the user to try out the platform.

![](https://cdn-images-1.medium.com/max/3200/0*quZL96ju8ehXmatR.png)

Since the guiding principles of alerts and popups remain unchanged, we recommend you look at the video below, which deep dives into its essentials.
{% youtube C4YT0BT_wbQ  %}
As far as the Pyppeteer library is concerned, the *dialog* class provides methods that help handle popups and alerts. Dialog objects are dispatched by page via the dialog event.

The *dismiss()* method is below the dialog box. On similar lines, the *accept(promptText: str = ‘’)* method lets you accept the dialog. You can also pass a *promptText* (format: string), which is the text to enter in the prompt.

**Demonstration**

For demonstrating the usage of dialog boxes (or alerts & popups) with Pyppeteer, we would be automating interactions with the different alerts in [LambdaTest Playground — Alert Demo](https://www.lambdatest.com/selenium-playground/javascript-alert-box-demo?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=webpage).

![](https://cdn-images-1.medium.com/max/3200/0*s9scvIXAbTsJ1tzT.png)

The scenarios are automated with Chromium installed on a local machine (*EXEC_PLATFORM = local*). However, the same tests would also work seamlessly with Pyppeteer on LambdaTest cloud (*EXEC_PLATFORM = cloud*)

    import asyncio
    import pytest
    from pyppeteer.errors import PageError
    from urllib.parse import quote
    import json
    import os
    import sys
    from os import environ
    from pyppeteer import connect, launch
    
    
    # Documentation Link
    # https://miyakogi.github.io/pyppeteer/reference.html#dialog-class
    
    
    # Interesting reference question
    # https://stackoverflow.com/questions/75622322/
    # why-dialog-popup-alert-doesnt-dismiss-as-they-claim-by-using-pyppeteer-class
    
    
    # Scenario Handling Alerts
    dialog_test_url = 'https://www.lambdatest.com/selenium-playground/javascript-alert-box-demo'
    
    
    # Locators for different elements
    # 0 - JS Alert
    # 1 - Confirm Box
    # 2 - Prompt Box
    
    
    loc_alert_arr =  ['.my-30',
                   '.py-20.ml-10 .btn',
                   'section:nth-of-type(3) div:nth-of-type(3) .btn']
    
    
    test_message = 'LambdaTest is a great platform!'
    
    
    # Event event listener for handling JS alert dialog box
    def handle_js_dialog_box(dialog):
       asyncio.ensure_future(dialog.accept())
       print(f"Dialog message: {dialog.message}")
    
    
    # Event event listener for handling confirm dialog box
    def handle_confirm_accept_dialog_box(dialog):
       asyncio.ensure_future(dialog.accept())
       print(f"Dialog message: {dialog.message}")
    
    
    # Event event listener for handling confirm dialog box
    def handle_confirm_dismiss_dialog_box(dialog):
       asyncio.ensure_future(dialog.dismiss())
       print(f"Dialog message: {dialog.message}")
    
    
    # Event event listener for handling prompt dialog box
    def handle_confirm_prompt_dialog_box(dialog):
       asyncio.ensure_future(dialog.accept(test_message))
       print(f"Dialog message: {dialog.message}")
    
    
    @pytest.mark.asyncio
    @pytest.mark.order(1)
    async def test_handling_js_alerts(page):
       await page.goto(dialog_test_url)
    
    
       # Can be changed with non-blocking sleep
       await asyncio.sleep(1)
    
    
       page.on('dialog', handle_js_dialog_box)
    
    
       elem_alert = await page.querySelector(loc_alert_arr[0])
    
    
       # Click on the located element
       await elem_alert.click()
    
    
       # Wait for the event loop to process events
       await asyncio.sleep(2)
    
    
    @pytest.mark.asyncio
    @pytest.mark.order(2)
    async def test_handling_confirm_accept_alerts(page):
       await page.goto(dialog_test_url)
    
    
       # Can be changed with non-blocking sleep
       await asyncio.sleep(1)
    
    
       page.on('dialog', handle_confirm_accept_dialog_box)
    
    
       # Confirm Alert
       elem_alert = await page.querySelector(loc_alert_arr[1])
    
    
       # Click on the located element
       await elem_alert.click()
    
    
       # Wait for the event loop to process events
       await asyncio.sleep(2)
    
    
    @pytest.mark.asyncio
    @pytest.mark.order(3)
    async def test_handling_confirm_dismiss_alerts(page):
       await page.goto(dialog_test_url)
    
    
       # Can be changed with non-blocking sleep
       await asyncio.sleep(1)
    
    
       page.on('dialog', handle_confirm_dismiss_dialog_box)
    
    
       await asyncio.sleep(2)
    
    
       # Dismiss Alert
       elem_alert = await page.querySelector(loc_alert_arr[1])
    
    
       # Click on the located element
       await elem_alert.click()
    
    
       # Wait for the event loop to process events
       await asyncio.sleep(2)
    
    
    @pytest.mark.asyncio
    @pytest.mark.order(4)
    async def test_handling_prompt_alerts(page):
       await page.goto(dialog_test_url)
    
    
       # Can be changed with non-blocking sleep
       await asyncio.sleep(1)
    
    
       page.on('dialog', handle_confirm_prompt_dialog_box)
    
    
       await asyncio.sleep(1)
    
    
       # Prompt Alert
       elem_alert = await page.querySelector(loc_alert_arr[2])
    
    
       # Click on the located element
       await elem_alert.click()
    
    
       page.on('dialog', handle_confirm_dismiss_dialog_box)
    
    
       await asyncio.sleep(2)

**Code Walkthrough**

Four separate event listeners are created for automating interactions with the dialog boxes:

![](https://cdn-images-1.medium.com/max/2000/1*VzUa6pVYPoP8Iu0LTGBn5w.png)

Let’s deep dive into different test scenarios bifurcated based on the interactions they have with the respective alert box.

* **Test Method: test_handling_js_alerts()**

After navigating to the [LambdaTest Alert Demo](https://www.lambdatest.com/selenium-playground/javascript-alert-box-demo?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=webpage) website, the *page.on() *method in Pyppeteer is used for attaching event listeners to the current page. The syntax of the *page.on()* method is below:

    await page.on(event, callback)

– *event:* Event name (string) to listen for.
– *callback:* Callback function to be executed when the specified event fires.

Since dialog objects are dispatched by the page via the dialog event, the method listens to that particular event. The callback function in the current scenario is *handle_js_dialog_box(dialog)*.



![](https://cdn-images-1.medium.com/max/2000/0*hIrz-LjzSUImVAIy.png)

Here is the implementation of the callback function:

![](https://cdn-images-1.medium.com/max/2344/0*QkMXWIfQXY8RLyZN.png)

The asyncio.ensure_future() method of the asyncio module is used for background execution of the accept() method. In short, when the callback function is fired, it accepts the dialog box. The message() method of the dialog class prints the message attached to the current dialog box.

Next, locate the Click Me button associated with JavaScript Alerts on the page.

![](https://cdn-images-1.medium.com/max/3200/0*VGvW_v8FVphw_SQL.png)

![](https://cdn-images-1.medium.com/max/3200/0*pmYQM0LroPeBUsTi.png)

Since the element is located using the CSS Selector, we use the querySelector() method to get the element (i.e. elem_alert) matching that locator.

![](https://cdn-images-1.medium.com/max/2076/0*zyRX2o_dpJ4Kpd4D.png)

Finally, the *click()* method of the button element is invoked so that it opens up the dialog box.

![](https://cdn-images-1.medium.com/max/2000/0*qLi21bsakCmT4RU7.png)

With this, the dialog event is also fired, after which the respective callback function [i.e., handle_js_dialog_box(dialog)] is executed non-blocking.

**Test Method: test_handling_confirm_accept_alerts()**

Like before, the page.on() method in Pyppeteer is used for attaching event listeners to the current page.

![](https://cdn-images-1.medium.com/max/2208/0*osQMpHrSpdM7CLoI.png)

Here is the implementation of *handle_confirm_accept_dialog_box()*:

![](https://cdn-images-1.medium.com/max/2312/0*J3rr9SW7wg6uBDKf.png)

Next, locate the **Click Me** button associated with the *Confirm box* on the page.

![](https://cdn-images-1.medium.com/max/3200/0*iMeMP2J75uy2Dckf.png)

Like earlier, the *click()* method of the button element is invoked; after which the respective callback function [i.e., *test_handling_confirm_accept_alerts(dialog)*] is executed in a non-blocking manner.

![](https://cdn-images-1.medium.com/max/2444/0*lHdOOqqyTRnKx4sQ.png)

**Test Method: test_handling_confirm_dismiss_alerts()**

Like before, the *page.on()* method in Pyppeteer is used for attaching event listeners to the current page.

![](https://cdn-images-1.medium.com/max/2244/0*N8Rz-zp95p7QvG4T.png)

Since the dialog box will be dismissed, the *dismiss()* method of the *Dialog* class is invoked in the callback function.

![](https://cdn-images-1.medium.com/max/2312/0*5XGrKQ9USYffspov.png)

Next, locate the **Click Me** button associated with the *Confirm box* on the page.



![](https://cdn-images-1.medium.com/max/3200/0*_KaNNDfe7M4blmep.png)

Like earlier, the *click()* method of the button element is invoked, after which the respective callback function [i.e., *test_handling_confirm_dimiss_alerts(dialog)*] is executed in a non-blocking manner.

![](https://cdn-images-1.medium.com/max/2444/0*Isx1eB5u3c8T9z5T.png)

**Test Method: test_handling_prompt_alerts()**

Like before, the *page.on()* method in Pyppeteer is used for attaching event listeners to the current page.

![](https://cdn-images-1.medium.com/max/2208/0*bsTBeL-0GztMIUG0.png)

Since there is a text message that needs to be entered in this prompt box, the respective message (i.e., *test_message*) is passed to the *accept()* method.

![](https://cdn-images-1.medium.com/max/3200/0*fzj2fm9a3-bkt4I9.png)

![](https://cdn-images-1.medium.com/max/2312/0*g1rFgj1QNTJqFmZQ.png)

The **Click Me** button associated with the *Prompt box* uses the CSS Selector property. The respective callback function [i.e., *handle_confirm_dismiss_dialog_box(dialog)*] is executed once the dialog box appears after the click operation on the button element.

![](https://cdn-images-1.medium.com/max/3200/0*UQ-02HdgG-cTD4jq.png)



![](https://cdn-images-1.medium.com/max/3200/0*0EBDLo-MFqEvfDWi.png)

**Execution**

Invoke the command make *pyppeteer-dialog-box* on the terminal with the Chromium browser instantiated on the local machine. As seen below, the test execution is successful.

![](https://cdn-images-1.medium.com/max/3200/0*eNUcvhDW88d4BfJd.png)

Shown below are the execution snapshots that indicate that the interactions with the dialog (or alert) boxes were successful.

![](https://cdn-images-1.medium.com/max/3200/0*WuVuALVFt4vJjHR0.png)

*Execution: Handling of Alert box*

![](https://cdn-images-1.medium.com/max/3200/0*h3F1-FHqpqdDVWQv.png)

***Execution: Handling of Confirm box (Accept)***

![](https://cdn-images-1.medium.com/max/3200/0*-weWZSRE5x0cbYml.png)

***Execution: Handling of Confirm box (Dismiss)***

![](https://cdn-images-1.medium.com/max/3200/0*5MlrNNMAPWBokhZp.png)

***Execution: Handling of Prompt box (promptText)***

Though alerts that open up a *dialog* box, popups (or pop-ups) are mere window(s) that appear on top of the existing content of the page. Automating interactions with popups involves locating the window using appropriate locators (i.e., *CSS Selector, XPath*.

After that, we use appropriate coroutines (i.e., *waitForXPath*, *waitForSelector*) to detect the presence of the popup element. Once detected, you can interact with it by clicking buttons or performing other actions.
>  **Learn to implement Explicit Wait command in [Selenium WebDriverWait](https://www.lambdatest.com/blog/selenium-webdriverwait/?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=blog) class in Java. Improve web testing by precisely waiting for elements!**

## Handling dynamic content with Pyppeteer

Most modern-age websites (including SPAs) leverage JavaScript for loading and updating content on the page. AJAX (Asynchronous JavaScript and XML) retrieves data from the server and updates relevant sections (or parts) on the page. This means that the content on the page is loaded with a reading of the entire page.

Not only content, the WebElements can also be dynamic. These WebElements are invoked at run-time and dynamically added to the page. It is important to note that there is a difference between dynamic elements and hidden elements (*display: none, visibility: hidden*).

A WebElement can never be hidden as well as dynamic. For instance, a web page could have a *collapsed* dropdown element (or menu) that becomes visible dynamically based on the interactions with the element. [Handling dynamic dropdown elements with Selenium](https://www.lambdatest.com/blog/handling-dropdowns-in-selenium-webdriver-java/), Pyppeteer, and other frameworks (or libraries) needs to be in an automation engineer’s arsenal!

For instance, YouTube is a prominent website that leverages dynamic loading of content to showcase video content to its end-users. As seen below, content (including videos, and meta-data) on the [LambdaTest YouTube channel ](https://www.youtube.com/@LambdaTest/videos?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=youtube)is loaded dynamically when the user performs a scroll operation.

![](https://cdn-images-1.medium.com/max/3200/0*VsM_f_IkUFuwyqdE.png)

*YouTube: Dynamic Content Example*

![](https://cdn-images-1.medium.com/max/3200/0*mFa3hp_j98mLoumS.png)

*YouTube: Dynamic Content Example*

For a demonstration of dynamic content handling with Pyppeteer, we would be automating interactions with WebElements on the following websites:

* LambdaTest eCommerce Playground

* Scraping Club Infinite Scroll

Along with dynamically loaded content, the LambdaTest eCommerce Playground also [lazy loaded images for better website performance](https://www.lambdatest.com/blog/how-to-lazy-load-images-javascript/?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=blog).

![](https://cdn-images-1.medium.com/max/3200/0*yI8vm2AL7CC0elmR.png)

*Lazy Loading of images*

**Demonstration**

For demonstrating dynamic content handling with Pyppeteer, we automate the following test scenario on Chrome installed on a local machine (*EXEC_PLATFORM = local*) or LambdaTest cloud grid (*EXEC_PLATFORM = cloud*)

**Test Scenario 1**

 1. Navigate to [LambdaTest eCommerce Playground](https://ecommerce-playground.lambdatest.io/).

 2. Scroll into the *Nikon D300* Product to be added in the Cart.

 3. Assert if the target URL does not match the [expected URL](https://ecommerce-playground.lambdatest.io/index.php?route=product/product&product_id=63).

**Test Scenario 2**

 1. Navigate to [LambdaTest eCommerce Playground](https://ecommerce-playground.lambdatest.io/).

 2. Click on the *third tab* in the carousel.

 3. Assert if the target URL does not match the [expected URL](https://ecommerce-playground.lambdatest.io/index.php?route=product/product&product_id=30).

**Test Scenario 3**

 1. Navigate to [Scraping Club Infinite Scroll](https://scrapingclub.com/exercise/list_infinite_scroll/).

 2. Click on the *Short Chiffon Dress* product.

 3. Assert if the target URL does not match the [expected URL](https://scrapingclub.com/exercise/list_basic_detail/93926-C/).

**Test Scenario 4**

 1. Navigate to [Scraping Club Infinite Scroll](https://scrapingclub.com/exercise/list_infinite_scroll/)

 2. Scroll to the *Top with Tie* product.

 3. Assert if the target URL does not match the [expected URL](https://scrapingclub.com/exercise/list_basic_detail/94967-A/).

Though all four test scenarios appear similar, the difference lies in how the content (including images) is presented on the website. In one of the scenarios, we would also be demonstrating scroll operations that come in handy when automating interactions with dynamically loaded content.

**Implementation**

Here is the implementation of all the test scenarios. We will be dissecting the code in further sections of the blog!

    import asyncio
    import pytest
    from pyppeteer.errors import PageError
    from urllib.parse import quote
    import json
    import os
    import sys
    from os import environ
    from pyppeteer import connect, launch
    
    
    exec_platform = os.getenv('EXEC_PLATFORM')
    
    
    # Get username and access key of the LambdaTest Platform
    username = environ.get('LT_USERNAME', None)
    access_key = environ.get('LT_ACCESS_KEY', None)
    
    
    test1_url = 'https://ecommerce-playground.lambdatest.io/'
    test2_url = 'https://scrapingclub.com/exercise/list_infinite_scroll/'
    
    
    # Usecase - 1
    # loc_ecomm_1 = ".order-1.col-lg-6 div:nth-of-type(1) > div:nth-of-type(1) > div:nth-of-type(1) > div:nth-of-type(1) > div:nth-of-type(1) div:nth-of-type(1) > img:nth-of-type(1)"
    loc_ecomm_1 = "[aria-label='1 / 2'] div:nth-of-type(1) > [alt='Nikon D300']"
    target_url_1 = "https://ecommerce-playground.lambdatest.io/index.php?route=product/product&product_id=63"
    
    
    # Usecase - 2 (Click on e-commerce sliding banner)
    loc_ecomm_2 = "[alt='Canon DSLR camera']"
    target_url_2 = "https://ecommerce-playground.lambdatest.io/index.php?route=product/product&product_id=30"
    
    
    # Usecase - 3 Automating interactions on https://scrapingclub.com/exercise/list_infinite_scroll/
    loc_infinite_src_prod1 = ".grid .p-4 [href='/exercise/list_basic_detail/93926-C/']"
    target_url_3 = "https://scrapingclub.com/exercise/list_basic_detail/93926-C/"
    
    
    # Usecase - 4 Automating interactions on https://scrapingclub.com/exercise/list_infinite_scroll/
    # when the images are lazy loaded
    loc_infinite_src_prod2 = "div:nth-of-type(31) > .p-4 [href='/exercise/list_basic_detail/94967-A/']"
    target_url_4 = "https://scrapingclub.com/exercise/list_basic_detail/94967-A/"
    
    
    # Set timeout in ms
    timeOut = 60000
    
    
    async def scroll_to_element(page, selector):
       # Scroll until the element is detected
       await page.evaluateHandle(
           '''async (selector) => {
               const element = document.querySelector(selector);
               if (element) {
                   element.scrollIntoView();
               }
           }''',
           selector
       )
    
    
       return selector
    
    
    async def scroll_carousel(page, scr_count):
       for scr in range(1, scr_count):
           elem_next_button = "#mz-carousel-213240 > ul li:nth-child(" + str(scr) + ")"
           await asyncio.sleep(1)
           elem_next_button = await page.querySelector(elem_next_button)
           await elem_next_button.click()
    
    
    # Replica of https://github.com/hjsblogger/web-scraping-with-python/blob/
    # main/tests/beautiful-soup/test_infinite_scraping.py#L67C5-L80C18
    
    
    async def scroll_end_of_page(page):
       start_height = await page.evaluate('document.documentElement.scrollHeight')
    
    
       while True:
           # Scroll to the bottom of the page
           await page.evaluate(f'window.scrollTo(0, {start_height})')
    
    
           # Wait for the content to load
           await asyncio.sleep(1)
    
    
           # Get the new scroll height
           scroll_height = await page.evaluate('document.documentElement.scrollHeight')
    
    
           if scroll_height == start_height:
               # If heights are the same, we reached the end of the page
               break
    
    
           # Add an additional wait
           await asyncio.sleep(2)
    
    
           start_height = scroll_height
    
    
       # Additional wait after scrolling
       await asyncio.sleep(2)
    
    
    @pytest.mark.asyncio
    @pytest.mark.order(1)
    async def test_lazy_load_ecomm_1(page):
    
    
       # The time out can be set using the setDefaultNavigationTimeout
       # It is primarily used for overriding the default page timeout of 30 seconds
       page.setDefaultNavigationTimeout(timeOut)
       await page.goto(test1_url,
           {'waitUntil': 'load', 'timeout': timeOut})
      
       # Set the viewport - Apple MacBook Air 13-inch
       # Reference - https://codekbyte.com/devices-viewport-sizes/
       # await page.setViewport({'width': 1440, 'height': 770})
    
    
       await asyncio.sleep(2)
    
    
       if exec_platform == 'local':
           # Scroll until the element is detected
           elem_button = await scroll_to_element(page, loc_ecomm_1)
    
    
           # await page.click(elem_button)
    
    
           # Wait until the page is loaded
           # https://miyakogi.github.io/pyppeteer/reference.html#pyppeteer.page.Page.waitForNavigation
           navigationPromise = asyncio.ensure_future(page.waitForNavigation())
           await page.click(elem_button)
           await navigationPromise
       elif exec_platform == 'cloud':
           elem_button = await page.waitForSelector(loc_ecomm_1, {'visible': True})
           await asyncio.gather(
               elem_button.click(),
               page.waitForNavigation({'waitUntil': 'networkidle2', 'timeout': 30000}),
           )
      
       # Assert if required, since the test is a simple one; we leave as is :D
       current_url = page.url
       print('Current URL is: ' + current_url)
    
    
       try:
           assert current_url == target_url_1
           print("Test Success: Product checkout successful")
       except PageError as e:
           print("Test Failure: Could not checkout Product")
           print("Error Code" + str(e))
    
    
    @pytest.mark.asyncio
    @pytest.mark.order(2)
    async def test_lazy_load_ecomm_2(page):
       carousel_len = 4
    
    
       # The time out can be set using the setDefaultNavigationTimeout
       # It is primarily used for overriding the default page timeout of 30 seconds
       page.setDefaultNavigationTimeout(timeOut)
       await page.goto(test1_url,
           {'waitUntil': 'load', 'timeout': timeOut})
      
       # Set the viewport - Apple MacBook Air 13-inch
       # Reference - https://codekbyte.com/devices-viewport-sizes/
       # await page.setViewport({'width': 1440, 'height': 770})
    
    
       await asyncio.sleep(2)
    
    
       # Approach 1: Directly click on the third button on the carousel
       # elem_carousel_banner = await page.querySelector("#mz-carousel-213240 > ul li:nth-child(3)")
       # await asyncio.sleep(1)
       # await elem_carousel_banner.click()
    
    
       # Approach 2 (Only for demo): Serially click on every button on carousel
       await scroll_carousel(page, carousel_len)
      
       await asyncio.sleep(1)
    
    
       # elem_prod_1 = await page.querySelector(loc_ecomm_2)
       elem_prod_1 = await page.waitForSelector(loc_ecomm_2, {'visible': True})
       await asyncio.gather(
           elem_prod_1.click(),
           page.waitForNavigation({'waitUntil': 'networkidle2', 'timeout': 60000}),
       )
      
       # Assert if required, since the test is a simple one; we leave as is :D
       current_url = page.url
       print('Current URL is: ' + current_url)
    
    
       try:
           assert current_url == target_url_2
           print("Test Success: Product checkout successful")
       except PageError as e:
           print("Test Failure: Could not checkout Product")
           print("Error Code" + str(e))
    
    
    
    
    @pytest.mark.asyncio
    @pytest.mark.order(3)
    async def test_lazy_load_infinite_scroll_1(page):
       # The time out can be set using the setDefaultNavigationTimeout
       # It is primarily used for overriding the default page timeout of 30 seconds
       page.setDefaultNavigationTimeout(timeOut)
       await page.goto(test2_url,
           {'waitUntil': 'load', 'timeout': timeOut})
      
       # Set the viewport - Apple MacBook Air 13-inch
       # Reference - https://codekbyte.com/devices-viewport-sizes/
       # await page.setViewport({'width': 1440, 'height': 770})
    
    
       await asyncio.sleep(1)
    
    
       elem_prod1 = await page.querySelector(loc_infinite_src_prod1)
    
    
       await asyncio.gather(
           elem_prod1.click(),
           page.waitForNavigation({'waitUntil': 'networkidle2', 'timeout': 60000}),
       )
    
    
       # await asyncio.sleep(1)
       # await elem_carousel_banner.click()
    
    
       # elem_button = scroll_to_element(page, loc_infinite_src_prod1)
       # print(elem_button)
       # await asyncio.sleep(2)
       # await elem_button.click()
      
       # Assert if required, since the test is a simple one; we leave as is :D
       current_url = page.url
       print('Current URL is: ' + current_url)
    
    
       try:
           assert current_url == target_url_3
           print("Test Success: Product checkout successful")
       except PageError as e:
           print("Test Failure: Could not checkout Product")
           print("Error Code" + str(e))
     
    @pytest.mark.asyncio
    @pytest.mark.order(4)
    async def test_lazy_load_infinite_scroll_2(page):
       # The time out can be set using the setDefaultNavigationTimeout
       # It is primarily used for overriding the default page timeout of 30 seconds
       page.setDefaultNavigationTimeout(timeOut)
    
    
       # Tested navigation using LambdaTest YouTube channel
    
    
       # await page.goto("https://www.youtube.com/@LambdaTest/videos",
       await page.goto(test2_url,
           {'waitUntil': 'load', 'timeout': timeOut})
      
       # Set the viewport - Apple MacBook Air 13-inch
       # Reference - https://codekbyte.com/devices-viewport-sizes/
       # await page.setViewport({'width': 1440, 'height': 770})
    
    
       await asyncio.sleep(1)
    
    
       await scroll_end_of_page(page)
    
    
       await page.evaluate('window.scrollTo(0, 0)')
    
    
       await asyncio.sleep(1)
    
    
       # elem_prod = await page.querySelector(loc_infinite_src_prod2)
    
    
       # asyncio.sleep(1)
    
    
       # await asyncio.gather(
       #     elem_prod.click(),
       #     page.waitForNavigation({'waitUntil': 'load', 'timeout': 60000}),
       # )
    
    
       elem_button = await scroll_to_element(page, loc_infinite_src_prod2)
    
    
       await asyncio.sleep(1)
    
    
       # await page.click(elem_button)
    
    
       await asyncio.gather(
           page.click(elem_button),
           page.waitForNavigation({'waitUntil': 'networkidle2', 'timeout': 60000}),
       )
    
    
       # Assert if required, since the test is a simple one; we leave as is :D
       current_url = page.url
       print('Current URL is: ' + current_url)
    
    
       try:
           assert current_url == target_url_4
           print("Test Success: Product checkout successful")
       except PageError as e:
           print("Test Failure: Could not checkout Product")
           print("Error Code" + str(e))

**Code Walkthrough (Test Scenario — 1)**

In the test [*test_lazy_load_ecomm_1()](https://github.com/hjsblogger/web-automation-with-pyppeteer/blob/main/tests/handling-dynamic-content/test_page_class_lazy_loaded_content.py#L108)*, we first navigate to the test URL using the *goto()* method. The navigation timeout is set to 60000 ms, overriding the default timeout of 30000 ms.

![](https://cdn-images-1.medium.com/max/3052/0*HGhEurXaVTUf6G8g.png)

Once the page is loaded, we use the *querySelector()* method to get the element matching the specified CSS Selector.

![](https://cdn-images-1.medium.com/max/3200/0*Wh9Cy9N8dDJVyyug.png)

Once the element is located, the *scrollIntoView()* method of the Element’s interface (in JavaScript) is used to scroll into view of the specified element.

![](https://cdn-images-1.medium.com/max/2548/0*Rs6EQx8xgySyvoCB.png)

![](https://cdn-images-1.medium.com/max/2376/0*e7C4nTJ6wHKBZDb7.png)

The *asyncio.ensure_future()* method of the *asyncio* module is used to convert the promise (i.e., *navigationPromise*) returned by *waitForNavigation()* into an asynchronous task.

![](https://cdn-images-1.medium.com/max/3200/0*XtR42ETmiyxnFqMN.png)

The *click()* method clicks on the element in the previous step. Finally, *await navigationPromise* waits for the completion of the navigation operation (i.e., the page would have loaded completely by the end of this step).

![](https://cdn-images-1.medium.com/max/3200/0*kRktGr2XFfjxIKiv.png)

![](https://cdn-images-1.medium.com/max/2000/0*wz-sZzg7i5xqhvyr.png)

Assert if the post-navigation URL does not match with the target URL.

![](https://cdn-images-1.medium.com/max/2884/0*s_tDWDgfF4NVdFvs.png)

**Code Walkthrough (Test Scenario — 2)**

After navigating to the test URL, we invoke the user-defined *scroll_carousel()* method that takes two input parameters — *page instance & carousel count* (i.e., *carousel_len = 4*).

![](https://cdn-images-1.medium.com/max/2916/0*-sRHgzw4e4L2ef-F.png)

As shown below, the carousel contains 3 items, our target element is located in the final scroll.

![](https://cdn-images-1.medium.com/max/3200/0*hm4fXSU9NcZjHbHL.png)

Here is the implementation of the *scroll_carousel()* method:

![](https://cdn-images-1.medium.com/max/3200/0*dO_5oM_2PxLUQqZb.png)

The end count of the carousel is set to 4. Hence, the for loop executes in the *1..4* range. The **next **button on carousel is located using the CSS Selector:

* **Button 1:** #mz-carousel-213240 > ul li:nth-child(1)

* **Button 2:** #mz-carousel-213240 > ul li:nth-child(2)

* **Button 3:** #mz-carousel-213240 > ul li:nth-child(3)

![](https://cdn-images-1.medium.com/max/3200/0*gtOyp4jfGXD2RuFZ.png)

At each step, the *click()* method on the located button element invokes the click operation. Since the loop runs from *1* through *3*, the final click is performed on the last item (or button) in the carousel.

![](https://cdn-images-1.medium.com/max/3200/0*oJc8bQmSR_x-g2zi.png)

Now that we are at the target location in the carousel, the element is located via the CSS Selector. The option *visible* is set to *True* so that the *waitForSelector()*
method waits for the located element to be visible on the page.

![](https://cdn-images-1.medium.com/max/3200/0*nF56MvkFA7Vx7y9A.png)

![](https://cdn-images-1.medium.com/max/2916/0*28zDYZ1TAE2KMrf8.png)

Finally, the *asyncio.gather()* of the *asyncio* library is called for executing the *click()* method on the located element and waiting for the navigation to complete [i.e., *waitForNavigation()* ].

![](https://cdn-images-1.medium.com/max/3052/0*ljNAy6M0z55WZucZ.png)

Like before, an *assert* is raised if the current URL does not match with the [target URL](https://ecommerce-playground.lambdatest.io/index.php?route=product/product&product_id=63).

![](https://cdn-images-1.medium.com/max/2884/0*iBguYFH7g-HyEnii.png)

**Code Walkthrough (Test Scenario — 3)**

After navigating to the [test URL](https://scrapingclub.com/exercise/list_infinite_scroll/), we locate the element via the CSS property — .*grid .p-4 [href=’/exercise/list_basic_detail/93926-C/’]*. The *querySelector() *method returns the element (*i.e., elem_prod1*) matching the selector.

![](https://cdn-images-1.medium.com/max/3200/0*FdgUsgkb4eDRdUb0.png)

The *click()* method on *elem_prod1* performs a click on the element. As always, we wait for a maximum of 60000 ms for the navigation to complete.

![](https://cdn-images-1.medium.com/max/3052/0*ka1O0HRBRWqfbjuK.png)

**Code Walkthrough (Test Scenario — 4)**

In this test scenario, the target image (or element) is lazy-loaded and available only after 2~3 scrolls.

![](https://cdn-images-1.medium.com/max/3200/0*ynZrHVs53fAVU4-U.png)

After the test URL is loaded, the next step is to ensure that all the content on the page has loaded completely. For this, we perform a vertical scroll operation till the end of the page is reached. The logic of [scrolling a page in Selenium](https://www.lambdatest.com/blog/scroll-a-webpage-in-selenium-using-java/?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=blog) is very similar to that used in Pyppeteer!

*For vertical scrolling, we have created a user-defined method named [scroll_end_of_page()](https://github.com/hjsblogger/web-automation-with-pyppeteer/blob/main/tests/handling-dynamic-content/test_page_class_lazy_loaded_content.py#L81). I have ported the code from my [existing implementation](https://github.com/hjsblogger/web-scraping-with-python/blob/main/pageobject/helpers.py#L100), which I* used for my other web scraping blog. 🙂

Once the page is loaded (in maximized mode), we first get the scrollable height using the JavaScript function *document.documentElement.scrollHeight*. The JavaScript logic is executed using the *page.evaluate()* method.

![](https://cdn-images-1.medium.com/max/3020/0*4oskVjWnjVgEIxCS.png)

Next, a *While* loop is run where a vertical scroll is performed using the *Window.scrollTo()* method (in JS) with *document.documentElement.scrollHeight* as the input argument. Since it is a JavaScript function, it is executed using the *evaluate()* method of the *Page *class.

![](https://cdn-images-1.medium.com/max/2444/0*t0Ilys4fGoUxWEM_.png)

To witness the *Window.scrollTo()* method in action, open the Test URL in the browser. Then trigger the following commands in the browser console:

* *document.documentElement.scrollHeight*

* *window.scrollTo(0, document.documentElement.scrollHeight)*

The above two actions are performed till we reach the end of the document (or page). This is when we break from the while loop!

![](https://cdn-images-1.medium.com/max/2548/0*QpoxQapQBV3r6bIE.png)

Now that the page load is complete, we scroll to the top of the page by executing the JS function *window.scrollTo(0, 0) *via the *evaluate()* method of Pyppeteer.

![](https://cdn-images-1.medium.com/max/2000/0*w4-ueuDIkFT1nTeo.png)

Next, we locate the target element using the CSS Selector *div:nth-of-type(31) > .p-4 [href=’/exercise/list_basic_detail/94967-A/’]* and scroll into the view using the user-defined function *scroll_to_element()*.

![](https://cdn-images-1.medium.com/max/3200/0*EdQzw9jmgO2yfuY1.png)

![](https://cdn-images-1.medium.com/max/2548/0*3b470CW_OUTyxq3f.png)

*Implementation of scroll_to_element() method*

![](https://cdn-images-1.medium.com/max/3200/0*iudlQSXZNesM7eIL.png)

Finally, a click is initiated on the element (i.e., *elem_button*) located in the earlier step. As usual, the *waitForNavigation()* helps in waiting for the page navigation to complete.

![](https://cdn-images-1.medium.com/max/3188/0*dXbiIuyX19-PhmA7.png)

The current page URL is obtained via the *page.url* attribute. An assert is raised if the URL does not match with the expected URL!

![](https://cdn-images-1.medium.com/max/2884/0*-Gll455rHen9EzFj.png)

**Execution**

Invoke the command *make pyppeteer-dynamic-content* on the terminal with the Chromium browser instantiated on the local machine (*EXEC_PLATFORM = local*). You can also perform web scraping using Chromium (or Chrome) on the LambdaTest cloud grid (*EXEC_PLATFORM = cloud*).

As seen below, all four test scenarios are executed in parallel using the [*pytest-xdist](https://pypi.org/project/pytest-xdist/)* plugin for parallel execution with pytest & Pyppeteer.

![](https://cdn-images-1.medium.com/max/3200/0*Rf33zM8ptz2-oAvp.png)

The execution snapshots of the test scenarios are below:

![](https://cdn-images-1.medium.com/max/3200/0*6eKTmud8a0c1goW_.png)

*Execution: Test Scenario — 1*

![](https://cdn-images-1.medium.com/max/3200/0*OJ8a2UrFYh88cjVk.png)

*Execution: Test Scenario — 2*

![](https://cdn-images-1.medium.com/max/3200/0*ZKNiwjaShJDXRNzL.png)

*Execution: Test Scenario — 3*

![](https://cdn-images-1.medium.com/max/3200/0*XPekmwJUrxW-IOi8.png)

*Execution: Test Scenario — 4*

We also executed the same tests on the LambdaTest Grid and the execution was successful!

![](https://cdn-images-1.medium.com/max/3200/0*XaNwdXxedKIBD0EF.png)

Shown below are the execution screenshots from [LambdaTest Automation Dashboard](https://automation.lambdatest.com/build?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=webpage):

![](https://cdn-images-1.medium.com/max/3200/0*Re5nDs3-A4mlFYov.png)

*Parallel Test Execution (In Progress)*

![](https://cdn-images-1.medium.com/max/3200/0*cp_txYfWzbPGksse.png)

*Test Execution (Completion)*

Now that we have introduced the nuances of [parallel testing](https://www.lambdatest.com/blog/what-is-parallel-testing-and-why-to-adopt-it/?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=blog) in this section let’s do a detailed deep-dive in the further sections of the blog!

## Web scraping with Pyppeteer

Python is one of the most prominent languages used extensively for web scraping. For starters, web scraping uses automated tools (or libraries) to extract meaningful information from websites.

Competitor analysis, Data analysis, and Lead generation are some of the main uses of web scraping. Apart from Pyppeteer, Selenium, and Beautiful Soup can also scrap static and dynamic web content. If you want to learn about the fine nuances of web scraping, please refer to this detailed blog that deep-dives into [web scraping using Python](https://www.lambdatest.com/blog/web-scraping-with-python/?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=blog).

For demonstration web scraping with Pyppeteer, I would be porting scraping logic that was used for scraping product content from LambdaTest eCommerce Playground.

**Demonstration**

As mentioned above, we would be scraping product meta-data (i.e., Name, Price, and Image Link) from the eCommerce website.

![](https://cdn-images-1.medium.com/max/3200/0*sjuI-YyV5swD_jeq.png)

*LambdaTest E-Commerce Playground*

The scraping logic with Pyppeteer can be executed on local Chromium (i.e., *EXEC_PLATFORM=local*) as well as Chrome/Chromium on LambdaTest Cloud Grid (i.e., *EXEC_PLATFORM=cloud)*.

    import asyncio
    import pytest
    from pyppeteer.errors import PageError
    from urllib.parse import quote
    import json
    import os
    import sys
    from os import environ
    from pyppeteer import connect, launch
    
    
    test_url = 'https://ecommerce-playground.lambdatest.io/index.php?route=product/category&path=57'
    
    
    @pytest.mark.asyncio
    async def test_ecommerce_scraping(page):
       for iteration in range(1,6):
           meta_data_arr = []
    
    
           curr_test_url = test_url + "&page=" + str(iteration)
           meta_data_arr = await scrap_ecommerce(page, curr_test_url)
           print('\n')
           print("Product Page = " + curr_test_url)
           print("*********************************************************************************************************\n")
           await print_scrapped_content(meta_data_arr)
    
    
    async def scrap_ecommerce(page, curr_test_url) -> list:
       await page.goto(curr_test_url, {'waitUntil': 'domcontentloaded'})
    
    
       rows = await page.querySelectorAll('.product-layout.product-grid.no-desc.col-xl-4.col-lg-4.col-md-4.col-sm-6.col-6')
    
    
       meta_data_arr = []
       # Extract information from the selected elements
       for row in rows:
           link = await row.querySelector('a.carousel.d-block.slide')
           name = await row.querySelector('h4.title')
           price = await row.querySelector('span.price-new')
    
    
           link_href = await page.evaluate('(element) => element.getAttribute("href")', link)
           product_name = await page.evaluate('(element) => element.textContent', name)
           product_price = await page.evaluate('(element) => element.textContent', price)
    
    
           # Create a dictionary of the meta-data of the items on e-commerce store
           meta_data_dict = {
               'product link': link_href,
               'product name': product_name,
               'product price': product_price
           }
    
    
           meta_data_arr.append(meta_data_dict) 
      
       return meta_data_arr
    
    
    # # Pagination - 1:5
    # # Page 1: https://ecommerce-playground.lambdatest.io/index.php?route=product/category&path=57&page=1
    # # Page 5: https://ecommerce-playground.lambdatest.io/index.php?route=product/category&path=57&page=5
    
    
    # Replica of https://github.com/hjsblogger/web-scraping-with-python/blob/main/pageobject/helpers.py#L146
    async def print_scrapped_content(meta_data):
       for elem_info in meta_data:
           print(elem_info)

**Code Walkthrough**

To get started, we set up the test URL that comprises of the base URL followed by the page number (i.e. &*page=< page-number >*).

![](https://cdn-images-1.medium.com/max/3200/0*rr8UcymyT5sAeRLN.png)

![](https://cdn-images-1.medium.com/max/3200/0*rhbq_RlfO1n94O7E.png)

As seen above, there are a total of 5 pages in the product category. Hence, the scraping implementation is executed in an iterative loop [from 1 .. 6].

![](https://cdn-images-1.medium.com/max/3200/0*AgM1CycKQJAxoSzf.png)

Here are the two main coroutines (or methods) that are used in scraping the test website:

* [*scrap_ecommerce()](https://github.com/hjsblogger/web-automation-with-pyppeteer/blob/main/tests/web-scraping-content/test_scraping_with_pyppeteer.py#L25)* — Method that scraps content [or meta-data] from the current URL and stores it in a list

* [*print_scrapped_content()](https://github.com/hjsblogger/web-automation-with-pyppeteer/blob/main/tests/web-scraping-content/test_scraping_with_pyppeteer.py#L57) *— Prints the content scrapped by the scrap_ecommerce() method

Let’s look at each of them in more detail!

The *scrap_ecommerce()* method takes two parameters:

* *page* — browser page that is created using the *newPage()* method

* *curr_test_url* — URL (example) from which the content needs to be scrapped using Pyppeteer

The *goto()* method takes the test *URL* as the input parameter. Along with it, the *waitUntil* option is set to *domcontentloaded *(i.e. wait for a maximum of 30000 ms till the firing of the *DOMContentLoaded *event).

![](https://cdn-images-1.medium.com/max/2816/0*XfkZoyH78NZVnmOo.png)

The product page contains 15 products, where the *product meta-data* is within a common CSS Selector
*(.product-layout.product-grid.no-desc.col-xl-4.col-lg-4.col-md-4.col-sm-6.col-6)*

![](https://cdn-images-1.medium.com/max/3200/0*c5TWmxPxJc_lhKf6.png)

Hence, 15 elements (on every product page) match the element located using the above CSS Selector. The *querySelectorAll()* method of the Page class returns a list that contains elements matching the provided CSS Selector. The list will be further used to iteratively scrap information (i.e., name, cost) of each product.

![](https://cdn-images-1.medium.com/max/3200/0*itfkHj-RrwvJZMsU.png)

Next, we run a *for *loop (1 .. 16) to scrap information of all the 15 products (or elements) under the *div* located in the previous step.

![](https://cdn-images-1.medium.com/max/3200/0*Q_qpi60xhp8_4HRW.png)

The **product link** is obtained by locating the respective element using the *querySelector()* method. As seen below, the first argument is the tag that must be searched for (i.e., a — anchor tag), and the second is the CSS *Class *attribute.

![](https://cdn-images-1.medium.com/max/2580/0*RN2cztIWwhHzjvGE.png)

![](https://cdn-images-1.medium.com/max/3200/0*ik3GxSvgQA8VXoS_.png)

Now that we have located the element, the link (or href) to the product page is obtained by executing a JS-function (on browser) via the evaluate() method of the Page class.

![](https://cdn-images-1.medium.com/max/2532/0*RYVsAHNfbW8XyhmJ.png)

The JS function uses the arrow function syntax and takes a parameter named *element*. The *getAttribute()* method of the *Element* interface returns the value of the *href* attribute of the element.

![](https://cdn-images-1.medium.com/max/3200/0*otF_W6yUMPdeOkeG.png)

![](https://cdn-images-1.medium.com/max/3200/0*5o7QfMyuTzH4uaa4.png)

The **product name** is also scraped by first locating the respective element using the *querySelector()* method. The first argument is the *h4* tag, and the second argument is the Class Name locator *title*.

![](https://cdn-images-1.medium.com/max/3200/0*1iUIxnFiV6ARVP15.png)

![](https://cdn-images-1.medium.com/max/2000/0*EDzyzzOgI32HuLSo.png)

Once the element is located, the text content is obtained using the JavaScript method — *textContent()* on the located element (i.e., *element.textContent*). It uses the arrow function syntax where the element (i.e., *name*) is passed as an input parameter.

![](https://cdn-images-1.medium.com/max/3200/0*0PlpoEyfUy2a-Z8P.png)

![](https://cdn-images-1.medium.com/max/3052/0*C7IzzNseAIs2Q_5s.png)

Lastly, the **price** of the respective product is scraped by first locating the element using the *querySelector()* method. As seen below, the text under the element with Class Name *price-new* provides the product price.

![](https://cdn-images-1.medium.com/max/3200/0*sTNyeNUS8oX36-0V.png)

![](https://cdn-images-1.medium.com/max/2140/0*vCUs89egdRyVEbVg.png)

Like before, the text content is obtained using the JavaScript method — *textContent()* (or *element.textContent*). It uses the arrow function syntax where the element (i.e., *price*) is passed as an input parameter.

![](https://cdn-images-1.medium.com/max/3120/0*SFLldvfByHpdGRO6.png)

Now that we have scraped the required information (or meta-data), a dictionary (i.e., *meta_data_dict*) is created and appended to the list (i.e., *meta_data_arr*).

![](https://cdn-images-1.medium.com/max/2884/0*mqvm3vDEnmDMTHBL.png)

The above-mentioned steps are repeated for all the other product pages (i.e., Product Page 1 → Product Page 5), and the scraped data is displayed on the screen.

![](https://cdn-images-1.medium.com/max/3200/0*T7LheIRTxCR7dd3i.png)

**Execution**

Invoke the command *make pyppeteer-web-scraping* on the terminal with the Chromium browser instantiated on the local machine (*EXEC_PLATFORM = local*). You can also perform web scraping using Chromium (or Chrome) on the LambdaTest cloud grid (*EXEC_PLATFORM = cloud*).

As seen below, the Chrome browser was successfully instantiated on the local machine.

![](https://cdn-images-1.medium.com/max/3200/0*9sqFZX09fg9j5Gci.png)

Here are the screenshots of the scrapped content of [Product Page 1](https://ecommerce-playground.lambdatest.io/index.php?route=product/category&path=57&page=1) and [Product Page 5](https://ecommerce-playground.lambdatest.io/index.php?route=product/category&path=57&page=5).

![](https://cdn-images-1.medium.com/max/3200/0*hyG8Je4qq3M0iLW7.png)

![](https://cdn-images-1.medium.com/max/3200/0*cxFLFDix_2pIMIQq.png)

With this, we have successfully scraped a website using Pyppeteer! As always, use web scraping with utmost caution and responsibility. 🙂

## Parallel execution with Pyppeteer

Pyppeteer does not have built-in support for parallel test execution. However, tests can be run parallel with Pyppeteer by leveraging other test automation frameworks (or libraries) offering support.

Before you parallelize the tests, it is important to follow the test execution best practices, the most important being test isolation (or independent tests).

In the section where we demonstrated dynamic content handling with Pyppeteer, we used the pytest plugin *pytest-xdist* to trigger the tests in parallel. If you are new to pytest, check out the [pytest tutorial](https://www.lambdatest.com/learning-hub/pytest-tutorial?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=learning_hub) that deep-dives into the basic nitty-gritty of the framework.

The video below shows how to use the *pytest-xdist* plugin with Selenium, but the learnings also apply to Pyppeteer!
{% youtube m0XcQcb25p8 %}
You can subscribe to the [LambdaTest YouTube Channel](https://www.youtube.com/c/LambdaTest?sub_confirmation=1?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=youtube) for more tutorials on [Selenium testing](https://www.lambdatest.com/selenium?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=webpage), [Playwright testing](https://www.lambdatest.com/playwright-testing?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=webpage), [Cypress testing](https://www.lambdatest.com/learning-hub/cypress-tutorial?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=youtube), and more.

*pytest-parallel* is another popular pytest plugin that can be used instead of *pytest-xdist*. The major advantage of the *pytest-parallel* plugin is its thread-safe approach and improved performance due to the usage of non-blocking IO for HTTP requests. Though we could have also chosen unittest (or PyUnit), features like fixtures, parameterized testing, and advanced assertions make pytest the de-facto choice for parallel testing.

As a part of the installation, we had already installed the *pytest-xdist* plugin via requirements.txt

    pytest
    pyppeteer
    pytest-asyncio
    pytest-xdist
    pytest-order
    py

**Demonstration**

For demonstrating parallel execution with Pyppeteer using pytest, we have created two Python files that have their implementation of fixtures and tests. Setting the *EXEC_PLATFORM* to local lets you run tests in parallel on the local machine, whereas setting *EXEC_PLATFORM* to cloud runs tests on the LambdaTest cloud grid.

![](https://cdn-images-1.medium.com/max/2000/1*ihiI9sBEsyvWqYdv5vQ23Q.png)

Since the overall implementation steps remain largely unchanged, we would be avoiding code walkthrough for this section!

    import asyncio
    import pytest
    from pyppeteer.errors import PageError
    from urllib.parse import quote
    import json
    import os
    from os import environ
    from pyppeteer import connect, launch
    
    
    exec_platform = os.getenv('EXEC_PLATFORM')
    
    
    # Get username and access key of the LambdaTest Platform
    username = environ.get('LT_USERNAME', None)
    access_key = environ.get('LT_ACCESS_KEY', None)
    
    
    cloud_capabilities = {
       'browserName': 'chrome',
       'browserVersion': 'latest',
       'LT:Options': {
           'platform': 'Windows 11',
           'build': '[Build] Duckduckgo Search using Pyppeteer',
           'name': 'Duckduckgo Search using Pyppeteer',
           'resolution': '1920x1080',
           'user': username,  # Replace with your LT_USERNAME
           'accessKey': access_key,  # Replace with your LT_ACCESS_KEY
           'network': True,
           'video': True,
           'console': True,
           'headless': False
       }
    }
    
    
    @pytest.fixture(scope='function')
    async def browser():
       if exec_platform == 'local':
           browser = await launch(headless = False, args=['--start-maximized'])
       elif exec_platform == 'cloud':
           capability = quote(json.dumps(cloud_capabilities))
           print('Initializing test:: ', cloud_capabilities['LT:Options']['name'])
    
    
           browser = await connect(
               browserWSEndpoint=f'wss://cdp.lambdatest.com/puppeteer?capabilities={capability}'
           )
       yield browser
       await asyncio.sleep(1)   
    
    
    # Pytest fixture for page setup
    @pytest.fixture(scope='function')
    async def page(browser):
       page = await browser.newPage()
       yield page
      
       await page.close()
       await asyncio.sleep(1)
       await browser.close()
    
    
    @pytest.mark.usefixtures('page')
    
    
    @pytest.mark.asyncio
    async def test_search_1(page):
       # asyncio.set_event_loop(asyncio.new_event_loop())
       await page.goto('https://www.duckduckgo.com')
       # Maximize the page
       await page.setViewport({'width': 1920, 'height': 1080})
       element = await page.querySelector('[name="q"]')
       await element.click()
       await element.type('LambdaTest')
       await asyncio.gather(
           page.keyboard.press('Enter'),
           page.waitForNavigation()
       )
    
    
       await asyncio.sleep(1)
    
    
       title = await page.title()
    
    
       try:
           assert title == 'LambdaTest at DuckDuckGo', 'Expected page title is incorrect!'
           await page.evaluate('_ => {}', f'lambdatest_action: {json.dumps({ "action": "setTestStatus", "arguments": { "status": "passed", "remark": "Title matched" } })}')
       except PageError as e:
           await page.evaluate('_ => {}', f'lambdatest_action: {json.dumps({ "action": "setTestStatus", "arguments": { "status": "failed", "remark": str(e) } })}')
    
    
       await asyncio.sleep(1)
    
    
       return title

    import asyncio
    import pytest
    from pyppeteer.errors import PageError
    from urllib.parse import quote
    import json
    import os
    from os import environ
    from pyppeteer import connect, launch
    
    
    exec_platform = os.getenv('EXEC_PLATFORM')
    
    
    # Get username and access key of the LambdaTest Platform
    username = environ.get('LT_USERNAME', None)
    access_key = environ.get('LT_ACCESS_KEY', None)
    
    
    cloud_capabilities = {
       'browserName': 'chrome',
       'browserVersion': 'latest',
       'LT:Options': {
           'platform': 'Windows 10',
           'build': '[Build] Brave Search using Pyppeteer',
           'name': 'Brave Search using Pyppeteer',
           'resolution': '1920x1080',
           'user': username,  # Replace with your LT_USERNAME
           'accessKey': access_key,  # Replace with your LT_ACCESS_KEY
           'geoLocation': 'US',
           'network': True,
           'video': True,
           'console': True,
           'headless': False
       }
    }
    
    
    @pytest.fixture(scope='function')
    async def browser():
       if exec_platform == 'local':
           browser = await launch(headless = False, args=['--start-maximized'])
       elif exec_platform == 'cloud':
           capability = quote(json.dumps(cloud_capabilities))
           print('Initializing test:: ', cloud_capabilities['LT:Options']['name'])
    
    
           browser = await connect(
               browserWSEndpoint=f'wss://cdp.lambdatest.com/puppeteer?capabilities={capability}'
           )
       yield browser
       await asyncio.sleep(1)
    
    
    # Pytest fixture for page setup
    @pytest.fixture(scope='function')
    async def page(browser):
       page = await browser.newPage()
       yield page
      
       await page.close()
       await asyncio.sleep(1)
       await browser.close()
    
    
    @pytest.mark.usefixtures('page')       
    @pytest.mark.asyncio
    async def test_search_2(page):
       await page.goto('https://search.brave.com')
      
       # Maximize the page
       await page.setViewport({'width': 1920, 'height': 1080})
      
       element = await page.querySelector('#searchbox')
       await element.click()
       await element.type('LambdaTest')
       await asyncio.gather(
           page.keyboard.press('Enter'),
           page.waitForNavigation()
       )
    
    
       await asyncio.sleep(1)
    
    
       title = await page.title()
    
    
       try:
           assert title == 'LambdaTest - Brave Search', 'Expected page title is incorrect!'
           await page.evaluate('_ => {}', f'lambdatest_action: {json.dumps({ "action": "setTestStatus", "arguments": { "status": "passed", "remark": "Title matched" } })}')
       except PageError as e:
           await page.evaluate('_ => {}', f'lambdatest_action: {json.dumps({ "action": "setTestStatus", "arguments": { "status": "failed", "remark": str(e) } })}')
    
    
       await asyncio.sleep(1)
    
    
       return title

**Execution**

Invoke the command *make pytest-pyppeteer* on the terminal to trigger the execution. As seen in the execution snapshot on the ***local machine***, the *-n 2* option starts two processes in parallel, with tests in both files executing in separate processes.

It is important to note that tests in separate files are executed in parallel, whereas multiple tests in the same file run in a serial manner.

![](https://cdn-images-1.medium.com/max/3200/0*EvH70-a_LN3SEZ6l.png)

Here is the browser output, which indicates that the tests were executed successfully.

![](https://cdn-images-1.medium.com/max/3200/0*JP2XLENo0GbRDat0.png)

*Execution — Test Scenario: 1*

![](https://cdn-images-1.medium.com/max/3200/0*YnQPa25U_zKtuzGW.png)

*Execution — Test Scenario: 2*

You can find the LambdaTest username and access key in the [LambdaTest Password & Security page](https://accounts.lambdatest.com/security?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=webpage). Shown below is the execution on the LambdaTest cloud, which was triggered by setting EXEC_PLATFORM to *cloud*.

![](https://cdn-images-1.medium.com/max/3200/0*yFGmPhXzCsCYstyZ.png)

As seen on the dashboard, the tests in both the Python files are running in parallel.

![](https://cdn-images-1.medium.com/max/3200/0*R66gpEuYUNraV3L3.png)

As seen below, the test execution was successful.

![](https://cdn-images-1.medium.com/max/3200/0*0miT34emO6pW2-zc.png)

*Execution on LambdaTest — Test Scenario: 1*

![](https://cdn-images-1.medium.com/max/3200/0*heAGwRNxuvcltYQr.png)

*Execution on LambdaTest — Test Scenario: 2*

## Difference between Pyppeteer and Puppeteer

As stated in the official Pyppeteer documentation, Pyppeteer is an unofficial port of Puppeteer. Both frameworks provide high-level APIs that let you interact with the Elements in a document (or page) when opened on Chrome (or Chromium browsers).

## Language-specific differences

Python-based Pyppeteer & JavaScript-based Puppeteer are very much similar, barring aside some high-level differences. For the Pyppeteer demonstration on LambdaTest, we used the endpoint that is also used for Puppeteer:

![](https://cdn-images-1.medium.com/max/3200/0*tCdJUeNCaf7Cd28H.png)

Since Pyppeteer is based on Python, it leverages the *asyncio* library for writing concurrent code using the *async/await* syntax. As Puppeteer is based on JavaScript, it leverages *Promises* that are objects representing the completion (or failure) of an asynchronous operation.

## Keyword arguments for options

As seen so far, Pyppeteer accepts both dictionary & keyword arguments for options.

![](https://cdn-images-1.medium.com/max/2076/0*eQEo45AUcRN3G_p1.png)

On the other hand, Puppeteer uses object for passing options to methods.

![](https://cdn-images-1.medium.com/max/2000/0*aOTkji1yKJYzZRTj.png)

## The evaluate() method

The *page.evaluate()* method in Pyppeteer takes a string of JavaScript.

![](https://cdn-images-1.medium.com/max/3020/0*YRObpo8oHlgrbvwK.png)

On the other hand, the *page.evaluate()* method in Puppeteer takes JavaScript raw function (or string of JavaScript expression).

![](https://cdn-images-1.medium.com/max/2140/0*xeTUW3kBqBsF4tnQ.png)

There are a few more differences between Pyppeteer and Puppeteer, I recommend going through the Pyppeteer vs Puppeteer comparison for more information. Even though the Pyppeteer project is no longer under maintenance, it is still used by the community!

If you are looking for options, you can switch to other frameworks like Puppeteer or Selenium. However, the choice depends on the project complexity and technical expertise of the said library (or framework).

## It’s A Wrap

Thanks for making it this far! In this super-exhaustive [Python tutorial](https://www.lambdatest.com/learning-hub/python-tutorial?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=learning_hub) with Puppeteer, we deep-dived into almost all the use cases (or scenarios) that can be realized with the Pyppeteer library. Like other Python-based test automation frameworks like pytest, unittest, etc., Pyppeteer can be leveraged for [web automation](https://www.lambdatest.com/learning-hub/web-automation?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=learning_hub) and scraping of pages.

Lastly, switching (or porting) from Pyppeteer to Puppeteer/Selenium should be considered in scenarios where you want to harness the potential the other frameworks (or libraries) offer. You can also leverage Pyppeteer on LambdaTest since it lets you run tests at scale, that too on different versions of Chrome (or Chromium).

The integration of Pyppeteer with LambdaTest provides a seamless environment for executing scripts, ensuring compatibility testing on a diverse set of browser versions. This combination not only enhances the scalability of your testing but also allows you to validate the performance and functionality of your applications in a wide range of browser environments. LambdaTest’s infrastructure, combined with Pyppeteer’s capabilities, provides a robust solution for efficient and comprehensive [web application testing](https://www.lambdatest.com/learning-hub/web-application-testing?utm_source=devto&utm_medium=organic&utm_campaign=feb_05&utm_term=bh&utm_content=learning_hub).

Happy Testing 🙂

## Frequently Asked Questions (FAQs)

### What is Pyppeteer in Python?

Pyppeteer is a Python library that provides a high-level API to control headless browsers using the Chrome DevTools Protocol. It is essentially a Python port of the popular Node library Puppeteer, which is maintained by the Chrome team at Google.

### What is the difference between Pyppeteer and Puppeteer?

Pyppeteer and Puppeteer are sibling libraries designed for different programming languages, Python and JavaScript, respectively. Yet, they aim to provide a high-level interface to control headless browsers through the Chrome DevTools Protocol.

Puppeteer, developed by Google for Node.js, has been widely adopted in the JavaScript community, offering features like web scraping and automated testing. On the other hand, Pyppeteer is the Python adaptation of Puppeteer, allowing Python developers to leverage similar capabilities using Python syntax and asyncio. Despite their language differences, both libraries empower developers to automate tasks in headless browsers effectively.