How to run Playwright tests with Selenium Grid versions after `4.5`
Tags: playwright, selenium, testing, java
Published: 2023-02-18 16:35:04

Compared to other automation testing libraries, [Playwright](https://playwright.dev) is relatively new, but it supports writing tests in different languages.

If you're using Java, Playwright can take advantage of the Selenium Grid to scale test execution, helping you save time and increase efficiency.


#### TL;DR;
- Playwright has a limitation that restricts running tests in Selenium Grid to only the version prior to [`4.5.2-20221021`](https://github.com/SeleniumHQ/docker-selenium/releases/tag/4.5.2-20221021) (at the time of this writing - February/2023).
- Selenium 4 has passed through a redesign of its architecture. After a long period of stabilization, a long-standing intermittent issue was fixed in version `4.5.2-20221021`.
- If Playwright tests only work with versions prior to `4.5.2-20221021`, it won't be able to benefit from the latest improvements made by the Selenium team to the grid or any browser version upgrades.
A workaround for the issue is to use a specific property -`Djdk.httpclient.websocket.intermediateBufferSize`, passed as an argument to the `selenium-server.jar`.

## Problem

Although Playwright is still relatively new compared to Selenium, it generally offers faster test execution times. In a "real-work" test that I used to compare the two tools, the results were as follows:

| Spec                       | Library    | Number of tests | Duration     |
|----------------------------|------------|-----------------|--------------|
| selenium.specs.LoginSpec   | Selenium   | 32              | 1:31 minutes |
| playwright.specs.LoginSpec | Playwright | 32              | 43 seconds   |

The example above is a simple test run from a local machine, without using Selenium Grid.

According to the Playwright website, tests written with Playwright can run against Selenium Grid, as documented [here](https://playwright.dev/docs/selenium-grid). However, as of February 2023, there is currently an issue with Selenium Grid versions `4.5.2-20221021` or later, which can cause errors when attempting to use Playwright with Selenium Grid.

- [Cannot run playwright tests on Selenium Grid](https://github.com/microsoft/playwright/issues/18892)

If Playwright tests can run only with Selenium Grid version prior to this, then it means it is running only with older versions of Google Chrome or Microsoft Edge:

| Spec         | Supported versions in `4.5.0` | Supported versions in latest (`4.8.0`) |
|--------------|-------------------------------|---------------------------------------|
| Chrome       | 106.0.5249.119                | 110.0.5481.77                         |
| ChromeDriver | 106.0.5249.61                 | 110.0.5481.77                         |
| Edge         | 106.0.1370.47                 | 110.0.1587.41                         |
| EdgeDriver   | 106.0.1370.47                 | 110.0.1587.41                         |

## History

When **Selenium 4** was release, it changed completely how Selenium Grid is structured. It has totally changed its design, based on distributed components to handle messages between the `hub` and the `nodes`.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/m0l98empv2owkjhcgy6x.png)

After its release, a critical intermittent issue received significant attention, as it was difficult to reproduce and persisted for almost two years before being resolved.
- [https://github.com/SeleniumHQ/selenium/issues/9528#issuecomment-947994758](java.util.concurrent.TimeoutException thrown at random netty read timeouts with RemoteWebDriver)

This fix was officially released on `docker-selenium` project on version `4.5.2-20221021`. The exact same version that started to present issues with the execution of tests written with Playwright.

## Fix (Workaround)

The fix was reported by [EndPositive](https://github.com/EndPositive) in an [issue](https://github.com/SeleniumHQ/selenium/issues/11408#issuecomment-1345401948) opened for the Selenium team.

The issue seems to be related to WebSocket communication, with messages larger than the default configuration of Selenium Grid causing problems.

The suggestion is to increase the WebSocket buffer size passing the parameter `Djdk.httpclient.websocket.intermediateBufferSize` to the `selenium-server.jar`.

Using [docker-selenium](https://github.com/SeleniumHQ/docker-selenium) images, it is possible to do as shown in the example below:

```docker-compose.yml
version: "3"

services:
  selenium-hub:
    image: selenium/hub:4.8.0-20230210
    container_name: selenium-hub
    ports:
      - "4442:4442"
      - "4443:4443"
      - "4444:4444"
    environment:
      - JAVA_OPTS=-Djdk.httpclient.websocket.intermediateBufferSize=3000000
      - SE_EVENT_BUS_HOST=selenium-hub

  chrome:
    image: selenium/node-chrome:4.8.0-20230210
    shm_size: 2gb
    depends_on:
      - selenium-hub
    environment:
      - JAVA_OPTS=-Djdk.httpclient.websocket.intermediateBufferSize=3000000
      - SE_EVENT_BUS_HOST=selenium-hub
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443
      - HUB_HOST=selenium-hub
      - HUB_PORT=4444
      - SE_NODE_GRID_URL=http://localhost:4444
```

To run Playwright tests against Selenium Grid, the environment variable `SELENIUM_REMOTE_URL` must be declared:
```
export SELENIUM_REMOTE_URL=http://localhost:4444
```

After that, tests can be run in different ways:
```
./gradlew test
```
Or
```
./mvnw test
```
Or with other build tools of preference.

## Final consideration

It seems that Playwright uses Selenium Grid, but Selenium Grid does NOT officially support Playwright. The problem to run Playwright tests with the newest version of Selenium Grid were raised in Selenium GitHub project, however it hasn't been investigated so far to be fixed once it cannot be reproduced with tests implemented using Selenium (and Selenium `WebDriver`).

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/urju6k70lzfl6dvdect3.png)