Exploring the Various Retry APIs of Playwright for Robust Testing
Tags: playwright, javascript
Published: 2023-05-28 15:51:25

## Global Retries

### Configuring Global Retries

Playwright provides a built-in global retry mechanism for test cases. This means that when a test fails, Playwright automatically retries the test up to the configured number of times before marking it as a failure. To set the global retry ability, you can use the `retries` option in the Playwright config file (`playwright.config.ts`):

```typescript
import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
});
```

This code snippet configures retries only when running tests in a continuous integration (CI) environment. You can override this setting by using the `--retries` flag when running tests from the command line:

```bash
npx playwright test --retries=1
```

### Configuring Retries per Test Block

If you need more granular control over retries, you can configure them for individual test blocks or groups of tests. To do this, use the `test.describe.configure()` function:

```typescript
import { test } from '@playwright/test';

test.describe('Playwright Test', () => {
  test.describe.configure({ retries: 5 });

  test('should work', async ({ page }) => {
    // Your test code here
  });
});
```

This configuration allows the specified test block to be retried up to 5 times before being marked as a failure.

## Auto-waiting and Retrying

### Built-in Auto-waiting Mechanism

Playwright has a built-in auto-waiting and retry mechanism for locators (e.g., `page.getByRole()`) and matchers (e.g., `toBeVisible()`). This mechanism continuously runs the specified logic until the condition is met or the timeout limit is reached, helping to reduce or eliminate flakiness in your tests. For instance, you don't need to manually specify a wait time before running some code, such as waiting for a network request to complete.

To learn more about the specific timeout limits, refer to the [Playwright timeout documentation](https://playwright.dev/docs/test-timeouts).

### Custom Conditions with Retrying and Polling APIs

Sometimes, you might need to wait for a condition unrelated to the UI, such as asynchronous processes or browser storage updates. In these cases, you can use Playwright's [Retrying](https://playwright.dev/docs/test-assertions#retrying) and [Polling](https://playwright.dev/docs/test-assertions#polling) APIs to explicitly specify a condition that is awaited until it is met.

#### Using the Retry API

The `Retry` API uses a standard `expect` method along with the `toPass(options)` method to retry an assertion within the `expect` block. If the assertion fails, the `expect` block is retried until the timeout limit is reached or the condition passes. The example below demonstrates waiting for a value to be written to local storage:

```typescript
import { test } from '@playwright/test';

test('runs toPass() until the condition is met or the timeout is reached', async ({ page }) => {
  await expect(async () => {
    const localStorage = await page.evaluate(() => JSON.stringify(window.localStorage.getItem('user')));
    expect(localStorage).toContain('Tim Deschryver');
  }).toPass();
});
```

#### Using the Poll API

The `Poll` API is similar to the `Retry` API, but it uses the `expect.poll()` method instead of a standard `expect` block. The `expect.poll()` method also returns a result, which is used to invoke the matcher. The example below demonstrates waiting for a process state to be completed:

```typescript
import { test } from '@playwright/test';

test('runs expect.poll() until the condition is met or the timeout is reached', async ({ page }) => {
  await expect
    .poll(async () => {
      const response = await page.request.get('https://my.api.com/process-state');
      const json = await response.json();
      return json.state;
    })
    .toBe('completed');
});
```

Both the `Retry` and `Poll` APIs can be configured with custom timeout and interval durations:

```typescript
import { test } from '@playwright/test';

test('runs toPass() until the condition is met or the timeout is reached', async ({ page }) => {
  await expect(async () => {
    // Your test code here
  }).toPass({ intervals: [1000, 1500, 2500], timeout: 5000 });
});

test('runs expect.poll() until the condition is met or the timeout is reached', async ({ page }) => {
  await expect
    .poll(async () => {
      // Your test code here
    }, { intervals: [1000, 1500, 2500], timeout: 5000 })
    .toBe('completed');
});
```

## Test Retries in Worker Processes

### How Worker Processes Work

Playwright Test runs tests in worker processes, which are independent OS processes orchestrated by the test runner. These workers have identical environments and start their own browsers. When all tests pass, they run in order in the same worker process. However, if any test fails, Playwright Test discards the entire worker process along with the browser and starts a new one. Testing continues in the new worker process, beginning with the next test.

### Enabling Retries in Worker Processes

When you enable retries, the second worker process starts by retrying the failed test and continues from there. This approach works well for independent tests and guarantees that failing tests can't affect healthy ones.

To enable retries, you can use the `--retries` flag or configure them in the configuration file:

```bash
npx playwright test --retries=3
```

```typescript
import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: 3,
});
```

Playwright Test categorizes tests as follows:

*   "passed" - tests that passed on the first run;
*   "flaky" - tests that failed on the first run but passed when retried;
*   "failed" - tests that failed on the first run and all retries.

## Detecting Retries at Runtime

You can detect retries at runtime using the `testInfo.retry` property, which is accessible to any test, hook, or fixture. The example below demonstrates clearing server-side state before retrying a test:

```typescript
import { test, expect } from '@playwright/test';

test('my test', async ({ page }, testInfo) => {
  if (testInfo.retry) {
    await cleanSomeCachesOnTheServer();
  }
  // Your test code here
});
```

## Configuring Retries for Specific Groups or Files

You can specify retries for a specific group of tests or a single file using the `test.describe.configure()` function:

```typescript
import { test, expect } from '@playwright/test';

test.describe(() => {
  test.describe.configure({ retries: 2 });

  test('test 1', async ({ page }) => {
    // Your test code here
  });

  test('test 2', async ({ page }) => {
    // Your test code here
  });
});
```

## Grouping Dependent Tests with `test.describe.serial()`

For dependent tests, you can use `test.describe.serial()` to group them together, ensuring they always run together and in order. If one test fails, all subsequent tests are skipped. All tests in the group are retried together. While it's usually better to make your tests isolated, this technique can be useful when you need to run tests in a specific order.

```typescript
import { test } from '@playwright/test';

test.describe.serial.configure({ mode: 'serial' });

test('first good', async ({ page }) => {
  // Your test code here
});

test('second flaky', async ({ page }) => {
  // Your test code here
});

test('third good', async ({ page }) => {
  // Your test code here
});
```

## Reusing a Single Page Object Between Tests

By default, Playwright Test creates an isolated `Page` object for each test. However, if you'd like to reuse a single `Page` object between multiple tests, you can create your own in the `test.beforeAll()` hook and close it in the `test.afterAll()` hook:

```typescript
import { test, Page } from '@playwright/test';

test.describe.configure({ mode: 'serial' });

let page: Page;
test.beforeAll(async ({ browser }) => {
  page = await browser.newPage();
});

test.afterAll(async () => {
  await page.close();
});

test('runs first', async () => {
  await page.goto('https://playwright.dev/');
});

test('runs second', async () => {
  await page.getByText('Get Started').click();
});
```

## Conclusion

In summary, Playwright offers various retry APIs to make your tests more resilient and less flaky. The built-in retry mechanism for locators and matchers covers most daily use cases. However, for assertions that need to wait for external conditions, you can use the explicit retry and polling APIs. Additionally, you can utilize the global retry mechanism for test cases to handle inconsistencies caused by conditions beyond your control.

By incorporating these retry strategies into your testing workflow, you can ensure a more robust and reliable testing experience, leading to higher-quality software and happier end-users.