testing-guidelines/playwright.md

Playwright Tips

Gradio uses playwright to interact with gradio applications programmatically to ensure that both the frontend and backend function as expected. Playwright is very powerful but it can be a little intimidating if you haven't used it before. No one on the team is a testing expert so don't be afraid to ask if you're unsure how to do something. Likewise, if you learn something new about playwright, please share with the team!

Tip 1 - Retrying Assertions

Playwright tests are written imperatively - first type into this textbox, then click this button, then check this textbox has this output. This is nice because it matches how users interact with Gradio applications. However, playwright carries out these steps much faster than any human can! This can cause you to check whether a textbox has the correct output before the server is finished processing the request.

For this reason, playwright ships with some retrying assertions. These assertions will retry until they pass or a timeout is reached, by default 5 seconds. So even if playwright checks a DOM element before the server is done, it gives the server a chance to finish by retrying.

An example of a retrying assertion is toBeChecked. Note that you can manually increase the timeout as well:

// 5 seconds
await expect(page.getByTestId('checkbox')).toBeChecked({timeout?: 5000});

An example of a non-retrying assertion is isChecked:

await expect(page.getByTestId('checkbox').isChecked())

Sometimes there may not be a retrying assertion for what you need to check. In that case, you can retry any custom async function until it passes using toPass (docs).

await expect(async () => {
  const response = await page.request.get('https://api.example.com');
  expect(response.status()).toBe(200);
}).toPass();

Tip 2 - Don't rely on internal network calls to check if something is done

Internal network calls are not visible to the user, so they can be refactored whenever. If we have tests that rely on a request to a given route finishing before moving on, for example, they will fail if we ever change the route name or some other implementation detail.

It's much better to use a retrying assertion that targets a visible DOM element with a larger timeout to check if some work is done.

Avoid this:

const uploadButton = page...
await uploadButton.click();
await page.waitForRequest("**/upload?*");
await expect(page.getByTestId("file-component")).toHaveValue(...)

Do This:

const uploadButton = page...
await uploadButton.click();
await expect(page.getByTestId("file-component")).toHaveValue(..., {timeout?: 5000});

Tip 3 - Use the playwright trace viewer

Whenever a test fails locally, playwright will write out some details about the test to the test-results directory at the top level of the repo.

You can view the trace using the following command:

npx playwright show-trace test-results/<directory-name>/trace.zip

You can see a "video" of the failing test, a screenshot of when it failed, as well as all the network calls and console messages.

If a test fails on CI, you can obtain the same trace by downloading the artifact from github actions.

1. From the failing Github Actions page, go to the Summary page
2. Scroll down to the bottom to where it says Artifacts
3. Click on playwright-screenshots to download a zip archive.
4. Unzip it and use the show-trace command.

Tip 4 - Playwright can write the test for you

You can write the basic skeleton of the test automatically by just interacting with the UI!

First, start a gradio demo from the command line. Then use the following command and point it to the URL of the running demo:

npx playwright codegen <url>

This will open up a Chromium session where each interaction with the page will be converted into a playwright accessor.

NOTE: Only copy the test("test-name", ....) not the imports. For playwright to work when running in the gradio CI, test and expect need to be imported from @gradio/tootils.