End-to-end testing with our Chaos Stream Proxy
Tags: testing, tutorial, javascript, opensource
Published: 2022-06-22 15:55:51

Testing the behavior of a video player when there is **something wrong with the video source** used to be a tedious task: You had to either manually create or hunt down different broken streams to test different errors, and even then you couldn't always know for sure that the same error would always occur with the same timing.

To address this problem, Eyevinn has released its **Chaos Stream Proxy** as [open-source](https://github.com/Eyevinn/chaos-stream-proxy)! As the name suggests, this is a **very handy tool** for proxying an adaptive bitrate stream and **deterministically** introducing corruptions to it. 

This tutorial will demonstrate how to use the Proxy in the end-to-end testing of the open-source [Eyevinn WebPlayer](https://github.com/Eyevinn/web-player), using [Playwright](https://playwright.dev/).

## Using the Chaos Stream Proxy
A demo of the Chaos Stream Proxy, which we'll use for the purpose of this tutorial, is currently running [here](https://chaos-proxy.prod.eyevinn.technology).

The Proxy supports both HLS **and** MPEG-DASH streaming formats, and easily lets us add different corruptions by adding a stringified JSON object as a **query parameter** to the proxied URL. If we take this HLS URL:

```
https://maitv-vod.lab.eyevinn.technology/VINN.mp4/master.m3u8
```
it can be played with no problems, for example in the Eyevinn WebPlayer, [hosted here](https://web.player.eyevinn.technology/). But, if we proxy it through the Chaos Stream Proxy, and set `&statusCode=[{i:*,code:404}]`:

```
https://chaos-proxy.prod.eyevinn.technology/api/v2/manifests/hls/proxy-master.m3u8?url=https://maitv-vod.lab.eyevinn.technology/VINN.mp4/master.m3u8&statusCode=[{i:*,code:404}]
```
we'll instead get `404` errors for all segments (because we set `i:*`; it's also possible to specify individual segments).

For specifying other types of corruptions, check out [the project's README](https://github.com/Eyevinn/chaos-stream-proxy).

## EPAS and the Eyevinn WebPlayer
The Eyevinn WebPlayer implements [Eyevinn Player Analytics Specification (EPAS)](https://github.com/Eyevinn/player-analytics-specification), which is an open specification that defines a standard for implementing analytics in any video/audio player. This means that we have a format for event reporting that we expect the WebPlayer to follow. 

User interactions are easy enough to create for the sake of validating the EPAS implementation, and thanks to the Chaos Stream Proxy, manifest related errors are now also a piece of cake!

## Setting up Playwright
Playwright is a powerful end-to-end test runner, which we're using to automatically test our code in a real browser environment. At the time of writing, we're adding a dependency for `"@playwright/test": "^1.19.2"` in our `package.json`(this is a Node.js project). We also add the script `"test:e2e": "playwright test"`.

Our `playwright.config.ts` is in the project root, and looks like this: 

``` TypeScript
import { PlaywrightTestConfig, devices } from '@playwright/test';

const config: PlaywrightTestConfig = {
  forbidOnly: !!process.env.CI,
  retries: process.env.CI ? 2 : 0,
  testDir: "tests/",
  workers: 3,
  use: {
    trace: 'on-first-retry',
    // Necessary to get the media codecs to play video (default 'chromium' doesn't have them) 
    channel: 'chrome'
  },
  webServer: {
    command: 'npm run examples',
    port: 1234,
    reuseExistingServer: !process.env.CI,
  },
  projects: [
    {
      name: 'chromium',
      use: { ...devices['Desktop Chrome'] },
    },
  ],
};
export default config;
```

`testDir` points to the folder where we put our test `.spec.ts` files; `examples` in the `npm run examples` command is a folder in our root directory, in which we put the HTML files that Playwright will use.

**Note** that `channel: 'chrome'` is necessary for our tests to run; the default setting is `chromium`, [which doesn't include the necessary media codecs for video playback ](https://playwright.dev/docs/browsers#when-to-use-google-chrome--microsoft-edge-and-when-not-to). This may cause issues with automated workflows and running tests on Firefox and Webkit, but works locally in Chrome.

In our `examples` folder, we add an `index.html` HTML file that includes a link to our Chaos Stream Proxy HTML:

```HTML
<!DOCTYPE html>
<html>

<head>
  <meta charset="utf-8">
  <meta http-equiv="X-UA-Compatible" content="IE=edge">
  <title>Examples</title>
</head>
<body>
  <a href="chaos-proxy/index.html">Chaos Stream Proxy Example</a>
</html>
```
As well as the Chaos Stream Proxy HTML `index.html` file in the sub-folder `chaos-proxy`:

```HTML
<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
  <meta http-equiv="X-UA-Compatible" content="IE=edge">
  <title>Chaos Stream Proxy Example</title>
  <script async src="../../packages/web-player-component/dist/web-player.component.js"></script>
</head>
<body>
  <!-- VOD: With segment delay of 1500ms and response code 400 on sixth (response of 400 will be sent after 1500ms): -->
  <eyevinn-video
    source="https://chaos-proxy.prod.eyevinn.technology/api/v2/manifests/hls/proxy-master.m3u8?url=https://maitv-vod.lab.eyevinn.technology/VINN.mp4/master.m3u8&delay=[{i:5,ms:1500}]&statusCode=[{i:5,code:400}]"
    muted autoplay>
  </eyevinn-video>
  <!-- VOD: With response of status code 404 on all segments: -->
  <eyevinn-video
  source="https://chaos-proxy.prod.eyevinn.technology/api/v2/manifests/hls/proxy-master.m3u8?url=https://maitv-vod.lab.eyevinn.technology/VINN.mp4/master.m3u8&statusCode=[{i:*,code:404}]"
  muted autoplay>
</eyevinn-video>
</body>
</html>
```

This file imports the compiled version of [Eyevinn WebPlayer Component](https://dev.to/video/creating-a-video-player-web-component-3lgg) from the repository, which let's us embed the WebPlayer as web components with the `<eyevinn-video>` tags.

In our `tests` folder, we add `chaos-proxy.specs.ts`, which will be run automatically when we run the `npm run test:e2e` script:

```TypeScript
import { test } from '@playwright/test';

  test('player sends error events when loading corrupt streams', async ({page}) => {
    const [request] = await Promise.all([
      page.goto('/chaos-proxy/index.html'),
      page.waitForRequest(req => req.url().match('https://sink.epas') && req.method() === 'POST' && req.postDataJSON().event === "warning" && req.postDataJSON().payload.code === "400"),
      page.waitForRequest(req => req.url().match('https://sink.epas') && req.method() === 'POST' && req.postDataJSON().event === "warning" && req.postDataJSON().payload.code === "404"),
    ]);
  });
```

We make asynchronous calls to the [goto](https://playwright.dev/docs/api/class-page#page-goto) and [waitForRequest](https://playwright.dev/docs/api/class-page#page-wait-for-request) methods, available on the `page`object. We check that the POST [requests](https://playwright.dev/docs/api/class-request) conform to the expected format: If both matching requests are made within 30 seconds, the test passes!

We run the test with `npm run test:e2e`, and this should be the result: 
![Test ran successfully](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qfj73q04apkgx2k7x78w.png) 

Success!

##Bonus: Playwright Test for VSCode and debug mode 
Playwright is developed my Microsoft, so perhaps it shouldn't come as a surprise that there is [an excellent VS Code extension available](https://marketplace.visualstudio.com/items?itemName=ms-playwright.playwright).

With this extension, it's a breeze to run individual tests in isolation, add breakpoints and more! 


![Playwright Test for VS Code screenshot](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5457mfqi6e39kcs1joex.png)

It also gives easy access to debug mode, which lets us view and interact with the test in the test runner browser: 

![Debug mode screenshot](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qh5oj6n04wknqjzbfh79.png)