Container Queries: Cross-Resolution Testing
Tags: ember, javascript, testing, showdev
Published: 2020-06-08 13:07:49

_Originally published on [crunchingnumbers.live](https://crunchingnumbers.live)_

_Failure_ to test is why I started to work on [ember-container-query](https://github.com/ijlee2/ember-container-query/).

A few months ago, my team and I introduced [ember-fill-up](https://github.com/chadian/ember-fill-up) to our apps. It worked well but we noticed something strange. Percy snapshots that were taken at a mobile width would show `ember-fill-up` using the desktop breakpoint. They didn't match what we were seeing on our browsers.

For a while, we ignored this issue because our CSS wasn't great. We performed some tricks with `flex` and `position` that could have affected Percy snapshots. Guess what happened when we switched to `grid` and improved the document flow. We still saw incorrect Percy snapshots.


## 1. Whodunit? ember-qunit.

To eliminate `ember-fill-up` as a suspect, I used modifiers to recreate the addon. To my surprise and distress, using modifiers didn't fix the problem. After many trial-and-errors, I found the culprit: `ember-qunit`.

<figure>
<img src="https://crunchingnumbersdotlive.files.wordpress.com/2020/06/meme-1.jpg" alt="" width="600" height="300" />
<figcaption>TIL Morpheus never said "What if I told you." Were the past 20 years real?</figcaption>
</figure>

By default, `ember-qunit` [scales the test window](https://github.com/emberjs/ember-qunit/blob/master/vendor/ember-qunit/test-container-styles.css#L32-L37) so that your app fits inside its test container.

```css
#ember-testing {
  width: 200%;
  height: 200%;
  transform: scale(0.5);
  transform-origin: top left;
}
```

What does this mean? When you write tests, you cannot trust DOM render decisions that are based on width or height. Decisions made by media queries and addons like `ember-container-query`, `ember-fill-up`, `ember-responsive`, and `ember-screen`. Because what your test saw differed from what you saw on a browser, you might have had to mock a service (fake the window size) to get certain elements to (dis)appear.

<figure>
<img src="https://crunchingnumbersdotlive.files.wordpress.com/2020/06/test-comparisons.jpg" alt="" width="600" height="415" />
<figcaption>Percy snapshots on top row are what we get with default <code>ember-qunit</code>. The snapshots on the middle row are what we want. There seem to be larger differences in smaller windows. (<a href="https://crunchingnumbersdotlive.files.wordpress.com/2020/06/test-comparisons.jpg" target="_blank" rel="noreferrer noopener">Full screen image</a>)</figcaption>
</figure>

Fortunately, there is an [escape hatch](https://github.com/emberjs/ember-qunit/blob/master/vendor/ember-qunit/test-container-styles.css#L39-L44). We can apply the `.full-screen` class to the test container (`#ember-testing-container`) to undo the scaling.

```css
.full-screen #ember-testing {
  position: absolute;
  width: 100%;
  height: 100%;
  transform: scale(1);
}
```

As an aside, this class is applied when we enable [development mode](https://github.com/ember-cli/ember-cli-qunit/pull/120), a relatively unknown feature.

In my opinion and conjecture, we (Ember community) didn't really notice this problem and fix it because we were used to writing tests only at 1 resolution: the 1440 × 900 px desktop. We are also prone to designing the web for desktop first. Had we been able to test multiple resolutions easily, I think today's state of testing would be even better.


## 2. Cross-Resolution Testing

How can you test your app and addon at multiple resolutions?

We need to be able to mark tests that can only be run at one resolution. After all, there will be user workflows that make sense only on mobile or tablet, for example. My team and I followed the Octane craze and introduced **filters** that look like decorators:

```javascript
// Mobile only
test('@mobile A user can do X in Dashboard');

// Tablet only
test('@tablet A user can do X in Dashboard');

// Any resolution
test('A user can do X in Dashboard');
```

Let's go over how to update your test setup, configure CI, and write a Percy test helper to allow these filters.

I'll use [GitHub Actions for CI](https://crunchingnumbers.live/2020/03/17/ci-with-github-actions-for-ember-apps/). Describing each line of code can get boring so I'll convey the idea and simplify code in many cases. I encourage you to look at [ember-container-query](https://crunchingnumbers.live/2020/03/17/ci-with-github-actions-for-ember-apps/) to study details and use my latest code.


### a. testem.js

We'll start by updating `testem.js`. It is responsible for setting the window size.

The idea is to dynamically set the window size based on an environment variable. I'll call this variable `DEVICE`.

```javascript
const FILTERS = {
  mobile: '/^(?=(.*Acceptance))(?!(.*@tablet|.*@desktop))/',
  tablet: '/^(?=(.*Acceptance))(?!(.*@mobile|.*@desktop))/',
  desktop: '/^(?!(.*@mobile|.*@tablet))/'
};

const WINDOW_SIZES = {
  mobile: '400,900',
  tablet: '900,900',
  desktop: '1400,900'
};

const { DEVICE = 'desktop' } = process.env;

const filter = encodeURIComponent(FILTERS[DEVICE]);
const windowSize = WINDOW_SIZES[DEVICE];
const [width, height] = windowSize.split(',');

module.exports = {
  test_page: `tests/index.html?filter=${filter}&amp;width=${width}&amp;height=${height}`,
  browser_args: {
    Chrome: {
      ci: [
        `--window-size=${windowSize}`
      ]
    }
  }
};
```

From lines 15-16, we see that `DEVICE` decides how tests are run. In QUnit, we can use regular expressions to filter tests. I used **lookaheads** to say, "When `DEVICE=mobile`, only run application tests with `@mobile` filter or application tests without any filter." I decided to run rendering and unit tests only when `DEVICE=desktop` because they are likely independent of window size.

On line 20, the query parameters `width` and `height` are extra and have an important role. I'll explain why they are needed when we write the test helper for Percy.


### b. Reset Viewport

Next, we need to apply the `.full-screen` class to the test container.

There are two options. We can create a test helper if there are few application test files (likely for an addon), or an initializer if we have many (likely for an app).

```javascript
// Test helper
export default function resetViewport(hooks) {
  hooks.beforeEach(function() {
    let testingContainer = document.getElementById('ember-testing-container');
    testingContainer.classList.add('full-screen');
  });
}


// Initializer
import config from 'my-app-name/config/environment';

export function initialize() {
  if (config.environment === 'test') {
    let testingContainer = document.getElementById('ember-testing-container');
    testingContainer.classList.add('full-screen');
  }
}

export default {
  initialize
}
```


### c. package.json

The last step for an MVP (minimum viable product) is to update the test scripts.

Since Ember 3.17, `npm-run-all` has been available to run scripts in parallel. I'll assume that you also have `ember-exam` and `@percy/ember`.

```json
{
  "scripts": {
    "test": "npm-run-all --parallel test:*",
    "test:desktop": "percy exec -- ember exam --test-port=7357",
    "test:mobile": "DEVICE=mobile percy exec -- ember exam --test-port=7358",
    "test:tablet": "DEVICE=tablet percy exec -- ember exam --test-port=7359"
  }
}
```

In addition to setting `DEVICE`, it's crucial to use different port numbers. Now, we can run `yarn test` to check our app at 3 window sizes. If you have disparate amounts of tests for desktop, mobile, and tablet, you can set different `--split` values so that you allocate more partitions to one window size. For example, 4 partitions to desktop, 2 to mobile, and 1 to tablet.


### d. CI

Your code change may depend on what features your CI provider offers and how many `ember-exam` partitions you used to test a window size. I don't know what your CI looks like right now, so I'll make a handwave.

In `ember-container-query`, I didn't split tests into multiple partitions. There simply weren't that many. As a result, I was able to use `matrix` to simplify the workflow:

```yaml
jobs:
  test-addon:
    strategy:
      matrix:
        device: [desktop, mobile, tablet]
    steps:
      - name: Test
        uses: percy/exec-action@v0.3.0
        run:
          custom-command: yarn test:${{ matrix.device }}
```


### e. Test Helper for Percy

The end is the beginning is the end. We want to write a test helper for Percy, the thing that launched me into a journey of discovery.

In its simplest form, the test helper understands filters and knows the window size. It also generates a unique snapshot name that is human readable.

```javascript
import percySnapshot from '@percy/ember';

export default async function takeSnapshot(qunitAssert) {
  const name = getName(qunitAssert);
  const { height, width } = getWindowSize();

  await percySnapshot(name, {
    widths: [width],
    minHeight: height
  });
}

function getName(qunitAssert) { ... }

function getWindowSize() {
  const queryParams = new URLSearchParams(window.location.search);

  return {
    height: Number(queryParams.get('height')),
    width: Number(queryParams.get('width'))
  };
}
```

On line 13, I hid implementation details. The idea is to transform QUnit's `assert` object into a string.

Line 16 is the interesting bit. Earlier, when we updated `testem.js`, I mentioned passing width and height as query parameters. I tried two other approaches before.

In my first attempt, I stored `process.env.DEVICE` in `config/environment.js` and imported the file to the test helper file. From `WINDOW_SIZES`, one can find out width and height from `DEVICE`. For QUnit, this worked. For Percy, it did not. Since `v2.x`, Percy doesn't hook into the Ember build pipeline so `DEVICE` was `undefined`.

In my second attempt, I used `window.innerWidth` and `window.innerHeight` to get direct measurements. `innerWidth` gave the correct width, but `innerHeight` turned out to be unreliable. Because I wanted to test at multiple widths and multiple heights, I rejected this approach as well.


## 3. How to Run Tests

After we make these changes, an important question remains. How do we run tests locally?

- `yarn test` to run all desktop, mobile, and tablet tests in parallel
- `yarn test:desktop --server` to run all desktop tests with `--server` option
- `DEVICE=mobile ember test --filter='@mobile A user can do X in Dashboard'` to run a particular test


## 4. What's Next?

On the long horizon, I'd like us to reexamine and change why we are currently limited to testing 1 resolution. Ember's testing story is an already amazing one. I believe the ability to test multiple resolutions (and do so easily without taking 5 steps like above) would make that story even better.

For nearer goals, I'd like us to iron out a couple of issues in overriding `ember-qunit`:

- Even with `.full-screen`, the test container's height can be off if we use `--server` to launch the test browser. If assertions sometimes fail due to incorrect window size, it's harder to separate true and false positives.
- Visiting `localhost:4200/tests` to start tests will also throw off the test container's width and height. It may be impractical to ask developers to run tests with `--server` because this does not launch Ember Inspector.

We need to look at allowing cross-resolution testing for `ember-mocha` as well.


## 5. Notes

Special thanks to my team Sean Massa, Trek Glowacki, and Saf Suleman for trying out a dangerously unproven, new testing approach with me.