Repository: flutter
Issue: Golden tests paint different pixels using same Docker image on different platforms
Labels: a: tests,framework,P2,team-framework,triaged-framework

A notorious issue with golden tests is that pixels are painted differently on different platforms. Mac, Windows, and Linux host platforms will paint slightly different shadows, text anti-aliasing, etc. This poses a systemic issue when developers on a project work on different host machines, which is pretty much any project that involves 2+ developers.

To get around this issue, when developers on our project run golden tests locally, they do so through a Docker image, based on Ubuntu. The goal is to avoid any pixel differences when the same painting calls are run on different host machines.

Unfortunately, we're seeing slight pixel differences even with this approach. We generate goldens with the Docker image on Windows, and then we run the goldens with the Docker image on Mac, and we get a random smattering of mismatched pixels.

Why would we get any difference in pixel output at all when we're running the same Ubuntu Docker image? Can something be done to prevent that? I know that we can choose to ignore small pixel count mismatches, but it's unclear how many pixels we should ignore. Sometimes this issue is only a few pixels, and in other cases we've seen 30 pixel and 90 pixel mismatches. The robust answer is to have some approach that truly runs the same painting calls no matter where it's executed.

Here are two examples of this behavior. The first example has a single pixel mismatch. The second example has a three pixel mismatch.

Isolated Diff 1:
![super_textfield_ios_toolbar_pointing_down_isolatedDiff](https://github.com/flutter/flutter/assets/7259036/b5cc988d-4737-467d-9ad2-55adbcfe7478)

Masked Diff 1:
![super_textfield_ios_toolbar_pointing_down_maskedDiff](https://github.com/flutter/flutter/assets/7259036/09989a28-568c-4ec2-aa3d-01a10a00b20b)

Master Image 1:
![super_textfield_ios_toolbar_pointing_down_masterImage](https://github.com/flutter/flutter/assets/7259036/e88a704a-e452-4d84-9647-c16c44e8ccf1)

Test Image 1:
![super_textfield_ios_toolbar_pointing_down_testImage](https://github.com/flutter/flutter/assets/7259036/3324997a-194e-4cbe-88ce-ebbe4b4324c6)

Isolated Diff 2:
![super_textfield_ios_toolbar_pointing_up_isolatedDiff](https://github.com/flutter/flutter/assets/7259036/f9258acb-45d9-4e1d-b0bd-2925abc8122d)

Masked Diff 2:
![super_textfield_ios_toolbar_pointing_up_maskedDiff](https://github.com/flutter/flutter/assets/7259036/d01274ac-f897-49a0-a9cc-01e2c6797f0c)

Master Image 2:
![super_textfield_ios_toolbar_pointing_up_masterImage](https://github.com/flutter/flutter/assets/7259036/e22eba07-69f9-469d-a663-98e5269cb17c)

Test Image 2:
![super_textfield_ios_toolbar_pointing_up_testImage](https://github.com/flutter/flutter/assets/7259036/4072bcef-0255-41a7-8c60-a0e99bda95ba)

Here's our Dockerfile:

```
FROM ubuntu:latest

ENV FLUTTER_HOME=${HOME}/sdks/flutter 
ENV PATH ${PATH}:${FLUTTER_HOME}/bin:${FLUTTER_HOME}/bin/cache/dart-sdk/bin

USER root

RUN apt update

RUN apt install -y git curl unzip

# Invalidate the cache when flutter pushes a new commit.
ADD https://api.github.com/repos/flutter/flutter/git/refs/heads/master ./flutter-latest-master

RUN git clone https://github.com/flutter/flutter.git ${FLUTTER_HOME}

RUN flutter doctor

# Copy the whole repo.
# We need this because we use local dependencies.
COPY ./ /super_editor
```

We use a Dart script to run the tests in the Docker image. The relevant segment is this:

```dart
// Runs the container.
    //
    // --rm: Removes the container when it exits.
    //
    // --workdir: Sets the working directory to /super_editor/super_editor in the container.
    await _runProcess(
      exe: 'docker',
      arguments: [
        'run',
        '--rm',
        ...volumeMappings,
        '--workdir',
        '/super_editor/super_editor',
        'supereditor_golden_tester',
        'flutter',
        'test',
        testDirectory,
        ...rest,
        ...cmdArguments,
      ],
      description: 'Golden tests',
    );```