Repository: GoogleChrome/lighthouse-ci

Issue #570: Upload - Assertion failure for assertions with level error not actual checking passed value
Created: 2021-03-12T10:10:07Z
Describe the bug
When uploading assertion results to LHCI server. It is incorrectly setting the github status check and marking it as failed.
Looking at the code It is checking for 'error' level but then not checking the actual passed parameter. So it seems that if any assertions set to error will always fail the Github status check.
To Reproduce
Steps to reproduce the behavior:

Collect tests
Audit Tests
Upload Tests with GithubToken
See Error

Expected behavior
When the assertion is marked as not passed then it should mark the github status as failed. Not just because it's been set as an error level assertion.
Logs/Screenshots
If applicable, add LHCI logs or screenshots to help explain your problem.
+ ls .lighthouseci
assertion-results.json
lhr-1615540607976.html
lhr-1615540607976.json
lhr-1615540612869.html
lhr-1615540612869.json
lhr-1615540617782.html
lhr-1615540617782.json
+ cat .lighthouseci/assertion-results.json
[
  {
    "name": "minScore",
    "expected": 0.98,
    "actual": 0.99,
    "values": [
      0.99,
      0.99,
      0.99
    ],
    "operator": ">=",
    "passed": true,
    "auditProperty": "performance",
    "auditId": "categories",
    "level": "error",
    "url": "http://run-server:5000/"
  },
  {
    "name": "minScore",
    "expected": 0.95,
    "actual": 1,
    "values": [
      1,
      1,
      1
    ],
    "operator": ">=",
    "passed": true,
    "auditId": "cumulative-layout-shift",
    "level": "warn",
    "url": "http://run-server:5000/",
    "auditTitle": "Cumulative Layout Shift",
    "auditDocumentationLink": "https://web.dev/cls/"
  },
  {
    "name": "minScore",
    "expected": 0.95,
    "actual": 1,
    "values": [
      1,
      1,
      1
    ],
    "operator": ">=",
    "passed": true,
    "auditId": "first-contentful-paint",
    "level": "warn",
    "url": "http://run-server:5000/",
    "auditTitle": "First Contentful Paint",
    "auditDocumentationLink": "https://web.dev/first-contentful-paint/"
  },
  {
    "name": "minScore",
    "expected": 0.95,
    "actual": 1,
    "values": [
      1,
      1,
      1
    ],
    "operator": ">=",
    "passed": true,
    "auditId": "interactive",
    "level": "warn",
    "url": "http://run-server:5000/",
    "auditTitle": "Time to Interactive",
    "auditDocumentationLink": "https://web.dev/interactive/"
  },
  {
    "name": "minScore",
    "expected": 0.95,
    "actual": 0.97,
    "values": [
      0.97,
      0.97,
      0.97
    ],
    "operator": ">=",
    "passed": true,
    "auditId": "largest-contentful-paint",
    "level": "warn",
    "url": "http://run-server:5000/",
    "auditTitle": "Largest Contentful Paint",
    "auditDocumentationLink": "https://web.dev/lighthouse-largest-contentful-paint/"
  },
  {
    "name": "minScore",
    "expected": 0.95,
    "actual": 1,
    "values": [
      1,
      1,
      1
    ],
    "operator": ">=",
    "passed": true,
    "auditId": "speed-index",
    "level": "warn",
    "url": "http://run-server:5000/",
    "auditTitle": "Speed Index",
    "auditDocumentationLink": "https://web.dev/speed-index/"
  },
  {
    "name": "minScore",
    "expected": 0.95,
    "actual": 1,
    "values": [
      1,
      1,
      1
    ],
    "operator": ">=",
    "passed": true,
    "auditId": "total-blocking-time",
    "level": "warn",
    "url": "http://run-server:5000/",
    "auditTitle": "Total Blocking Time",
    "auditDocumentationLink": "https://web.dev/lighthouse-total-blocking-time/"
  }
]+ npm run test:perf:upload -- --githubToken=$GITHUB_TOKEN

> @domain-internal/cypress@0.0.0-377 test:perf:upload /drone/src/github.domain.net/web-engineering/cypress
> lhci upload "--githubToken={{GITHUB_TOKEN}}"

Saving CI project cypress (78b524fa-0249-4c64-a084-82f6b61c5e56)
Saving CI build ({{BUILDTOKEN}})
Saved LHR to https://lighthouse-ci.domain.io/ (a754f305-4152-4ab7-89f6-d8f1228f6293)
Saved LHR to https://lighthouse-ci.domain.io/ (f547531f-38cc-4ae5-9b3c-149b6fc7a522)
Saved LHR to https://lighthouse-ci.domain.io/ (54ebf667-27a4-48bb-afa1-6ff8849c8257)
Done saving build results to Lighthouse CI
View build diff at https://lighthouse-ci.domain.io/app/projects/cypress/compare/{{BUILD_TOKEN}}
GitHub token found, attempting to set status...
GitHub accepted "failure" status for "lhci/url/".

Environment (please complete the following information):

OS: Linux
Browser chrome
Version [Chrome: 89.0.4389.82, LighthouseCI: 0.7.0]

Additional context
Will raise a PR with a suitable fix.

Duplicate issue #588: GitHub statuses marked as failed when they pass
Created: 2021-04-19T17:16:58Z
Describe the bug
With assert.includePassedAssertions: true, GitHub statuses on PR are marked as "failed" even when they pass.
I've narrowed down to the upload run command checking for error levels instead of if the assertion passed when preparing the github status messages.
When assert.includePassedAssertions is true, passed assertions are saved in .lighthouseci/assertion-results.json. So loadAssertionResults() returns passed and failed assertions.
So the following two lines:
const failedResults = group.filter(result => result.level === 'error');
const warnResults = group.filter(result => result.level === 'warn');
should be the following:
const failedResults = group.filter(result => result.level === 'error' && !result.passed);
const warnResults = group.filter(result => result.level === 'warn' && !result.passed);
To Reproduce
Steps to reproduce the behavior:

Setup a GitHub action with a LHCI_GITHUB_TOKEN
Use assertions with includePassedAssertions: true
Visit pages that pass the assertions

Expected behavior

GitHub status per URL is green.

Actual behavior

GitHub status per URL are red.

More context
Partial contents of .lighthouseci/assertion-results.json
[
  {
    "name": "minScore",
    "expected": 0.9,
    "actual": 0.93,
    "values": [
      0.93,
      0.93,
      0.93
    ],
    "operator": ">=",
    "passed": true,
    "auditProperty": "performance",
    "auditId": "categories",
    "level": "error",
    "url": "https://example.com"
  }
]

Closing comment (2021-04-19T17:31:46Z): thanks for filing @charlespwd! this is a duplicate of #570 and will be fixed by #575, we'll keep tracking over there