visualcheck 0.3.1
Visual regression checking with figma-first baselines, pixel+SSIM diffing, ignore regions, and HTML reports.
License: MIT
Requires Python: >=3.9
Keywords: visual-regression, playwright, ssim, qa, figma

# visualcheck

Visual regression checking with:

- **Figma-first baselines** (optionally synced via Figma API)
- **Runtime baselines** (auto-create when missing; never overwrite by default)
- **Pixel diff % + SSIM** (with optional resize-on-mismatch + WARN)
- **Ignore regions** (mask dynamic areas via selectors + explicit rects)
- **Self-contained HTML report** + `report.json`

## Install

```bash
pip install visualcheck
playwright install chromium
```

## Config (`visualcheck.yaml`)

```yaml
project: consumer_website
suite: daily_sanity

envs:
  prod: "https://example.com"

views:
  desktop_profiles: ["desktop_1440x900", "macbook_1440x900"]
  mobile_devices: ["iPhone 15 Pro Max", "iPhone 13 mini", "Pixel 7"]

pages:
  - id: home
    url: "/"
    wait_for: "body"
    full_page: true

# Optional flows (multi-step user journeys)
flows:
  - id: search_flow
    start_url: "/"
    steps:
      - action: click
        selector: "text=Search"
      - action: wait
        ms: 500
    snapshots:
      - id: after_search
        wait_for: "body"
        full_page: true

baseline:
  # Locked resolution order:
  # figma -> runtime -> create runtime baseline (if enabled)
  priority: ["figma", "runtime"]
  create_if_missing: true
  never_overwrite: true
  on_created: "INFO"  # INFO|WARN|FAIL
  # Optional legacy mode: copy figma image into runtime when figma is used.
  # Default behavior keeps runtime as website-captured truth.
  seed_runtime_from_figma: false

compare:
  resize_on_mismatch: true
  mismatch_level: "WARN"  # WARN|FAIL
  thresholds:
    max_pixel_diff_pct: 0.10
    min_ssim: 0.995
  # Optional per-snapshot override + region-level checks
  by_snapshot:
    home:
      max_pixel_diff_pct: 0.15
      min_ssim: 0.992
      regions:
        - name: header_strict
          x: 0
          y: 0
          width: 1440
          height: 120
          max_pixel_diff_pct: 0.03
          min_ssim: 0.998
  # Optional ignore-region suggestions from diff output
  suggest_ignore_regions:
    enabled: true
    min_area: 2500
    max_regions: 5

capture:
  # prefer Chrome channel (fallback to bundled Chromium if unavailable)
  browser_channel: "chrome"
  extra_wait_ms: 1000
  scrolls: 5
  scroll_delay_ms: 700
  retries: 1
  retry_delay_ms: 700
  # Optional page stabilization before screenshot
  stability_checks: 2
  stability_interval_ms: 250
  stability_timeout_ms: 3000
  screenshot_timeout_ms: 90000

stability:
  # auto-rerun unstable snapshots and choose final outcome by consensus
  reruns_on_warn: 1
  reruns_on_fail: 2
  require_consensus: "best"   # best|median|majority

quarantine:
  snapshots:
    - "search_flow__after_search"

owners:
  by_snapshot:
    home: "web-platform-team"
    "desktop_1440x900:pricing": "checkout-team"

ignore_regions:
  global:
    selectors: ["#cookie-banner", ".chat-widget"]
  by_snapshot:
    home:
      rects:
        - {x: 0, y: 0, width: 300, height: 120}

# Optional Figma sync (writes into visual_baseline/<project>/<suite>/figma/...)
# figma:
#   token_env: FIGMA_TOKEN
#   file_key: "<FIGMA_FILE_KEY>"
#   frames:
#     - id: home
#       node_id: "123:456"
#       view_id: "desktop_1440x900"   # optional
```

## Commands

### Run full check

```bash
visualcheck run --env prod
```

Outputs:

- Baselines:
  - `visual_baseline/<project>/<suite>/figma/<view>/<snapshot>.png`
  - `visual_baseline/<project>/<suite>/runtime/<view>/<snapshot>.png`
- Run artifacts:
  - `test_report/<project>/visual_runs/<run_id>/current/...`
  - `test_report/<project>/visual_runs/<run_id>/diff/...`
  - `test_report/<project>/visual_runs/<run_id>/report/report.html`
  - `test_report/<project>/visual_runs/<run_id>/report/report.json`
  - `test_report/<project>/visual_runs/<run_id>/report/report.junit.xml`

### Capture only

```bash
visualcheck capture --env prod --out current
```

### Diff only

```bash
visualcheck diff --baseline visual_baseline/myproj/mysuite/runtime --current current --out report
```

### Sync Figma baselines

```bash
export FIGMA_TOKEN="..."
visualcheck figma-sync
# force refresh selected frames:
visualcheck figma-sync --force --only home,pricing
# auto-map page/flow snapshot ids to figma frame names:
visualcheck figma-sync --match-by-name
```

### Compare strictly against Figma baseline

```bash
visualcheck figma-diff --env prod
```

### Approve current run as runtime baseline (explicit)

```bash
visualcheck approve --env prod --run-id 20260207_235500
# Overwrite existing runtime baselines only with:
visualcheck approve --env prod --run-id 20260207_235500 --force
```

### Validate config

```bash
visualcheck validate-config --config visualcheck.yaml
```

### Cleanup old run artifacts

```bash
visualcheck cleanup --config visualcheck.yaml --keep-last 20 --dry-run
visualcheck cleanup --config visualcheck.yaml --keep-last 20
```

### Quarantine management

```bash
visualcheck quarantine-list --config visualcheck.yaml
visualcheck quarantine-add --config visualcheck.yaml --snapshot search_flow__after_search
visualcheck quarantine-remove --config visualcheck.yaml --snapshot search_flow__after_search
```

## Baseline rules (locked)

For each snapshot + view:

1) If a **Figma baseline** exists → use it
   - visualcheck will attempt a best-effort `figma-sync` when `figma` is listed in `baseline.priority` so remote Figma baselines are consulted before falling back.
2) Else if a **runtime baseline** exists → use it
3) Else → capture and **create runtime baseline** (controlled by `baseline.create_if_missing`)

Notes on configuration and behavior

- `baseline.priority` controls the preference order. Example:
  `priority: ["figma","runtime"]` (default in examples) — visualcheck will try Figma first.
- By default a Figma baseline is used directly for comparison (reported baseline path will point at `.../figma/...`).
- By default, Figma does **not** auto-create runtime baseline copies.
  Runtime remains website-captured truth (created from current captures when missing).
- If you want legacy behavior (copy Figma into runtime), set:

```yaml
baseline:
  seed_runtime_from_figma: true
```
- If you want Figma to become the authoritative runtime baseline (copied into the `runtime/` folder and then used), set:

```yaml
baseline:
  figma_authoritative: true
```

- This avoids accidental comparisons against an old runtime image when Figma is the source of truth.
- If a figma baseline is missing locally and `figma` appears in priority, visualcheck will attempt to fetch it via the Figma API (best-effort). If the fetch fails (token/permission), it falls back to runtime per `priority`.

Figma metadata in report

- When a snapshot resolves to a Figma frame, report rows include `figma.file_key`, `figma.node_id`, and a direct `figma.frame_url`.

Report upgrades

- HTML report includes status summary cards, client-side status filters, top regressions, baseline/current slider, and optional ignore-region suggestions.
- Rows include plain-language issue summary and recommended action to make pixel triage faster.
- Report includes flake score, root-cause hint, snapshot owner, quarantine status, and likely-introduced-after commit hint.
- History files are stored under `test_report/<project>/history/` for trends, flake scoring, and change correlation.


## Use from code (framework integration)

If you already navigate with **Playwright**/**Selenium** in your own framework and just want visualcheck to **capture + baseline + diff + report**, use the code API.

> Baselines still go to `visual_baseline/<project>/<suite>/...` and the HTML report goes to `test_report/<project>/visual_runs/<run_id>/report/report.html`.

### Playwright (sync) example

```python
from playwright.sync_api import sync_playwright

# NOTE: API object names may evolve; refer to the package docs in case of changes.
from visualcheck.api import VisualCheck

vc = VisualCheck(
    project="my_project",
    suite="daily_sanity",
    env="prod",
    base_url="https://example.com",
    view_id="desktop_1440x900",
    baseline_priority=["figma", "runtime"],
    create_if_missing=True,
    ignore_selectors=["#cookie-banner", ".chat-widget"],
)

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page(viewport={"width": 1440, "height": 900})

    page.goto("https://example.com/", wait_until="domcontentloaded")
    vc.check(snapshot_id="home", page=page)

    page.goto("https://example.com/pricing", wait_until="domcontentloaded")
    vc.check(snapshot_id="pricing", page=page)

    vc.finalize()
    print("Report:", vc.report_html)

    browser.close()
```

### Selenium example

```python
from selenium import webdriver
from visualcheck.api import VisualCheck

vc = VisualCheck(
    project="my_project",
    suite="daily_sanity",
    env="prod",
    base_url="https://example.com",
    view_id="desktop_1440x900",
)

driver = webdriver.Chrome()
driver.set_window_size(1440, 900)

driver.get("https://example.com/")
vc.check(snapshot_id="home", selenium_driver=driver)

driver.get("https://example.com/pricing")
vc.check(snapshot_id="pricing", selenium_driver=driver)

vc.finalize()
print("Report:", vc.report_html)

driver.quit()
```

## Notes

- Ignore regions are masked with a solid color before diffing.
- If screenshot sizes differ and `compare.resize_on_mismatch=true`, current is resized to baseline size and a warning is recorded.

## License
MIT


## Library & CLI usage (examples)

### Programmatic API (Python)

The package exposes a simple programmatic API to integrate with Playwright or Selenium.

Example (Playwright):

```python
from visualcheck.runner import run_visualcheck
from visualcheck.config import load_config

cfg = load_config(Path('examples/wakefit.yaml'))
result = run_visualcheck(cfg, env='prod', headed=False)
print('Report saved at', result.get('report_html'))
```

### CLI usage

- Sync Figma baselines:

```bash
export FIGMA_TOKEN="<your_token>"
visualcheck figma-sync --config examples/wakefit.yaml
```

- Run full check:

```bash
visualcheck run --env prod --config examples/wakefit.yaml
```

### Notes on FIGMA token & env handling

- The library reads Figma token from the environment variable name specified in config (defaults to `FIGMA_TOKEN`).
- Keep secrets out of git; use host-level secrets (e.g. `~/.openclaw/secrets.env`) or a CI secret store.