Arisole Eval (arisole-eval)
Independent postprocessor for LeRobot evaluation artifacts (eval_info.json).
Affiliation: Built by Arisole (Neufin OÜ).
Not an official LeRobot or Hugging Face partnership or endorsement.
Five-minute quickstart
pip install arisole-eval==0.1.4
arisole quickstart --out-dir ./arisole-quickstart
Requires Python ≥ 3.11. No Arisole API key, GPU or robot. The bundled CloseFridge sample is educational OBSERVED_SIMULATION evidence (n=1 → expect INSUFFICIENT_DATA).
0.1.4: Source distribution includes gate YAML fixtures under tests/fixtures/configs/ so the full test suite runs from the published sdist without a private monorepo checkout. Gate logic unchanged from 0.1.3.
0.1.3: Gate evidence uses fully labelled seed-matched pairs only. INCOMPARABLE / missing success labels no longer inflate the CI evidence threshold or regression denominator; incomplete label coverage returns INSUFFICIENT_DATA (exit 2 by default).
0.1.2: Correctly parses multi-task per_task[].metrics.successes lists. Paired PASS_TO_FAIL analysis requires shared episode seeds; otherwise unpaired rates with uncertainty are reported.
Your checkpoints
arisole evaluate --baseline ./policy-v1 --candidate ./policy-v2 --out result.json
arisole report result.json
arisole gate result.json --config path/to/reliability.yml
Optional cloud upload: arisole upload result.json (requires ARISOLE_API_URL + ARISOLE_API_KEY).
Tutorial for Hub discussions: see monorepo docs/arisole-eval-hf-tutorial.md.
Educational sample — CloseFridge latency (OBSERVED_SIMULATION)
Verified open-loop demo action replay in RoboCasa/MuJoCo (seed 42, n=1):
| Field | Value |
|---|---|
| Evidence class | OBSERVED_SIMULATION |
| Transition | FAIL_TO_FAIL |
| Gate | INSUFFICIENT_DATA (paired_episodes=1 < 20) |
| Physical robot | No |
| SmolVLA Benchmark 001 | Not this pack (Benchmark 001 remains PENDING) |
This sample shows how the evaluator handles limited evidence. It is not:
- Successful robot task execution
- A validated checkpoint improvement
- A real-world robot result
- A statistically reliable benchmark
- An official LeRobot result
Source methodology: public evidence report
Developer docs: https://www.arisole.com/developers/lerobot
LeBoard reliability case study (independent)
Independent Arisole analysis of the public Apache-2.0 dataset rogation/leboard-training-runs (revision f25f8a7603fcca023d16694dd01b9cb8280741b8). This card does not host or mirror that dataset.
- Case study: https://www.arisole.com/developers/leboard-case-study
- Reproducibility bundle (scripts, SHA-256 manifest, outputs): https://github.com/stealthg0dd/arisole-leboard-reliability
Caveats: episode transition counts are exploratory (position pairing). Published arisole-eval reports unpaired comparisons for these multi-task files (no per-episode seeds). Not a LeBoard / LeRobot / Hugging Face endorsement.
License
Apache License 2.0 — see package LICENSE / NOTICE. Upstream dataset and simulator licenses still apply to referenced RoboCasa / LeRobot / LeBoard assets.