growing-intelligence/probe56-counterfact

Probe56 — CounterFact reproducibility package

Llama-3.1-8B · 1,000 edits · 492 nearby facts · 5,209 held-out

Method	Edits fixed	Nearby facts broken	General benchmarks
ROME	7 / 1,000	492 / 492	-33.12 pp
MEMIT	916 / 1,000	48 / 492	-0.10 pp
AlphaEdit	990 / 1,000	202 / 492	-1.96 pp
Probe56	841 / 1,000	0 / 492	0.00 pp

What this repository is

An encrypted runtime plugin and a runner. Run it on the unmodified base model and you get the numbers in the Probe56 row. You cannot open it: the plugin is encrypted and signed with the Growing Intelligence signing key, and the runner decrypts it in memory only. It shows the result. It does not show the method.

• probe56_package_v4/probe56.gix — the encrypted, signed plugin
• probe56_package_v4/run_probe56 — Linux x86-64 executable (4 GB, self-contained; no Python, no pip, no internet)
• probe56_package_v4/cf_harness.py — the open scoring harness (scoring only), the same one that scored ROME, MEMIT and AlphaEdit
• probe56_package_v4/data/ — the 1,000 edits, 500 controls and 5,209 held-out questions
• probe56_package_v4/expected.json, MANIFEST.json, SHA256SUMS — the certified values and the hash of every file

Requirements: one 80 GB NVIDIA GPU (H100 class), CUDA 12.8 driver, and the unmodified meta-llama/Llama-3.1-8B as a local directory (gated by Meta; steps in probe56_package_v4/README.md). Its file hashes are checked against the certificate before anything runs.

One command

hf download growing-intelligence/probe56-counterfact --local-dir . && cd probe56_package_v4 && sha256sum -c SHA256SUMS && chmod +x run_probe56 && ./run_probe56 --model /path/to/llama-3.1-8b --plugin probe56.gix --out results

About 30 minutes on an H100. The run ends with RUN_PROBE56 PASS and the numbers: edits 841, all known controls kept, 0 broken, held-out identical to the unmodified model.

The full comparison

Report, the five baseline reproducibility packages, the signed certificate and the Merkle root: https://github.com/growing-intelligence/counterfact-comparison

---

Growing Intelligence · Pardes Hanna, Israel · growing-intelligence.com Method patent-pending (USPTO 64/029,741, 64/048,143). Results public. Method private.