The Anatomy of Silent Data Corruption: GPU Error Pattern Study and Modeling Guidance
Paper • 2605.04213 • Published
Request access to card-check
The runner, the reference files and the evidence are released to named engineers at organisations that run GPU fleets. Requests are approved by hand, usually within a few hours.
Evaluation only. By requesting access you agree to the terms in LICENSE: you may run the software and reference files on hardware you own or control to evaluate whether it computes correctly, and send the resulting report to Unorganized Machines Inc. You may not redistribute the software or reference files, publish the reference hashes or any digest taken from them, use them commercially, create derivative reference files, or use them to build or operate a competing service, without written agreement.
Log in or Sign Up to review the conditions and access this model content.
card-check
For a given GPU model at a given driver and PyTorch version, this repository holds the exact output a healthy card produces for one training step. You run the same step on your card and compare one hash. It takes seconds per card and needs no network.
Why this exists
GPUs sometimes compute a wrong answer and raise no error. Google's Gemini reports expect silent data corruption "to impact training every week or two" and catch it with "lightweight deterministic replay" and "per-device intermediate checksums" [1]. ByteDance found in 2026 that fleets' synthetic microbenchmarks "miss over 60% of defective devices"; deterministic replay of the failing iteration located them [2]. A second ByteDance system found 18 incidents in 35 million GPU hours [3]. Meta counted 37 numerical or silent-corruption interruptions on a 32,000-GPU run [4]. Gate-level fault injection on a production-class GPU model produced NaN or infinity in 1.01% of corruptions; NaN checks catch almost nothing [5]. Fleet checks measure speed and connectivity; this one measures whether the arithmetic is right.
What it checks
One real training step of a small transformer: two layers, width 768, 12 heads, a vocabulary of 50,304, 8 sequences of 1,024 tokens, fp32, AdamW. Fixed seed, fixed inputs, on your card, with the pinned PyTorch build and the determinism settings the reference was made with. The step is defined by the reference file, not by the runner; the runner has no numbers of its own. The result is hashed twice over: every parameter after the step, both optimizer moments and the loss go into one hash, and every tensor of the step gets a hash of its own, so a difference can be placed at the first tensor that differs. The one hash is compared with the reference for your exact card model, driver, PyTorch build, SM count and MIG mode. MATCH means your card produced the same bytes a healthy card produced. MISMATCH means it did not. Three plain numbers travel with every run, the loss, the largest gradient and the largest parameter, so a small structured difference can be told from a large random one. The report also carries the card's identity: SM count, ECC and MIG modes, clocks, temperature and power, before and after the step.
What it does not check
One step exercises the parts of the chip that step exercises. This catches persistent faults and many intermittent ones. A fault in a path this step does not touch will not show. Memory errors, thermal throttling and link faults are not what this measures; keep running your existing checks for those. A card that agrees with itself but not with the reference may be a healthy card in a configuration we have not recorded; the output tells the two apart.
Run it
cd card-check
python -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt
python check.py
python check.py --all --repeat 20
On Debian and Ubuntu, python -m venv needs the python3-venv package; any other way of making a
virtual environment is fine, the check only needs the pinned torch in it.
requirements.txt pins torch 2.13.0 built for CUDA 13.0, from PyTorch's own wheel index: the build
the references were made with (the wheel PyPI serves for 2.13.0 today is the same build; the two
differ only in their version label). A different PyTorch build is a different configuration and
prints NO-REF by design; its hashes are still recorded.
The first run of check.py prints one line per visible card and writes out/report.json.
The second command is for a card you suspect: twenty runs and a summary line. Then send
out/report.json to the address on your invitation. It holds hashes, the card's identity and
telemetry, and three floats per run; no data and no weights leave your node. You get back a
written report on your fleet: which cards agree with the reference, which agree only with
themselves, and where each difference begins.
Reading the output
[card 0] NVIDIA H100 PCIe driver 580.126.09 torch 2.13.0+cu130 MATCH ref=H100-PCIe_580.126.09_torch-2.13.0+cu130
[card 0] NVIDIA H100 PCIe driver 580.126.09 torch 2.13.0+cu130 MISMATCH ref=H100-PCIe_580.126.09_torch-2.13.0+cu130 got=1faa6f1a want=0faa6f1a first=layer0.attn_out
[card 0] NVIDIA H100 PCIe driver 580.126.09 torch 2.13.0+cu130 NO-REF ref=none got=1faa6f1a
The three lines above are real output of an H100 PCIe; the second came from a test in which a digit of the expected step hash and a digit of the layer0.attn_out hash were changed, and the third from a run against a folder holding only another driver's reference.
MATCH: the card computed what a healthy card computes. Nothing to do.
MISMATCH: run the same card again with --repeat 20. If it disagrees with itself, the fault
is intermittent; drain the card. If it agrees with itself but not with the reference, check
that the driver and PyTorch version equal the reference's exactly; if they do, the card
produces wrong arithmetic; drain it. The first= field names the first tensor that differs,
in the order the step computes them.
NO-REF: your card, driver or PyTorch version is not covered yet. The step still runs and its
hashes are still recorded, with ref=none. Send the report; once we have recorded that
configuration on healthy cards, a reference follows and your report is read against it.
NO-REF (configuration differs: ...) means a reference exists for your card model, driver and
PyTorch build, but the card reports another SM count or MIG mode than the reference (a MIG slice
carries its parent's name with fewer SMs and computes different bits), or the MIG mode is unknown
because nvidia-smi is missing. That is another configuration, read against itself only.
With --repeat, each card prints one line per run and then a summary:
[card 0] summary: agrees with itself: yes (1 distinct digest); agrees with reference: 20/20
agrees with itself: no means the card is not deterministic: an intermittent fault. Drain it.
Where the references come from
Each reference was made by running this same check on a card of that model, at that driver
and PyTorch version, installed the way this README installs it, on cloud instances rented for
the purpose, and kept only when every run agreed. The reference file records the card, the
date, the runs and instances that agreed, the step definition, and the expected hash of every
tensor. References are added as configurations are recorded; the current list is the
contents of references/: an NVIDIA L4 (driver 580.178.04), an NVIDIA L40S (580.126.09) and an
NVIDIA H100 PCIe (580.126.09), all at torch 2.13.0+cu130, each from twenty-five runs of the step
in one session on 2026-09-24, one digest per card. One check takes about 12 s on the L4 and under 10 s on the L40S and the H100,
model construction and hashing included. The measurements behind the method are in EVIDENCE.md.
Terms
Evaluation only. See LICENSE.
[1] Gemini Team, Google. Gemini: A Family of Highly Capable Multimodal Models. arXiv:2312.11805, section 3, Training Infrastructure ("every week or two"); and Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities. arXiv:2507.06261, the pre-training infrastructure section, "Split-Phase SDC Detection".
[2] Wenxin Zheng et al. SDCs in the Wild: Characterizing and Diagnosing SDC-defective GPUs in Production LLM Training. OSDI 2026, the abstract ("miss over 60% of defective devices") and the localization phase, which deterministically replays the problematic iteration.
[3] Kinman Lei et al. Safeguarding LLM Training at Scale: Online SDC Detection and Insights from 35 Million GPU Hours. OSDI 2026, the abstract (18 SDC incidents and 13 faulty GPUs in a deployment of 3.5 x 10^7 GPU-hours).
[4] Omkar Salpekar et al., Meta. Training LLMs with Fault Tolerant HSDP on 100,000 GPUs. arXiv:2602.00277, Table 1 (Numerics/Silent Data Corruption: 37 interruptions, 5.5%, during pre-training on 32K GPUs).
[5] Chung-Hsuan Tung et al. The Anatomy of Silent Data Corruption: GPU Error Pattern Study and Modeling Guidance. DSN 2026 (Industry Track), arXiv:2605.04213, the abstract and Observation 1 (NaN and infinities are 1.01% of 600 M observed corruptions).
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
Papers for unorganized-machines/card-check
Training LLMs with Fault Tolerant HSDP on 100,000 GPUs
Paper • 2602.00277 • Published
Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
Paper • 2507.06261 • Published • 69
Gemini: A Family of Highly Capable Multimodal Models
Paper • 2312.11805 • Published • 50