HELM 1B: C4 2B attention operand experiments
Six eager HELM runs use the same width-910, 16-layer model and the 61,000-step 2B recipe. Scaled4/Scaled8 and published Unit4/Unit8 emulate E2M1/E4M3 rounding of the operands to both attention geometry operations, in blocks of 32. The rest of training is BF16 with FP32 parameters; these are not packed 4-bit/8-bit model weights. Native BF16 Unit seeds 7/99 have no rounding hooks and no FP32 geometry policy.
| Run | Native outcome | Actual terminal step | Final C4 / WikiText PPL |
|---|---|---|---|
| scaled4 | nonfinite | 19600 | No final PPL; see last finite evaluation in RESULTS.json |
| scaled8 | nonfinite | 37800 | No final PPL; see last finite evaluation in RESULTS.json |
| unit_bf16_s7 | nonfinite | 21300 | No final PPL; see last finite evaluation in RESULTS.json |
| unit_bf16_s99 | finished | 61000 | 42.14531938222052 / 150.1752538901007 |
| unit4 | nonfinite | 100 | No final PPL; see last finite evaluation in RESULTS.json |
| unit8 | nonfinite | 500 | No final PPL; see last finite evaluation in RESULTS.json |
Only BF16 seed99 completed all 61,000 steps. Its final C4/WikiText PPL is 42.14531938222052 / 150.1752538901007. Other runs ended through the native loss/pre-clip-gradient-norm nonfinite guard. Their JSONs retain intentional NaNs and original bytes. Early Unit4/Unit8 have no evaluation or saved weights. The last finite model exports for Scaled4, Scaled8 and BF16 seed7 are explicitly NOT FINAL, from saved steps 19,500 / 37,500 / 21,000. Their native keys encode the planned 61,000-step horizon; the suffix and CHECKPOINTS.json give the actual saved step.
All six native cells.d JSONs, per-run evaluation/health CSVs, the combined 68-point curve and compact scientific logs are included. Full operator logs and optimizer resumes stay outside this model package. The original 20B comparison is a separate package.
See RESULTS.json for last finite evaluation scores and profiles/ for each run. Lowp checkpoints require the corresponding E2M1/E4M3 hooks for downstream evaluation; loading their keys alone does not install those hooks. See LOADING.md and CHECKPOINTS.json.
The pinned scientific code is PR2 at 5edb20302017712ac97bf39374f66db80f94bc51,
with HELM revision 7501deca8f413848bfef804be64ce874b72a3cd7. The source repository is private; use the provided
version when reproducing its model configuration and attention wrappers.
The native C4 tensor matches SHA256
03c0c8663b10d51b0117ae97264e085e95c9c90ef332f2765ed4087844dddb7c;
WikiText matches 2759990ff21d21050c8e223d855cd785b8a5c2b2d8f02fafbb6b8d4023ae3609.