zimplex/helm-c4-20b-checkpoints

HELM C4 20B checkpoints and seed observations

These are native research checkpoints for a 1B HELM model: 16 blocks, width 910, 966,957,719 parameters, fixed curvature c=1. Training uses the eager C4 20B recipe with BF16 autocast and FP32 parameters. Attention is not quantized in these runs.

Available results

Run	Observed outcome	C4 PPL	WikiText PPL	Available state
Scaled init, seed 1234	Finished 152,580 steps	19.198618161946133	52.69278098379149	Native final model
Published init + FP32 geometry, seed 1234	Finished 152,580 steps	20.370028561152527	50.02364864030602	Native final _pol model
Published init, seed 1234	Nonfinite first flagged at 2,280; detected at 2,300	unavailable	unavailable	Last saved finite model at step 2,000
Published init, seed 7	Finite through requested 10,000-step cap	36.28621540996052	84.48696841726898	Native step-10,000 training resume
Published init, seed 99	Finite through requested 10,000-step cap	37.023797483960436	88.28929618013538	Native step-10,000 training resume

The seed-7/99 scores are at step 10,000, not final 20B scores. These seeds are censored observations; their later full-budget behavior is unknown. The original seed-1234 failure happened before its first scheduled evaluation. Its JSON intentionally retains NaN scores and an empty evaluation curve. The step-2,000 model is not a completed 20B checkpoint.

The published-init _pol run also completed the full budget. It protects cinner and lorentzian_centroid in FP32 while the rest of the BF16 recipe, initialization, schedule and budget remain unchanged. See geometry_policy.json and the runtime selection below. Checkpoint tensors do not serialize the method wrappers.

The original full-horizon attempt's final verifier crashed before final artifact upload. These final weights come from a verified continuation of the saved step-152,000 state: the last 580 updates were repeated with the same model, policy, seed, optimizer/RNG, data and full learning-rate schedule. This is a new continuation result; it is not claimed to recover identical original node-local final bytes.

The forthcoming quantized 2B experiments are not present in this repository.

Recipe and provenance

• HELM source: Graph-and-Geometric-Learning/helm, revision 7501deca8f413848bfef804be64ce874b72a3cd7.
• Trainer recipe revision: 5f968bd8d023298728e3ac43d6fefe64f786de8e.
• Sequence length 2,048, effective batch 64, 152,580 planned optimizer steps: 19,998,965,760 processed tokens for a completed full run.
• AdamW (0.9, 0.95), weight decay 0.1, pre-clip norm limit 1.0, peak LR 1e-3, 3,051-step warmup, cosine decay to 1e-4.
• C4 and WikiText evaluation every 5,000 steps plus the final step, using 256 held-out sequences each. The supplied tokenizer is NousResearch/Meta-Llama-3-8B, EOS ID 128001.
• Python 3.12.3, Torch 2.11.0+cu128, CUDA 12.8, NVIDIA B200; TF32 off.

Native per-run JSON is in cells.d/ and is preserved byte for byte, including its original worker-local paths. Curves and selected scientific log lines are provided separately. Internal operational/custody receipts are excluded.

Terminal verifier recovery for the capped seeds

Both seed-7/99 trainers completed their 10,000-step evaluation and resume save and joined all eight ranks. Our terminal verifier then incorrectly required every saved model-state tensor to be FP32; the native model has a valid Boolean causal attn_mask buffer. The worker failed after the scientific outputs were already durable. A separate verification confirmed the original parameter/buffer inventory, causal mask, finite tensors, optimizer, random-generator state, full-budget LR and saved evaluations. The published resume bytes are unchanged. This recovery did not rerun training or cast the saved resume state.

Native checkpoint loading

Final/last-saved model files under checkpoints/ use:

{"key": "the native experiment key", "state_dict": {...}}

Use the pinned HELM source and a compatible Torch environment. A minimal state read, which does not require CUDA, is:

from huggingface_hub import hf_hub_download
import torch

path = hf_hub_download(
    repo_id="zimplex/helm-c4-20b-checkpoints",
    filename="checkpoints/ckpt_scaled_bf16_d910_c1_lr0.001_acc1_s1234_st152580_ev5000_corpc420b_perm_cos_mb8_w8.pt",
)
checkpoint = torch.load(path, map_location="cpu", weights_only=True)
assert set(checkpoint) == {"key", "state_dict"}
print(checkpoint["key"])

For model construction and inference, follow the full pinned training recipe and HELM source, including its distributed/module initialization. These checkpoints were produced on eight GPUs; a standalone model loader has not been qualified here. Learned normalizer weights are already included; do not rescale them after loading. This is the native research checkpoint format; the example above verifies and reads its payload.

Selecting the _pol geometry precision

For the _pol model, select the same two protected methods in downstream code. The following is the method wrapper used by the pinned trainer; it does not construct a model or change the learned weights:

import torch
from helm.hypercore.manifolds import Lorentz

def protect_fp32(cls, name):
    original = getattr(cls, name)
    def wrapped(self, *args, **kwargs):
        with torch.autocast(device_type="cuda", enabled=False):
            args = tuple(x.float() if torch.is_tensor(x) else x for x in args)
            kwargs = {k: (x.float() if torch.is_tensor(x) else x)
                      for k, x in kwargs.items()}
            return original(self, *args, **kwargs)
    setattr(cls, name, wrapped)

protect_fp32(Lorentz, "cinner")
protect_fp32(Lorentz, "lorentzian_centroid")

Apply this policy explicitly before using the _pol model. The remaining training operations used BF16 autocast, with FP32 parameters. Follow the pinned recipe for model/distributed initialization; this snippet is not a standalone model loader.

Files under resumes/ have the original training-state payload with model, opt, sampler/generator state, CPU/CUDA RNG state and evaluation history. They are for restoration through the original trainer, not the two-key model loader above. Preserve the 152,580-step schedule when continuing a capped seed.

Checksums and health evidence

CHECKSUMS.json records public filenames, byte counts and SHA-256 hashes. eval_curves/ includes full available trajectories and Scaled's exact matched step-5,000/10,000 values. health/ contains original seed-1234 health intervals wholly before step 2,280. Its later 2,001–2,300 aggregate crosses that boundary, so it is not included as strictly pre-failure evidence.