Experiment1-B-Signed

A single continuous pretraining run from random initialization over the whole FineWeb-Edu 10B GPT-2 corpus, in one strictly monotonic pass. The repository contains custom PyTorch source and raw safetensors weights; it is not a drop-in Transformers model.

Status

• Status: running
• Processed tokens: 0 of 9,851,371,520 (0.00%)
• Training token positions in the committed run: 0
• Next zero-based training step: 0 of 18790
• Run ID: 9be699e5-45d3-4a0d-bf65-fdff9dc46375
• GPU: NVIDIA A100-SXM4-40GB
• Parameters/optimizer state: FP32; forward autocast: BF16
• Attention: signed dot products divided by 1 + sum(abs(scores)); no attention softmax
• Tokens per optimizer step: 524,288
• Data: kjj0/finewebedu10B-gpt2 at 440d9d739f970c5d80492e12e81d63d60622ca30, shards 000001 through 000099, read in order
• Usable sequences: 300,645; trained on 300,640, leaving 5 that cannot fill a whole optimizer step
• Activation: trainable xIELU
• MLP: latent_gated
• Validation: fixed held-out prefix, 4 x 2048 tokens every 100 updates (sampled loss, not full-shard evaluation)
• Attention window: progressive 2,304 to 8,192 tokens, then held at maximum
• Learning rate: 0 warmup steps, constant, then a single cooldown over the final 3,152 steps at the true dataset end

Resuming

Every run reads checkpoints/latest/manifest.json, restores the newest validated checkpoint, and continues from the exact sequence it stopped on. There are no stage boundaries: the optimizer, the learning-rate schedule, and the data cursor all carry across sessions untouched. A checkpoint is published every 250,000,000 processed tokens. Only matching signed-attention recipes can resume. Work after the latest durable checkpoint may be recomputed after a Colab disconnect. A monotonic cursor prevents repeating committed training windows; it does not establish that the corpus itself has no duplicate text.

Repository layout

• model/model.safetensors: final standalone model weights
• checkpoints/steps/tokens-<n>-step-<s>/: full resumable checkpoints
• checkpoints/final: the completed run
• checkpoints/latest/manifest.json: pointer to the newest valid checkpoint
• training/: exact source and resolved training configuration
• runtime/: environment metadata
• metrics/training_metrics.jsonl: per-step training metrics

FineWeb-Edu dataset shards are not included.