Semantic GEAR experiment backups
PyTorch checkpoints, exports, evaluations, and restoration files for two MsTok GEAR-style semantic-supervision experiments on small_owt (context 256), using a frozen RoBERTa-base teacher and eight B200 GPUs.
Browse the complete backup · Restore instructions · Checksum inventory · Remote verification
Saved runs
| Run | Status | Key full checkpoints |
|---|---|---|
semantic-gear-full-v1 |
Original LR schedules; intentionally stopped for restart | Evaluated milestones 5,000 / 10,000 / 17,192 / 25,000; last resumable checkpoint 28,000 |
semantic-gear-matched-lr-v1 |
Completed 34,384 steps and all evaluations | Evaluated milestones 5,000 / 10,000 / 17,192 / 25,000 / 34,384 |
Each evaluated milestone includes a full training checkpoint and ncp.pt, vqvae.pt, and vqvae-teacher.pt component exports. Full checkpoints include the semantic projector, optimizer, scheduler, EMA state, and per-rank RNG state. best_checkpoint.pt means lowest validation loss, not best generation PPL.
Matched-LR generation evaluation
GPT-2 Large; 5 sampling seeds, 128 generations per seed, supplied level zero. Truncated sampling uses top-k 50 / top-p 0.95. Uncertainties are standard errors across sampling seeds, not training seeds.
| Step | Random generation PPL | Truncated generation PPL |
|---|---|---|
| 10,000 | 219.77 ± 1.84 | 71.02 ± 0.69 |
| 17,192 | 185.07 ± 2.63 | 57.37 ± 1.61 |
| 25,000 | 173.60 ± 2.29 | 55.50 ± 1.11 |
| 34,384 | 185.82 ± 5.45 | 55.86 ± 1.09 |
The matched-LR recipe gives codec, generator, and semantic projector peak LR 5e-4, minimum LR 1e-5, 300 warmup steps, and decay through 34,384 updates. All generated samples and repetition/diversity summaries are included. Inspected outputs still contain broken grammar and repetition; generation PPL alone does not establish text quality or isolate RoBERTa's contribution.
Reproducibility and scope
- Source snapshots reproduce every file covered by the original run manifests' source SHA-256s. Embedded credentials in four unrelated legacy shell scripts were redacted; full Git history was excluded to avoid exposing historical credentials.
- Run configs, code/data hashes, qualification results, teacher revisions, training and evaluation logs, W&B histories, and environment records are included.
- Dataset and public pretrained weights are already hosted remotely. External dependencies records immutable revisions and dataset hashes checked against these runs.
- Redundant rolling intermediate checkpoints and duplicate epoch-named export directories are omitted. All evaluated checkpoints, final/latest states, and validation-selected best checkpoints are retained.
- These archives contain Python/PyTorch checkpoint objects. Restore in the recorded environment and use the repository's evaluation/inference scripts with matching generator and codec exports.