MicroLlama V2: full checkpoints and training artifacts
For ordinary inference, use keeeeenw/MicroLlama2-base. That lightweight release contains the final NVIDIA 50B weights, tokenizer, current Transformers adapter, model card and Python example.
This archive preserves the full NVIDIA training checkpoint history and exact tokenized data. An unrestricted repository download can fetch a very large amount of data. Download only the specific checkpoint or data archives you need. Model and inference usage, evaluation results and the reported $880 compute spend are described in the base model card.
Completed NVIDIA run
- Final checkpoint: step 381,824, 50,000,054,177 valid next-token targets.
- Full final restart payload: 11 files, 4,661,856,785 bytes, including Adam state and rank RNG/data cursors.
- Immutable final revision:
527ab4101332284704765e1b2f3cfb577609bf56. - Final checkpoint manifest SHA-256:
9968e0d8db74a6618750ca38c71e68332c3947921f8dd75a591415afd965840c. - Final training and held-out validation completed. The base model card distinguishes benchmarked earlier checkpoints from the exact final weights.
Contents
checkpoint/: full restart checkpoints; known historical gaps remain.datasets/tokenized-v1/: 138 token archives, 6,800 chunks, tokenizer/document boundaries and checksums; about 92.6 GB compressed in total. Do not download all of it by default.tokenizer/: exact tokenizer and source provenance.reproduction/four-source-v1/: saved catalogs, policy and frozen validation settings. Historical interim resume pointers remain historical records; the final checkpoint is identified above.sources.json,LICENSES.md,publication.json: original data sources, terms and publication records. A publication record is not a current training-status feed.
Data stages were FineWeb-Edu through 10B targets; 60% FineWeb-Edu / 40% DCLM-Edu to 23.774B; then 48% FineWeb-Edu / 32% DCLM-Edu / 15% Cosmopedia v2 / 5% FineMath 4+ for future targets through the finite 50B endpoint. Cosmopedia is a public synthetic corpus. Upstream dataset terms continue to apply; this archive does not replace them with a blanket license.
The technical report will be linked shortly. Training code and reproduction instructions: MicroLlama2-training, currently private for release review; the author will enable public access. Full restart training requires that implementation, not just the inference adapter. Restore saved optimizer, RNG and source-cursor state for continuation rather than restarting data order from the beginning.
This repository was previously named keeeeenw/MicroLlama2. The rename preserves its commit history and file contents; existing historical paths identify the same archived artifacts. The separate MicroLlama2-base release has no datasets or Adam states.