ulmentflam/gpt2-124m-fineweb-mojo

GPT-2 124M, trained from scratch on FineWeb (llm.mojo)

A GPT-2 124M (d12: 12 layers, 12 heads, 768 channels) language model trained from scratch (random initialization, no GPT-2 weight warm-start) for one full epoch on the FineWeb classic 10B-token sample, using llm.mojo — a Mojo/MAX port of Andrej Karpathy's llm.c.

This repo exists primarily as a training-run artifact / reproducibility reference for the llm.mojo project, not as a state-of-the-art model. At this scale and data budget, GPT-2 124M produces locally-coherent but not highly capable text.

Architecture

Params	124M
Layers	12
Heads	12
Channels	768
Context length	1024
Vocab	50257 (GPT-2 BPE), padded to 50304 internally
Precision (training)	bf16

Architecture and tokenizer are unmodified GPT-2 (GPT2LMHeadModel / GPT2Tokenizer from gpt2).

Training

• Initialization: random (from scratch, no pretrained warm-start)
• Data: HuggingFaceFW/fineweb, sample-10BT config (FineWeb "classic" 10B-token sample), tokenized with the GPT-2 BPE tokenizer
• Epochs: 1 full epoch = 19,552 steps
• Effective batch size: 524,288 tokens/step
• LR schedule: cosine decay, 6e-4 peak
• Precision: bf16
• Final validation loss: 3.2807
• Hardware: single NVIDIA GB10
• Training code: github.com/ulmentflam/llm.mojo

This run hit two unrelated hardware crashes on the training box during the run; both were recovered from checkpoint without loss of training progress. See docs/ai/ in the training repo for the full incident writeup (in progress at time of publishing this checkpoint).

Files in this repo

• Root (this directory): a standard Hugging Face transformers GPT2LMHeadModel export (safetensors, bf16 weights) — use this for everyday inference with transformers.
• llm_mojo_raw_checkpoint/model_19552.bin: the original raw llm.mojo checkpoint (llm.c-compatible binary format, 249MB), for reproducibility with the original llm.c / llm.mojo training and inference toolchains. This is the exact byte-for-byte checkpoint produced by the training run — the safetensors export above is derived from it, not the other way around.

About the raw checkpoint's magic number

llm.mojo checkpoints use their own format magic number (20240520, see MODEL_MAGIC in llmm/checkpointing.mojo) to distinguish them from upstream llm.c's own checkpoints (magic 20240326), even though the 256-int32 header layout and parameter blob order are otherwise byte-identical. If you want to run third_party/llm.c/dev/eval/export_hf.py (upstream llm.c's own HF converter) directly against model_19552.bin, you'll need a magic-patched copy first — the llm.mojo repo provides scripts/export_to_hf.py to do exactly that (and nothing else): it makes a temporary patched copy, never touches the source file, and is what produced the safetensors export published in this repo. This is intentional, documented behavior, not a format inconsistency.

Usage

from transformers import GPT2LMHeadModel, GPT2Tokenizer

model = GPT2LMHeadModel.from_pretrained("ulmentflam/gpt2-124m-fineweb-mojo")
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")

inputs = tokenizer("The quick brown fox", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=40, do_sample=False)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

AutoModelForCausalLM / AutoTokenizer work equally well since this is a standard GPT-2 architecture:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("ulmentflam/gpt2-124m-fineweb-mojo")
tokenizer = AutoTokenizer.from_pretrained("gpt2")

To load the raw llm.mojo/llm.c-format checkpoint instead (e.g. for continued training or evaluation with the original toolchains), download llm_mojo_raw_checkpoint/model_19552.bin and follow the llm.mojo or llm.c instructions for loading a model_*.bin checkpoint.

Dataset license and attribution

Training data is the sample-10BT config of HuggingFaceFW/fineweb, licensed under ODC-BY 1.0 (Open Data Commons Attribution License). Per that dataset's card: the underlying content is derived from CommonCrawl web crawl archives, and use of the dataset should credit both the FineWeb project and CommonCrawl as the original source, and remains subject to CommonCrawl's own terms of use for the underlying crawled content.

Model license

This model's weights are released under the same ODC-BY terms as the training data, given the model is a direct statistical function of it.

Citation / acknowledgments

• Training code: llm.mojo (Mojo/MAX port of llm.c)
• Original reference implementation: llm.c by Andrej Karpathy
• Dataset: FineWeb by HuggingFaceFW