UDGAM LABS · Training checkpoints · 152M parameters, every surviving state

Laghu-Checkpoints

The raw training states behind Laghu, a 152M-parameter chat model trained from scratch for $146 of pretraining compute: model weights, EMA weights and optimiser states from the end of pretraining and of the 32K stretch, all three supervised fine-tuning runs (1, 2 and 3 passes), the reinforcement-learning run that failed its gate, and the run logs. Together with the code on GitHub, they let you resume training, re-run the fine-tuning, or study how the model changed between stages.

Try it: the chat model · code on GitHub · udgamlabs.com/models/laghu

Quick start

import torch
from huggingface_hub import hf_hub_download

p = hf_hub_download("udgamlabs/Laghu-Checkpoints", "pretrain/step_076294/model_ema_076294.pt")
state = torch.load(p, map_location="cpu", weights_only=True)   # a state dict for nanochat's GPT model

These are nanochat files (torch.save), not transformers checkpoints. Load them with the training code in udgamlabs/laghu (nanochat/), or convert one to the transformers format with python -m scripts.export_hf --checkpoint-dir <folder> --weights final --out <dir>. Ready-to-use transformers weights are in Laghu, Laghu-Base and Laghu-Base-32K, and in the hf/ folders below.

What is here

Folder	Stage	Files
pretrain/step_076000/	pretraining, 99.6%	model_076000.pt (weights), ema_076000.pt (EMA), optim_076000_rank{0-7}.pt (one optimiser shard per GPU of the 8-GPU run), meta_076000.json
pretrain/step_076294/	end of pretraining	as above, plus model_ema_076294.pt (the EMA weights as a state dict; this is Laghu-Base)
pretrain/logs/, pretrain/summary.txt		the main run's log (every step's loss, validation every 2,000 steps)
ctx32k/step_001750/, ctx32k/step_001900/	32K stretch	model_*.pt, optim_*_rank0.pt, meta_*.json; step 1,900 is Laghu-Base-32K and the start of SFT
ctx32k/logs/, ctx32k/eval/		training log; recall and short-text checks
sft/1-pass/, sft/2-passes/, sft/3-passes/	supervised fine-tuning	ckpt/ (weights and optimiser at the halfway and final steps), hf/ (transformers export of the final step), logs/; 2 passes is the released Laghu
rl-r1-failed/	reinforcement learning (GRPO + distillation from Gargi), 40 steps	hf/ export, eval/, logs/; it failed its pre-registered gate (dev GSM8K 8.6% → 8.2%, bar 11.6%) and is published for study only

File formats:

• model_*.pt and model_ema_*.pt: state dicts (the EMA ones in fp32), loadable with torch.load(..., weights_only=True).
• ema_*.pt: the EMA as a list of fp32 tensors in model.parameters() order, used to resume the EMA.
• optim_*.pt: optimiser state dicts (Muon for matrices, Adam for embeddings, output and scalars), loadable with weights_only=True.
• meta_*.json: model config, run settings, data-loader position and validation scores at that step. Measure a checkpoint yourself rather than reading scores from its meta file.

Results

Measured on the exported weights (details on the model cards and in the technical report):

Checkpoint	Bits per byte on new text (lower is better)	Notes
pretraining step 76,294, EMA (Laghu-Base)	0.919	SmolLM2-135M 0.941, Falcon-H1-Tiny-90M-Base 0.979
32K step 1,900 (Laghu-Base-32K)	0.921	hidden-code recall 96-100% up to 16K tokens
SFT 1 / 2 / 3 passes	held-out SFT loss 1.724 / 1.784 / 2.036	2 passes was picked by a generation-based rule (60.0% judge win rate over 1 pass)

Where it falls short

• Most of the pretraining trajectory is missing. Checkpoints were saved every 2,000 steps, but the uploader kept only the newest two on the main branch, and a storage clean-up on 2026-10-21 deleted the older versions (steps 2,000-74,000) and about six earlier 32K checkpoints. That was our mistake. Only the late states here survive, so this repository cannot show how Laghu learned over the whole run.
• The 32K stretch states have one optimiser shard (one GPU); pretraining states have eight. Resuming on a different GPU count needs re-sharding.
• The RL run is a failed experiment, not an improvement.

How it was trained

Pretraining: 76,294 steps × 524,288 tokens (40.0B token positions) at 2,048 context on 8× A100 80GB, 13.2 hours, $146.26; Muon; token bags for the first 25% of steps; cool-down data from 80%; EMA from 90% (β 0.998). 32K stretch: 1,900 steps at 32,768 context on one RTX PRO 6000. SFT: 41,467 conversations, sequence 16,384, batch 262,144 tokens. Every setting and script: GitHub.

Intended use and limitations

Research on training dynamics, continued pretraining, fine-tuning and reproduction. The weights carry no safety tuning beyond light SFT data.

Licence

Apache-2.0. Copyright 2026 Udgam Labs LLP. Training-data credits: DATA_LICENSES.

Citation

@misc{udgamlabs2026laghu,
  title  = {Laghu: a fully open 152M chat model trained from scratch},
  author = {Prasad, Jitesh K},
  year   = {2026},
  howpublished = {Udgam Labs},
  url    = {https://huggingface.co/udgamlabs/Laghu}
}

---

About Udgam Labs. Udgam Labs is an independent AI research lab. We work from first principles: before we build, we understand why things work, whether we are training a model, fine-tuning one or applying AI to a new problem. Udgam (उद्गम) is Sanskrit for origin. udgamlabs.com