noOneCodes

/

tinystories-gpt-124m

Like 0

The card says 124M parameters, but the checkpoint actually stores 162,419,712 (162.4M).

What I found loading model_final.pth:

• 173 tensors total: 12 att.mask buffers (786,432 elements — not parameters) and 161 real parameter tensors.
• Both tok_emb.weight (50257, 768) and out_head.weight (50257, 768) are present and are separate tensors — they are not tied.
• 162,419,712 × 4 bytes (fp32) = 649,678,848 B, which matches the 652,889,393 B file size up to ~3.2 MB of pickle overhead.

The "124M" figure comes from CodingGPTFr.py, which computes a "tied-equivalent" count by subtracting out_head from the total:

total_params_gpt2 = total_params - sum(p.numel() for p in model.out_head.parameters())

That gives 162,419,712 − 38,597,376 = 123,822,336 ≈ 124M. But the model never actually ties the weights — tok_emb and out_head are independent nn modules, so the saved checkpoint carries both. The gap is exactly one vocab_size × emb_dim = 38,597,376, i.e. the checkpoint is ~31% larger than the card implies.

Two ways to make the card and artifact agree:

1. Actually tie the weights in the model (e.g. self.out_head = nn.Linear(emb, vocab, bias=False); self.out_head.weight = self.tok_emb.weight) and re-export — the checkpoint would then genuinely be ~124M.
2. Update the card to say 162M (or "124M (tied-equivalent), 162M as stored").

Minor, separate note: the code sets context_length=1024 but the saved pos_emb is (256, 768), so the checkpoint's context is 256 — the README's "256" is the one that's right.