keyzersoze04/language-entropy-b4

Language-Entropy: B4 checkpoints and arrays

Reproduction files for B4 (structural abstention) in the Language-Entropy research program: three subword tokenizers, running no model and using no labels, recover 75% of the abstention power of the model's own predictive entropy.

What is here

checkpoints/: eight small decoder-only language models (8 layers, d_model 512, 25.2M non-embedding parameters), trained from scratch on BabyLM, one per tokenizer arm and seed.

file	tokenizer arm	seed	size
checkpoints/bpe32k_s0.pt	BPE, 32k vocabulary	0	239 MB
checkpoints/bpe32k_s1.pt	BPE, 32k vocabulary	1	239 MB
checkpoints/char_s0.pt	character (one token per character)	0	112 MB
checkpoints/char_s1.pt	character (one token per character)	1	112 MB
checkpoints/uni32k_s0.pt	unigram, 32k vocabulary	0	239 MB
checkpoints/uni32k_s1.pt	unigram, 32k vocabulary	1	239 MB
checkpoints/uni8k_s0.pt	unigram, 8k vocabulary	0	139 MB
checkpoints/uni8k_s1.pt	unigram, 8k vocabulary	1	139 MB

b4_bits/: what code/b4_abstain.py consumes, all per character on the BabyLM validation split:

• bits_<arm>_s<seed>.npy: realised bits (8 files)
• ent_<arm>_s<seed>.npy: the model's predictive entropy (8 files)
• cov_<arm>_s<seed>.npy: coverage mask, which characters received a prediction (8 files)
• seg_<arm>.npy: the arm's segmentation, as a token label per character (4 files, seed-independent)

SHA256SUMS: a checksum for every file.

Reproducing B4

1. Clone the GitHub repo and install requirements.txt.
2. Place checkpoints/*.pt in results/tok01_models/ and b4_bits/*.npy in results/b4_bits/.
3. Either re-score from the arrays: B4_ARMS=uni8k,uni32k,bpe32k python code/b4_abstain.py (the 75% figure), or regenerate the arrays from the checkpoints with python code/b4_bits.py first. Tokenizers are rebuilt deterministically from BabyLM by the code; they are not stored here.

The checkpoints are PyTorch pickles (torch.load(..., weights_only=False)), as the code loads them. Load only files whose checksum matches SHA256SUMS.

License

MIT.