ezrawinston/gptbert-sambal

GPT-BERT (SAMBAL) - ablated arm

Models for Learning syntax without semantics: Disentangled tiny language models
Ezra Winston and Zico Kolter, ICML 2026

• Paper: OpenReview (PDF)
• Code and reproduction recipes: https://github.com/ezrawinston/sambal

The SAMBAL models: GPT-BERT trained on the SAMBAL-ablated 10M-word corpus, in which content words are systematically substituted while grammatical structure is preserved. Their baseline counterparts, trained on the un-ablated corpus, are in gptbert-babycosmofine.

Weights

All reported results for this arm come from the EMA weights.

Files

file	what it is
gptbert_sambal_long_ema.bin	long-regime model, EMA weights
gptbert_sambal_short_lr_0.007_seed_{0,1,2}_ema.bin	short-regime runs, EMA (3 seeds)
lora/lotr_sambal_*	LoRA adaptation on the small-domain text
lora/pubmed_sambal_*	LoRA adaptation on the PubMed split

Loading

These are plain state_dict files, not transformers-format checkpoints - AutoModel.from_pretrained will not load them. The architecture, config, and tokenizer all live in the GitHub repo (lm/gpt-bert/); to load and evaluate the checkpoints, follow reproduce/icml2026/README.md.

Model details

• Architecture: GPT-BERT (Charpentier & Samuel, 2024, arXiv:2410.24159), small config - 12 layers, hidden size 384, 6 attention heads, vocabulary 8192.
• Tokenizer: gpt-bert-babylm-small - the corpus is the only difference.
• Training data: ezrawinston/babycosmofine-sambal
  • see that card for how the corpus is produced and regenerated.
• Short-regime runs: learning rate 0.007, seeds 0/1/2.
• LoRA runs: rank 32, alpha 16, embeddings also trainable, learning rate 1e-3, checkpoint selected by best dev ppl.

License

These models are released under an MIT license.

Intended use

Research artifacts for studying how lexical ablation of a pretraining corpus affects what a language model learns. These are small models trained on 10M words; they are not instruction-tuned and are not intended for deployment.

Citation

@inproceedings{winston2026syntax,
  title     = {Learning syntax without semantics: Disentangled tiny language models},
  author    = {Winston, Ezra and Kolter, J. Zico},
  booktitle = {Proceedings of the 43rd International Conference on Machine Learning},
  series    = {Proceedings of Machine Learning Research},
  volume    = {306},
  year      = {2026},
  publisher = {PMLR},
  url       = {https://openreview.net/forum?id=p7HVrmZwWB}
}