instanton-hybrid

A 40.2M-parameter GPT-BERT masked–causal hybrid trained entirely from scratch on the official BabyLM 2026 Strict-Small corpus (≤10M words).

Part of the instanton family — small models trained in minutes-to-hours, submitted to the BabyLM 2026 Challenge.

Parameters	40,170,648
Architecture	GPT-BERT (8 layers, 512 hidden, 8 heads, 1706 FFN)
Vocabulary	16,384 (byte-level BPE, trained only on the in-budget corpus)
Training data	Official BabyLM 2026 Strict-Small, 9,972,806 words
Word exposure	10 epochs ≈ 99.7M words (165.3M subword tokens)
Pretrained weights used	None — random init, per challenge rules

Training objective

Following Charpentier & Samuel (2024), BERT or GPT: why not both?, the model is trained on a 50/50 blend of masked- and causal-LM objectives. Upstream splits the two objectives across DDP ranks; this model was trained on a single GPU, so the blend is reproduced in-process: half the micro-batches of each optimizer step come from a masked dataset and half from a causal dataset, gradients accumulate, then one step is taken.

Recipe

LAMB, max lr 0.007, cosine schedule with 1.6% warmup / 1.6% cooldown, weight decay 0.1, grad clip 2.0, z-loss 1e-4, mask ratio 0.30→0.15, sequence length ramped 128→256→512 at 70%/90% of training, 16,384 tokens per step, seed 42, bf16. ~2 hours on one A100-80GB.

Checkpoints

Nineteen word-exposure revisions are published as git branches, as required by the challenge: chck_1M … chck_9M, chck_10M, chck_20M … chck_100M, plus main (final). Load any of them with revision=.

Usage

The repository ships a vendored modelling wrapper, so trust_remote_code=True is required. It loads under transformers 4.51.x; 5.x is not supported.

from transformers import AutoTokenizer, AutoModelForMaskedLM

tok = AutoTokenizer.from_pretrained("qyxu1994/instanton-hybrid", trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained(
    "qyxu1994/instanton-hybrid", trust_remote_code=True)          # or revision="chck_10M"

AutoModelForCausalLM also works and is what the age-of-acquisition evaluation uses. Everything else should be scored as a masked LM (pseudo-log-likelihood / mntp), which is how the numbers below were produced.

Evaluation

Official BabyLM 2026 pipeline, mntp backend for zero-shot / GLUE / reading and causal for AoA.

Task	Score
BLiMP	72.13
BLiMP supplement	60.86
EWoK	50.15
Entity tracking (filtered)	19.87
COMPS	52.96
Global-PIQA	37.14
(Super)GLUE	67.58
Reading	6.35
Age of acquisition	0 (r = −0.11, p = 0.081)
Leaderboard NLP average	51.53
Leaderboard Overall	40.78

For comparison, the best causal decoder in this family (27M-parameter GPT-2, same 10M-word budget) reaches BLiMP 67.29 and GLUE 63.01 — the hybrid objective is worth roughly +4.8 BLiMP and +4.6 GLUE.

Known limitation

Like every model in this family, instanton-hybrid correlates negatively with children's age-of-acquisition ordering, so it scores 0 on that benchmark. This is a property of the model rather than of the measurement: the checkpoint sweep is complete (19/19, 152,095 surprisal observations) and tokenizer fragmentation was ruled out (215 of 237 scored CDI words are single tokens, and controlling for subword count leaves the correlation unchanged).

Intended use

A research artifact for studying sample-efficient language acquisition. It is small, trained on <10M words, and is not suitable for deployment or for generating text people will rely on.

Citation

Data mixture and recipe are described in the accompanying BabyLM 2026 report, Reweighting Child-Directed and Conversational Data for Sample-Efficient BabyLM Pretraining. The architecture is from Charpentier & Samuel (2024).