P7

Model Details

Model name: P7
Challenge: BabyLM 2026
Track: Strict-Small
Backend: causal
Repository: moral1/babylm_strict_small_2026
Final full-eval checkpoint: 80M (words_seen=80,010,749)
Training checkpoints: 1M-100M cumulative words
100M branch: uses the strictly compliant final_compliant endpoint (words_seen=99,992,923)
Evaluation pipeline commit SHA: 3d57ddc8c40ee795c0b5e41b3a20251a9457a593
Final predictions JSON SHA256: 1e41f721edd432379a8d253dc4936e5a7ea4f003662d037fa9d8c927f77ceefe

Architecture

Base architecture: GPTBERTForMaskedLM
Model type: gpt-bert
Parameter count: 132037444
Hidden size: 768
Layers: 12
Attention heads: 12
Vocabulary size: 16384

Tokenizer

Tokenizer type/source: self_trained_strictsmall
Tokenizer training words: 2000168
Tokenizer files: tokenizer.json, tokenizer_config.json, special_tokens_map.json

Training

Training dataset budget: BabyLM Strict-Small 10M
Submitted full-eval checkpoint: 80M checkpoint selected from the training trajectory
Random seed: 73
Training objective: NEEDS_USER_CONFIRMATION: see babytutor_run_config.json
Checkpoint trajectory branches: chck_1M through chck_9M, then chck_10M through chck_100M every 10M words

The originally overshooting 100,002,969-word checkpoint is not used as the official 100M branch. The chck_100M branch points to the strict compliant endpoint with 99,992,923 words seen.

Intended Use

This model is intended for research and BabyLM 2026 Strict-Small evaluation workflows.

Limitations

The model was trained under the BabyLM Strict-Small data budget and is not intended as a general-purpose production language model. Outputs may be unreliable, biased, or inappropriate, and the model should be evaluated in context before any downstream use.

Data and License

Training data follows the BabyLM 2026 Strict-Small challenge data constraints. Dataset licensing and attribution follow the BabyLM challenge materials. Model code and weights are provided for research and evaluation use.

Downloads last month: 4