qmichalwarda/qoat-tiny-wiki30-8m

qoat-tiny Wiki30 8M

The best fully benchmarked checkpoint from the qoat-tiny BDH campaign, published at the author's request on 2026-10-09.

This is a native residual BDH with independent matrices at each of four depths, 256-dimensional embeddings, four heads, 2,560 neurons per layer, a 2,048-token BPE vocabulary, and tied input/output embeddings. It is a BDH ablation, not Pathway's proprietary BDH-CQ implementation.

Complete Tiny-ML benchmark

Measurement	Result
Unique parameters	8,388,608
BLiMP	68.78955223880597% (46,089 / 67,000)
ARC-Easy test	36.111111111111114% (858 / 2,376)
WikiText-2 historical byte perplexity	2.577376629901653
WikiText-2 direct NLL/byte perplexity	2.577371521608232
Leaderboard efficiency	74.35688093090441
Selected cumulative training targets	2,087,845,888

The canonical evaluation uses ordinary FP32 inference, a fresh state for every independent example/context window, context length 256, all 67 BLiMP tasks, the complete ARC-Easy test, and all 1,292,008 UTF-8 bytes of the joined WikiText-2 raw test. ARC uses unnormalized choice log likelihood. The Wiki historical conversion uses 477,691 encoded tokens and 477,690 scored targets. evaluation-report.json contains exact dataset revisions, scores and provenance. canonical_evaluate.py is the unchanged evaluated scoring source.

The score estimates rank 28/86 on board revision 3795f68a4373d5581da21ae5168e0774fca00f2f; this is not an accepted leaderboard position. Competitor scores are author-reported.

Native loading

Download this repository, then run from its directory:

import torch
from load_model import load_model

model, tokenizer = load_model(device="cpu")
ids = tokenizer.encode("The small model").ids
with torch.inference_mode():
    logits = model(torch.tensor([ids]))
print(tokenizer.decode([logits[0, -1].argmax().item()]))

This repository uses the original native PyTorch architecture and state dict, not an AutoModel conversion. Inputs must fit the trained 256-token context; there is no persistent inference cache or automatic BOS/EOS insertion. load_model.py checks every retained component against the original checksums. For authenticated downloads, pass your token as HF_TOKEN and explicitly pass it to Hugging Face download clients.

Training and selection

The selected lineage contains 1,785,856,000 original FineWeb-Edu targets, 100,663,296 targets of a 85/15 FineWeb-Edu/WikiText-103 continuation, and 201,326,592 targets of a 70/30 continuation. These are cumulative training targets, not unique corpus tokens. The final selected phase step is 12,288. The original streams contain 993,579,418 FineWeb-Edu tokens and 177,414,754 WikiText-103 tokens. Full ancestry, tokenizer/data pins, and training recipes are retained in the training reports and the original archive.

Checkpoint selection used WikiText-2 validation under a FineWeb-Edu retention constraint, followed by ARC validation (208/570), before canonical evaluation. This is a single-seed result with reused validation data and repeated campaign visibility of canonical benchmarks. Literal-overlap filtering does not establish absence of all upstream, semantic, or tokenizer contamination. No untouched-test or contamination-free claim is made.

Exact artifact

• Model artifact: 93b7d47b-b947-4125-904c-b84dc8c50a1c
• Canonical run: 2ebe2ace-4d63-4377-a64d-97873e940747
• Original archive SHA256: c4abaa5448b2d0c45d31bde85f4ef618a0cec113a8d263d79e46706d2ec0ea61
• Canonical report SHA256: 438f6615aaa942c4461f5144b277035c5deb7457c50811bd41356d91e1327610

selected-model.zip is the exact evaluated archive, including nested parent archives. Convenient extracted files accompany it. Authored code and weights are MIT licensed; upstream architecture notices are retained in third_party/. Dataset source-text rights remain with their respective authors.