nur-dev/strata-headquotient-q25

STRATA-HeadQuotient Q25 Model and Evaluation Package

This repository contains the physical Q25 export accompanying:

N. Kadyrbek and M. Mansurova, "STRATA-HeadQuotient: Functional Localization of One Quarter of Global KV Heads with Typed Predicate-Graph Computation at 8k Context," submitted to Machine Learning and Knowledge Extraction, 2026.

It complements the compact analytical artifact dataset. Hugging Face is the sole public maintenance location; no GitHub mirror is required.

Model Boundary

The backbone is a controlled 554M-parameter multilingual decoder with 24 layers, model width 1024, 16 query heads and 16 KV heads in a one-to-one standard multi-head-attention layout, learned absolute positions, a 1024-token local window, RMSNorm, and SwiGLU feed-forward blocks.

The static Q25 export contains:

• 288 full-history GLOBAL heads;
• 81 LOCAL heads;
• 15 final-layer LOCAL_GRAPH heads with event-scoped typed programs.

Thus, 25% of heads stop storing full-history KV, but only 3.91% of all heads execute graph programs. Including the 1024-token local windows, token-KV bytes are reduced by 21.875% before graph state. Ordinary PPL evaluation disables graph reads, so it evaluates localization rather than graph necessity for PPL.

Included Artifacts

checkpoints/dense_model.pt          dense true-8k reference
checkpoints/q25_export.pt           physically exported Q25 checkpoint
scoped_adapter/adapter.pt           event-scoped typed graph adapter
selection/Q25/plan.json             exact 96-head static assignment
evaluation/q25_470_documents.npz    one 8192-token window from 470 documents
evaluation/heldout-5lang-1m/        packed source for semantic confirmation
tokenizer/strata_sp48k.model        fixed SentencePiece tokenizer
source/                              minimal transitive source snapshot
artifacts/                           frozen result and provenance records
replications/                        two fresh Q25 checkpoints, audits, plans, and evaluations
verify_export.py                     structural and cryptographic verification
evaluate_release.py                 dense/Q25 PPL and semantic confirmation

The evaluation material contains token IDs rather than redistributed article text. Its provenance, language counts, deterministic document hashes, and licensing boundary are documented in EVALUATION_DATA_LICENSE.md.

Verify the Physical Export

Python 3.11 and PyTorch 2.5.1 are required.

python verify_export.py

The verifier checks every file hash, both checkpoint hashes, the 96-head assignment, the 81/15 mode split, all 384 exported head modes, the 470-document token bundle, and the expected parameter tensors without loading the model onto a GPU.

Re-run the Confirmation

Install the pinned environment and run on a CUDA device with enough memory for the dense and Q25 models:

cd source
uv sync --frozen
cd ..
uv run --project source python evaluate_release.py --device cuda:0

The evaluator reconstructs the physical export, loads the two checkpoints, evaluates the same 470 document-disjoint 8192-token windows, computes paired document-bootstrap intervals, and regenerates the untouched 2000-case semantic controls. The full run is intentionally substantial; --documents 8 --semantic-examples 64 is available only as an installation smoke test.

Expected complete-run results are recorded in artifacts/postreview.json:

Measure	Frozen result
Q25/dense PPL ratio	1.001463
95% document-bootstrap interval	1.001315--1.001611
4--8k PPL ratio	1.002006
Typed semantic execution	1998/2000
Untyped semantic execution	374/2000
Wrong-role execution	0/2000

Small floating-point variation is possible across CUDA and PyTorch builds. The scientific non-inferiority condition is the upper 95% PPL-ratio interval below 1.03, not bitwise equality of GPU reductions.

Cached Prefill and Decode

artifacts/cached_decode.json records a separate inference-cache profile at 2k, 4k, and 8k prefixes for batch sizes 1, 4, 8, and 16. At an 8160-token prefix, Q25 retained 78.125% of dense persistent KV bytes. Cached decode throughput ranged from 1.003 times dense at batch 1 to 1.237 times dense at batch 16; prefill ranged from 0.926 to 1.018 times dense. This is a model-core profile on one NVIDIA L40 with a fixed 32-token continuation, not an end-to-end serving benchmark.

Training and Selection Reproduction

The source snapshot includes the registration, H0 evaluation, scoped-adapter training, 6,903-pair interaction audit, mixed-integer selection, static frontier export, post-review evaluation, and independent-replication drivers. The original Wikipedia snapshot is not duplicated here. Re-running training or selection therefore requires preparing the source corpus described by config/headquotient.json and the article; all local data locations in the public config are represented by relative paths or source-data:// identifiers.

The release also contains two fresh seeded Q25 campaigns. Each used a new graph-adapter initialization, disjoint diagnostic and interaction material, a complete 6,903-pair and 128-triple audit over the frozen 118-head candidate pool, a fresh mixed-integer selection, and a physical export. Their 96-head selections differ from the original but reproduce the PPL non-inferiority and typed-path effects. They do not independently rediscover the 118-head candidate pool.

Corrected Matched-Capacity Attribution

The original development comparator enabled all 16 final-layer graph heads in dense H0 but 15 in Q25. artifacts/matched_h0_reanalysis.json and replications/summary.json record the corrected comparison, where H0 receives exactly the graph-head/program capacity of each Q25 export. The original and two fresh 95% marginal-effect intervals all include zero. The supported result is therefore a reproducible global-to-local quotient plus a causally operative typed graph path; the artifacts do not establish that graph computation took over a function from the localized attention heads or caused PPL preservation.

Intended Use and Limitations

This is a research checkpoint for reproducing the accompanying architecture study. It is not a production assistant or a general-purpose factual system. The demonstrated boundary is one backbone and 8k context. Q30 failed; production-serving efficiency, compiler-inclusive acceleration, 16k/32k behavior, and broad factual reliability are not established.

Licenses

The authors' checkpoint, source snapshot, plans, and generated evaluation records are released under CC BY 4.0; see LICENSE. The tokenized Wikipedia evaluation material remains subject to the Wikimedia/Wikipedia source terms summarized in EVALUATION_DATA_LICENSE.md. Third-party dependencies retain their own licenses.

Citation

Until an article DOI is assigned:

@article{kadyrbek2026strata,
  author  = {Kadyrbek, Nurgali and Mansurova, Madina},
  title   = {STRATA-HeadQuotient: Functional Localization of One Quarter of Global KV Heads with Typed Predicate-Graph Computation at 8k Context},
  journal = {Machine Learning and Knowledge Extraction},
  year    = {2026},
  note    = {Manuscript submitted for publication}
}