hotdog-27B

An open binary classifier trained with Tinker on Qwen3.8-27B (dense 27B). Give it text and a yes/no question; it returns a probability and a decision. It reads the logits for A = false and B = true at the same token position, without generating reasoning. The selected model is a rank-16 LoRA adapter.

Training code · Benchmark results · Protocol and provenance

Benchmarks

The same frozen model was evaluated once against Jev 1.13.0 on both sets, with complete coverage and no subsequent tuning. These are September 22, 2026 results for this model and comparator version.

Evaluation	hotdog-27B	Jev	Accuracy difference, paired 95% CI
Independent test, 1,631 decisions / 1,002 groups	1,603/1,631 (98.28%)	1,587/1,631 (97.30%)	+0.98 pp [+0.19, +1.79]
Public text JevBench, 74 native-binary decisions / 62 groups	60/74 (81.08%)	62/74 (83.78%)	−2.70 pp [−12.20, +5.63]

Evaluation	Our scalar Brier ↓	Jev scalar Brier ↓	Our ECE ↓	Jev ECE ↓
Independent test	0.013175	0.026884	0.58%	6.40%
Public native-binary JevBench	0.125090	0.106305	11.95%	6.54%

The model beats Jev on this independent test mixture. It does not establish a JevBench win. The 74 decisions cover only the native-binary subset of the public text JevBench repository, not its full composite score, private tests, or the different interactive-agent benchmark currently at jevbench.dev.

Scalar Brier is the mean of (p_yes − y)². ECE is top-label confidence calibration with 10 equal-width bins; the 74-decision estimate is noisy. Confidence intervals use 10,000 paired cluster bootstrap replicates, keeping related decisions together. The independent test uses disjoint documents/scenario groups but shares source datasets with training. Latency is not compared because serving transports differ. Full metrics, paired uncertainty, source composition, frozen hashes, and limitations are in the linked JSON reports.

Run the released model

The Hugging Face release contains the unchanged Tinker-format adapter. It is not a directly loadable PEFT adapter: linear-attention Q/K/V layouts need the official Tinker merger. The included loader checks the adapter hashes, downloads the pinned base, applies every adapter tensor, and uses the same singleton BF16 eager / FP32 two-token readout as the measured model.

Use a Linux CUDA machine with enough memory for the dense 27B backbone and merge workspace; the evaluated deployment used an H200. The adapter is about 483 MB, but downloading/loading the base requires tens of GB more.

git clone https://github.com/mrmps/tinker-binary.git
cd tinker-binary
uv venv --python 3.12 .venv
uv pip install --python .venv/bin/python -r requirements-inference.txt

git clone https://github.com/thinking-machines-lab/tinker-cookbook.git artifacts/modal/tinker-cookbook
git -C artifacts/modal/tinker-cookbook checkout 1e53aa3d1cdd6389b3290c2574641eccc0503242

PYTHONPATH=artifacts/modal/tinker-cookbook .venv/bin/python public_inference.py \
  --text 'Parcel weight: 12 kg.' \
  --question 'Is the parcel weight strictly greater than 7 kg?'

The loader downloads public weights without a Tinker or Hugging Face token. Temperature is fixed at 1.0 and the threshold at 0.5. It rejects inputs longer than 8,192 tokens rather than silently truncating. Different precision, batching, attention kernels or readout can change probabilities. Standard text-generation pipelines do not reproduce this scoring protocol.

GPU portability evidence records exact agreement with the frozen backend on three new synthetic inputs, including repeated inference and input validation checks. This checks loader parity, not additional benchmark accuracy. To repeat those checks from the source checkout and write a JSON artifact:

PYTHONPATH=artifacts/modal/tinker-cookbook .venv/bin/python verify_public_inference.py

Training and data

train.py implements Tinker's multi-target custom BCE, scalar Brier, and combined losses. sweep.py orchestrates parallel runs; configs/ contains the 16 recipes. The search used 661 successful optimizer steps and 51,840 example exposures. The selected model's ancestry consumed 13,017 distinct rows / 16,384 exposures: a fresh 27B run followed by the selected step 96 of a low-rate continuation. It was selected by lowest development Brier among candidates within one percentage point of the best development accuracy. The development set had 1,468 decisions. No fitted calibration was applied; the released model uses temperature 1.0.

Data combines public natural examples from Kev's pinned training partition, independently generated OpenAI examples with blinded audits, and executable procedural rules. data_pipeline/import_kev.py reproduces the train-only import and verifies upstream and converted hashes. build.py, audit_natural.py and compositional.py contain generation, cleaning and grouping logic.

JevBench is evaluation-only. Its examples, labels, errors and derived variants were not training/development inputs or selection criteria. Blind exact/near matching checked public overlap. Unknown backbone/teacher pretraining and unavailable private benchmark data cannot be certified contamination-free.

Use TINKER_API_KEY for training and OPENAI_API_KEY for generation/audits in an ignored .env. Source datasets have mixed licenses, including unknown/unspecified terms; raw training, development and independent-test records are not included. The Apache license for this code/model does not relicense those datasets. Source license metadata is embedded in data_pipeline/import_kev.py.

This is an executable research code release, not a claim of byte-for-byte training reproduction from public files alone. Exact independent-test replication needs the retained frozen corpus; new API generation/audits may produce different examples. Four early runs lack archived trainer-source bytes, and no per-run Git commit baseline was recorded. The published source/config/data hashes document those limits explicitly.

The historical commands were:

.venv/bin/python sweep.py configs/sweep-v1 --concurrency 4
.venv/bin/python sweep.py configs/continuation-v2 --concurrency 2
.venv/bin/python train.py configs/backbone-v2/candidate.json
.venv/bin/python train.py configs/final-development-pass/candidate.json

Reproduction requires assembling the expected data paths and replacing the REPLACE_WITH_YOUR_PARENT_CHECKPOINT placeholders with your own Tinker parents. Run names are immutable: use new names or a fresh workspace. The final recipe ran 117 steps, but the released checkpoint is step 96. Do not describe the selected checkpoint as having consumed the entire final pass.

Evaluation and playground code

evaluation/evaluate.py contains the native-binary JevBench preparation, Jev baseline collection, frozen candidate evaluation, scoring and paired bootstrap. Pin upstream JevBench to the revision in benchmarks/reproducibility.json before preparing it. Keep all benchmark examples under evaluation/sealed/, outside training inputs. Public files include aggregate results, not raw test examples or per-example predictions.

serve.py, playground.html, predict.py and modal_inference.py preserve the original local comparison playground and measured private Modal runtime. The playground supports independent model/Jev responses, yes probabilities, copyable JSON and keyboard submission. Configure your own Modal deployment, model manifest and TYPESAFE_API_KEY to use it; the author's private deployment and credentials are not shared. Use public_inference.py to load the published model directly. E2E scripts cover real API/playground behavior and save artifacts.

The public snapshot removes local credential-file defaults, absolute report paths and private parent checkpoint IDs. Frozen numerical prediction files are unchanged. Historical hashes refer to the original experiment; release hashes and portability verification are published separately.

Attribution and license

Apache-2.0 for code and the released adapter; see NOTICE.

• Qwen3.8-27B: pinned dense backbone.
• Tinker and its cookbook: training service and official weight merger.
• Kev: natural training-data assembly and binary framing. No Kev weights were used.
• AutoJev: backbone/training research. No AutoJev weights or private corpus were used.
• Public text JevBench: held-out native-binary evaluation only.