N-Q-26M-JEV

A 26M-class decision model: a choice, a score or a yes/no with calibrated probabilities, in one forward pass

This is a research release, not a product. The model is good at the kinds of decisions it was trained on and well calibrated there; on new kinds of questions it is close to chance (see Honest results). Treat it as a fast, cheap classifier for the listed task families, not as a general reasoner.

What this is

A decision model in the style of TypeSafe AI's Jev: instead of generating text, it reads a state (any text), a question and a list of options, and returns a distribution over the options in a single forward pass. Three primitives:

primitive	returns
choice	the picked option, the full distribution, the confidence
score	a level on an ordered scale of 2–10 tiers, the distribution
noul	P(yes) for a yes/no question

It is N-Q-26M-Chat (Gated DeltaNet + gated attention, Russian and English) with a linear decision head. The language-model head is kept, so the checkpoint can still generate text.

Usage

from transformers import AutoModel, AutoTokenizer
tok = AutoTokenizer.from_pretrained("q-project/N-Q-26M-JEV", trust_remote_code=True)
jev = AutoModel.from_pretrained("q-project/N-Q-26M-JEV", trust_remote_code=True).eval()

state = "Write a Python function that returns the n-th Fibonacci number."
print(jev.decide(tok, state, {
    "topic": {"kind": "choice", "question": "What is this request about? Pick the topic.",
              "options": ["coding", "math", "writing", "cooking", "travel", "health"]},
    "level": {"kind": "score", "question": "How difficult is this request?", "tiers": ["easy", "medium", "hard"]},
}))
# {'topic': {'choice': 'coding', 'probs': {'coding': 0.673, 'math': 0.326, ...}, 'confidence': 0.673},
#  'level': {'level': 'medium', 'probs': {'easy': 0.258, 'medium': 0.666, 'hard': 0.077}, ...}}

jev.noul(tok, "Как сварить борщ?\n---\nВозьмите свёклу, капусту, картофель и бульон; варите 1.5 часа.",
         "Отвечает ли ответ на вопрос пользователя?", labels=("нет", "да"))     # 0.988
jev.noul(tok, "Как сварить борщ?\n---\nПитон — это язык программирования с динамической типизацией.",
         "Отвечает ли ответ на вопрос пользователя?", labels=("нет", "да"))     # 0.014

• All questions about one state run as one batch; each question sees only the state and itself.
• symmetric=True (default) also scores the options in reverse order and averages: the backbone is causal, so a single pass lets later options see earlier ones. symmetric=False is a strict single forward pass.
• Input layout, if you want to call the head directly: <bos><USER>state<EOT><R12>question<R11>option_1<R11>…option_K<R11>; the head reads the hidden state at the <R11> that closes each option (NQ4ForDecision.option_logits).

Honest results

Held-out test splits, symmetric mode. ECE = expected calibration error (0 = confidence matches accuracy).

task	options	accuracy	ECE	chance / majority
does this reply answer the question (yes/no)	2	98.2%	0.010	50%
which of 4 replies answers the question	4	99.0%	0.005	25%
news topic (AG News)	4	88.8%	0.017	25%
request topic (24 topics, 6 offered)	6	81.1%	0.029	17%
toxicity, Russian (yes/no)	2	80.0%	0.052	50%
sentiment (RU 3 classes + EN 2 classes)	2–3	78.1%	0.033	~40%
request difficulty easy / medium / hard	3	69.4%	0.028	63% (always "medium")
which of two answers is better	2	68.7%	0.055	50%
entailment / neutral / contradiction (XNLI RU+EN)	3	64.5%	0.026	33%
answer quality bad / okay / excellent	3	56.6%	0.042	33%

Task types never seen in training (zero-shot):

task	options	accuracy	chance
emotion (EN, dair-ai/emotion)	6	29.9%	17% (majority class 35%)
ruOpenBookQA (MERA)	4	26.3%	25%
PARus, cause / effect (MERA)	2	49.0%	50%

At 26M parameters the model learns every task family it is trained on, with honest probabilities, but it does not become a general decision model: new kinds of questions are answered near chance. Questions that need world knowledge or reasoning (OpenBookQA, PARus) are out of reach at this size.

A failure you will meet: "Доставили не тот размер, хочу вернуть деньги, а поддержка не отвечает уже третий день!" is classified as neutral (0.59) rather than negative (0.36) — the Russian sentiment training data is noisy.

Training

Full fine-tune of N-Q-26M-Chat plus the new head, log-loss over the options (a proper scoring rule, which is what keeps the probabilities calibrated). Option order is shuffled for choice and randomly reversed for score/noul.

stage	data	V100 time
v1	161k examples, 6 task families from saiga_scored (topic, quality, difficulty), preference pairs (saiga_preferences, UltraFeedback) and SmolTalk/GrandMaster replies (reply matching, yes/no match)	1.4 h
v2 (this checkpoint)	+ XNLI ru/en, RU sentiment, SST-2, RU toxicity, AG News: 242k examples, 10 task families, continued from v1	1.9 h

Adding the new families did not hurt the old ones (all within ±1–2 points) and lifted the new ones from their zero-shot level (NLI 38% → 65%, sentiment 44% → 78%, toxicity 36% → 80%, news 67% → 89%).

Packed single-file format (model.qpack)

Same QPACK1 container as the other N-Q releases (89 MiB): a folder with model.qpack and no *.safetensors loads through AutoModel.from_pretrained(..., trust_remote_code=True); python qpack.py --selftest model.qpack prints SELFTEST 5/5.

Limitations

• Near chance on task types it was not trained on; no world knowledge or multi-step reasoning.
• Not a calculator: the exact <CALC> circuits of the chat model work only when generating text.
• Russian sentiment and answer-quality labels are noisy, and so are the model's judgements there.
• Causal backbone: use symmetric=True (default) to remove the option-order bias.
• No safety tuning. Research and education only.

License and data terms

Code and weights: Apache-2.0 (see LICENSE). Training data keeps its own terms — check each dataset card before commercial use: saiga_scored, saiga_preferences, UltraFeedback, SmolTalk, GrandMaster-PRO-MAX, XNLI, the RU sentiment collection, SST-2, ru_paradetox_toxicity, AG News; plus the pretraining data of N-Q-26M.