Sieve-2B

Sieve-2B is a decision model from the Sieve family. You give it a piece of text or JSON (the state) and typed questions, and it returns a calibrated probability for every option of every question. It reads the state once, scores the options directly, and never generates text.

It is a LoRA adapter (rank 8) and a scalar scorer head on the first 16 of 24 blocks of Qwen/Qwen3.5-2B-Base. The code is at github.com/sthanika-ai/Sieve.

Summary

Backbone	Qwen/Qwen3.5-2B-Base, blocks 1-16 of 24
Design	option fork
Trained weights	LoRA r=8, α=16 on the attention and MLP projections of blocks 1-16, and a scalar head
Question types	choice and noul
Decision Index	Full score 20.53 and public score 22.16 on the leaderboard (edition 0.3.2)
Temperature	T = 1.7144
Token limits	32,768 tokens for the state, the question and the longest option; trained on prompts up to 2,048
GPU memory	the weights take about 3 GB in bf16

How it works

Decision models that read several options in one pass can change their answer when the same options are given in a different order. Sieve cannot flip: the state and question are encoded once, that encoded state is copied once per option, and each option is scored as a clean continuation of its own copy, never seeing its siblings. This is a property of the architecture, not something learned: it holds by construction, and was verified in development at a numerical precision of ~1e-6 and as exactly 0 argmax flips across 600 real banking77 examples under option reshuffling.

The prefix-state fork (see sieve/fork/model.py) replicates DynamicCache -- both the attention KV cache and the Gated DeltaNet layers' recurrent state -- once per option, then runs every option as one right-padded batch of continuations. load_sieve and the Decision Index engine run it in bf16; load_sieve(..., dtype=torch.float32) gives fp32.

	parameters
run at inference (embeddings + blocks 1-16 of the base, + adapter + head)	1.43 B
trained by us (LoRA 3,637,248 + scorer head 1,053,697)	4.69 M
full base model (Qwen/Qwen3.5-2B-Base text model), for reference	1.88 B

Blocks 17-24 and the base model's text-generation head never run, so they are not loaded onto the GPU.

type	options	returns
choice	the caller's keys, optionally described	a probability per key
noul	yes / no, scored as a 2-option choice	P(yes)

Use

pip install "sieve-decisions[cuda,serve] @ git+https://github.com/sthanika-ai/Sieve"
pip install --no-build-isolation causal-conv1d==1.7.0   # recommended: the fused convolution kernel

from sieve import load_sieve, decide

m = load_sieve("sthanika-ai/Sieve-2B")
decide(m, {"subject": "Duplicate charge on invoice 4411", "body": "Billed twice. Refund today or we cancel."},
       {"department": {"type": "choice", "instructions": "Which team handles this?",
                       "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs and outages"}},
        "churn_risk": {"type": "noul", "instructions": "Does the customer threaten to cancel?"}})

The model's own call takes a plain list of options:

from sieve import load_sieve

s = load_sieve("sthanika-ai/Sieve-2B")   # downloads Qwen3.5-2B-Base once (~4.3 GB), applies this LoRA + head
s.choice(
    state="Customer: my invoice was charged twice and nobody answers the phone!",
    question="Which team should handle this?",
    options=["Billing", "Technical support", "Sales"],
)
# -> {"decision": "Billing", "index": 0, "confidence": 0.81,
#     "probs": {"Billing": 0.81, "Technical support": 0.16, "Sales": 0.03}}

Reordering options gives back the identical distribution (only the dict order changes), and the probs field is a genuine probability distribution over the options actually given.

• Loading. load_sieve downloads Qwen3.5-2B-Base once (~4.3 GB) and applies this LoRA and head.
• Server. sieve-serve --model sthanika-ai/Sieve-2B.
• Decision Index. python -m decision_index run --engine sieve.decision_index_engine:SieveEngine --option model=sthanika-ai/Sieve-2B --out runs/Sieve-2B.
• Context length. config.json sets max_len: 32768: inputs up to 32,768 tokens are scored. sieve.decide and the Decision Index engine refuse longer ones; the model's own choice() trims them from the front of the state. Training used prompts up to 2,048 tokens.

Results

Decision Index

On the Decision Index leaderboard (edition 0.3.2, board data of 2026-10-12), Sieve-2B has a Full score of 20.53 and a public score of 22.16. The Full score ranks the board: the public benchmarks count 20%, the maintainers' private tests of the same skills 50% and their private tasks from new domains 30%.

Our own run of the 0.2.1 suite was scored with the Decision Index reproduction kit on the full frozen suite (rebuilt byte-identical to the maintainers' files: selected rows sha256 b2b56d6f..., added rows 7429f3c9...): 150,317 scored requests, complete, coverage 1.00 in every area.

	Decision Index	raw	breadth	Knowledge & Reasoning	Language	Retrieval & Classification	Tools & Automation	Arts & Human Taste
Sieve-2B	21.74	39.91	20.55	10.6	19.7	28.5	35.3	17.8

• Coverage. All 150,317 scoreable requests over 44 benchmarks were run: 150,315 answered and 2 refused (their options have identical text), with no truncation and no error. The index averages 38 of the benchmarks.
• Files. sthanika-ai/Sieve-2B-decision-index-results -- scores.json, index.json, benchmark-summary.json, environment.json, status.json, and results.jsonl.gz (the kit's --compact form: payloads dropped so no suite text is republished; every row keeps its payload_sha256, and the file re-scores to the identical index).
• Engine. sieve.decision_index_engine:SieveEngine from github.com/sthanika-ai/Sieve with --option model=sthanika-ai/Sieve-2B (the run used the same engine code, then shipped in this repo as sieve_engine:SieveEngine), bf16, context limit 32,768 tokens, options mem_fraction=0.16, max_batch_tokens=49152 (these options reproduce the recorded probabilities bit-for-bit; defaults give the same decisions). 2 scored requests refused (options with identical text); 0 errors.
• Execution. 24 shards of exactly the rows pipeline runs, 7 concurrent processes on 2x A100 80GB, merged by run_id (details in environment.json). Single-process latency: median 85 ms, p95 223 ms (400 uniformly sampled suite requests, A100).
• Overlap with the suite. We checked every training row and the temperature-fitting pool against every suite row (inputs only: a shared 13-word run, or a whole 5-12-word field contained in the other side). 33 of the 150,317 scored rows (0.02%) share an input with the training data: BANKING77 10 (short customer queries that occur word-for-word in BANKING77's own train and test splits), HoVer 9 and RAGTruth 6 (shared source sentences with unrelated labels), ANLI 2, SATA-Bench 2, iSarcasmEval 2, BRIGHT 1, MMLU 1 (not in the index). No benchmark's test set was used for training. The temperature pool (used only to fit the single temperature, which never changes a decision) shares inputs with 85 suite rows, mostly ContractNLI clauses, CLINC150 utterances and MuSR. Every run_id is listed in runs/Sieve-2B/overlap_run_ids.json of the results dataset.

Other benchmarks

This checkpoint, single seed:

benchmark	Sieve-2B
MASSIVE (intent, K=59)	71.5
MMLU	60.8
SciQ	97.5
transfer-v4 (clean, out-of-domain)	73.2
held-out pool (unseen schemas, K up to 150)	77.7
banking77 (in-domain)	76.5
answer changes under option reordering	0.0%

Calibration

Calibration is built in, with no labels needed from you -- a single shipped temperature, fit once by us on data this checkpoint never trained or was tested on (a held-out pool of unseen schemas: clinc, newsgroups, yahoo, LEDGAR), and applied automatically by choice(). config.json's "temperature": 1.7144 is that value; loading the model already gets you this.

Measured effect on held-out benchmarks (Expected Calibration Error, lower is better; accuracy is provably unchanged, since a temperature only rescales confidence, never the decision):

benchmark	raw (T=1.0)	built in (T=1.71)
MMLU	0.135	0.040
transfer-v4 (clean)	0.114	0.074
transfer-v4 (all)	0.122	0.049
held-out calibration split	0.050	0.032
SciQ	0.013	0.018
banking77	0.079	0.082
MASSIVE	0.088	0.094

4 of 7 benchmarks improve, with the largest gains on MMLU and transfer-v4. SciQ, banking77 and MASSIVE get slightly worse -- one shared number can't correct every benchmark's own direction of miscalibration at once. If your use case is MASSIVE- or banking77-shaped specifically, the per-schema refit below will do better than either column here.

Optional, sharper upgrade if you have your own labels: call s.fit_schema(name, examples) once with ~100 labelled (state, question, options, gold_index) examples for your specific kind of question, then pass schema=name to s.choice(...). This is a 2-parameter logistic refit of the confidence (never the decision itself):

benchmark	raw (T=1.0)	built in (T=1.71)	with 100-label fit_schema
MASSIVE	0.089	0.094	0.059
banking77	0.078	0.082	0.050
MMLU	0.134	0.040	0.063
transfer-v4 (clean)	0.116	0.074	0.072
held-out calibration split	0.049	0.032	0.042
pool (unseen schemas)	0.087	--	0.041

fit_schema is never required -- the model is fully usable, and already reasonably calibrated, without it.

Latency

Single process, one request at a time, bf16, idle A100: median 85 ms, p95 223 ms (400 uniformly sampled Decision Index requests).

Training

• Base. Qwen/Qwen3.5-2B-Base, a hybrid of Gated DeltaNet (linear-attention) and full-attention blocks. Only its first 16 of 24 blocks ever run (read_layer=16).
• Adapter. LoRA, r=8, α=16, on the attention and MLP projections of blocks 1-16 (3,637,248 parameters), plus a scalar scorer head (LayerNorm -> 512 -> 1, 1,053,697 parameters).
• Recipe. From scratch, 2 epochs, lr 5e-5, training prompts up to 2,048 tokens. An earlier run at lr 2e-4 (4x higher) scored 13.8 points lower on MMLU with everything else held constant -- evidence that a lower rate preserves more of the base model's own knowledge during LoRA fine-tuning. See "Caveats": this finding is from a single seed pair.
• Data, per epoch (26,241 rows):
  • ~18,200 rows from ten public datasets (banking77, boolq, ag_news, multi_nli, sst5, yelp_review_full, trec, dbpedia_14, amazon_reviews_multi_en, imdb; listed above), with policy-rule and contrast examples;
  • 7,999 synthetic option-comparison questions generated by us (pick the largest / second-largest / median / closest-to-target / sum of a set of numbers or expressions). No outputs of other models were used for training or labels.
• The prompt lists every option, in an order fixed by a hash of each option's own text -- the order depends only on which options are present, never on their order or value, which is what keeps the invariance above exact while still letting the model see the whole option set.

Limitations

• Single seed. Earlier multi-seed runs of the same recipe family showed a spread of roughly +/-0.2-1.5 points on real benchmarks; this exact checkpoint has not been replicated on a second seed.
• The lr 5e-5 vs 2e-4 finding is from one paired comparison, not a multi-seed sweep.
• MMLU and transfer-v4 are the weakest results (general knowledge and out-of-domain decisions); this is the known, open gap for this model size.
• Requires transformers>=5.17 for the qwen3_5 architecture and peft>=0.21.

Files

file	contents
adapter_model.safetensors	the LoRA adapter (rank 8)
head.safetensors	the scalar scorer head
config.json	backbone, read layer, temperature, context limit and training settings
tokenizer.json, tokenizer_config.json, vocab.json, merges.txt	the Qwen3.5 tokenizer

License

Apache-2.0 for the adapter and head in this repository, and for the code at github.com/sthanika-ai/Sieve. The Qwen3.5 base model is Apache-2.0. The public training datasets each keep their own license (see the datasets: field above and each dataset's own card); several of them (ag_news, yelp_review_full, imdb, sst5, trec, amazon_reviews_multi) do not state a licence on their Hub cards -- check their original terms before commercial use.