litillabs/arbit-1.7b

Arbit

Arbit is a 1.7B model that picks from the options you give it and tells you how sure it is. One forward pass, one probability per option. Nothing to parse.

On DecisionBench 1.0 it is first on the English suite (87.51 vs Bosun's 87.29) and first overall (87.18 vs 84.90). On the reasoning board it is third (81.08). It is a LoRA fine-tune of Qwen/Qwen3-1.7B with a small decision head on top.

What it does

Most models answer by writing. Then your code reads the paragraph and hopes it understood. A decision model skips that. You hand it the situation, the question and the allowed answers; it hands back a score for each answer. Three kinds of question:

• Yes or no. Is this passage relevant? Is this comment a threat?
• Pick one. Which of 77 support queues? Which of 255 records is the same person? Which tool next?
• Rate it. How severe is this comment? How risky is this code change?

The answers are whatever you say they are, at request time. So the same model routes a support ticket, matches a person across 255 records, classifies a tax form page, or picks an agent's next move. And because you get a probability on every option, you can run it the way you'd run a junior: act on the confident calls, kick the close ones up to a person.

Results

DecisionBench 1.0, 23,900 rows, 43 tasks, dataset revision 071b7b2, official evaluator. Full coverage, no unsupported rows, no errors. The other rows in the table are reviewed records from Hanno-Labs/decision-bench-results as of 2026-10-22. Those ran at dataset revision b7c8107; 071b7b2 adds optional compact fields to the same frozen rows, and this run does not use them. ECE and NLL are over all scored rows, as in those records.

Model	Params	Overall	English suite	Reasoning	ECE	NLL
LiTiL Labs - Arbit 1.7B	2.06B	87.18	87.51	81.08	0.057	0.544
Hanno Labs - Bosun v3.1 1.7B	2.03B	84.90	87.29	39.67	0.050	0.478
mohit67890 - imajev-4b	4.78B	79.69	79.65	80.58	0.069	0.694
OpenAI - GPT-5.6 Luna	–	69.90	69.04	86.17	0.203	3.154

Overall is the benchmark's primary metric: accuracy over all 23,900 rows. English suite and Reasoning are the leaderboard's two featured boards, DecisionBench(eng, v1) (22,700 rows) and DecisionBench(Reasoning, eng, v1) (the other 1,200), scored the way the leaderboard scores them. Unsupported and error rows count as misses in all three. Params is each record's parameter_count. Bold is the best in the column.

By question type. The English suite on the left, the reasoning rows on the right. On the English suite the three of us are within a point on every type; the reasoning rows are where the models separate.

By task. The ten applied tasks and the four reasoning tasks are 89% of the rows. The other 27 tasks are pooled on the last line.

Use it

Stock Transformers and PEFT. pip install torch transformers peft safetensors, then:

from transformers import AutoModel

model = AutoModel.from_pretrained("litillabs/arbit-1.7b", trust_remote_code=True)

state = {"text": "My card was declined at the grocery store this morning."}
questions = {
    "route": {
        "type": "choice",
        "instructions": "Route this customer message to the right support queue.",
        "criteria": {
            "001 Declined Card Payment": "The card was declined when paying.",
            "002 Card Not Working": "The card does not work at all.",
            "003 Lost Or Stolen Card": "The card is lost or was stolen.",
        },
    }
}
answers = model.predict(state, questions)
print(answers["route"]["answer"])         # 001 Declined Card Payment
print(answers["route"]["probabilities"])  # {'001 Declined Card Payment': 0.9999, '002 Card Not Working': 0.0001, '003 Lost Or Stolen Card': 0.0}

Unsloth's FastDecisionModel loads it too:

from unsloth import FastDecisionModel

model, tokenizer = FastDecisionModel.from_pretrained("litillabs/arbit-1.7b", subfolder="checkpoint", max_seq_length=27648)
FastDecisionModel.for_inference(model)
answers = FastDecisionModel.predict(model, tokenizer, state, questions)

• Question types: choice (pick one), score (an ordered scale; criteria is the list of levels) and noul (yes or no). One call can ask several questions about one state.
• State: any JSON object.
• Input limit: 27,648 tokens, the training length. Past that the end of the state is cut. Questions and options never are.
• Loading: the first load pulls the base, unsloth/Qwen3-1.7B at revision 6262b50, then applies the LoRA adapter and the decision head from checkpoint/. Default is float32; dtype="auto" gives bfloat16 on a CUDA GPU that supports it.
• Jev-style serving: model.systemone(request) takes a /v1/systemone request body (model, state, questions) and returns model, answers and usage.
• Benchmark adapter: arbit-hf-remote-code-v1, the DecisionBench adapter this result ran with. It loads the model as above with trust_remote_code, at a pinned revision, in bfloat16 on CUDA, and renders each benchmark row as one question in the format shown. When every option's description is a record with the same fields, the field names are listed once and each option lists its values. Nothing is dropped.

How it works

The backbone reads the state, the instruction and every option in one pass. Then a small head, the same design as Cloudflare's Clef, looks at the backbone's output over every question and option and scores all the options together instead of judging each one on its own. Softmax those scores and you have the distribution. One pass, done.

Training

• Base: Qwen3-1.7B in 16-bit (the unsloth/Qwen3-1.7B mirror of Qwen/Qwen3-1.7B, revision 6262b50), a rank-128 LoRA adapter and a Clef-design decision head. Trained with Unsloth on single NVIDIA GPUs, 27,648-token inputs.
• Data: public datasets from the task families DecisionBench covers, listed below, plus synthetic tasks from our own generators. No DecisionBench evaluation rows were used in training.

Data	License
PolyAI/banking77	CC-BY-4.0
clinc/clinc_oos	CC-BY-3.0
Tevatron/msmarco-passage (MS MARCO)	MS MARCO terms
tasksource/esci	Apache-2.0
NortheasternUniversity/big_patent	CC-BY-4.0
google/civil_comments	CC0-1.0
nguha/legalbench	Per task; see the LegalBench task cards
Team-ACE/ToolACE	Apache-2.0
nvidia/Nemotron-PII	CC-BY-4.0
zal-analytics-core/dutch-law-enforcement-entity-resolution-dataset	CC-BY-4.0
dreamerdeo/finqa (FinQA)	MIT (FinQA release)
dgslibisey/MuSiQue	CC-BY-4.0 (MuSiQue release)
tasksource/folio (FOLIO)	CC-BY-SA-4.0
kyotofin/tax-doc-classifier on GitHub (data/criteria.json, revision 6afcf70)	Apache-2.0; IRS-derived, 17 U.S.C. § 105

License

Apache-2.0. Built on Qwen/Qwen3-1.7B (Apache-2.0), fine-tuned with LoRA and a decision head whose design follows Cloudflare/clef (Apache-2.0). clef/joint_schema_model.py is Cloudflare's reference code, vendored unmodified with its license.

Author

Aaron Kelly, Litil