mohameddhameem/systemone-decider-e4b-qp

SystemOne-Decider E4B qp

A router between a small and a large language model for multi-hop questions: a Strands Decider (E4B) with its LoRA and pointer head fine-tuned to answer one typed question, "Which model should answer this question?" Given the question and the small model's passages (before the small model answers), it returns P(large model). Send the question to the large model when that probability is at or above threshold in systemone.json, which is chosen on validation.

• Small model: Qwen2.5-1.5B-Instruct on nano-jev's top 2 HotpotQA paragraphs. Large model: Qwen2.5-7B-Instruct on all 10.
• Label: the large model is needed: the small one is wrong and the large one is right (lenient scoring).
• Seeds: seed 0 is in the root folder; seeds 1 and 2 are in seed1/ and seed2/.
• Base: StrandsAgents/strands-decider-E4B-gemma4-v1-2610, on google/gemma-4-E4B-it, which loads on first use.
• Code: github.com/mohameddhameem/systemone-decider (systemone_decider.route.PairRouter).

Results

The SLR split of HotpotQA's dev set: 4,814 training, 1,110 validation and 1,481 test questions. Cost is the AgentTTS Table 5 price per answering call (Qwen2.5-1.5B 0.5, Qwen2.5-7B 10); the judge's own call is priced the same way. AIQ is the area under the accuracy-vs-cost curve. Intervals are 95% paired bootstrap over the test questions.

	AIQ	With its own cost
Seed 0 (root folder)	0.681 [0.660, 0.703]	0.645 [0.623, 0.665], 4.27 a call
Single seeds, mean ± sd	0.681 ± 0.001	0.645
Three-seed ensemble (mean P)	0.682	0.609 (three calls)
Three-seed ensemble, the 1,303 questions nano-jev never saw	0.679
Entropy threshold on the small model	0.665
Logistic-regression router on the question and passages (before)	0.668
Best logistic-regression router at the same decision point	0.669
Random	0.638
Oracle	0.758

Seed 0 minus each baseline:

	Before its own cost	With it
Entropy threshold	+0.016 (95% CI +0.003 to +0.027, better)	-0.021 (95% CI -0.034 to -0.009, worse)
Logistic-regression router on the question and passages (before)	+0.013 (95% CI +0.002 to +0.019, better)	-0.024 (95% CI -0.036 to -0.016, worse)

At its published threshold (0.6732), seed 0 on the test set: 64.89% at a cost of 1.10 a question, plus 4.27 for the judge's call, sending 6.3% of questions to the large model. Always-large: 66.64% at a cost of 10.00 a question; always-small: 60.90% at a cost of 0.50 a question. The threshold keeps accuracy within 1% of always-large's on validation; on test, seed 0 lands 2.6% below it, at 54% of always-large's cost with the judge's call included.

Use

pip install git+https://github.com/mohameddhameem/systemone-decider

from systemone_decider.route import PairRouter

# seed 0; subfolder="seed1" or "seed2" for the others
router = PairRouter.from_pretrained("mohameddhameem/systemone-decider-e4b-qp")
# passages: ["title: text", ...], the small model's top 2
p = router.p_large(question, passages)
use_large = router.route(question, passages) == "large"

The judge runs the Strands Decider's E4B torso, so a GPU is needed for useful speed. On a GPU (bf16, the judge's default batches of 16), these folders reproduce every seed's saved scores on all 1,481 test questions exactly.

Training

The Strands Decider's LoRA (rank 48) and pointer head, fine-tuned with grouped-softmax cross-entropy on the 4,814 training questions: lr 0.0001, batch 16, up to 2 epochs, with the epoch of lowest NLL on a 10% hold-out kept. Training started from StrandsAgents/strands-decider-E4B-gemma4-v1-2610 at revision 1344d44397c457f3be34b8ff15c593b41248c6bf. training.json has the full record and the loss per epoch.

Limits

Trained and tested on HotpotQA with one model pair. The threshold is chosen on validation for this pair; re-pick it on your own validation data before using another pair or dataset.

Licence

Apache-2.0, as the Strands Decider and google/gemma-4-E4B-it are. HotpotQA, the training data, is CC BY-SA 4.0. Whether its share-alike terms apply to weights trained on it is not settled, and this card does not claim that they do not.