jacobbeckdev/trigon-qwen3-4b-lms-agents

trigon qwen3-4b-lms-agents v1.1

A typed-decision model: send state and a map of typed questions (choice, yes/no, score), get one calibrated distribution per question in a single prefill pass. Served by the trigon gateway (docs/self-host.md).

What it is

• Backbone: Qwen/Qwen3-4B at 1cfa9a7208912126459214e8b04321603b3df60c (Apache-2.0), frozen; fetched from its own repository, never re-published here.
• This bundle: LoRA rank 16 and an answer readout (adapter.pt): each answer is scored as the backbone's own continuation, w * log p(answer) + residual, w = 0.303 / 0.482 (choice / yes-no); build trigon-lm-score-qwen3-4b-qwen3-4b-lms-agents-s0+lm-score-v1.
• Trained: 1 epochs, lr 0.0001, seed 0, on cuda in 3.0 h.
• Licence: Apache-2.0 for this bundle. Every training corpus is green-tier (docs/data.md); share-alike and research-only corpora evaluate only and never train.

Trained on

Corpus	Cases	Calibration cases	Licence	Attribution
banking77	1,500	500	CC BY 4.0	Banking77 (Casanueva et al., 2020), PolyAI. CC BY 4.0. https://github.com/PolyAI-LDN/task-specific-datasets
goemotions	1,238	412	Apache-2.0	GoEmotions (Demszky et al., 2020), Google Research. Apache-2.0. https://github.com/google-research/google-research/tree/master/goemotions
helpsteer2	844	281	CC BY 4.0	HelpSteer2 (Wang et al., 2024), NVIDIA. CC BY 4.0. https://huggingface.co/datasets/nvidia/HelpSteer2
measuring_hate_speech	844	281	CC BY 4.0	Measuring Hate Speech (Kennedy et al., 2020; Sachdeva et al., 2022), UC Berkeley D-Lab. CC BY 4.0. https://huggingface.co/datasets/ucberkeley-dlab/measuring-hate-speech
hatexplain	1,107	368	MIT	HateXplain (Mathew et al., AAAI 2021), Punyajoy Saha and co-authors. MIT (repository LICENSE, Copyright (c) 2020 Punyajoy Saha); the authors' dataset card states CC BY 4.0. https://github.com/punyajoy/HateXplain at 01d742279dac
teacher-workflows	1,400	0	Apache-2.0	Generated and labelled by Qwen2.5-7B-Instruct (Qwen Team, Alibaba Cloud), Apache-2.0, huggingface.co/Qwen/Qwen2.5-7B-Instruct at a09a35458c70. Teacher labels buy coverage, not calibration.
teacher-local	1,120	0	Apache-2.0	Generated and labelled by Qwen3.6-35B-A3B (Qwen Team, Alibaba Cloud), Apache-2.0, huggingface.co/Qwen/Qwen3.6-35B-A3B, served by ollama; each record names the teacher and its ollama digest. Teacher labels buy coverage, not calibration.
mind2web-train	1,202	400	CC BY 4.0	Mind2Web (Deng et al., 2023), The Ohio State University. CC BY 4.0. https://huggingface.co/datasets/osunlp/Mind2Web
wanli	1,500	500	CC BY 4.0	WANLI (Liu et al., 2022), University of Washington and the Allen Institute for AI. CC BY 4.0. https://huggingface.co/datasets/alisawuffles/WANLI
jailbreak-train	1,199	399	Apache-2.0	jailbreak-classification (Jack Hao, 2023). Apache-2.0. https://huggingface.co/datasets/jackhhao/jailbreak-classification at 2f2ceeb39658
prompt-injections-train	410	136	Apache-2.0	prompt-injections (deepset, 2023). Apache-2.0. https://huggingface.co/datasets/deepset/prompt-injections at 4f61ecb038e9
aegis2-train	1,500	500	CC BY 4.0	Aegis 2.0 AI Content Safety Dataset (Ghosh et al., 2025), NVIDIA. CC BY 4.0. https://huggingface.co/datasets/nvidia/Aegis-AI-Content-Safety-Dataset-2.0 at d86bb8bedff5
teacher-agents	2,426	0	Apache-2.0	Generated and labelled by Qwen3 mixture-of-experts models (Qwen Team, Alibaba Cloud), Apache-2.0, served by ollama; each record names the teacher and its ollama digest. Teacher labels buy coverage, not calibration.

Held out: corpora clinc150, boolq, circa. Every website in the web-action evaluation is excluded from mind2web-train.

Calibration was fitted on 4,140 held-out cases from banking77, goemotions, helpsteer2, measuring_hate_speech, hatexplain, mind2web-train, wanli, jailbreak-train, prompt-injections-train, aegis2-train, with option sets reshaped as training reshapes them. Teacher labels are never calibration evidence.

v1.1 (2026-10-14): yes/no answers calibrated. v1 served them raw. The rule that chooses a calibrator checked its fit on unshuffled answers -- other corpora than it was fitted on -- and counted every question of a case, so GoEmotions' seven questions a text cancelled the other corpora's overconfidence. Refit with both fixed on the same 4,140 cases on the serving GPU: a yes/no temperature of 1.2946 (held-out check 0.0420 -> 0.0275). The weights and the Choice temperature are v1's: the fixed rule would also have declined the Choice temperature, and the public decision benchmark's knowledge slices, which no calibration corpus resembles, need it. On that benchmark, mean ECE over 11 slices 0.073 -> 0.068, accuracy unchanged (Aegis 0.072 -> 0.034, agent transcripts 0.082 -> 0.059).

Evaluation

Generality (scripts/generality.py, 1,000 cases per task, fixed seed):

Task	Accuracy	Chance	ECE	ECE floor p95	Order agreement
banking77/declared	0.793	0.013	0.028	0.045
banking77/shift	0.847	0.020	0.031	0.037
banking77/renamed	0.836	0.020	0.029	0.035
clinc150	0.926	0.020	0.115	0.039
boolq	0.866	0.500	0.046	0.027
banking77/order	0.851	0.020	0.032	0.035	0.924

Web actions (scripts/webact.py, Mind2Web, websites unseen in training):

Steps	Operation	Element	Step success	ECE (floor p95)
600	0.885	0.685	0.605	0.066 (0.053)

Single seed: a measurement, not a certification (CLAUDE.md).

Use it

curl -LO <host>/models/bundles/qwen3-4b-lms-agents/v1.1/adapter.pt  # + SHA256SUMS files
sha256sum -c SHA256SUMS
trigon serve --backend torch --weights adapter.pt  # TRIGON_*_PATH for the calibrators