jacobbeckdev/trigon-qwen3-4b-lms-yn

trigon qwen3-4b-lms-yn v1

A typed-decision model: send state and a map of typed questions (choice, yes/no, score), get one calibrated distribution per question in a single prefill pass. Served by the trigon gateway (docs/self-host.md).

What it is

• Backbone: Qwen/Qwen3-4B at 1cfa9a7208912126459214e8b04321603b3df60c (Apache-2.0), frozen; fetched from its own repository, never re-published here.
• This bundle: LoRA rank 16 and an answer readout (adapter.pt): each answer is scored as the backbone's own continuation, w * log p(answer) + residual, w = 0.639; build trigon-lm-score-qwen3-4b-qwen3-4b-lms-yn-s0+lm-score-v1.
• Trained: 1 epochs, lr 0.0001, seed 0, on cuda in 1.4 h.
• Licence: Apache-2.0 for this bundle. Every training corpus is green-tier (docs/data.md); share-alike and research-only corpora evaluate only and never train.

Trained on

Corpus	Cases	Calibration cases	Licence	Attribution
banking77	1,400	600	CC BY 4.0	Banking77 (Casanueva et al., 2020), PolyAI. CC BY 4.0. https://github.com/PolyAI-LDN/task-specific-datasets
goemotions	1,050	600	Apache-2.0	GoEmotions (Demszky et al., 2020), Google Research. Apache-2.0. https://github.com/google-research/google-research/tree/master/goemotions
helpsteer2	525	600	CC BY 4.0	HelpSteer2 (Wang et al., 2024), NVIDIA. CC BY 4.0. https://huggingface.co/datasets/nvidia/HelpSteer2
measuring_hate_speech	525	600	CC BY 4.0	Measuring Hate Speech (Kennedy et al., 2020; Sachdeva et al., 2022), UC Berkeley D-Lab. CC BY 4.0. https://huggingface.co/datasets/ucberkeley-dlab/measuring-hate-speech
hatexplain	875	600	MIT	HateXplain (Mathew et al., AAAI 2021), Punyajoy Saha and co-authors. MIT (repository LICENSE, Copyright (c) 2020 Punyajoy Saha); the authors' dataset card states CC BY 4.0. https://github.com/punyajoy/HateXplain at 01d742279dac
teacher-workflows	1,400	0	Apache-2.0	Generated and labelled by Qwen2.5-7B-Instruct (Qwen Team, Alibaba Cloud), Apache-2.0, huggingface.co/Qwen/Qwen2.5-7B-Instruct at a09a35458c70. Teacher labels buy coverage, not calibration.
teacher-local	1,120	0	Apache-2.0	Generated and labelled by Qwen3.6-35B-A3B (Qwen Team, Alibaba Cloud), Apache-2.0, huggingface.co/Qwen/Qwen3.6-35B-A3B, served by ollama; each record names the teacher and its ollama digest. Teacher labels buy coverage, not calibration.
mind2web-train	1,002	600	CC BY 4.0	Mind2Web (Deng et al., 2023), The Ohio State University. CC BY 4.0. https://huggingface.co/datasets/osunlp/Mind2Web
wanli	1,400	600	CC BY 4.0	WANLI (Liu et al., 2022), University of Washington and the Allen Institute for AI. CC BY 4.0. https://huggingface.co/datasets/alisawuffles/WANLI

Held out: corpora clinc150, boolq, circa. Every website in the web-action evaluation is excluded from mind2web-train.

Calibration was fitted on 4,732 held-out cases from banking77, goemotions, helpsteer2, measuring_hate_speech, hatexplain, mind2web-train, wanli, with option sets reshaped as training reshapes them. Teacher labels are never calibration evidence.

Evaluation

Generality (scripts/generality.py, 1,000 cases per task, fixed seed):

Task	Accuracy	Chance	ECE	ECE floor p95	Order agreement
banking77/declared	0.812	0.013	0.070	0.034
banking77/shift	0.862	0.020	0.066	0.028
banking77/renamed	0.838	0.020	0.077	0.029
clinc150	0.919	0.020	0.021	0.028
boolq	0.842	0.500	0.044	0.037
banking77/order	0.859	0.020	0.059	0.029	0.937

Web actions (scripts/webact.py, Mind2Web, websites unseen in training):

Steps	Operation	Element	Step success	ECE (floor p95)
600	0.882	0.678	0.600	0.068 (0.053)

Single seed: a measurement, not a certification (CLAUDE.md).

Use it

curl -LO <host>/models/bundles/qwen3-4b-lms-yn/v1/adapter.pt  # + SHA256SUMS files
sha256sum -c SHA256SUMS
trigon serve --backend torch --weights adapter.pt  # TRIGON_*_PATH for the calibrators