intent-router-zh-setfit-v1

A SetFit intent router for AI coding-agent traffic, fine-tuned from Qwen/Qwen3-Embedding-0.6B. It classifies a user prompt into 21 intents across three domains (coding / ops / general_control) plus out_of_scope, and ships with a v3 routing policy (confidence rejection + top-2 flip rules + precision-first keyword overrides) for end-to-end routing.

基于 Qwen3-Embedding-0.6B 微调的中英双语意图路由器，面向 AI 编码助手流量。 21 个意图，覆盖 coding（9 类）/ ops（8 类）/ general_control（3 类）+ out_of_scope。 附带 v3 路由策略（置信度拒判 + top2 翻转 + 高精度关键词硬路由）。 已知限制： (1) ops 领域（8 个意图）目前不可用——训练样本太少（每类 31-129 条），模型尚未学到可分性，请勿将 ops 预测用于生产路由； (2) 对"独立短指令"类 coding 输入泛化差。详见 Limitations。

Release Status

Domain	Status	Notes
coding (9 intents)	Partially usable	Usable on long, agent-style inputs (acc ~73%); not usable on short standalone instructions (acc ~12%)
ops (8 intents)	Not usable	Training samples too few (31-129 per class); predictions must not be trusted or used for routing
general_control (3 intents)	Insufficiently validated	Gold-set metrics only; no independent test

If you only need the coding intents, restrict the label set or treat any ops prediction as untrusted. A future release will retrain the ops head with more data.

Intended Use

• Route incoming prompts of an AI coding assistant to specialized downstream handlers. (The ops intents exist in the label set but are not usable in this release — see Release Status.)
• Best suited for long, context-rich, agent-style prompts (system prompts, requests with code/diff/stack-trace context). Accuracy on such inputs is ~73% (see Evaluation).
• For optimal classification performance, use a maximum input length of 1024 tokens. Inputs longer than this are truncated before classification.
• Not reliable for short standalone commands ("写一个防抖函数") — such inputs should be caught by the built-in confidence rejection (out_of_scope) instead of being routed.

Labels

Domain	Intents
coding (9)	code_authoring, code_modification, code_repair, code_review, code_explanation, test_generation, code_search, architecture_consultation, dependency_management
ops (8)	deployment, infrastructure_provisioning, monitoring_query, incident_response, pipeline_operation, config_change, security_operation, log_analysis
general_control (3)	context_specification, workflow_control, general_inquiry
fallback	out_of_scope

Training Data: Original Sample Set (n = 3203)

Label entries of the original training split, with per-class sample counts. The counts below are the raw, pre-oversampling distribution and explain the release status above: ops intents have at most 129 samples each, and dependency_management / test_generation are also critically underrepresented within coding.

Label (internal)	Source annotation	Domain	Samples	Status
code_review	Code Review	coding	741	adequate
code_modification	Code Modification	coding	420	adequate
code_explanation	Code Explanation	coding	368	adequate
code_authoring	Code Authoring	coding	249	adequate
code_repair	Code Repair	coding	187	adequate
code_search	Code Search	coding	158	adequate
context_specification	Context Specification	general_control	155	adequate
architecture_consultation	Architecture Consultation	coding	154	adequate
log_analysis	Log Analysis	ops	129	insufficient
out_of_scope	Out of Scope	fallback	125	adequate
config_change	Config Change	ops	94	insufficient
general_inquiry	General Inquiry	general_control	70	marginal
workflow_control	Workflow Control	general_control	54	marginal
monitoring_query	Monitoring Query	ops	46	insufficient
pipeline_operation	Pipeline Operation	ops	45	insufficient
test_generation	Test Generation	coding	45	insufficient
deployment	Deployment	ops	36	insufficient
security_operation	Security Operation	ops	32	insufficient
incident_response	Incident Response	ops	32	insufficient
dependency_management	Dependency Management	coding	32	insufficient
infrastructure_provisioning	Infrastructure Provisioning	ops	31	insufficient

Notes on the distribution:

• All 8 ops intents fall below 130 samples; 6 of them below 50. This is the direct reason the ops domain is declared not usable in this release.
• Within coding, dependency_management (32) and test_generation (45) match their observed F1 of ~0 on the independent test set.
• Training applied sqrt-inverse class weighting and oversampling to a minimum of 150 samples per class, which compensates the head but cannot create evidence the body never saw.

Usage

import os
os.environ["SETFIT_DEVICE"] = "cpu"          # CPU inference is the reference target
import torch
from setfit import SetFitModel

model = SetFitModel.from_pretrained("snival/intent-router-zh-setfit-v1", trust_remote_code=True)
model.to(torch.device("cpu"))

# Qwen3 embedding: lasttoken pooling, left padding, max_length 1024
body = model.model_body
body.max_seq_length = 1024
body.tokenizer.model_max_length = 1024
transformer = body[0]
transformer.max_seq_length = 1024
transformer.tokenizer.model_max_length = 1024
transformer.tokenizer.padding_side = "left"

probas = model.predict_proba([
    "帮我 review 一下这个 PR，重点关注线程安全",
    "Traceback (most recent call last): ... NullPointerException at OrderService.java:148",
])

Classification examples

Load the labels and display names from the model repository, then sort each probability vector to print the full classification signal:

import json

with open("labels.json", encoding="utf-8") as file:
  metadata = json.load(file)

labels = metadata["labels"]
display_names = metadata["display_names"]

prompts = [
  """Review this session-handling diff before merge. The original raised on cache miss;
  the new version falls back to DB. Review checklist: 1) does the fallback re-check
  expiry, 2) cache stampede on hot tokens, 3) is the failure mode fail-open or
  fail-closed. Give a merge / request-changes verdict with reasoning:

  diff --git a/src/auth/session.py b/src/auth/session.py
  index 3f2a1b9..9c8d2e1 100644
  --- a/src/auth/session.py
  +++ b/src/auth/session.py
  @@ -41,7 +41,10 @@ class SessionStore:
       def refresh(self, token: str) -> Session:
           session = self._cache.get(token)
           if session is None:
  -            raise SessionNotFound(token)
  +            session = self._db.load(token)
  +            if session is None:
  +                raise SessionNotFound(token)
  +            self._cache[token] = session
           session.expires_at = now() + self.ttl
           return session
  """,
]

for prompt, scores in zip(prompts, model.predict_proba(prompts)):
  print(f"\nPrompt: {prompt}")
  ranked = sorted(enumerate(scores.tolist()), key=lambda item: item[1], reverse=True)
  for rank, (index, score) in enumerate(ranked, start=1):
    intent = labels[index]
    print(f"{rank}. {display_names[intent]} (`{intent}`): {score:.2%}")

Measured raw-classifier output

The following full ranking was measured locally with the bundled model on CPU at a maximum input length of 1024 tokens. It is raw classifier output, not the final result after applying the v3 routing policy:

Prompt: Review this session-handling diff before merge. ...
1. Code Review (`code_review`): 82.23%
2. Security Operation (`security_operation`): 2.40%
3. Workflow Control (`workflow_control`): 2.39%
4. Architecture Consultation (`architecture_consultation`): 1.83%
5. Dependency Management (`dependency_management`): 1.64%
6. General Inquiry (`general_inquiry`): 1.46%
7. Code Search (`code_search`): 1.41%
8. Deployment (`deployment`): 1.23%
9. Incident Response (`incident_response`): 0.98%
10. Out of Scope (`out_of_scope`): 0.93%
11. Infrastructure Provisioning (`infrastructure_provisioning`): 0.64%
12. Code Repair (`code_repair`): 0.61%
13. Config Change (`config_change`): 0.60%
14. Monitoring Query (`monitoring_query`): 0.41%
15. Code Explanation (`code_explanation`): 0.37%
16. Pipeline Operation (`pipeline_operation`): 0.35%
17. Test Generation (`test_generation`): 0.17%
18. Log Analysis (`log_analysis`): 0.17%
19. Code Modification (`code_modification`): 0.08%
20. Context Specification (`context_specification`): 0.05%
21. Code Authoring (`code_authoring`): 0.05%

This high-confidence code_review result is representative of the model's intended long, context-rich coding inputs. For production routing, pass the full probability vector to the v3 policy described below so confidence rejection, keyword rules, and top-2 flip rules are applied.

End-to-end routing (recommended)

The repository includes labels.json, metrics.json and rejection_policy.json. Apply the v3 policy on top of predict_proba output:

1. keyword hard-routing (high-precision regex overrides, e.g. traceback → code_repair),
2. top-2 flip rules for historically confused near-neighbor pairs,
3. confidence rejection: top_score < 0.15 (or short text ≤ 3 chars) → out_of_scope.

Reference implementation of decide_route() is available in the training project (routing_policy.py); the policy JSON is self-contained so the rules can be reimplemented in any stack.

Quantized ONNX variants (edge deployment)

The fp32 body (model.safetensors, 2.3 GB) is heavy for edge gateways, so two int8 ONNX builds of the encoder (transformer + lasttoken pooling + L2 normalize) are included. Both were verified end-to-end against the fp32 reference on the 611-sample gold test set (max_length=128, CPU):

Variant	Files	Size	Agreement with fp32 predictions	Gold accuracy	Relative CPU latency
Weight-only int8 (recommended)	encoder-woq8.onnx + encoder-woq8.onnx.data	615 MB	98.9%	identical to fp32	~2× slower
Dynamic int8	encoder-int8.onnx	600 MB	78.4%	-0.3pp vs fp32	~25% faster than fp32

• Choose encoder-woq8 when routing decisions must match the fp32 model (mean embedding cosine 0.999). Only MatMul weights (per-block) and the token embedding table (per-row) are int8; activations stay fp32.
• Choose encoder-int8 (dynamic quantization of weights and activations) when latency matters more than exact routing parity. Prediction flips vs fp32 concentrate on low-confidence samples, which the v3 confidence rejection largely discards anyway.

Both variants share the same inference contract (input_ids / attention_mask → sentence_embedding) and reuse head_coef.npy / head_intercept.npy / head_meta.json for classification — see ONNX_INFERENCE.md for a complete onnxruntime example. The source fp32 weights above remain the training/checkpoint format of record.

Evaluation

Gold test set (in-distribution, n=611, real agent prompts)

Metric	Closed-set	Routed (v3 policy)
Accuracy	0.558	0.563
Macro-F1	0.296	0.278 (in-scope)
Domain accuracy (coding/ops/general)	—	0.791
OOS precision / recall / F1	—	0.329 / 0.920 / 0.484

By prompt length (closed-set): 800+ chars: 0.733 (n=277) · 100–800: 0.392 (n=148) · <100: 0.430 (n=186)

Independent synthetic coding test (out-of-distribution stress test, n=360, 9 coding intents × 40)

A fully independent set (zero leakage vs train/gold), hand-authored with controlled diversity: zh 36% / en 44% / mixed 20%; easy 60% / medium 30% / hard 10%; 30% long texts (800+ chars with code, diffs, stack traces).

Metric	Closed-set	Routed
Accuracy	0.117	0.203
Macro-F1	0.134	0.246
Weighted-F1	0.134	0.246

Per-class routed F1 (best → worst): code_review 0.548 · code_repair 0.333 · code_search 0.256 · code_modification 0.254 · code_authoring 0.222 · test_generation 0.222 · architecture_consultation 0.188 · code_explanation 0.187 · dependency_management 0.000.

By difficulty (routed acc): easy 0.181 / medium 0.241 / hard 0.222. By language (routed acc): zh 0.256 / en 0.133 / mixed 0.260.

Top confusion pairs (closed-set): code_authoring → architecture_consultation (13×), code_modification → architecture_consultation (11×), code_modification → code_search (10×), code_review → architecture_consultation (9×).

This set is dominated by short standalone instructions (median 204 chars) — the model's weakest region. Routed gain comes almost entirely from keyword rules; 29% of samples were honestly rejected as out_of_scope by the confidence threshold.

Ops domain: not evaluated, declared unusable

No ops test set was built for this release because the ops training data is too scarce for the model to have learned reliable boundaries. We deliberately declare the 8 ops intents unusable rather than publish misleading numbers. They remain in the label set for forward compatibility — treat any ops prediction as untrusted.

CPU speed (Intel i9-14900KF, 32 threads)

Cold start 0.3s · single-sample latency p50/p90/p99 = 108/489/647ms · throughput 4.4 samples/s (batch=8)

Limitations

• Short standalone prompts are out-of-distribution. Training data was dominated by long agent-style prompts (median ~600 chars). Short imperative inputs collapse to a narrow region of the embedding space and produce near-uniform predictions. Do not deploy for general short-prompt routing without augmenting short-instruction training data.
• dependency_management and test_generation are the weakest intents (F1 ≈ 0 on the synthetic test); code_review is the strongest (routed F1 0.55).
• The 8 ops intents are currently unusable: their training sample counts are too small, so the model has not learned to separate them. Do not use ops predictions for routing; they are kept in the label set only for forward compatibility and will be retrained in a future release.
• Confusion concentrates on semantic near-neighbors: code_authoring ↔ architecture_consultation, code_modification ↔ code_repair, code_search ↔ code_explanation.

Training

• Method: SetFit (contrastive body fine-tuning + weighted linear head)
• Base: Qwen/Qwen3-Embedding-0.6B, max_length=1024, pooling=lasttoken, padding_side=left
• Data: 3203 real user/agent prompts (21 intents, oversampled to min 150/class); body trained on a capped class-balanced subset (2424), head on the full split
• Head: SetFitHead with sqrt-inverse class weighting, lr=5e-3, l2=0.01, 8 epochs; body 2 epochs, 8 iterations
• Trained on CUDA (RTX 5090 D); inference reference target is CPU

Files

File	Purpose
model.safetensors + transformer/tokenizer configs	Fine-tuned embedding body
encoder-woq8.onnx (+ .data), encoder-int8.onnx	Int8 quantized ONNX encoders for edge deployment (see Quantized ONNX variants)
ONNX_INFERENCE.md	onnxruntime inference guide for the quantized encoders
classification_head.pkl/.pt, model_head.pkl	SetFit head (native)
head_coef.npy, head_intercept.npy	Head weights as plain arrays (framework-free inference)
labels.json	Label list, label map, display names, intent→domain map
rejection_policy.json	v3 routing policy (thresholds, flip rules, keyword rules)
metrics.json	Full training-time evaluation record