coding-router — trained default artifact (v1.2, traffic-shaped union bank)

Changelog v1.2 (2026-08-01): closes the evidence gap that made v1.1 route all real interactive traffic to opus5_high. v1.1's bank held only long-form DeepSWE issue statements (110 texts, min 471 chars), so short opencode asks and tool-loop turns fell below sim_floor and abstained to the fallback arm — safe but zero saving. v1.2 rebuilds the bank as a union of three live-measured sources: DeepSWE-live (110 tasks, cells unchanged from v1.1) + LiveCodeBench-live (65 agentic tasks that cleared the ≥8/10-arm density rule; 27 hard tasks dropped, not imputed) + a new utility suite (32 short interactive asks — bugfix / feature / refactor / repo questions on 4 pinned OSS repos, each with a deterministic programmatic verifier, no LLM judging). All 10 live arms were measured on every source in-lane through the E2B harness (utility graded = fraction of deterministic checks passed). med_cost per arm is now the median of live per-task costs pooled across the three sources (e.g. opus5_high $5.46 → $2.57). sim_floor recomputed as the LOO-nearest-neighbour p10 over the union bank (0.459 → 0.470). lam re-selected train-side only (0.005 → 0.10) on 6 seeded group-clustered splits weighting sources 50 % utility / 30 % LCB / 20 % DeepSWE (assumed interactive-traffic mix, documented in provenance/selection.json); the higher lam is feasible because utility/LCB tasks are cheap and near-ceiling, and it is what now pulls short asks onto cheap arms. Tuned encoder, bank embedding space, and soft-vote T unchanged.

Serve-path verification (MLX, real build_trajectory → route_embedding): the 5-probe interactive replay that funneled 100 % to opus5_high under v1.1 now lands 4/4 short/mid/multi-turn probes on luna_high in-distribution; the French sonnet probe still abstains (off_distribution=True); the 7-rung difficulty ladder makes identical decisions to v1.1 on the hard rungs (rungs 5–6 → luna_high, rung 7 distributed-systems → opus5_high by abstention) — no escalation regression — and is invariant to a 12 k-char system preamble 7/7. Latency p50 15 ms / p95 26 ms.

Paired serve fix (required: coding-router ≥ b52d234): user messages are embedded BARE in the routing trajectory. The previous [user] "..." wrapper inflated short-text similarity against this bank's short utility rows so much that non-coding one-liners stopped abstaining; with bare content, coding asks match raw-to-raw (0.47–0.70, in-distribution → cheap arms) and non-coding text falls below the floor and abstains (measured: sonnet 0.352, weather 0.432, wedding 0.467 vs floor 0.470).

Honest numbers (union-bank, 6-seed group-clustered holdout, routed vs always-opus5_high):

source	n	routed graded	opus graded	Δgraded (95% CI)	cost ratio (95% CI)
utility (short interactive)	64	1.000	1.000	+0.000 [0.000, 0.000]	5.96× [3.46, 9.53]
LiveCodeBench	92	0.961	0.997	−0.036 [−0.100, +0.004]	6.02× [3.81, 8.90]
DeepSWE	160	0.908	0.926	−0.018 [−0.070, +0.048]	3.78× [2.74, 5.44]

Quality deltas' CIs contain zero on every source (statistical parity); the cost saving is the win, and it now applies to the short interactive traffic that v1.1 could not serve. lam was selected on inner train-side splits only; these holdouts were scored afterward. See provenance/eval_v12.json, verify_v12.json, ladder.json.

---

Changelog v1.1 (2026-08-01): the bank's published DeepSWE cells were replaced with the EXP-015 LIVE matrix (1,130 fresh trials, dense 10 arms × 113 tasks) after the June matrix was shown to be quality-stale live (e.g. luna_max 0.946 published → 0.687 live, effort ladders inverted) — a router voting over stale cells never escalates. Roster 41 → 10 live arms; per-arm med_cost and fallback (opus5_high, live f2p 0.951) from live cells; lam re-selected (0.005) on inner train-side splits under the same protocol; tuned encoder, bank embeddings, T, and sim_floor unchanged. Paired server fix: client system prompts are now excluded from the routing embedding (constant preamble bytes homogenized every request toward one arm).

The default routing artifact for coding-router: optimizes coding requests between big and small models — more usage at the same cost. Given a coding task (or a whole agent conversation), it picks the model+effort arm with the best measured cost/quality trade-off. Routing runs fully locally: the encoder below runs in-process.

This release replaces the kNN-over-base-embeddings artifact with a trained router (EXP-012 winning recipe reward_lcb_b0.2), shipped in two encoder formats plus one head/calibration payload.

Files

path	what
router.json	arm list + per-arm request kwargs, decision-rule params (T, lam, sim_floor), provenance (kind: "trained")
router.npz	emb — 110-task DeepSWE evidence bank in the tuned space; graded — 10×110 LIVE outcome matrix (EXP-015; the calibration side of the vote); med_cost — per-arm median live $/task
encoder-fp16/	merged (LoRA→base) Qwen3-Embedding-0.6B, fp16 safetensors — loads via sentence-transformers / transformers on CUDA or CPU
encoder-mlx-4bit/	the same merged encoder, 4-bit MLX (group size 64) for Apple Silicon — loads via mlx_embeddings

Both encoder dirs are the SAME model in two precisions; the bank and every query must be embedded by one of them (backend agreement measured below). The server picks MLX on Apple Silicon and torch elsewhere, and downloads only the directory it needs.

What the router is

Qwen3-Embedding-0.6B with LoRA r=16 on the attention projections, trained by exact expected reward −E_π[graded − λ·cost] over the LiveCodeBench 7×76 outcome matrix with a KL(π‖π_init) anchor (β=0.2), plus a trained soft-vote temperature T. At inference: embed the task → softmax(sims/T) vote over the DeepSWE bank → per-arm P(solve) → argmax of u = P(solve) − lam·med_cost. If no bank task is within sim_floor cosine similarity the router abstains: it escalates to the strongest arm rather than trust the vote off-distribution.

Final-mint policy: one training run with the sweep's exact code and seed; (checkpoint, lam) selected only on the six standard seeds' inner DeepSWE-train splits (the sweep's own selection discipline); the shipped bank is the full 110-task evidence. The artifact was never selected or early-stopped on the holdout numbers reported below.

Honest numbers

This shipped artifact, LIVE cells (EXP-015), 6-seed repo-split holdout: routed graded 0.9276 at $98.47/split vs always-opus5_high (the strongest live arm) 0.9509/$139.69 — graded delta −0.023, repo-clustered 95% CI [−0.066, +0.025] (contains zero: statistical parity at n=110), cost ratio 1.42× [1.25, 1.65]. lam was selected on inner train-side splits only; holdouts were scored afterwards.

The encoder recipe's original certification (EXP-012, June-matrix cells, 41 arms) for context — these numbers describe the RECIPE on the old cells, not this artifact:

policy	graded	cost/split	read
always-best-train arm (deployable baseline)	0.9336	$126.28	what you could actually deploy without hindsight
recipe, per-seed selection (sweep)	0.9484	$60.17	+0.015 graded, 2.1× cheaper
single-(step,λ) mint, same splits	0.9353	$50.21	per-seed selection's optimism removed
hindsight-best static arm (opus-5@high)	0.9635	$135.73	parity, NOT better

Quality-BEATING routing on this data is dead (held-out oracle analysis, EXP-018): the honest framing is parity-quality at a cost saving vs the strongest deployable policy. Live cells make the cost claim current; models keep drifting monthly, so the bank is expected to be refreshed from live re-benchmarks (the EXP-014/015 harness) rather than treated as frozen.

Difficulty ladder (through the real serve path, MLX)

7 rungs trivial → distributed-systems-hard, each run with and without a 12,000-char opencode-scale system preamble: decisions identical 7/7 (system messages are excluded from the routing embedding), 4 distinct arms across the ladder (luna_high on easy/medium, sol_xhigh/terra_high on hard in-bank tasks, opus5_high), trivial one-liners abstain to opus5_high by design (logged as off_distribution). Table in provenance/ladder.json.

Backend agreement (fp16 vs MLX 4-bit)

Over 20 probe texts (DeepSWE issues, LCB tasks, short interactive prompts) against the live-cell bank: max per-arm |ΔP(solve)| = 0.0340; decisions and abstentions agreed on 18/20 probes (provenance/agreement_live.json). The two disagreements are utility-ranking flips near the decision boundary — adjacent tiers whose utility margin is smaller than the 4-bit quantization noise (luna_medium↔luna_high at |ΔP|=0.0005, and luna_high↔opus5_high on one probe): with only 10 live arms the utility gaps are tighter than under the old 41-arm bank (which agreed 20/20). Treat backend choice as part of the artifact's identity; both flips escalate to an equal-or-stronger arm on MLX, never a weaker one. Cross-backend embedding cosine runs 0.91–0.97.

Scope warning

Validated on repo-issue-shaped inputs (p50 ≈ 2,000 chars, long-horizon SWE tasks). Short one-line prompts sit below sim_floor and abstain to the strongest arm — safe, but no saving. Always check Decision.off_distribution.

Rollback

The previous kNN artifact keeps loading with the same product code (kind defaults to "knn"); pin the prior revision of this repo to roll back.