eeahugs/query-intent-setfit-v1

EEA Query Intent — setfit-v1

Multilingual query-intent classifier for the European Environment Agency website search. Decides whether a search query (28 languages) is AI-eligible (a natural-language question, an exploratory request, or a factual claim) or no-AI (keyword/document retrieval, or anything uncertain). Only eligible queries may trigger an AI-summary LLM call; everything else — including low confidence, errors, and timeouts — fails closed (no AI).

• Architecture: SetFit linear head (384 -> 5 labels) on intfloat/multilingual-e5-small (MIT, 112M params)
• Labels: question, exploratory, claim, retrieval, unknown
• Routing: binary — AI-eligible when the sum of the three eligible-label probabilities is >= 0.95 (see manifest.json); otherwise abstain
• Input: the serving pipeline casefolds queries; training text is casefolded

Usage

The production deployment is a FastAPI service (eea-query-intent) that loads this repo and exposes POST /v1/classify / GET /health. To load the model directly:

from setfit import SetFitModel
model = SetFitModel.from_pretrained("eeahugs/query-intent-setfit-v1")
# probabilities over: question, exploratory, claim, retrieval, unknown
# AI-eligible when P(question)+P(exploratory)+P(claim) >= 0.95

manifest.json (a serving-contract file, not a standard SetFit file) carries model_version, labels, eligible_labels, and abstain_threshold.

Results (frozen exam v2, 23,718 rows, threshold 0.95)

metric	value
average eligible recall	0.859
worst-language no-AI false positive	0.005 (Maltese) — inside the 1% gate
average abstention	0.546

Known accepted scope: out-of-domain general-knowledge queries and some short English fact-lookups abstain (safe failure: no summary, never a wrong one). Maltese/Irish/Icelandic are the weakest languages (accepted gap).

Training data provenance (publication disclosure)

stratum	source	license note
~50,700 rows	in-house Gemma 31B native generation, 17 languages	internal, AI-generated
~16,800 rows	GPT-5.6-luna native generation, 6 languages (cs el et hu lt lv)	OpenAI terms govern redistribution of the outputs
~5,800 rows	NLLB-200-1.3B machine translation, sl + sv	NLLB-200 is CC-BY-NC-4.0 (non-commercial)
~4,500 rows	legacy v3 English-anchor translations, 11 languages	NLLB/opus-mt (Maltese model is CC-BY-SA-4.0)
~4,700 rows	hand-authored English short bank, NLLB/opus-translated	see above

81,404 training rows total, 2 epochs, seed 3 of 3 (selected on a held-out calibration set). This model is published under the MIT license; the upstream data-source licenses in the table above (NLLB-200 CC-BY-NC-4.0, opus-mt CC-BY-SA-4.0) still apply if you reuse or redistribute the underlying training data.

Reproducibility

The training recipe is in configs/setfit-v1.yaml of the eea-query-intent repository (backbone revision 614241f, 2 epochs, batch 64, head LR 1e-2, backbone LR 1e-5, seed 3). The exact training data is published as eeahugs/query-intent-setfit-v1-train (CC-BY-NC-4.0). The frozen training mix, calibration set, and exams are SHA-256-pinned in that repository's data/MANIFEST.json.