jonasaise/laya-swedish-preference-gate

Laya Swedish Preference Gate (v4)

A ~10 ms judge for Swedish text. One forward pass of a fine-tuned Laya decision model (mmBERT-base backbone) answers one of five questions, as a "System 1" filter in front of a slow LLM judge.

Question	Input	Output
preference	prompt (or English source) + two responses	calibrated P(response A is better)
naturalness	two responses (prompt optional)	P(A is more natural, idiomatic Swedish)
quality	prompt + one response	1–5 score
natural_swedish	one response	P(natural Swedish vs translationese)
grammatical (new)	one text	P(grammatically correct); compare two texts, see limitations

Earlier releases: revision="v3", revision="v2", revision="v1".

Code, the full ablation and every number below: github.com/joenaess/laya-swedish-preference-gate. Training and evaluation pairs: jonasaise/laya-swedish-gate-v4-data.

What changed from v3

v4 adds three kinds of training data, chosen in a controlled ablation (same recipe, seed and hyperparameters; only the data differs; all tests held out):

• DaLAJ-GED (Superlim 2): learner sentences vs the same sentence corrected, for naturalness and the new grammatical question.
• SweFAQ (Superlim 2): a question's own answer vs other answers from the same FAQ page, for preference.
• Europarl (Språkbanken): professional Swedish translations vs Kimi-k3 machine translations of English speeches, plus adequacy negatives (another paragraph's translation, a half translation, a word-shuffled translation, the untranslated English).

Held-out test (share where the reference wins)	v3	v4
Fluent-vs-fluent probe, preference	79.9%	78.6%
DPO validation, preference	98.9%	98.7%
liu-nlp native vs OPUS-MT, naturalness	44.3%	52.2%
FLORES+ human vs OPUS-MT, preference with source	48.5%	73.8%
WMT24++ human vs OPUS-MT, preference with source	54.3%	74.6%
WMT24++ human vs Kimi-k3, preference with source	43.1%	58.9%
DaLAJ corrected vs learner, naturalness	55.9%	89.7%
GerLangMod-sv original vs verb misplaced, naturalness	83.3%	95.5%
SweFAQ own answer vs other, preference	60.3%	88.9%
ScaLA-sv MCC (EuroEval's metric), single-response	0.146	0.341

Calibration: preference T = 1.590, naturalness T = 0.379, quality T = 1.121, natural_swedish T = 1.405, grammatical T = 1.449. ECE on the DPO validation pairs 0.0142.

A trap we hit on the way. Human-vs-MT pairs alone teach a gate that "the candidate least like the source wins": the intermediate arms preferred a fluent translation of the wrong sentence, half translations and nonsense. The adequacy negatives fix it (FLORES+ devtest, English source in the prompt, share where the reference wins):

The reference against …	v3	without adequacy negatives	v4
the reference of a different sentence	91.0%	0.0%	78.0%
LFM2-350M's translation (pilot's lowest chrF)	97.0%	3.5%	98.0%
only the first half of the reference	72.0%	7.5%	100.0%
the untranslated English	20.5%	100.0%	100.0%
the untranslated English (naturalness, no prompt)	87.5%	38.5%	99.5%

On 4,400 translations by 22 small LLMs, v4's translation score agrees with chrF at Spearman 0.42 per translation (v3: 0.28).

Pilot: a quick proxy for EuroEval Swedish

Scoring 100 Swedish answers per model with quality and averaging, over 22 small open LLMs with Swedish results on the EuroEval leaderboard: Spearman with the Swedish rank 0.74 [0.39, 0.88], 0.62 [0.11, 0.86] after controlling for model size. A pilot on models up to about 4B parameters, not a calibrated predictor.

Quickstart

uv pip install "git+https://github.com/joenaess/laya-swedish-preference-gate"

from laya_gate.predictor import SwedishPreferenceGate

gate = SwedishPreferenceGate("jonasaise/laya-swedish-preference-gate")

r = gate.predict_preference(
    prompt="Hur fungerar fotosyntesen?",
    chosen_candidate="Fotosyntesen är den process där växter omvandlar ljus till energi.",
    rejected_candidate="Det är när solen lyser på träd och gör magi.",
)
# {"preference": "candidate_a", "p_chosen": <float>, "is_valid_margin": <bool>, ...}

# translation quality: put the English source in the prompt
t = gate.predict_preference(
    prompt="Översätt följande text till svenska:
The meeting was postponed.",
    chosen_candidate="Mötet sköts upp.",
    rejected_candidate="Mötet ställdes in.",
)

s = gate.score_quality(prompt="Hur fungerar fotosyntesen?", response="Växter gör mat av solljus.")
g = gate.score_grammar("Människor är snälla där och alla varor är billiga.")
a, b = (gate.score_grammar(x)["p_correct"] for x in ("Hon har bott här.", "Hon har bo här."))

Limitations

• grammatical over-flags as a stand-alone verdict. At 0.5 it marks 40.4% of fluent LLM answers and 59.9% of WMT24++ references as incorrect (it was trained on single learner sentences). Compare two versions of a text instead. On lowercased text without punctuation (GerLangMod-sv) it is at chance (47.8%).
• Wrong-content translations still win sometimes: v4 prefers a fluent translation of a different sentence in 22.0% of FLORES+ pairs (v3: 9.0%).
• Pairwise preference, not fact-checking; 1,024-token window; one seed per training arm.
• Kimi-k3 is not a neutral reference for its own output: on WMT24++ it prefers the human translation over OPUS-MT in 81.0% of pairs but over its own translation in only 24.5%.

Training data and licenses

Weights: Apache-2.0. Base model convaiinnovations/laya: apache-2.0. Training data:

• jonasaise/dpo-swedish-interpreted (mit) and argilla/ultrafeedback-binarized-preferences-cleaned (mit).
• Europarl Swedish-English: Koehn (2005), Europarl: A Parallel Corpus for Statistical Machine Translation, MT Summit X; enriched and distributed by Språkbanken, https://doi.org/10.23695/mdp4-2p72, CC BY 4.0.
• DaLAJ-GED (CC BY 4.0) and SweFAQ (CC BY-SA 4.0) from Superlim 2 (sbx/superlim-2), Språkbanken.
• Machine translations by Helsinki-NLP/opus-mt-en-sv (evaluation only) and Kimi-k3.

Evaluation only: liu-nlp/translationese-opensubtitles (Apache-2.0), FLORES+, WMT24++ (Apache-2.0), ScaLA-sv (mteb/scala_sv_classification), GerLangMod (Apache-2.0).

@software{laya_swedish_preference_gate,
  title  = {Laya Swedish Preference Gate},
  author = {Lind, Jonas},
  year   = {2026},
  url    = {https://huggingface.co/jonasaise/laya-swedish-preference-gate}
}