HydraQE

HydraQE is an end-to-end, reference-free speech translation quality estimator created for the IWSLT 2026 Speech Translation Metrics shared task. It takes English source speech and a candidate German or Chinese translation and returns three quality estimates on a nominal 0–100 scale:

• da: trained toward direct human assessment scores.
• metricx_ref: trained toward MetricX reference-based pseudo-labels.
• xcomet_ref: trained toward XCOMET reference-based pseudo-labels.

Higher scores indicate better estimated translation quality. Outputs are not clipped and can fall outside 0–100.

The model was trained on out-of-English translation directions; the speech encoder supports other languages but has not been evaluated on them.

Usage

Inference code is provided in the GitHub repository:

git clone https://github.com/OSU-slatelab/HydraQE.git
cd HydraQE
python -m pip install -r requirements.txt
python -m pip install --no-deps .

from hydraqe import HydraQE

qe = HydraQE.from_pretrained("osu-slatelab/HydraQE")
scores = qe.score(
    "source.wav",
    "Die zu bewertende Systemübersetzung.",
)
print(scores)
# {'da': ..., 'metricx_ref': ..., 'xcomet_ref': ...}

Scoring inputs from a JSONL file is supported with a CLI utility. Select one head or average all heads with --head:

hydraqe \
  --input examples.jsonl \
  --output predictions.jsonl \
  --head mean

Each output row contains the resulting scalar in score.

The loader downloads the pinned Qwen3-ASR base model, loads this repository's LoRA and QE-specific weights, and merges LoRA by default for inference.

Development-set results

This checkpoint was selected using the IWSLT 2026 dev set.

Head	Level	en-de	en-zh	Average
DA	Segment	32.5%	39.9%	36.2%
DA	System	99.7%	69.2%	84.5%
MetricX-ref	Segment	31.5%	38.8%	35.2%
MetricX-ref	System	98.9%	83.5%	91.2%
XCOMET-ref	Segment	32.7%	36.2%	34.5%
XCOMET-ref	System	97.2%	36.0%	66.6%

Citation

HydraQE is described in the IWSLT 2026 system paper:

@inproceedings{krahn-fosler-lussier-2026-hydraqe,
    title = {{H}ydra{QE}: {OSU}{'}s Submission for the {IWSLT} 2026 Speech Translation Metrics Shared Task},
    author = {Krahn, Kevin and Fosler-Lussier, Eric},
    booktitle = {Proceedings of the 23rd International Conference on Spoken Language Translation ({IWSLT} 2026)},
    year = {2026},
    month = jul,
    address = {San Diego, USA (in-person and online)},
    publisher = {Association for Computational Linguistics},
    pages = {323--331},
    url = {https://aclanthology.org/2026.iwslt-1.37/},
    doi = {10.18653/v1/2026.iwslt-1.37}
}

License

Apache-2.0. The base model is also distributed under Apache-2.0.