A sk QE : Question Answering as Automatic Evaluation for Machine Translation (2025)

Abstract: How can a monolingual English speaker determine whether an automatic translation in French is good enough to be shared? Existing MT error detection and quality estimation (QE) techniques do not address this practical scenario. We introduce AskQE, a question generation and answering framework designed to detect critical MT errors and provide actionable feedback, helping users decide whether to accept or reject MT outputs even without the knowledge of the target language. Using ContraTICO, a dataset of contrastive synthetic MT errors in the COVID-19 domain, we explore design choices for AskQE and develop an optimized version relying on LLaMA-3 70B and entailed facts to guide question generation. We evaluate the resulting system on the BioMQM dataset of naturally occurring MT errors, where AskQE has higher Kendall’s Tau correlation and decision accuracy with human ratings compared to other QE metrics.

## introduction how can a monolingual english speaker determine whether an automatic translation of covid-19 protocol in french is good enough to be shared? in such high-stakes settings, inaccurate translations can lead to confusion, conversation breakdowns @xcite , and even life-threatening risks @xcite @xcite . for instance, even rare inaccurate translations in clinical instructions can pose significant risks to patient health @xcite . however, accurately assessing machine translation (mt) quality is significantly more challenging for monolingual speakers than for bilinguals, as they rely on translations in a language they do not understand @xcite . to bridge this gap, we need quality estimation (qe) feedback to help users assess mt quality @xcite . ## askqe we present an overview of askqe in figure 2 . we first generate a list of questions conditioned on the source ( §3.1), generate answers for each question based on the source or the backtranslated mt output ( §3.2), and compute the answer overlap ( §3.3). all prompts are in appendix a. ## contratico: controlled error generation by perturbation to simulate high-stakes settings, we use tico-19 @xcite , a mt dataset containing covid-19-related content translated from english into 36 languages. 6 since the original tico-19 dataset only provides english source and reference translations in the target language, we construct a dataset with eight synthetic perturbations across five language pairs, contratico, to assess the impact of askqe design in a controlled setting. formally, given a reference translation y ref , we prompt gpt-4o to perturb y ref with specific error, which results in a perturbed translation y tgt . we define eight linguistic perturbations, categorized by their level of severity into either minor or critical, based on the potential implications of the translation error in practice @xcite . typological errors, for instance, have minimal influence on mt quality @xcite , whereas errors that alter the semantics of the translation, such as those introducing misleading information, have a significantly greater impact @xcite . all eight perturbations are applied to each reference translation. we show detailed examples for each perturbation in appendix critical. these errors significantly changes the original meaning and usually appear in a highly visible or important part of the content. we define three types of critical perturbations: to validate the impact of different perturbations, we show that xcomet-qe @xcite and cosine similarity scores between the original source x src and the perturbed translation y tgt decrease as severity increases in appendix c.1. we first optimize the askqe metric by evaluating five llms across three qg variants and selecting llama-3 70b @xcite with nli as the best-performing configuration ( §6.1). next, we validate this setup by demonstrating that metric scores decrease for more severe perturbations ( §6.2) and exhibits strong correlations with established qe metrics ( §6.3). ## results on biomqm our initial experiments on the contratico dataset were conducted in a controlled setting. we extend this analysis with the biomqm dataset with more naturally occurring translation errors and additional language pairs. 7.1 askqe generalizes to biomqm error detection. since biomqm dataset contains multiple errors (1.95 on average) per segment, we assign the error severity of an mt output based on its highest level error (e.g., if the mt contains both critical and minor errors, it is categorized as critical). as shown in ## conclusion in this work, we introduce askqe, a question generation and answering framework designed to identify critical translation errors and provide actionable feedback for users relying on mt outputs in a language they do not understand. using the con-tratico dataset ( §4) across five language pairs, we validate the effectiveness of our method for critical error detection ( §6.2) and correlation with established qe metrics ( §6.3). analysis of generated questions shows that most focus on extent or definition while remaining faithful to the source context ( §6.4). we further show that askqe generalizes well to realistic scenarios with naturally occurring translation errors in the biomqm dataset, achieving stronger correlations with human judgments ( §7.1) and improving mt acceptance decisions compared to other qe metrics ( §7.2). these findings highlight the promise of qg/qa framework for mt quality assessment and qe feedback, calling for future work to explore optimal strategies such as enhancing context integration during qg and expanding to multi-turn qa.