Perturbation-based QE : An Explainable, Unsupervised Word-level Quality Estimation Method for Blackbox Machine Translation (2023)

Abstract: Quality Estimation (QE) is the task of predicting the quality of Machine Translation (MT) system output, without using any gold-standard translation references. State-of-the-art QE models are supervised: they require human-labeled quality of some MT system output on some datasets for training, making them domain-dependent and MT-system-dependent. There has been research on unsupervised QE, which requires glass-box access to the MT systems, or parallel MT data to generate synthetic errors for training QE models. In this paper, we present Perturbation-based QE - a word-level Quality Estimation approach that works simply by analyzing MT system output on perturbed input source sentences. Our approach is unsupervised, explainable, and can evaluate any type of blackbox MT systems, including the currently prominent large language models (LLMs) with opaque internal processes. For language directions with no labeled QE data, our approach has similar or better performance than the zero-shot supervised approach on the WMT21 shared task. Our approach is better at detecting gender bias and word-sense-disambiguation errors in translation than supervised QE, indicating its robustness to out-of-domain usage. The performance gap is larger when detecting errors on a nontraditional translation-prompting LLM, indicating that our approach is more generalizable to different MT systems. We give examples demonstrating our approach’s explainability power, where it shows which input source words have influence on a certain MT output word.

## introduction machine translation (mt), with the aim of translating text from a source language to a target language, has been increasingly adopted in different real-world scenarios, ranging from translations in healthcare areas to translations in the legal domains @xcite . in many of these applications, errors in translation output could cause serious harm to the users, e.g., translation errors leading to wrong medical diagnoses in healthcare or wrong judgment in court. therefore, it is important to let the users know how much they can trust a translation, by providing them with some quality assessment of the mt output. this is not always straightforward due to the lack of gold-standard human translations, or the mismatch between evaluation data and real-world usage. as a result, researchers have been looking into quality estimation. quality estimation (qe) is the task of predicting the quality of mt system output without access to reference translations. state-of-the-art qe systems are built in a supervised manner, where they require human-labeled quality assessment on mt output for training @xcite . this approach has 2 drawbacks: the labeled qe data is costly to obtain, and the trained qe models would only know about the types of error that are presented in the training data. supervised qe models are likely to underperform in unfamiliar settings @xcite , e.g., when evaluating the output of a new mt system on a new dataset from a different domain. consequently, there has been research into unsupervised qe, where the human-labeled assessment data is no longer required @xcite . these works either require glass-box information of the mt system (e.g., output log probabilities or attention scores), or a large amount of parallel mt data to create synthetic qe data for training. this is problematic for language pairs with low-resourced mt data, or when the mt system is kept blackbox, which is the current trend of some widely-discussed api-only large language models. in this paper, we propose an unsupervised word-level qe approach to evaluate blackbox mt systems, termed perturbation-based qe. our motivation is inspired by a known problem: when uncertain, mt systems rely on spurious correlations learnt from the training data to generate translation @xcite . we assume that, when outputting a translation token, if the mt system relies on too many parts of the source sentence, it is likely that the system is exploiting irrelevant correlations, thus the output token is unreliable. consider the @xmath0: "my friend has a ph.d. degree, and now she is a professor." → "meine freundin hat einen doktortitel, und sie ist jetzt eine professorin.". the translation word "freundin" should only depends on "friend" and "she", where "friend" indicates the meaning and "she" indicates the gender form. the output word being influenced by more source words would indicate that the mt system is focusing on the wrong part of the input sentence. broadly speaking, in perturbation-based qe, we perturb words in the source sentences one by one to find out which source words influence a single output word. if an output word is influenced by too many source words, then it is predicted as a bad translation. due to its simplicity, our approach does not require human-labeled qe data, nor parallel mt data, nor glass-box access to the evaluated mt system. additionally, our qe approach comes with explainability power: it shows which source words affect each output word in the translation, thus can be used as an indication of the wrong correlations that is inherent in the mt system. to summarize, our contributions are as follows: ## related work quality estimation (qe) aims to predict the quality of machine translation (mt) output, either at the sentence level or word level. for word-level qe, the goal is to predict whether each word in the translation is correct. state-of-the-art word-level qe methods are supervised @xcite , i.e., requiring labeled data for training, which is costly to obtain. additionally, supervised qe is likely to be domain-dependent and mt-system-dependent, as they do not aware of errors not occurring in the training data @xcite . unsupervised qe overcomes the need for labeled data. several works perform unsupervised qe by using glass-box features from the mt systems @xcite @xcite @xcite . as an example, @xcite proposed unsupervised qe using the output probability distribution and the attention mechanism from encoder-decoder mt models. therefore, their methods are model-specific. @xcite excludes the need for human-labeled data and mt glass-box access by creating synthetic data to train qe models. the synthetic data is generated by aligning candidate mt translations to the target references to find errors, or rewriting target reference sentences using a masked language model to introduce errors. these methods require a large and diverse amount of parallel mt data (i.e., source sentences and the goldstandard translations), which is not always available for different domains and language pairs. additionally, these methods are also likely to be domain-dependent and mt-system-dependent, as the qe model is trained on the output of pre-selected mt systems on pre-selected mt data. researchers also focus on quality estimation from the explainability perspective. @xcite propose using integrated gradients from mt models (i.e., glass-box information) to quantify how important each source word is to the output translated words. the method is then used for qe by detecting under-translated source words that have low importance to the output translation. @xcite also quantify the contribution of each source word on the output translation using glass-box information from transformer-based mt models, which is the layer-wise tokens attributions. here the source words' contribution can also be used to detect under-translated source words, or to assess the quality of the whole translation. another line of research is on explainable sentence-level qe, where the word-level error scores are provided as the explanation for the predicted sentence-level score @xcite . explanations can be extracted by using methods such as lime @xcite or shap @xcite on top of sentence-level qe models, or building interpretable models that output both sentence-level quality and word-level explanations @xcite . in contrast to the previous works on quality estimation, perturbation-based qe does not require labeled qe data, parallel mt data, nor glass-box access to the evaluated mt system. from the explainability perspective, our approach provides a new type of explanation for targetside word-level qe, i.e., the information on which source words affect each translated word. ## perturbation-based quality estimation in this section, we describe perturbation-based qe. recall our motivation: if the mt system relies on too many tokens in the source sentence to output a translation token, it is likely that the system is exploiting irrelevant correlations, thus the translation token is unreliable. perturbation generation (step i figure 1 ): we first perform perturbation to the source sentence. the subset of source words to perturb, which is a hyperparameter choice, is one of the following: (1) content words, i.e., noun, verb, adjective, adverb, pronoun, determined by nltk part-of-speech tagging @xcite ; (2) all words including functional words such as "a", "an", "the"; or (3) all tokens including non-word tokens such as punctuation marks. for each perturbed source word s i , we mask it out from the source sentence and use a language model to generate n best replacements. the language masking model can be bert @xcite , roberta @xcite or distilbert @xcite (choice of the language masking model is a hyperparameter). john met his wife in the hot spring of 1988. ## conclusion we proposed an unsupervised word-level quality estimation method, termed perturbationbased qe. our method does not rely on labeled qe data nor parallel mt data, masking it more domain-independent and system-independent to find mt errors that cannot be foreseen. this advantage is supported by our experiment on finding gender bias and word-sensedisambiguation erroneous translation from a nontraditional translation-prompting llm. our approach is not sensitive to hyperparameter settings, thus less dependent on labeled data for hyperparameter tuning. our approach is also explainable: it shows which source words affect an output translated word. additionally, our approach is mt-system-agnostic and works for blackbox systems. overall, perturbation-based qe, as an unsupervised method, still falls behind supervised qe on in-domain and known-mt-system settings, but outperforms supervised qe on zero-shot settings and on out-domain and unseen-mt-system settings. as future work, it can be extended to assess the quality of other tasks, such as question answering or summarization, in the same manner: minimally perturb the input and analyze changes in the output.