Pushing the Right Buttons: Adversarial Evaluation of Quality Estimation (2021)

Abstract: Current Machine Translation (MT) systems achieve very good results on a growing variety of language pairs and datasets. However, they are known to produce fluent translation outputs that can contain important meaning errors, thus undermining their reliability in practice. Quality Estimation (QE) is the task of automatically assessing the performance of MT systems at test time. Thus, in order to be useful, QE systems should be able to detect such errors. However, this ability is yet to be tested in the current evaluation practices, where QE systems are assessed only in terms of their correlation with human judgements. In this work, we bridge this gap by proposing a general methodology for adversarial testing of QE for MT. First, we show that despite a high correlation with human judgements achieved by the recent SOTA, certain types of meaning errors are still problematic for QE to detect. Second, we show that on average, the ability of a given model to discriminate between meaning-preserving and meaning-altering perturbations is predictive of its overall performance, thus potentially allowing for comparing QE systems without relying on manual quality annotation.

## introduction quality estimation (qe) is the task of predicting the quality of machine translation (mt) output in the absence of human reference translation. recent qe models based on multilingual pre-trained representations @xcite have shown impressive results achieving up to 0.9 pearson correlation with human judgements of translation quality at sentence level @xcite . not unlike other nlp systems, qe systems are typically tested on held-out datasets. on the one hand, such evaluation usually requires collecting additional human judgements and thus cannot be easily extrapolated to a different usage scenario, for example, a new language pair. on the other hand, evaluation on a given test set can hide performance issues related to the phenomena that are underrepresented in the data but are critical to the reliable performance of the system. finally, a single statistic capturing overall performance does not provide any insights on the strengths and weaknesses of a given approach. as a way to overcome these limitations, we explore adversarial evaluation for qe. specifically, we introduce two types of changes to high-quality mt outputs: meaning-preserving perturbations (mpps) and meaning-altering perturbations (maps). intuitively, we expect a strong qe system to assign lower scores to the sentences containing maps compared to the sentences with mpps. based on this intuition, we devise experiments to systematically test a set of five different qe systems by comparing the scores they produce for sentences containing mpps and maps. we use the difference in the predicted scores as a way of detecting specific problems as well as for assessing the overall performance of the systems. our main findings foot_0 can be summarised as follows: in the remainder of this paper, we first discuss related work on probing for nlp (section 2). we then describe the dataset (section 3) and qe models used in our experiments (section 4). we introduce our probing setup and strategies in section 5 and present and discuss the results in section 6. ## related work very few studies have analysed the performance of qe models beyond correlation with human judgements on held-out datasets. to the best of our knowledge, the only work that analyses the behaviour of qe models is @xcite . on various datasets popularly used for training qe models, they show that they contain certain biases, such as a skew towards high-quality mt outputs and lexical artefacts that are picked up by the sota architectures, e.g., sentences with certain tokens tend to have high or low scores. they also show that qe models can perform very well on these datasets by encoding only the source or target sentences. by contrast, we study the behaviour of the models under specific linguistic conditions. our experiments show that the models are not sensitive to certain meaning errors, which is in line with @xcite 's assumption that sota qe models do not capture adequacy. for mt, various studies have shown that models can achieve high performance on clean data, they are very brittle to noisy inputs, where both synthetic (e.g. character flips) or natural (social media data) noise is used to probe models @xcite @xcite . for other nlp tasks, blackbox methods for adversarial evaluation have been proposed that apply meaning-preserving perturbations in order to test whether the models are sensitive to changes in the input @xcite . different from this line of work, we probe the robustness of qe models to spurious changes but also sensitivity to relevant changes, such as meaning errors. ribeiro et al. (2020) recently devised a general methodology for behavioural testing of nlp models. they generate a subset of simple examples meant to test general linguistic capabilities expected from an nlp system. however, the linguistic capabilities tested within this framework are not directly applicable to the qe task. they could not, for example, capture the ability of a qe system to detect omission errors or copy errors in translation. ## dataset the dataset used in this paper is a subset from the wmt 2020 quality estimation shared task 1, sentence-level prediction @xcite . this data consists of seven language pairs which can be classified as high-resource [english-german (en-de), english-chinese (en-zh)], medium-resource [russian-english (ru-en), romanian-english (ro-en), estonian-english (et-en)], and low-resource [sinhala-english (si-en), nepalese-english (ne-en)] pairs. except for ru-en, sentences are extracted solely from wikipedia. the ru-en data also contains additional sentences from reddit @xcite . the data was collected by machine translating sentences sampled from source-language articles using sota nmt models built using the fairseq toolkit @xcite . the data was annotated with a variant of direct assessment (da) scores @xcite by professional translators. each translation was rated with a score in 1-100, according to the perceived translation quality by at least three translators @xcite . the goal of qe systems built on this data is to predict a z-score normalised mean da for each test source-target pairs, which we further standardise between 0 and 1. in the original dataset, 9k sentences per language pair were randomly split in training (7k), validation (1k) and test (1k). in this study, we focus on probing the models by modifying the target side (translations) with various perturbations. to keep the experiments consistent across the language pairs, we only consider the five pairs with english as the target language. we use the standard training partition of the data to train our qe models. to evaluate our probes, the assumption made is that sentences with perturbations should lead to lower predicted qe scores than original sentences. however, this assumption only holds if we can ensure that the original sentences have high enough quality since perturbing very low-quality sentences with already very low scores would not necessarily lead to further degradations. therefore, we create a subset of the validation + test sets by applying the threshold of 0.7 on the standardised human (da) scores to reflect high quality, based on the definition of the da scores used as guidelines for annotators in this dataset. ## qe models we choose three categories of heavy-to lightweight models for sentence-level qe models: first, the sota transquest with three variants mono-transquest, siamesetransquest and multilingual-transquest @xcite ; second, the lstm-based predictor-estimator approach @xcite and third, the unsupervised method sentsim @xcite . ## probing strategies in this section, we introduce the rationale for two types of probes: meaning-preserving and meaningaltering perturbations. we then describe each perturbation and discuss the experimental setup for the probing. we define a meaning-preserving perturbation (mpp) as a small change in the target-side translation that might affect the translation but should not affect the overall meaning of the sentence. for example, removing punctuation marks from the translated sentence should not affect the meaning conveyed by the text. by contrast, meaning-altering perturbations (map) should alter the meaning conveyed by the translation, for example, replacing a random word with its antonym or randomly replacing a content word. by introducing maps, we focus on probing models for whether they capture (lack of) adequacy in translations. given that sota qe models are based on pre-trained representations obtained from strong language models, it has been hypothesised that they could be biased by the fluency of translations @xcite . we, therefore, design two types of perturbations: mpps, which might affect fluency but not adequacy, and maps, which affect adequacy. perturbations are only introduced in the translations to mimic translation errors. we have chosen perturbations that can be introduced using automated methods, ## results and discussion in this section, we discuss the results obtained from our probing experiments using various qe models. ## conclusions in this work, we have proposed a methodology for analysing the performance of qe systems beyond correlation with human judgements. we have devised a set of perturbations to probe both the robustness of qe models towards changes in the input that do not affect sentence meaning and their sensitivity to meaning errors in translation. first, by applying the proposed methodology to a set of qe systems of varying accuracy, we are able to detect specific failures that cannot be detected by computing correlations between predicted scores and human judgements. second, we have shown that, on an average, the ability of a given model to discriminate between the two types of perturbations is predictive of its overall performance, thus allowing us to compare qe systems without relying on manual quality annotation. our choice of specific perturbations was motivated by the errors that occur in neural mt and the potential weaknesses of qe models. in the future, we plan to extend this set by including perturbations that capture other critical mt errors. furthermore, we plan to study whether the proposed perturbations can be used at training time to improve the ability of qe systems to detect critical errors in translation. figure 4 : difference between the predicted qe scores for original sentences and each perturbation for all language pairs (y-axis->mt -x, where x is the perturbation as labelled on the x-axis), using unsupervised sentsim method. figure 5 : difference between the predicted qe scores for original sentences and each perturbation for all language pairs (y-axis->mt -x, where x is the perturbation as labelled on the x-axis), using multilingualtransquest architecture. figure 6 : difference between the predicted qe scores for original sentences and each perturbation for all language pairs (y-axis->mt -x, where x is the perturbation on the x-axis), using the siamesetransquest architecture.