Translation Error Detection as Rationale Extraction (2022)

Abstract: Recent Quality Estimation (QE) models based on multilingual pre-trained representations have achieved very competitive results in predicting the overall quality of translated sentences. However, detecting specifically which translated words are incorrect is a more challenging task, especially when dealing with limited amounts of training data. We hypothesize that, not unlike humans, successful QE models rely on translation errors to predict overall sentence quality. By exploring a set of feature attribution methods that assign relevance scores to the inputs to explain model predictions, we study the behaviour of state-of-the-art sentence-level QE models and show that explanations (i.e. rationales) extracted from these models can indeed be used to detect translation errors. We therefore (i) introduce a novel semi-supervised method for word-level QE; and (ii) propose to use the QE task as a new benchmark for evaluating the plausibility of feature attribution, i.e. how interpretable model explanations are to humans.

## introduction quality estimation (qe) is the task of predicting machine translation (mt) quality at inference time, when no gold standard human translation is available @xcite . qe can be framed as a word-level or a sentencelevel task. both tasks have numerous practical applications, such as deciding whether a given mt output can be published without editing, highlighting potential critical errors. current qe approaches fine-tune powerful representations from pre-trained multilingual encoders such as bert @xcite or xlm-r @xcite . in the recent shared task on qe at wmt2020 @xcite these approaches have achieved very high performance at predicting sentence-level translation quality (up to 0.9 pearson correlation with human judgements for some language pairs). how-ever, as evidenced by these results, the accuracy of word-level prediction still leaves room for improvement. this is partly due to the limited amount of training data. word-level error annotation is especially time-consuming and expensive, as it requires work from bilingual experts. in this work we introduce a new semi-supervised approach to word-level qe that removes the need of training data at word level. to achieve this, we propose addressing qe as a rationale extraction task @xcite . explainability is a broad area aimed at explaining predictions of machine learning models @xcite . rationale extraction methods achieve this by selecting a portion of the input that justifies model output for a given data point. in translation, human perception of quality is guided by the presence of translation errors @xcite . we hypothesize that sentence-level qe models also rely on translation errors to make predictions. if that is the case, explanations for sentence-level predictions can be used to detect translation errors, thus removing the need for word-level labeled training data. to extract model explanations, we use post hoc rationale extraction methods @xcite which try to explain the predictions of a given model (as opposed to modifying its architecture or introducing constraints during training), since one of our goals is to study to what extent existing qe models rely on the same information as humans to make predictions. at the same time, by using word-level errors as explanations for sentence-level qe scores, we introduce a new benchmark for evaluating explainability methods. recent work has introduced various datasets for measuring the agreement between rationales extracted from nlp models and those provided by humans @xcite . qe is different from these datasets in various important aspects. first, it is a regression task, as opposed to binary or multiclass text classification mainly explored in previous work. second, it is a multi-lingual task where the output score captures the relationship between source and target sentences. finally, manual annotation of translation errors is a practical task with a long tradition in mt research and translation studies @xcite , and thus offers an interesting alternative to human explanations collected specifically for evaluating rationale extraction methods. our main contributions are: word-level qe is typically addressed as a binary classification task, where the qe model needs to predict a binary label indicating whether a word is correct or wrong for each word in the mt output @xcite . as illustrated in figure 1 (left), some supervised approaches use both sentence-level and word-level objectives in a multi-task setting, which results in superior performance @xcite . methods that do not require word-level training data either need access to the mt model @xcite , or still treat the problem as a supervised task but use synthetically generated data for supervision @xcite . rationale extraction for nlp sota nlp models based on deep neural networks achieve high performance in a variety of tasks, often at the cost of interpretability @xcite . recent work aims to address this issue by focusing on two different goals. on the one hand, the aim is to produce justifications for model predictions that are plausible to the users, in order to increase users' trust @xcite . on the other hand, the aim is to reveal the inner workings of the model and faithfully explain its predictions, so the explanation can be useful to model developers @xcite . typically, explainability methods operate by selecting a portion of the input that justifies model prediction for a single data point. this can be done either by modifying the model architecture, or by trying to explain the predictions of a given model. the first type of approaches (a.k.a. rationalization by construction) involves imposing restrictions on the generated rationales to satisfy certain constraints, e.g. compactness @xcite . note that such restrictions often result in lower performance and indeed are not guaranteed to explain the behaviour of an unconstrained model @xcite . the second type of approaches (the so called post hoc) usually rely on feature attribution methods, which assign an importance value to each input feature of a network @xcite . these methods do not allow for introducing useful biases during training, but focus on faithfully explaining model behaviour. feature attribution has a long tradition in image recognition tasks @xcite and only recently have been applied to some nlp tasks, most commonly text classification @xcite . qe is fundamentally different from text classification where clues are typically separate words or phrases @xcite which often can be considered independently of the rest of the text. this independence assumption does not hold for the task of evaluating translation quality where a word cannot be identified as a clue (e.g. translation error) without considering the surrounding context. sentence model train data sent test instance word x t x s word model sentence model train data sent test instance x t x s attributions layers furthermore, sota nlp models based on contextualized representations for input words make rationale extraction especially challenging, as the representation for a given word can encode not only the word identity but also its interactions with other words in the text. recent work has revealed various interesting properties that characterize the information flow through hidden layers in deep transformer models @xcite @xcite . we provide additional insights on this topic in section 5.2. ## translation error prediction as rationale extraction we propose framing semi-supervised word-level qe as rationale extraction from sentence-level qe models. instead of training a dedicated supervised model for word-level prediction, we propose deriving word-level scores from a strong sentence-level qe model by extracting explanations for model predictions (see figure 1 (right)). given a trained sentence-level qe model and the test data, rationale extraction methods detect the parts of the input that are relevant for model predictions on a sampleby-sample basis. we hypothesize that words with the highest relevance scores should correspond to actual translation errors on word-level. ## conclusion in this work, we propose a new semi-supervised approach for word-level qe by exploring feature attribution methods. we show that for well performing models our results approach performance of supervised methods. we also consider the qe as rationale extraction task as a new benchmark for plausibility-based evaluation of explainability methods. we hope this work will encourage further research on improving the efficiency of word-level qe models with lightly supervised methods. this work opens many directions for future research: from improving the achieved results by tuning linear weights to combine attributions to hidden states at different layers, to exploring different underlying architectures and sentence-level training objectives.