Explaining Errors in Machine Translation with Absolute Gradient Ensembles (2021)

Abstract: Current research on quality estimation of machine translation focuses on the sentence-level quality of the translations. By using explainability methods, we can use these quality estimations for word-level error identification. In this work, we compare different explainability techniques and investigate gradient-based and perturbation-based methods by measuring their performance and required computational efforts. Throughout our experiments, we observed that using absolute word scores boosts the performance of gradient-based explainers significantly. Further, we combine explainability methods to ensembles to exploit the strengths of individual explainers to get better explanations. We propose the usage of absolute gradient-based methods. These work comparably well to popular perturbation-based ones while being more time-efficient.

## introduction building trustworthy and reliable machine translation (mt) systems has been a broad topic in natural language processing (nlp) for the past decade. advances in research have led to dominant architectures like the bert transformer models @xcite , which got widely adapted by the nlp community for a variety of tasks, including automated mt. pretraining models @xcite @xcite on generic corpora has simplified the deployment of performant, yet easily adaptable models, and therefore became a central part of new translation systems which show great progress in terms of their translation quality while maintaining reasonable computational costs @xcite . traditionally, estimating the quality of a given translation requires human supervisors who are able to identify weaknesses in the translation, 1 code for this paper is available at https: //github.com/sinisterthaumaturge/ metascience-explainable-metrics attribute wrongly translated parts and correct them manually @xcite . reference-free (i.e. without access to a reference translation) quality estimation (qe) tries to solve this costly and time-consuming process by providing models that are able to assign quality-scores automatically @xcite . transquest (tq), presented by @xcite , is a quality estimation framework which won the sentence-level direct assessment shared task in wmt 2020, thus we will utilize tq primarily in this work as our target model to be explained. this year's 'explainable quality estimation' shared task @xcite focuses on the evaluation of current qe by using different explainability methods. as the organizers propose, the identification of translation errors therefore should be seen as an explainability problem. explanations are expected to provide insights into the connection between a given input-output-pair so that humans are able to easily understand the explanation and, if necessary, take action. further, analyzing a set of predictions of a system with explainability methods helps with comparing the system's decision with human reasoning, ultimately assessing trust in the system @xcite . in this work, we will compare five explainability techniques on this year's task dataset (section 3). we experiment with perturbation-based methods as well as gradient-based methods (section 4) and propose a simple, yet effective ensembling technique to improve the overall classification performance. our goal is to provide an overview of each approach's capabilities in terms of classification performance, but also in the context of the computational overhead required for each approach. ## related work explainability is already a key research topic for computer vision (cv), but there are many rising efforts to make nlp models explainable as well @xcite . it can be helpful for different tasks, e.g. automated fact-checking for public health @xcite . frameworks for interpretability were already developed for cv, e.g. innvestigate or in-terpretml @xcite but specific ones for nlp are also on the rise, like allennlp interpret @xcite . essential for ensuring trust in models are evaluation metrics that are more aligned with human perception of 'goodness'. for this reason, there are rising efforts in research for the creation of such metrics for various nlp tasks. existing popular evaluation metrics for text generation tasks like rouge @xcite ) for text summarization and bleu @xcite for mt base their measures on statistical similarity rather than evaluating semantic similarity. several alternative metrics aim to tackle these shortcomings: unsupervised metrics moverscore @xcite and its reference-free extension xmover-score (xms) @xcite ) that generates better aligned cross-lingual embeddings on basis of which translation quality can be measured more similarly to how humans do it. it should be possible to derive valuable information from sentence-level qe scores for word-level translation error identification. until now there are no existing approaches for explaining errors in mt based on qe scores that we know of. however, in consideration of the 'explainable quality estimation' shared task that is part of emnlp 2021, we expect new approaches that address this problem. ## data we conduct our experiments on the multilingual quality estimation and automatic post-editing dataset (mlqe-pe) @xcite . the training and development data for this shared task consists of romanian-english (ro-en) and estonian-english (et-en) language pairs where word-level gold labels and sentence-level gold scores are provided in addition to source sentences and mt outputs. data is shown exemplary in figure 1 . our approach is to estimate word-level scores in an unsupervised fashion to show specific errors in translation. therefore, we use explanations on sentence-level scores to get word-level scores, which are then evaluated with the given gold standard word-level scores in the development set. the training and development data consists of 7000 and 1000 tokenized sentences respectively for both estonian-english and romanian-english. sentence-level scores range in [0, 100] where higher scores indicate better translations. we predict these with the qe model. tokens for the sentence quality score (word-level labels) are rated binary, 1 being relevant and thus responsible for low-quality scores and 0 being correct tokens. test data consists of further annotated et-en and ro-en sentences. zero-shot test sets for german-chinese (de-zh) and russian-german (ru-de) language pairs are also provided in addition, which neither contain word-nor sentence-level annotations. we did not use the training set at all as the models were all pre-trained and evaluated the explanations on the development and test set. ## discussion the dependency on the performance of qe models is the main limitation of our approaches. explanations are generated in a pipeline fashion where potential errors will be propagated into the explanation models. mtq and xms work quite well already, but there is still room for improvement, especially for language pairs that are not that similar. it is likely that we could achieve better explanations with better qes. another limitation of absolute gradient-based methods is their model-awareness where explainers need access to a qe model's training procedure in order to calculate the gradients. in cases where explanations should be generated in a black-box manner, perturbation-based methods like lime or shap are better suited. we showed that perturbation-based methods work generally well for predicting errors in mt based on qe and that existing gradient-based methods perform quite poorly in comparison. our proposed absolute gradient method is a simple extension of those existing methods, but with large performance improvements. however, absolute perturbations seem to worsen the performance of existing perturbation-based methods. explainer ensembles outperformed single explainers in all cases, where maximum ensembles generally worked best. absolute explanations also improved gradient-based ensembles. our experiments justify the popularity of perturbation-based explainers. nonetheless, gradient-based methods should not be overlooked. they are not only faster in comparison, but with the extension of absolute explanation ensembles can also perform better for the given task and are hence worth to consider. ## conclusion we showed that absolute gradient-based methods are worthy contenders to perturbation-based methods when it comes to generating plausible word-level explanations for mt. explainer ensembles also exploit the strengths of their individual members and yield better explanations, be they perturbation-based or gradient-based ones. gradient-based methods have the potential to be used in online applications given that they are more time-efficient than popular perturbation-based approaches, even as ensembles. black-box models however are better explained with regular perturbation-based methods. future work might explore training qe and explanation methods end-to-end, find better performing (multilingual) qe models, or train models on word-level information. one could also try to solve the given problem with recently proposed explanation methods that try to tackle problems of existing explainers. another way of improving the explanation scores might be using supervised ensemble methods on different explainers by using the training dataset to train e.g. a simple decision tree. the training dataset could be also used to finetune the qe models.