Bias Mitigation in Machine Translation Quality Estimation (2022)

Abstract: Machine Translation Quality Estimation (QE) aims to build predictive models to assess the quality of machine-generated translations in the absence of reference translations. While state-of-the-art QE models have been shown to achieve good results, they over-rely on features that do not have a causal impact on the quality of a translation. In particular, there appears to be a partial input bias, i.e., a tendency to assign high-quality scores to translations that are fluent and grammatically correct, even though they do not preserve the meaning of the source. We analyse the partial input bias in further detail and evaluate four approaches to use auxiliary tasks for bias mitigation. Two approaches use additional data to inform and support the main task, while the other two are adversarial, actively discouraging the model from learning the bias. We compare the methods with respect to their ability to reduce the partial input bias while maintaining the overall performance. We find that training a multitask architecture with an auxiliary binary classification task that utilises additional augmented data best achieves the desired effects and generalises well to different languages and quality metrics.

## introduction despite the great advances of machine translation (mt) models over the past years, the adequacy and fluency of the translations cannot be guaranteed. without access to a gold-standard reference translation, it can be difficult to validate the reliability of the mt model's predictions. to address this issue, the field of mt quality estimation (qe) emerged, aiming to develop models that can approximate the quality of machine-generated translations in a scalable way. however, recent research suggests that state-of-the-art qe approaches tend to over-rely on features that do not have a causal impact on the quality of a translation. in particular, there appears to be a partial input bias, i.e. a tendency to assign high quality scores to translations that are fluent and grammatical, even though they do not resemble the actual meaning of the source @xcite . building upon these findings, the objective of our work is to characterise and, most importantly, mitigate the partial input bias of qe models. we focus on the use of auxiliary training tasks to specifically target the observed biases while avoiding strong modifications of the original model as well as the expensive collection and manual labelling of additional training data. our efforts concentrate on testing and improving monotran-squest, the best-performing architecture in the shared task on sentence-level qe hosted as part of the fifth conference on machine translation (wmt 2020) @xcite . we work with the recently published multilingual qe dataset mlqe-pe @xcite , allowing us to test the generalisability of our approaches across different languages and quality scores. our main contributions are as follows: we group our approaches into four categories: multitask training with mixed languages, multitask training with additional augmented data, training with adversarial tasks and training with debiased focal loss. in the subsequent sections, we first present related work, followed by the analysis of the partial input bias. building upon the findings, we explain the four bias mitigation approaches in section 4 and discuss the results in section 5. ## bias analysis in the following, we will describe the dataset and baseline model used, show benchmark results and analyse the partial input bias in more detail. ## methodology based on the bias analysis, our goal is to find an effective and feasible way to reduce the impact of spurious correlations and overly dominant features. as outlined in the previous section, the two high-resource datasets (en-de and en-zh) clearly show the strongest partial input bias. they will therefore be at the centre of the bias mitigation efforts. all four methods presented hereinafter share the core idea of using auxiliary tasks to achieve this aim: the main task -qe -is combined with helper tasks designed to reduce known biases. at test time, the auxiliary tasks can be discarded. hereinafter, we introduce four approaches and the corresponding model architectures. the first two methods are tailored to combat the biased behaviour by supporting the model with additional data. in contrast, the two alternative, restrictive approaches actively penalise the model for learning unwanted behaviour. we define three criteria to ensure comparability between the approaches: a good solution should 1) mitigate the observed biases, 2) retain the prediction quality of the benchmark model, and 3) avoid computational overhead and interference with the original model's design. ## results in the following, we present and discuss the results of the experiments conducted. based on the analysis in section 3.3, the experiments concentrate on the two most biased datasets english-german and english-chinese, each in combination with the da and hter scores. for each of the four sections, we assess different hyperparameter configurations on the en-de validation set. a configuration is considered to be good if the bias is reduced and the overall performance is at least maintained. the most promising variant is then evaluated on the en-de and en-zh test set, to see if the method generalises across language domains. finally, we compare the four methods against one another and provide further analyses on the robustness of the best-performing model. foot_6 ## future work building upon the previously discussed results, we propose ideas for future work. considering the experimental design, the multitask architecture provides additional degrees of freedom that were not explored extensively, yet. for example, one could vary the amount of training per task or learn the training schedule as a parameter which adapts dynamically during the training process @xcite . in addition, the number of auxiliary tasks could be increased to two or more, mixing different task types. to further evaluate the generalisability of the proposed methods, experiments with additional datasets, low-resource language pairs as well as alternative qe architectures and language models could be conducted, too. going beyond the field of machine translation quality estimation, it would be interesting to see the methods applied in adjacent areas of nlp. for example, this could entail closely related settings, such as quality estimation for machine-generated text summaries, as well as the fields of nli and vqa, both of which face partial input biases. other observable biases could also be considered as candidates for the use of targeted bias reduction techniques, provided that it is possible to design a counterbalancing auxiliary task or isolate the bias well enough to deploy adversarial approaches. we think that if the latter scenario applies, the adapted debiased focal loss technique for regression could be worth further exploration, too. ## conclusion this paper expands on recent research which suggests that qe models are susceptible to learning spurious correlations. based on additional analysis, and inspired by related work in the fields of nli and vqa, we propose a range of auxiliary tasks that inform the main quality estimation task during training and are discarded at test time. first, we train the main quality estimation task together with additional, less biased data in a multitask setting. then, we explore adversarial training and debiased focal loss to directly target the partial input bias. we find that the former approaches yield more stable results than the latter and conjecture that this is due to the difficulty of isolating partial input bias effects from useful predictive information encoded in the translation. we show that our proposed multitask architecture multitransquest, especially when trained with additional shuffled wikimatrix data, generalises well across the two most biased language pairs and the two different quality scores. our method retains the overall prediction quality, reduces the observed biases significantly and increases the models' robustness in a zero-shot setting.