Improving Translation Quality Estimation with Bias Mitigation (2023)

Abstract: State-of-the-art translation Quality Estimation (QE) models are proven to be biased. More specifically, they over-rely on monolingual features while ignoring the bilingual semantic alignment. In this work, we propose a novel method to mitigate the bias of the QE model and improve estimation performance. Our method is based on the contrastive learning between clean and noisy sentence pairs. We first introduce noise to the target side of the parallel sentence pair, forming the negative samples. With the original parallel pairs as the positive sample, the QE model is contrastively trained to distinguish the positive samples from the negative ones. This objective is jointly trained with the regression-style quality estimation, so as to prevent the QE model from overfitting to monolingual features. Experiments on WMT QE evaluation datasets demonstrate that our method improves the estimation performance by a large margin while mitigating the bias.

## introduction quality estimation (qe) aims to predict the quality of machine translation automatically in the absence of reference translations. state-of-the-art qe model mostly falls into pre-trained model (ptm)based paradigm. in the latest qe evaluation tasks @xcite , nearly all top-performing systems adopt multilingual ptms as backbone. good as the ptm based qe performance is, recent researches @xcite reveal that state-of-the-art qe models are biased. to be specific, the models largely rely on spurious monolingual features, such as the fluency of the target sequence, or the complexity of the source sequence, without really capturing the † contribution during internship at bytedance inc. ‡ corresponding authors. 1 codes are available at https://github.com/ huihuichyan/awesomeqe-contrast bilingual semantic alignment. such monolingual features do not have a casual impact on the translation quality, and bias the qe results to a large extent. for example, as shown in figure 1 , a fluent and uncomplicated translation might be assigned with a high quality score even it does not resemble the actual semantics of the source sentence, while an adequate translation with complicated structure might be assigned as bad translation. @xcite recommends to counter with the bias by using a metric that represents adequacy well as labels. however, in their such annotated dataset, the bias is still striking, as revealed by @xcite . as an alternative, @xcite explores several multitask architectures, to support the qe task and discourage the model from learning the bias. in spite of their success on alleviating the bias in qe, the overall estimation performance is degraded. in other words, they mitigate the bias at the cost of qe performance. in this work, we present a new strategy to mitigate the bias of qe and meanwhile improve qe performance. our method is based on contrastive learning between clean and noisy sentence pairs. firstly, we add noise to the target side of the parallel sentence pair. we corrupt the target sentence with hand-crafted rules, and then use another mono-lingual pre-trained model to restore it. secondly, with the original sentence pair as the positive sample and the noisy sentence pairs as the negative samples, contrastive learning is assigned to the qe model as an auxiliary task. in this procedure, the proposed method reassures the qe model to focus on the bilingual alignment in addition to monolingual features, therefore mitigating the bias while upholding the qe performance. we perform experiments on mlqe-pe dataset @xcite and wmt19 qe evaluation dataset @xcite , including both high-, medium-and low-resource language pairs. our method is confirmed to improve the qe accuracy by a large as well as margin mitigate the bias. in particular, we further provide in-detail analysis about the bias of qe by creating two adversarial test sets. examination on these data reveals that our method strikes a compromise between qe performance and bias mitigation, avoiding bias mitigation from overriding the qe objective. our contributions can be summarized as follows: ## related work in contrast to the automatic mt evaluation metrics which is good at system level, qe is usually conducted in either sentence-level or word-level. in this work, we mainly concentrate on sentence-level qe, where the translation quality is measured with different schemes, such as human-targeted error rate (hter) @xcite or direct assessment (da) @xcite , and the qe model is supposed to provide a quality score for each mt output with its source alongside. quality estimation was proposed as early as in 2004 @xcite . after the emergence of bert, pre-trained models (ptms) become popular in the area of qe @xcite . by pre-training on massive multilingual text, ptms have learned various linguistic knowledge, and can be adapted to quality estimation task without further adjustment. in wmt21 and wmt22 qe evaluation tasks @xcite , nearly all top-performing team build the system on multilingual ptms, e.g. xlm-roberta @xcite , multilingual bert @xcite , etc. ptm-based method has become the defacto paradigm. despite the breakthroughs made in qe, the prediction of qe model is revealed to be biased to spurious features. @xcite showed that qe models have a tendency to over-rely on spurious correlations, which is partially due to skewed label distributions and statistical artifacts in qe datasets. in particular, they show the existence of a partial input bias, i.e. the tendency to predict the quality of a translation based on just the target sentence @xcite . to this end, they annotate and release a new dataset, but as shown in subsequent results of @xcite , the bias is still striking in their newly-released dataset. the most correlated work with us is @xcite , who also aims to investigate the bias mitigation of qe model. they find that the model as well as the annotators tend to over-rate the quality of fluent but inadequate translations. accordingly, they propose four auxiliary tasks to perform bias mitigation, two approaches use additional data to inform and support the main task, while the other two are adversarial to discouraging the model from learning the bias. although their methods could alleviate the bias, the estimation accuracy (measured with pearson correlation coefficient) of the qe model is degraded in most cases. another correlated work is @xcite , who firstly propose to apply contrastive learning on qe. but the contrastive learning is solely performed in a zero-shot manner, and they did not apply their method to mitigate the bias of qe. ## conclusion in this paper, we propose to improve translation quality estimation with bias mitigation. we first use pre-trained model to generate contrast samples, and then the qe model is trained to distinguish positive and negative samples. while previous methods mitigate the bias at the cost of estimation accuracy, our method achieves a compromise between bias mitigation and quality estimation. while current state-of-the-art qe models being proved to be biased to monolingual features, the bias could not be simple ruled out for the sake of overall estimation accuracy. in the future, we will dig deeper into this problem, to improve the robustness and generalizability of qe in real applications.