U ni TE : Unified Translation Evaluation (2022)

Abstract: Translation quality evaluation plays a crucial role in machine translation. According to the input format, it is mainly separated into three tasks, i.e. , reference-only, source-only and source-reference-combined. Recent methods, despite their promising results, are specifically designed and optimized on one of them. This limits the convenience of these methods, and overlooks the commonalities among tasks. In this paper, we propose , which is the first unified framework engaged with abilities to handle all three evaluation tasks. Concretely, we propose monotonic regional attention to control the interaction among input segments, and unified pretraining to better adapt multi-task training. We testify our framework on WMT 2019 Metrics and WMT 2020 Quality Estimation benchmarks. Extensive analyses show that our single model can universally surpass various state-of-the-art or winner methods across tasks.Both source code and associated models are available at https://github.com/NLP2CT/UniTE .

## introduction automatically evaluating the translation quality with the given reference segment(s), is of vital importance to identify the performance of machine translation (mt) models @xcite @xcite . based on the input contexts, translation evaluation can be mainly categorized into three classes: 1) reference-only evaluation ( ref ) approaches like bleu @xcite and bleurt @xcite , which evaluate the hypothesis by referring the golden reference at target side; 2) source-only evaluation ( src ) methods like yisi-2 @xcite and transquest @xcite , which are also referred as quality estimation (qe). these methods estimate the quality of the hypothesis based on the source sentence without using references; 3) source-reference-combined evaluation ( src+ref ) works like comet @xcite , where the evaluation exploits information from both source and reference. with the help of powerful pretrained language models (plms, @xcite , modelbased approaches (e.g., bleurt, transquest, and comet) have shown promising results in recent wmt competitions @xcite @xcite @xcite . nevertheless, each existing mt evaluation work is usually designed for one specific task, e.g., bleurt is only used for ref task and can not support src and src+ref tasks. moreover, those approaches preserve the same core -evaluating the quality of translation by referring to the given segments. we believe that it is valuable, as well as feasible, to unify the capabilities of all mt evaluation tasks ( ref , src and src+ref ) into one model. among the promising advantages are ease of use and improved robustness through knowledge transfer across evaluation tasks. to achieve this, two important challenges need to be addressed: 1) how to design a model framework that can unify all translation evaluation tasks? 2) how to make the powerful plms better adapt to the unified evaluation model? in this paper, we propose unite -unified translation evaluation, a novel approach which unifies the functionalities of ref , src and src+ref tasks into one model. to solve the first challenge as mentioned above, based on the multilingual plm, we utilize layerwise coordination which concatenates all input segments into one sequence as the unified input form. to further unify the modeling of three evaluation tasks, we propose a novel monotonic regional attention (mra) strat-egy, which allows partial semantic flows for a specific evaluation task. for the second challenge, a multi-task learning-based unified pretraining is proposed. to be concrete, we collect the high-quality translations and degrade low-quality translations of nmt models as synthetic data. then we propose a novel ranking-based data labeling strategy to provide the training signal. finally, the multilingual plm is continuously pretrained on synthetic dataset with multi-task learning manner. besides, our proposed models, named unite-mra and unite-up respectively, can benefit from finetuning with human-annotated data over three tasks at once, not requiring extra task-specific training. experimental results demonstrate the superiority of unite. compared to various strong baseline systems on each task, unite, which unifies ref , src and src+ref tasks into one single model, achieves consistently absolute improvements of kendall's τ correlations at 1.1, 2.3 and 1.1 scores on english-targeted translation directions of wmt 2019 metric shared task @xcite , respectively. meanwhile, after introducing multilingual-targeted support for our unified pretraining strategy, a single model named unite-mup also gives dominant results against existing methods on non-english-targeted translation evaluation tasks. furthermore, our method can also achieve competitive results over wmt 2020 qe task compared with the winner submission @xcite . ablation studies reveal that, the proposed mra and unified pretraining strategies are both important for model performance, making the model preserve the outstanding performance and multi-task transferability concurrently. ## related work in this section, we briefly introduce the three directions of translation evaluation. ## methodology as mentioned above, massive methods are proposed for different automatic evaluation tasks. on the one hand, it is inconvenient and expensive to develop and employ different metrics for different evaluation scenarios. on the other hand, separate models absolutely overlook the commonalities among these evaluation tasks, of which knowledge potentially benefits all three tasks. in order to fulfill the aim of unifying the functionalities on ref , src , and src+ref into one model, in this section, we introduce unite (figure 1 ). ## ablation studies in this section, we conduct ablation studies to investigate the effectiveness of regional attention patterns ( §5.1), unified training ( §5.2), and rankingbased data labeling ( §5.3). all experiments are conducted by following english-targeted setting. ## conclusion in the past decades, automatic translation evaluation is mainly divided into ref , src and src+ref tasks, each of which develops independently and is tackled by various task-specific methods. we suggest that the three tasks are possibly handled by a unified framework, thus being ease of use and facilitating the knowledge transferring. contributions of our work are mainly in three folds: