Multilingual Machine Translation Evaluation Metrics Fine-tuned on Pseudo-Negative Examples for WMT 2021 Metrics Task (2021)

Abstract: This paper describes our submission to the WMT2021 shared metrics task. Our metric is operative to segment-level and system-level translations. Our belief toward a better metric is to detect a significant error that cannot be missed in the real practice cases of evaluation. For that reason, we used pseudo-negative examples in which attributes of some words are transferred to the reversed attribute words, and we build evaluation models to handle such serious mistakes of translations. We fine-tune a multilingual largely pre-trained model on the provided corpus of past years’ metric task and fine-tune again further on the synthetic negative examples that are derived from the same fine-tune corpus. From the evaluation results of the WMT21’s development corpus, fine-tuning on the pseudo-negatives using WMT15-17 and WMT18-20 metric corpus achieved a better Pearson’s correlation score than the one fine-tuned without negative examples. Our submitted models,hyp+src_hyp+ref and hyp+src_hyp+ref.negative, are the plain model using WMT18-20 and the one additionally fine-tuned on negative samples, respectively.

## introduction recent studies of automatic evaluation is mostly based on the family models of bert @xcite . @xcite , @xcite , @xcite have shown a strong correlation with human judgement scores. however, we reported in our previous study @xcite , it is hard for bert based metrics to correctly evaluate the translation errors that are annotated with low direct assessment (da) score. upon the problems of evaluating poor quality translations, @xcite has attempted to solve the problem by creating a different human annotation set and corpus. compared to da, their idea is to make a clear definition of critical translation errors and let models learn the critical errors that can cause a serious misunderstanding. following the idea, we used pseudo-negative examples to train the evaluation model. since, empirically, the cases of the evaluation failure happens frequently with the nouns translation errors, we generated pseudo-negative sentences by transferring the attribute of nouns with word attribute transfer @xcite . this system is based on our previous work @xcite , with an extension with fine-tuning with the pseudo-negative examples. ## segment-level evaluation experiments our experiment was conducted on the development data for wmt21 metric task, which is randomly selected 10% of wmt20 mqm segement-level corpus. all the results were calculated by the pearson's correlation with the mqm segment scores. ## conclusion in this paper, we presented a bert-based multilingual evaluation metric that is boosted by pseudonegative examples to evaluate poor translations more precisely. our model leverages our previous work @xcite and have shown an improvement of pearson correlation when finetuning on the synthetic examples in the wmt15-17 and wmt18-20 corpus settings.