Fine-Tuned Machine Translation Metrics Struggle in Unseen Domains (2024)

Abstract: We introduce a new, extensive multidimensional quality metrics (MQM) annotated dataset covering 11 language pairs in the biomedical domain. We use this dataset to investigate whether machine translation (MT) metrics which are fine-tuned on human-generated MT quality judgements are robust to domain shifts between training and inference. We find that fine-tuned metrics exhibit a substantial performance drop in the unseen domain scenario relative to both metrics that rely on the surface form and pre-trained metrics that are not fine-tuned on MT quality judgments.

## introduction automatic metrics are vital for machine translation (mt) research: given the cost and effort required for manual evaluation, automatic metrics are useful for model development and reproducible comparison between research papers @xcite . in recent years, the mt field has been moving away from string-matching metrics like bleu @xcite towards fine-tuned metrics like comet @xcite , which start with pre-trained models and then fine-tune them on human-generated quality judgments. fine-tuned metrics have been the best performers in recent wmt metrics shared task evaluations @xcite and are recommended by the shared task organizers, who go so far as to say, "neural fine-tuned metrics are not only better, but also robust to different domains." @xcite . given the growing popularity of fine-tuned metrics, it is important to better understand their behavior. here, we examine the question of domain robustness of fine-tuned metrics. fine-tuned metrics contain extra parameters on top of the pretrained model which are initialized randomly (or to zero) and then fine-tuned on human-generated mt quality annotations. the primary source of those annotations is prior wmt metrics shared tasks, and domains in wmt are often carried over from year to year (e.g. news). this raises the question: are fine-tuned metrics in fact robust across any domain (including domains not seen in training)? or can their apparent strong performance be attributed in part to the artificially good domain match between training and test data? to answer these questions, we first collect human multidimensional quality metrics (mqm) annotations in the biomedical (bio) domain. vocabulary overlap and error analysis suggest that this new dataset is distinct from the domains used in wmt. this data covers 11 language pairs and 21 translation systems, with 25k total judgments. in addition to the mqm annotations, we also create new high-quality reference translations for all directions. we release this data publicly, along with code for replication of our experiments. 1 next, we examine how different types of metrics perform on our new bio test set relative to the wmt test set. we find that fine-tuned metrics have substantially lower correlation with human architecture metrics surface-form score tgt ref metric bleu chrf ter pre-trained+algorithm score src tgt ref model metric bertscore prism pre-trained+fine-tuned score src tgt ref llm metric comet unite bleurt pre-trained+prompt score src tgt ref llm metric gemba automqm ## new bio mqm dataset we create and release new translations and mqm annotations for the system submissions from 21 participants to the wmt21 biomedical translation shared task @xcite . to explore how different the bio domain is from the wmt22 metric task domains, we computed the vocabulary overlap coefficient between each domain. bio had the smallest average overlap with the wmt domains (0.436) compared to 0.507, 0.486, 0.507, and 0.582 for e-commerce, news, social, and conversation, respectively. see appendix a for full details and example sentences from each domain. ## conclusion and future work this paper investigated the performance of machine translation metrics across divergent domains. to this end, we introduced a new, extensive mqmannotated dataset covering 11 language pairs in the bio domain. our analysis showed that pre-trained+fine-tuned metrics (i.e. those that use prior human quality annotations of mt output) exhibit a larger gap between in-domain and out-ofdomain performance than pre-trained+algorithm metrics (like bertscore). further experiments showed that this gap can be attributed to the da and mqm fine-tuning stage. despite the gap between in-domain and out-ofdomain performance, comet is still the best performing metric on the bio domain in absolute terms. thus, our findings suggest potential directions for future work including collecting more diverse human judgments for pre-trained+fine-tuned metrics and exploring ways to improve the generalization of such metrics during fine-tuning.