A Holistic Approach to Reference-Free Evaluation of Machine Translation (2023)

Abstract: Traditional machine translation evaluation relies on reference written by humans. While reference-free evaluation gets rid of the constraints of labor-intensive annotations, which can pivot easily to new domains and is more scalable. In this paper, we propose a reference-free evaluation approach that characterizes evaluation as two aspects: (1) fluency: how well the translated text conforms to normal human language usage; (2) faithfulness: how well the translated text reflects the source data. We further split the faithfulness into word-level and sentence-level. Extensive experiments spanning WMT18/19/21 Metrics segment-level daRR and MQM datasets demonstrate that our proposed reference-free approach, ReFreeEval, outperforms SOTA reference-fee metrics like YiSi-2.

## introduction machine translation evaluation has conventionally relied on reference, where outputs are compared against translations written by humans. this is in contrast to the reference-free manner in which translation quality is directly assessed with the source text. reference-free evaluation @xcite @xcite has the potential to free the evaluation model from the constraints of labor-intensive annotations, allowing it to pivot easily to new domains. in this way, reference-free evaluation metrics are substantially more scalable and have lately been in the spotlight. the history of reference-free evaluation for mt can trace back to "qe as a metric" track of ˚equal contribution. : corresponding author. 1 https://github.com/cocacola-lab/ reference-free-evaluation-of-machine-translation. git wmt2019 metrics task @xcite . yisi-2 @xcite and xbertscore @xcite are embedding-based methods that adopt contextual word embeddings to calculate the lexical similarity between the source and candidate translation words. quality estimation @xcite system metrics such as uni+ @xcite and comet-qe @xcite also leverage contextual word embeddings and feed them into a feedforward network. however, they are trained to regress on human scores that are expensive to collect, and gross discrepancies exist when different humans are asked to label the scores. more challenging but worthwhile, we focus on dispensing with references as well as human scores. nevertheless, embedding-based methods are limited to token-level semantic similarity while neglecting sentence-level faithfulness @xcite . besides, it's difficult for word embeddings to discriminate matched word pairs from random ones @xcite . in addition, current reference-free evaluation methods rarely take fluency into account. for the unfluent candidates whose content is roughly consistent with the source, the embedding-based metrics can hardly discriminate and provide accurate evaluation scores 2 . moreover, the general goal of evaluation metrics is to estimate not only the semantic equivalence between source and candidate but also the general quality (i.e., fluency and naturalness) @xcite @xcite . in this work, we propose a holistic approach (i.e., refreeeval) to enhance the evaluation model in aspects of fluency and faithfulness, meanwhile on both word and sentence levels. with regard to fluency, we pose a data augmentation method and train a fluency discrimination module. for word-level faithfulness, we adopt a self-guided contrastive word-alignment method. for sentencelevel faithfulness, we execute knowledge distillation with sbert @xcite to capture more fine-grained semantics. our method builds on the framework of xbertscore. extensive experiments spanning wmt18/19/21 metrics @xcite @xcite segment-level darr and mqm datasets demonstrate that our proposed reference-free approach, refreeeval, outperforms sota reference-free metrics like yisi-2, sentsim and bertscore-mkd in most language directions. ## approach reference-free evaluation of mt can be characterized as two aspects: (1) fluency: how well it conforms to normal human language usage; and (2) faithfulness: how well the translated text reflects the source data. we assess faithfulness at different granularity: word level and sentence level. figure 1 is the illustration of our refreeeval method. ## conclusion we propose a reference-free evaluation approach refreeeval that comprehensively considers three aspects: fluency, word-level faithfulness, and sentence-level faithfulness. extensive experiments spanning datasets from wmt18/19/21 demonstrate the superiority of each module designed for each aspect. refreeeval, combining the above three modules, achieves a higher correlation with human judgments, outperforming current sota referencefree metrics like yisi-2, sentsim and bertscore-mkd in most language directions.