DATS core: Evaluating Translation with Data Augmented Translations (2023)

Abstract: The rapid development of large pretrained language models has revolutionized not only the field of Natural Language Generation (NLG) but also its evaluation. Inspired by the recent work of BARTScore: a metric leveraging the BART language model to evaluate the quality of generated text from various aspects, we introduce DATScore. DATScore uses data augmentation techniques to improve the evaluation of machine translation. Our main finding is that introducing data augmented translations of the source and reference texts is greatly helpful in evaluating the quality of the generated translation. We also propose two novel score averaging and term weighting strategies to improve the original score computing process of BARTScore. Experimental results on WMT show that DATScore correlates better with human meta-evaluations than the other recent state-of-the-art metrics, especially for low-resource languages. Ablation studies demonstrate the value added by our new scoring strategies. Moreover, we report in our extended experiments the performance of DATScore on 3 NLG tasks other than translation.

## introduction massive pretrained language models have brought significant improvement to nlg tasks @xcite . recent systems can even generate texts of higher quality than human-annotated ones @xcite . at the same time, standard metrics, such as bleu @xcite and rouge @xcite , for translation and summarization respectively, have not evolved for the past two decades @xcite . these metrics rely on surface lexicographic matches, making them particularly unsuitable for evaluating modern systems operating with embeddings at the semantic level that often generate paraphrases (ng and abrecht, 1 https://github.com/moussakam/datscore 2015). to address this issue, many metrics have been proposed @xcite , but none of them were widely adopted until the release of bertsore @xcite and moverscore @xcite . these metrics take advantage of large pretrained language models like bert @xcite , which are now being used in nearly all nlp tasks @xcite . in this work, we focus on the task of evaluating machine translation. we propose an extension of bartscore @xcite , a recent metric exploiting the bart seq2seq language model @xcite to evaluate the quality of generated text from various aspects. bartscore covers four evaluation facets: faithfulness, precision, recall, and f-score, derived from different generation directions between the source text, the hypothesis (the text generated by a system given the source), and the reference (the reference text for the generation, often provided by human annotators). the scores are obtained by pairing the three entities differently at the input or the output side of a trained seq2seq model for fetching conditional generation probabilities. based on bartscore, and motivated by the general idea and positive effect of data augmentation techniques, we found that adding augmented, translated copies of the source and reference texts in bartscore, can greatly help evaluate the quality of the hypothesis translation. we also propose two novel score averaging and term weighting strategies to improve the original score computing process of bartscore. results and ablation studies show that our metric datscore (data augmented translation score) outperforms the other recent state-of-the-art metrics, and our new scoring strategies are effective. moreover, the performance of datscore is also reported on three other nlg tasks than translation: data-to-text, summarization, and image captioning. to the best of our knowledge, no prior work has been done on leveraging data augmentation techniques for untrained nlg evaluation metrics. our work will help fill this gap. our contributions include: 1) inspired by bartscore, we developed datscore, incorporating augmented data translated from the source and reference texts. datscore is an untrained and unsupervised translation evaluation metric that offers a larger performance boost in evaluating low-resource language generation. in contrast to other widely adopted metrics, datscore can efficiently incorporate both the source and reference texts in the evaluation. ## datscore as mentioned in subsection 2.1, bartscore is not based on matching tokens nor their embeddings as the other evaluation metrics. instead, it uses a novel approach by framing the evaluation of generated text as a text generation problem. assuming first a pretrained seq2seq model is "ideal" (e.g., bart), bartscore directly uses the model's conditional probability of generating a provided target text y given a provided input text x, as the evaluation score of the generation direction @xmath0. for example, y corresponds to a translation hypothesis generated by any system, and x is the reference. if y is of high quality, then by providing the pair to the pretrained bart model, the estimated conditional generation probability (evaluation score) p (y |x) should be high. therefore, by placing differently the source (src), the reference (ref), and the hypothesis (hypo) in pair at the input or the output side of the trained seq2seq model for fetching conditional generation probabilities, bartscore considers three different generation directions illustrated as dashed arrows in figure 1 . the conditional probabilities associated with the directions are denoted as: precision (@xmath1, recall (@xmath2. additionally, an f-score, the arithmetic average of precision and recall. the score (conditional probability) for the gen-eration direction from a source sequence @xmath3, weighted log probability over all generation steps: our contributions consist of three modifications tailored to machine translation: ## other nlg tasks in addition to machine translation, our main focus, we evaluate datscore on other nlg tasks, including data-to-text generation, abstractive summarization, and image captioning. to work around the different modalities of source inputs represented in these tasks (e.g., not able to create a data augmented translation with an image), we adapt datscore to only consider 4 generation directions: hypo↔ref and hypo↔trans2. data-to-text. summarization. image captioning. we consider flickr8k @xcite and pascal-50s @xcite , two image captioning datasets. the former is annotated with scores from 1 to 4 assessing the relevance of the captions, and the latter is annotated with relative ranking (i.e., given two descriptions which one is better). finally, although not the top-performing metric across all tasks, datscore showed an overall stable and competitive performance. conversely, each of the other metrics fails in evaluating generations, at least in one of the tasks. for example, bertscore and moverscore have poor performance on the webnlg dataset. on the other hand, although bartscore is finetuned on an abstractive summarization dataset, it fails to correlate well with human judgment on realsumm and summeval. this finding suggests that datscore can be safely used to evaluate nlg systems in other tasks for different evaluation dimensions, regardless of being initially designed for machine translation evaluation. ## ablation study to validate our different choices with regard to datscore, we conducted ablation studies on: 1) the contributions of all 8 direction scores, results are illustrated in figure 2 . ## conclusion in this work, we proposed one of the first applications of data augmentation techniques to nlg evaluation. to obtain an evaluation score of the translation hypothesis, our developed metric datscore additionally leverages newly translated copies aug-mented from the source and reference texts. we also proposed two novel strategies for score averaging and term weighting to improve the original, naive score computing process of bartscore, on the basis of which our work is built. experimental results show that datscore achieved a higher correlation with human meta-evaluations, in comparison with the other recent state-of-the-art metrics, especially for those less represented languages other than english. moreover, ablation studies show the effectiveness of our newly proposed score computing approaches, and extended experiments showed an overall stable and competitive performance of datscore on more nlg tasks.