R o M e: A Robust Metric for Evaluating Natural Language Generation (2022)

Abstract: Evaluating Natural Language Generation (NLG) systems is a challenging task. Firstly, the metric should ensure that the generated hypothesis reflects the reference’s semantics. Secondly, it should consider the grammatical quality of the generated sentence. Thirdly, it should be robust enough to handle various surface forms of the generated sentence. Thus, an effective evaluation metric has to be multifaceted. In this paper, we propose an automatic evaluation metric incorporating several core aspects of natural language understanding (language competence, syntactic and semantic variation). Our proposed metric, RoMe, is trained on language features such as semantic similarity combined with tree edit distance and grammatical acceptability, using a self-supervised neural network to assess the overall quality of the generated sentence. Moreover, we perform an extensive robustness analysis of the state-of-the-art methods and RoMe. Empirical results suggest that RoMe has a stronger correlation to human judgment over state-of-the-art metrics in evaluating system-generated sentences across several NLG tasks.

## introduction automatic generation of fluent and coherent natural language is a key step for human-computer interaction. evaluating generative systems such as text summarization, dialogue systems, and machine translation is challenging since the assessment involves several criteria such as content determination, lexicalization, and surface realization @xcite . for assessing system-generated outputs, human judgment is considered to be the best approach. obtaining human evaluation ratings, on the other hand, is both expensive and time-consuming. as a result, developing automated metrics for assessing the quality of machine-generated text has become an active area of research in nlp. the quality estimation task primarily entails determining the similarity between the reference and hypothesis as well as assessing the hypothesis for grammatical correctness and naturalness. widely used evaluation metrics such as bleu @xcite , meteor @xcite , and rouge @xcite which compute the word-overlaps, were primarily designed for evaluating machine translation and text summarization systems. word-overlap based metrics, on the other hand, are incapable of capturing the hypotheses' naturalness and fluency. furthermore, they do not consider the syntactic difference between reference and hypothesis. in a different line of research, word mover distance (wmd) @xcite , bertscore @xcite and moverscore @xcite compute word embedding based similarity for evaluating systemgenerated texts. although these metrics employ the contextualized representation of words, they do not take the grammatical acceptability of the hypothesis and the syntactical similarity to the reference into account. to address these shortcomings, we propose rome, an automatic and robust metric for evaluating nlg systems. rome employs a neural classifier that uses the generated sentence's grammatical, syntactic, and semantic qualities as features to estimate the quality of the sentence. firstly, it calculates the earth mover's distance (emd) @xcite to determine how much the hypothesis differs from the reference. during the computation of emd, we incorporate hard word alignment and soft-penalization constants to handle various surface forms of words in a sentence, such as repeated words and the passive form of a sentence. secondly, using a semantically enhanced tree edit distance, the difference in syntactic structures between the reference and hypothesis sentences is quantified. thirdly, the metric incorporates a binary classifier to evaluate the grammatical accept-ability of the generated hypotheses. finally, the scores obtained from the preceding steps are combined to form a representation vector, which is subsequently fed into a self-supervised network. the network produces a final score, referred to as rome's output which represents the overall quality of the hypothesis statement. we investigate the effectiveness of our proposed metric by conducting experiments on datasets from various domains of nlg such as knowledge graph based language generation dataset (kelm @xcite ), dialogue datasets @xcite , the webnlg 2017 challenge dataset @xcite , structured data to language generation dataset (bagel @xcite and sfho-tel @xcite ). the capability of existing metrics to handle various forms of text has lately become a matter of debate in the nlp community @xcite @xcite . hence, we conduct an extensive robustness analysis to assess rome's performance in handling diverse forms of system-generated sentences. to verify our claim, we design the analysis based on the text perturbation methods used in @xcite and adversarial text transformation techniques from textfooler @xcite and textattack @xcite . empirical assessment on benchmark datasets and the robustness analysis results exhibit that rome can handle various surface forms and generate an evaluation score, which highly correlates with human judgment. rome is designed to function at the sentence level and can be used to evaluate english sentences in the current version of the implementation. in the future versions, we plan to extend rome by including more languages. we released the code and annotation tool publicly foot_0 . ## rome in rome, a neural network determines the final evaluation score given a reference-hypothesis pair. the network is trained to predict the evaluation score based on three features: semantic similarity computed by emd, enhanced ted, and the grammatical acceptability score. we explain these features in the following subsections. ## related work a potentially good evaluation metric is one that correlates highly with human judgment. among the unsupervised approaches, bleu @xcite , meteor @xcite and rouge @xcite are the most popular evaluation metrics traditionally used for evaluating nlg systems. although these metrics perform well in evaluating machine translation (mt) and summarization tasks, @xcite shows that none of the word overlap based metrics is close to human level performance in dialogue system evaluation scenarios. in a different line of work, word embedding based metrics are introduced for evaluating nlg systems @xcite . several unsupervised automated metrics were proposed that leverage emd; one of them is word mover's distance (wmd) @xcite . later, @xcite proposed an evaluation metric, incorporating wmd and wordembedding, where they used word alignment between the reference and hypothesis to handle the word-order problem. recently, @xcite introduced an emd-based metric we_wpi that utilizes the word-position information to tackle the differences in surface syntax in reference and hypothesis. several supervised metrics were also proposed for evaluating nlg. adem @xcite uses a rnn-based network to predict the human evaluation scores. with the recent development of language model-based pre-trained models @xcite proposed bertscore, which uses a pre-trained bert model for evaluating various nlg tasks such as machine translation and image captions. recently, @xcite proposed moverscore, which utilizes contextualized embedding to compute the mover's score on word and sentence level. a notable difference between moverscore and bertscore is that the latter relies on hard alignment compared to soft alignments in the former. unlike the previous methods, rome focuses on handling the sentence's word repetition and passive form when computing the emd score. furthermore, rome trains a classifier by considering the sentence's semantic, syntactic, and grammatical acceptability features to generate the final evaluation score. ## conclusion we have presented rome, an automatic and robust evaluation metric for evaluating a variety of nlg tasks. the key contributions of rome include 1) emd-based semantic similarity, where hard word alignment and soft-penalization techniques are employed into the emd for tackling repetitive words and passive form of the sentence, 2) semantically enhanced ted that computes the syntactic similarity based on the node-similarity of the parsed dependency trees, 3) grammatical acceptability classifier, which evaluates the text's grammatical quality, and 4) robustness analysis, which assesses the metric's capability of handling various form of the text. both quantitative and qualitative analyses exhibit that rome highly correlates with human judgment. we intend to extend rome by including more languages in the future.