GRUEN for Evaluating Linguistic Quality of Generated Text (2020)

Abstract: Automatic evaluation metrics are indispensable for evaluating generated text. To date, these metrics have focused almost exclusively on the content selection aspect of the system output, ignoring the linguistic quality aspect altogether. We bridge this gap by proposing GRUEN for evaluating Grammaticality, non-Redundancy, focUs, structure and coherENce of generated text. GRUEN utilizes a BERT-based model and a class of syntactic, semantic, and contextual features to examine the system output. Unlike most existing evaluation metrics which require human references as an input, GRUEN is reference-less and requires only the system output. Besides, it has the advantage of being unsupervised, deterministic, and adaptable to various tasks. Experiments on seven datasets over four language generation tasks show that the proposed metric correlates highly with human judgments.

## introduction automatic evaluation metrics for natural language generation (nlg) tasks reduce the need for human evaluations, which can be expensive and timeconsuming to collect. fully automatic metrics allow faster measures of progress when training and testing models, and therefore, accelerate the development of nlg systems @xcite @xcite . to date, most automatic metrics have focused on measuring the content selection between the human references and the model output, leaving linguistic quality to be only indirectly captured (e.g., n-gram and longest common subsequence in rouge-n and rouge-l respectively @xcite q1: grammaticality the summary should have no datelines, system-internal formatting, capitalization errors or obviously ungrammatical sentences (e.g., fragments, missing components) that make the text difficult to read. ## related work the growing interest in nlg has given rise to better automatic evaluation metrics to measure the output quality. we first review the widely used metrics for nlg tasks and then discuss available metrics for evaluating linguistic quality. ## proposed metric in this section, we describe the proposed linguistic quality metric in detail. we define the problem as follows: given a system output s with n sentences [s 1 , s 2 , ..., s n ], where s i is any one sentence (potentially among many), we aim to output a holistic score, y s , of its linguistic quality. we explicitly assess system output for the four aspects in we measure y g using two features: sentence likelihood and grammar acceptance. for a system output s, we first use the punkt sentence tokenizer @xcite to extract its component sentences s 1 , s 2 , ..., s n . then, for each sentence s @xmath0,1 , w i,2 , ..., w i,k ), a sequence of words w i,j , we measure its sentence likelihood score l i and grammar acceptance score g i by a bert model @xcite . 6 the choice of bert is to leverage the contextual features and the masked language model (mlm), which can best examine the word choice. however, bert can not be directly applied to get the likelihood of a sentence, as it is designed to get the probability of a single missing word. inspired by @xcite ; @xcite , we estimate l i by a unigram approximation of the words in the sentence: l @xmath1,j |w i,1 ..., w i,j-1 , w i,j+1 , ..., w i,k ). by such approximation, l i can be estimated by computing the masked probability of each word. to obtain the grammar acceptance score g i , we fine-tune the bert model on the corpus of linguistic acceptability (cola) @xcite , a dataset with 10,657 english sentences labeled as grammatical or ungrammatical from linguistics publications. finally, scores from both models (i.e., l i and g i ) are linearly combined to examine the grammaticality of the sentence s i . the final grammaticality score y g is obtained by averaging scores of all n component sentences: y @xmath2. @xcite , nonredundancy refers to having no unnecessary repetition, which takes the form of whole sentences or sentence fragments or noun phrases (e.g., "bill clinton") when a pronoun ("he") would suffice across sentences. to calculate the non-redundancy score y r , we capture repeated components by using four inter-sentence syntactic features: 1) string length of the longest common substring, 2) word count of longest common words, 3) edit distance, and 4) number of common words. we compute the four features for each pair of component sentences and there are n 2 such pairs in total. for each pair of sentences (s i , s j ), we count the number of times m i,j that these pairs are beyond a non-redundancy penalty threshold. the penalty threshold for each feature are: <80% string length of the shorter sentence, <80% word count of the shorter sentence, >60% string length of the longer sentence, @xmath3, respectively. finally, we get y @xmath4.1 * i,j m i,j . note that the non-redundancy penalty threshold and penalty weight are learned empirically from a held-out validation set. we discuss the effectiveness of each feature in detail in appendix b.1. ## empirical evaluation in this section, we evaluate the quality of different metrics on four nlg tasks: 1) abstractive text summarization, 2) dialogue system, 3) text simplification and 4) text compression. evaluating the metrics: we assess the performance of an evaluation metric by analyzing how well it correlates with human judgments. we, following existing literature, report spearman's correlation ρ, kendall's correlation τ , and pearson's correlation r. in addition, to tackle the correlation non-independence issue (two dependent correlations sharing one variable) @xcite , we report william's significance test @xcite , which can reveal whether one metric significantly outperforms the other. correlation type: existing automatic metrics tend to correlate poorly with human judgments at the instance-level, although several metrics have been found to have high system-level correlations @xcite @xcite . instance-level correlation is critical in the sense that error analysis can be done more constructively and effectively. in our paper, we primarily analyze the instance-level correlations and briefly discuss the system-level correlations. baselines: we compare gruen with the following baselines: the tac-2011 dataset, from the text analysis conference (tac), contains 4488 data instances (4.43 sentences or 94 tokens on average). it has 88 document sets and each document set includes 4 human reference summaries and 51 summarizers. we report correlation results on the readability score, which measures the linguistic quality according to the guideline in @xcite . results: instance-level correlation scores are summarized in ## discussion the discussion is primarily conducted for the text summarization task considering that gruen can measure multiple dimensions in