UNION : A n U nreferenced M etric for E valuating O pen-ended S tory G eneration (2020)

Abstract: Despite the success of existing referenced metrics (e.g., BLEU and MoverScore), they correlate poorly with human judgments for open-ended text generation including story or dialog generation because of the notorious one-to-many issue: there are many plausible outputs for the same input, which may differ substantially in literal or semantics from the limited number of given references. To alleviate this issue, we propose UNION, a learnable UNreferenced metrIc for evaluating Open-eNded story generation, which measures the quality of a generated story without any reference. Built on top of BERT, UNION is trained to distinguish human-written stories from negative samples and recover the perturbation in negative stories. We propose an approach of constructing negative samples by mimicking the errors commonly observed in existing NLG models, including repeated plots, conflicting logic, and long-range incoherence. Experiments on two story datasets demonstrate that UNION is a reliable measure for evaluating the quality of generated stories, which correlates better with human judgments and is more generalizable than existing state-of-the-art metrics.

## introduction significant advances have been witnessed with neural encoder-decoder paradigm @xcite , transformer-based architecture @xcite and large-scale pretraining models @xcite in a wide array of natural language generation (nlg) tasks including machine translation @xcite , story generation @xcite , and many more. however, the research is increasingly hindered by the lack of effec- since human evaluation is time-consuming, expensive, and difficult to reproduce, the community commonly uses automatic metrics for evaluation. previous studies in conditional language generation tasks (e.g., machine translation) have developed several successful referenced metrics, which roughly quantify the lexical overlap (e.g., bleu @xcite ) or semantic entailment (e.g., moverscore @xcite ) between a generated sample and the reference. however, such referenced metrics correlate poorly with human judgments when evaluating open-ended text generation @xcite due to the one-tomany nature @xcite , as illustrated in to address the one-to-many issue, unreferenced metrics are proposed to measure the quality of a generated sample without any reference. @xcite presented a learnable, unreferenced metric which measures the text quality by learning to distinguish human-written texts from generated samples. however, the discriminatorbased metric can easily lead to over-fitting to specific data @xcite or model bias since the quality of generated texts varies substantially across different nlg models. as a matter of fact, the generalization or robustness issue is critical for any learnable metrics. therefore, we propose union, a learnable unreferenced metric for evaluating open-ended story generation. union learns to distinguish human-written stories from negative samples autoconstructed by generating perturbations of humanwritten stories. it is trained without dependence on specific nlg models or any human annotation, making it more generalizable to distribution drift @xcite than the discriminator-based metric and those metrics which learn from human preference (e.g., adem @xcite ). to capture commonly observed issues in generated stories, such as repeated plots, conflicting logic, and inter-sentence incoherence, we adopt four negative sampling techniques to construct negative samples, including repetition, substitution, reordering, and negation alteration. in addition, we design an auxiliary reconstruction objective for union, which recovers the perturbation from a negative sample. this objective is shown to further improve the performance of union. our contributions are summarized as follows: i. we propose a learnable unreferenced metric union for evaluating open-ended story generation to alleviate the one-to-many issue of referenced metrics. union does not depend on any output of nlg models or human annotation. ## related work automatic evaluation is crucial for language generation tasks. we roughly divide existing metrics into referenced, unreferenced, and hybrid metrics, according to whether they rely on human-written references when calculating the metric score. referenced metrics usually measure how similar a generated text is to the reference text. therefore, they are developed mainly for conditional language generation tasks such as machine translation and text summarization, where plausible outputs are largely limited within the semantics of input. commonly used referenced metrics include wordoverlap based (e.g., bleu @xcite , rouge @xcite ) and embedding based metrics (e.g., bertscore @xcite , mover-score @xcite ). however, referenced metrics are reported to correlate poorly with human judgments in open-ended generation tasks including open-domain dialog generation @xcite and story generation, where the input contains only limited information for generation, and there are many plausible outputs for the same input, which can vary substantially in literal or semantics. unreferenced metrics measure the quality of a sample without any reference. the most classic unreferenced metric is perplexity, which measures how likely a sample is generated by a given language model trained on human-written texts. however, recent work has shown that natural language is rarely the most probable text @xcite , and perplexity is inadequate to measure quality @xcite . therefore, perplexity may not indicate the actual text quality well. discriminator-based metric @xcite measures how easily a discriminator distinguishes the generated samples from human-written texts. however, training such a discriminator can be easily over-fitted to a specific dataset, thereby leading to poor generalization and low correlation with human judgments @xcite . in addition to the above point-wise metrics which score an individual sample, @xcite proposed the fréchet infersent distance (fid) to evaluate the model-level quality and diversity of generated samples, by computing the fréchet dis-tance between the gaussian distribution fitted to human text embeddings and that to generated sample embeddings. however, in real data, the distribution of embeddings may be far from gaussian. recently, @xcite proposed to evaluate sample-level quality by comparing a pair of samples, and further adopted a skill rating system to evaluate model-level quality based on the samplelevel pair-wise comparison. however, it is unlikely to evaluate a single sample without access to its references. hybrid metrics combine referenced and unreferenced metrics. for open-domain dialog system evaluation, @xcite proposed a learnable metric adem to learn from the human-annotated score of a response given its post and ground truth. however, such a metric shows very poor generalization and is not robust to easy attacks such as simple word substitution or random word shuffle @xcite . furthermore, ruber and its variants @xcite evaluate a response by directly averaging a nonlearnable referenced embedding similarity score and a learnable unreferenced post-response relatedness score that is learned by applying negative sampling without human annotations. however, merely measuring input-output relatedness is not sufficient for evaluating long text generation, as the intrinsic coherence and consistency within the generated text is a critical factor. additionally, some metrics which learn from human preference achieve substantial results in conditional language generation, e.g., ruse @xcite and bleurt @xcite . ruse trained a regression model to score a reference-candidate pair using their sentence embeddings. and bleurt used multiple automatic metrics (e.g., bleu) as supervision signals for pretraining on synthetic data, and was fine-tuned on human judgments. however, bleurt heavily relies on the quality of automatic metrics, but there are yet no such reliable metrics for open-ended text generation. ## methodology union is expected to measure the overall quality of a generated story. in this section, we begin with common issues that can be observed in the output of nlg models. we then propose four negative sampling techniques based on the observations. afterward, we introduce how union is trained and used for story evaluation. the overall paradigm of union is shown in figure 1 . bert 𝒗 [,-.] 𝒗 𝒔1 𝒗 𝒔7 𝒗 𝒔2 𝒗 [.34] ## experiment we conducted extensive experiments to evaluate union on two story datasets. first, we compared union against existing text generation metrics. then, we assessed its generalization on distribution drifts, including dataset drift and quality drift. last, we measured the effect of each negative sampling technique with ablation studies. ## conclusion we present union, an unreferenced metric for evaluating open-ended story generation. union is trained to distinguish human-written stories from auto-constructed negative samples and to recover the perturbation in negative samples. extensive experiments show that union outperforms stateof-the-art metrics in terms of correlation with human judgments on two story datasets, and is more robust to dataset drift and quality drift. results also show the effectiveness of the proposed four negative sampling techniques. as future work, we will explore the similar idea of designing unreferenced metrics for dialog generation.