I magin E : An Imagination-Based Automatic Evaluation Metric for Natural Language Generation (2023)

Abstract: Automatic evaluations for natural language generation (NLG) conventionally rely on token-level or embedding-level comparisons with text references. This differs from human language processing, for which visual imagination often improves comprehension. In this work, we propose ImaginE, an imagination-based automatic evaluation metric for natural language generation. With the help of StableDiffusion, a state-of-the-art text-to-image generator, we automatically generate an image as the embodied imagination for the text snippet and compute the imagination similarity using contextual embeddings. Experiments spanning several text generation tasks demonstrate that adding machine-generated images with our ImaginE displays great potential in introducing multi-modal information into NLG evaluation, and improves existing automatic metrics’ correlations with human similarity judgments in both reference-based and reference-free evaluation scenarios.

## introduction a major challenge for natural language generation (nlg) is to design an automatic evaluation metric that can align well with human judgments. to this end, many approaches have been investigated. metrics that base on matching mechanisms such as bleu @xcite , me-teor @xcite , cider @xcite , have been widely adopted in the field. edit-distance based metrics, such as char-acter @xcite , wmd @xcite , smd @xcite , have also been explored. recently, bertscore @xcite and bleurt @xcite attempt to leverage bert @xcite to compare text embedding similarities, which correlates better with human judgments than previous methods. these automatic evaluation metrics make use of textual information from various angles extensively. but what happens in our minds when we read, comprehend, and evaluate text? research @xcite has found that, unlike commonly designed automatic evaluation methods that compare the generated candidates with the references on the text domain only, humans, in contrast, leverage visual imagination and trigger neural activation in vision-related brain areas when reading text. cognitive studies show that visual imagery improves comprehension during language processing @xcite @xcite . inspired by this imagination-based multi-modal mechanism in human text comprehension, we ask a critical research question: can machines create a visual picture of any underlying sentence, and use their imaginations to improve natural language understanding? the advances of recent pre-trained visionlanguage models such as clip @xcite provide an excellent opportunity for us to utilize the learned image-text representations. this enables us to explore the possibility of incorporating multi-modal information into nlg evaluation. in this work, we propose imagine, an imagination-based automatic evaluation metric for text generation. specifically, we first use the state-of-the-art text-to-image generator stablediffusion @xcite to visualize machine imagination from sentences, which is to generate descriptive images for the candidate text and the references. then we receive the imagine scores by computing two sets of similarity scores with the pre-trained clip model @xcite : the visual similarity of the generated images, and the cross-modal similarity between the text and the generated image. figure 1 shows an example. to understand the role the machine-generated images play in nlg evaluation, we conduct a series of experiments with imagine on multiple nlg tasks and datasets, including machine translation, text summarization, and sentence completion for open-ended text generation, aiming to answer the following questions: 1. how influential is imagine in nlg evaluation in terms of correlations with human judgments? can it provide additional reference information on top of existing metrics? 2. what are the applicable scenarios of introducing imagine to nlg evaluation? when and why do machine-generated images help? 3. what are the potentials and limitations of introducing machine-generated images with imag-ine to nlg evaluation? experimental results show that imagine can serve as a complementary evaluation metric to textbased ones, and adding imagine scores to existing metrics surprisingly improves most of the popular metrics' correlations with human performance on various text generation tasks. this holds for both reference-based evaluation and referencefree evaluation. we further conduct comprehensive quantitative analyses with case studies to verify its effectiveness. overall, imagine displays great potential in introducing multi-modal information into nlg evaluation. ## imagine this section describes how our imagine metric evaluates the similarity between two pieces of text with the help of machine imagination. figure 2 provides an overview of our method. ## conclusion we present imagine, a novel automatic evaluation metric for nlg that is based on machine imagination. our experiments on five datasets across three different nlg tasks demonstrate the potential of incorporating imagine similarity scores as a supplement to existing automatic nlg metrics, which can lead to improvement in their correlation with human evaluations in various scenarios. in the future, it is interesting to explore effective ways of visualizing abstract concepts, and how to incorporate machine imagination efficiently. we hope our work can contribute to the discussion and advancement of multi-modal studies.