COSM ic: A Coherence-Aware Generation Metric for Image Descriptions (2021)

Abstract: Developers of text generation models rely on automated evaluation metrics as a stand-in for slow and expensive manual evaluations. However, image captioning metrics have struggled to give accurate learned estimates of the semantic and pragmatic success of output text. We address this weakness by introducing the first discourse-aware learned generation metric for evaluating image descriptions. Our approach is inspired by computational theories of discourse for capturing information goals using coherence. We present a dataset of image–description pairs annotated with coherence relations. We then train a coherence-aware metric on a subset of the Conceptual Captions dataset and measure its effectiveness—its ability to predict human ratings of output captions—on a test set composed of out-of-domain images. We demonstrate a higher Kendall Correlation Coefficient for our proposed metric with the human judgments for the results of a number of state-of-the-art coherence-aware caption generation models when compared to several other metrics including recently proposed learned metrics such as BLEURT and BERTScore.

## introduction an investigation of the descriptions used with images on the web shows that image descriptions can have different functions and goals @xcite . for instance, captions may describe visible entities, activities and relationships, provide background information that goes beyond what's visible, or report the writer's own subjective reactions to what's displayed. by drawing on such diverse examples, image captioning models can learn the different inferential links between text and images and use that information at generation time to produce descriptions that can fulfill different discourse goals and inject the desired context into their output @xcite ; in this case where a caption that does not just describe the image but elaborates on it, our metric recognizes that the model output is potentially successful (photo credit: moorthy gounder) @xcite @xcite . so far, however, efforts to develop such expressive captioning models have been hindered by the lack of automatic metrics that can evaluate their output with respect to their information goals in context. previous approaches to automatic caption evaluation have mostly focused on n-gram measures of similarity to reference output @xcite ; such surface-level models fail to deal with the lexical and syntactic diversity of image descriptions. more recent approaches more closely approximate semantic similarity using word embedding-based techniques. these models show robust performance and achieve a higher correlation with human judgments than that of previous metrics. nevertheless, they too fail to generalize to the different kinds of content that successful descriptions may exhibit across different goals and contexts. that is, they cannot distinguish reasonable descriptions that happen to differ from reference output in their goals and perspective, from problematic descriptions that hallucinate inappropriate content or context. to bridge this gap, we present a coherence-aware embedding-based generation metric that learns to respect diverse discourse goals without penalizing captions that are purposefully generated to fulfill different purposes or communicate background information. figure 1 demonstrates this capability by presenting an example image and captions with different coherence labels together with their scores. our approach to modeling discourse goals is based on the framework of discourse coherence theory @xcite , which characterizes the inferences that give discourse units a coherent joint interpretation using a constrained inventory of coherence relations. in particular, we use the taxonomy for image-text coherence developed by @xcite , which for example includes visible, story and subjective relations between the text and the image. a description and an image stand in a visible relation if the text includes information that is recognizably depicted in the image. subjective captions react to the content of the image and story captions provide a free-standing description of the circumstances depicted in the image similar to the narration relation in text. our metric is learned in part from a new dataset of 4000 images with descriptions labeled with different coherence labels in this taxonomy. in inaugurating the study of coherence-aware generation metrics, we make the following specific contributions. in section 3 we present two different, annotated datasets for training and testing a coherence-aware metric. we present a model to score a generated caption given the image, reference caption, and the discourse goals of both these captions (section 4). we compare this metric to previous ones using a common methodology, ranking the performance of several different caption generation systems on out-of-domain images-relying on a new benchmark out-of-domain test set, which we publish, providing reference captions for a subset of openimages @xcite . our experiments demonstrate that among all these metrics, our proposed metric has the highest correlation with human judgments. ## related work there are diverse ways of characterizing the contributions of text and imagery. @xcite investigate the genre of image captions and @xcite study the persuasive implicit relationships between text and images. @xcite study the emotional links between text and images. @xcite present an annotated dataset of text and imagery that compares the information load in text and images. however, we build on works that study information-level inferences between discourse units in different modalities such as comic book panels @xcite , movie plots @xcite , and diagrammatic elements @xcite . in particular, we use @xcite 's relations that characterize inferences between text and images. coherence-aware models have benefited several nlp tasks such as gesture interpretation @xcite ), text summarization @xcite , machine comprehension @xcite . the majority of these works use rhetorical structure theory (rst) @xcite and penn discourse treebank (pdtb) @xcite datasets to learn and predict these relations between two adjacent text spans. in this line of work, we are the first to present a coherenceaware generation metric. the most widely used automatic evaluation metrics are ngram-based, which compute the exact number of ngram matches between reference and generated text @xcite . examples of such metrics that are commonly used for evaluating the output of captioning, translation and summarization models are bleu @xcite , rouge @xcite , and cider @xcite , . the major problem of the n-gram similarity metrics is that they give no credit to synonym matches of reference n-grams, even if those words are common and used appropriately in the generated text. embedding-based metrics such as bleurt @xcite and bertscore @xcite designed to address this limitation are closer to human ratings. bleurt is a data-intensive training scheme that is based on bert @xcite fine-tuned on human facade of a glass building. a pink flower bush in a garden. the underside of the arc de triomphe. close-up of a fly sitting on a daisy. man sitting by his artwork looking at a large statue of a man on a horse in a royal courtyard. woman with an umbrella reading a book sitting in the grass in front of a city skyline. cowboy on a horse and cowboy on the ground working together to lasso a calf in a pen. black and white artwork painted on a blue wall. ratings of generated text. bertscore, however, computes the similarity score as the average of cosine similarities between predicted tokens and their top matching reference tokens. these metrics however, do not respect the information goal and the purpose for which the model has generated the text. we address this problem by introducing the first coherence-aware generation metric. similar to spice @xcite and vifidel @xcite we use the information encoded in images. we further propose the addition of coherence relations that facilitate learning with fewer samples by a multimodal metric using pre-trained bert and vilbert. ## data collection we collect two datasets: human judgments for image captions that are generated by coherenceaware captioning systems using conceptual captions dataset; and ground-truth labels for the open images dataset. with conceptual captions corpora we fine-tune vilbert with ratings and show that addition of coherence relations can make automated scoring closer to human scoring. we use openimages corpora to reinforce that multimodality and coherence relations have significant contributions to scoring out-of-domain datasets, as well. protocol we hired two expert linguists for data annotation and designed an annotation website to facilitate the annotation procedure. they are native english speakers who identify themselves as of white and latino ethnicity. the code foot_0 of the annotation website, and the details of the protocol is publicly available. the study has been approved by our institution's human subject board. ## method the goal of a coherence-aware image captioning metric is to predict a score for the generated caption given the image, reference caption, and coherence relations of one generated caption and one reference caption. this metric function m can be formalized as predicting a score s as follows: ## experiments here, we describe the experimental setup to compare cosmic with other metrics. as outlined in section 3 and 4, we use the raccoon data to train our models, and coin to test cosmic and other metrics. we have several baseline metrics that we compare to, which can be found on ## results all n-gram based metrics, such as bleu and cider, fail to adapt to the out-of-domain groundtruth captions from coin. this results in a relatively flat distribution of system-level scores concentrated close to 0, and hence low correlation coefficients. cider has a highly negative kendall's τ , which denotes a strong negative association with human judgements. this is partly due to low (∼0.01) and hence noisy cider scores. (figure 4 provides example cases that illustrate this argument.) embedding-based methods, bleurt and bertscore, do not suffer from this limitation resulting in more meaningful scoring of systems and hence higher correlation with human scores. however, by design, both these metrics are agnostic to coherence-labels and the input image. cosmic, which is coherence-aware, obtains the highest correlation with human scores. cosmic vilbert has the highest kendall's correlation among all of our models. cosmic vanilla performs the second best among our models and it performs better than the rest of the models in terms of kendall's correlation. ## conclusion our work is the first step towards designing generation metrics that respect the information goal of the generated text. we observe that a small set of examples annotated with coherence relations can provide what is needed for learning a discourse-aware generation metric. our findings have implications for designing context-aware multimodal metrics with criteria that are closer to human ratings for evaluating machine-generated multimodal content. we have called attention to the challenge of learning robust generation metrics that can evaluate the output of the generation models considering the information goals. our findings suggest that fine-tuning vilbert-originally trained with millions of images-with a smaller sample of coherence relations and expert-annotated scoring, automated metrics can score generated captions closer to a human rating. the presented dataset provides the opportunity for future research in the area of image description generation, designing discourse-aware metrics, and multimodal content evaluation. we hope that coherence-aware text generation metrics could be used for learning better generation models (such as abstractive summarization or story generation) and could be deployed directly in machine learning pipelines to help in optimizing hyper-parameters. ultimately, it is intended to have a generalizable model that can use a labeling mechanism-not restricted to coherence labels-to improve applicability of generation metrics in different tasks.