GROOV i ST : A Metric for Grounding Objects in Visual Storytelling (2023)

Abstract: A proper evaluation of stories generated for a sequence of images—the task commonly referred to as visual storytelling—must consider multiple aspects, such as coherence, grammatical correctness, and visual grounding. In this work, we focus on evaluating the degree of grounding, that is, the extent to which a story is about the entities shown in the images. We analyze current metrics, both designed for this purpose and for general vision-text alignment. Given their observed shortcomings, we propose a novel evaluation tool, GROOViST, that accounts for cross-modal dependencies, temporal misalignments (the fact that the order in which entities appear in the story and the image sequence may not match), and human intuitions on visual grounding. An additional advantage of GROOViST is its modular design, where the contribution of each component can be assessed and interpreted individually.

## introduction generating a textual story that is plausible given a sequence of images is a challenging task involving aspects such as cross-modal interactions, temporal dependencies between linguistic and visual content, and causal reasoning. in the language-and-vision community, @xcite operationalized the task and released the visual storytelling dataset (vist), a collection of english stories created by speakers on top of 5-image visual sequences. several models have been proposed for the task of generating plausible stories for a given sequence, ranging from rnns @xcite to transformers, trained either end-to-end or leveraging additional knowledge-graphs @xcite . evaluating the quality of the automatically generated stories is extremely difficult: given the creative nature of the task (many stories could be sensible for a given image sequence), referencebased metrics like meteor @xcite or cider @xcite are not appropriate-they indeed poorly correlate with human judgments @xcite . moreover, a proper evaluation must consider multiple aspects, such as coherence, grammaticality and, importantly, visual grounding. yet, most evaluation metrics proposed specifically for visual storytelling do not consider the images at all @xcite . in this paper, we focus on evaluating a story's degree of grounding, that is, the extent to which a story is about the entities shown in the images. to the best of our knowledge, there is only one metric proposed to date for evaluating grounding in visual storytelling, the visual grounding scorer (rovist-vg) by @xcite . we carry out an extensive analysis of this metric and reveal that it has critical shortcomings. to overcome this, we propose a novel, modular evaluation tool, which we name groovist (grounding objects in visual storytelling). we show that groovist is robust to temporal misalignments, correlated with human intuitions about grounding, and easy to interpret. our code is available at: https://github.com/akskuchi/groovist ## analyses of existing metrics to assess the level of visual grounding of a story in visual storytelling, @xcite proposed rovist-vg. this metric is the output of a model pre-trained on the flickr30k entities dataset @xcite to learn the relationships between the nouns in a story and the regions of an image in a contrastive learning regime. for a @xmath0, @xmath1, rovist-vg extracts: from each image, the bounding boxes and corresponding visual features of its 10 most salient regions, using @xcite ; from the story, the glove @xcite representations of each noun in it. the pre-trained model receives these extracted embeddings (from glove and fasterrcnn) and returns the final representations t and i, respectively. the grounding score is then calculated using eq. ( 1 ) as the maximum cosine similarity between t and i, weighted by inverse document frequencies (idf ) of the nouns. 1 rovist to analyze the suitability of rovist-vg, we compare it to clipscore @xcite . clipscore has not been designed to evaluate visual storytelling. here, we use it to score each imagesentence pair independently in a story sequence. this approach is not ideal as it cannot capture temporal misalignments between a text and a visual content (e.g., an early sentence may be 'they were getting ready to go to the circus' but the circus may only appear later). however, since clipscore has been designed for general vision-text alignment, we expect it to be reasonably effective at capturing visual grounding at the image-sentence level. it corresponds to the cosine similarity between clip's @xcite representations of a sentence c and an image v (with 2.5 as re-scaling factor). next we explore how good the above metrics are at capturing grounding in visual storytelling data. ## groovist our analyses showed that rovist-vg has some important limitations as a metric for assessing the degree of visual grounding-both in stories and image captions. to overcome these issues, we propose groovist, a modular metric consisting of various components informed by insights from both clipscore and rovist-vg. these are: ## role of groovist components we test groovist on the same evaluation criteria used in section 2. from figure 2 and to understand the impact of groovist's components on the final grounding score, we conduct several experiments by both ablating the components and replacing them with plausible alternatives. ablations penalizing poorly grounded nps and concreteness weighting are the two components of groovist that can be ablated from the metric. ## evaluation of groovist to further evaluate the extent to which groo-vist captures intuitions on stories' degree of visual grounding, we compare our metric to human judgments. since no previous work collected human data for this specific purpose, we run a small data collection by asking 5 participants to rate a sample of the vist data. in particular, we ask participants to provide ratings for 100 randomly sampled @xmath2, @xmath3, using a 4point likert-like scale (instructions: "a score of 4 7 the resulting values are provided in appendix e. ## conclusion we proposed groovist, a novel reference-free metric for evaluating grounding in visual storytelling, an aspect that surprisingly is often overlooked in this task. we showed that existing metrics have serious shortcomings, and analyzed the strengths and limitations of our proposed metric. 9 human @xmath4. 10 the same results also hold when using mean and median. groovist is modular, highly interpretable, and aligned with human intuitions on visual grounding. preliminary results indicate that groovist is a suitable tool for evaluating automatically generated stories. we plan to test this aspect extensively in future work.