Not (yet) the whole story: Evaluating Visual Storytelling Requires More than Measuring Coherence, Grounding, and Repetition (2024)

Abstract: Visual storytelling consists in generating a natural language story given a temporally ordered sequence of images. This task is not only challenging for models, but also very difficult to evaluate with automatic metrics since there is no consensus about what makes a story ‘good’. In this paper, we introduce a novel method that measures story quality in terms of human likeness regarding three key aspects highlighted in previous work: visual grounding, coherence, and repetitiveness. We then use this method to evaluate the stories generated by several models, showing that the foundation model LLaVA obtains the best result, but only slightly so compared to TAPM, a 50-times smaller visual storytelling model. Upgrading the visual and language components of TAPM results in a model that yields competitive performance with a relatively low number of parameters. Finally, we carry out a human evaluation study, whose results suggest that a ‘good’ story may require more than a human-like level of visual grounding, coherence, and repetition.

## introduction visual storytelling is the task of generating a story for a sequence of several temporally-ordered images or video frames. for both human speakers and machine learning models, the task requires connecting the visual data causally, to generate a narrative consistent with the contents of the images. as for model-generated stories, evaluation is one of the key challenges due to the inherently creative nature of the task. since human-written stories are typically used to train visual storytelling modelsunder the assumption that these stories provide a good learning signal-most previous work evaluated model-generated stories by directly comparing them to human ones using pattern-matching metrics. however, this approach is simplistic as it ignores several key aspects of visual stories, such as their degree of visual grounding, their overall coherence, or how repetitive they are. this problem has only been addressed recently, with @xcite and @xcite proposing various metrics to take into account some of these crucial aspects. these methods assess the appropriateness of a generated story independently from its overlap with a ground-truth story for the same image sequence. given that the same image sequence can possibly give rise to many different stories, this type of higher-level evaluation that does not rely on text overlap is clearly desirable. nevertheless, we argue that measuring the degree of coherence or visual grounding of a story may not be sufficiently informative, as there are no standard conventions that determine the preferable level of such properties. to address this issue, in this work, we first propose an evaluation method that assesses the quality of generated stories in terms of their distance from human-written stories along several relevant dimensions, each measured by an available reference-free metric. using this method, we evaluate a range of models on the visual storytelling task, including models specifically designed and trained for this task, as well as-for the first time-foundation models pre-trained to achieve general-purpose language and vision abilities, which we test in a zero-shot manner. we show that llava @xcite , a powerful foundation model, performs best on the task, but only slightly better than tapm @xcite , a model designed for visual storytelling which is 50 times smaller than llava. second, given insights derived from our proposed distance-based evaluation method, we upgrade the visual and language components on tapm, resulting in a model that achieves comparable performance to llava with a significantly lower number of parameters. our results show that the stories generated by llava and the upgraded tapm model are very close to human stories regarding their degree of visual grounding, coherence, and repetition. to further make sense of this finding, we collect human judgments with regards to comparing human and model stories. the results of this qualitative study indicate that humans tend to prefer humanwritten stories by a significant margin despite the quantitative closeness we observe. in sum, we make the following contributions: ## problem formulation in this work, we take a human-centric approach and define the quality of generated stories in terms of their 'closeness' to stories produced by humans, regarding different dimensions that capture abstract properties of interest. concretely, we compute dimension-specific scores for model-and for human-generated stories, measure human-model distance per dimension, and aggregate these distances to derive an overall distance score. we posit that the lower the overall distance, the more the generated story complies with high-level features observed in human stories. deciding on which dimensions are most relevant to determine the quality of a story, let alone how to operationalize such dimensions, is not trivial. in this study, we consider the three aspects proposed by @xcite : visual grounding, coherence, and repetition. here we describe how they are operationalized as reference-free metrics, following which we formally define our proposed method. ## model analysis and improvements in section 4, we showed that the d hm obtained by the visual storytelling-specific tapm model is only slightly higher than that of llava, a 50times larger foundation model. in particular, we notice that llava has an advantage over tapm in two dimensions, i.e., visual grounding and coherence. we hypothesize that this advantage is due to the model's better language and vision backbone models-llava builds on a pre-trained, transformer-based language model and image processor. thus, we leverage the modular architecture of tapm and test whether we can obtain better results (lower distances) by replacing its original language and vision components with models similar to those embedded in llava, while keeping the number of parameters significantly lower. to test whether the results we obtain are consistent across datasets, we perform this analysis on both vist and vwp @xcite . ## qualitative analysis and discussion our results suggest that the stories generated by the best-performing models according to d hm are very close to human levels of visual grounding, coherence, and degree of repetition. to test whether this aligns with the perceived overall quality of the stories, we conduct a qualitative human evaluation. we consider the vist stories generated by the two models-tapm (+llama 2) and llavathat achieve the best performance in terms of d hm (lowest distances). for each model, we select 100 generated stories, that we randomly sample using the distribution of d hm values on the vist test set. 4 we then provide annotators @xmath0, model-@xmath1, and ask them to assess whether one is better than the other, or whether both are similarly fine or bad. five annotators unrelated to the project participated in the task. 5 figure 5 shows the judgments obtained from the annotators for the two models. we observe that yet, preference for human-written stories suggests that a good story involves more than just visual grounding, coherence, and limited repetition. to get more insight into this issue, we asked the annotators in the study to describe the properties they considered when comparing stories. some annotators reported that they employed their subjective world knowledge in differentiating between creative and hallucinated stories. also, stories that describe events without an overarching narrative were judged as bad despite being locally coherent, well-grounded, and non-repetitive. in addition, annotators generally preferred stories that contained phrases or sentences expressing emotions. we provide examples of these cases in appendix c, which illustrate the limitations of current evaluation metrics in fully capturing hallucinations, relevant emotions, and creative expressions in the generated stories. therefore, there is scope for a lot of further work in this evaluation domain. ## conclusion we proposed a novel human-centric method (d hm ) for evaluating the quality of model-generated stories in terms of their closeness to human-written stories along coherence, visual grounding, and nonredundancy. using our proposed method, we compared various models and found that the large foundation model llava obtains slightly better results than the (50 times smaller) best visual storytelling model tapm. we showed that upgrading the components of tapm boosts its performance (lowers its d hm ) across multiple datasets, confirming the advantage of leveraging last-generation language and vision pre-trained models. finally, we conducted a qualitative human evaluation to explore whether these quantitative findings align with the overall perceived story quality. we observed that human judgments align with our quantitative evaluation. yet, humans still prefer human-written stories over model-generated ones, which suggests that capturing coherence, visual grounding, and non-repetitiveness may not (yet) be the whole story.