R o V i ST : Learning Robust Metrics for Visual Storytelling (2022)

Abstract: Visual storytelling (VST) is the task of generating a story paragraph that describes a given image sequence. Most existing storytelling approaches have evaluated their models using traditional natural language generation metrics like BLEU or CIDEr. However, such metrics based on n -gram matching tend to have poor correlation with human evaluation scores and do not explicitly consider other criteria necessary for storytelling such as sentence structure or topic coherence. Moreover, a single score is not enough to assess a story as it does not inform us about what specific errors were made by the model. In this paper, we propose 3 evaluation metrics sets that analyses which aspects we would look for in a good story: 1) visual grounding, 2) coherence, and 3) non-redundancy. We measure the reliability of our metric sets by analysing its correlation with human judgement scores on a sample of machine stories obtained from 4 state-of-the-arts models trained on the Visual Storytelling Dataset (VIST). Our metric sets outperforms other metrics on human correlation, and could be served as a learning based evaluation metric set that is complementary to existing rule-based metrics.

## introduction visual storytelling (vst) is a natural language generation (nlg) task that aims to automatically generate a cohesive story given a sequence of images @xcite . the task is fundamental to the development of intelligent agents capable of understanding complex visual scenarios, and can be further applied to assist the visually impaired in understanding images on the web. recently, progress has been made on designing network architectures to accomplish the vst task but little work has been done to explore new metrics that automatically evaluate and quantify the errors produced by these systems. as to date, a majority of the past works on vst have used existing popular n-gram based metrics such as bleu, meteor, rouge, cider, and spice to evaluate their models @xcite @xcite . however, it is known that such metrics are unreliable for vst. figure 1 shows two machine generated stories for a photo sequence and their corresponding n-gram matching based metrics (bleu, cider, meteor, rouge-l and spice). evidently, the first candidate story is more repetitive and lacks a narrative style but achieves higher scores across a majority of the ngram based metrics in figure 1 . the second story however, has greater word diversity and is more expressive through its use of phrases like 'completely in disrepair'. relevant words like 'trip', 'countryside' and 'hills' are also used but are not rewarded since they are not mentioned in the gold story. the low level of agreement between human judgement and current automatic metrics may be because such metrics were originally developed to assess machine translation, summarization and image captioning tasks @xcite , which are significantly different problems to vst. specifically, vst is a multimodal task that firstly requires: 1) generating text relevant to the image content but unlike image captioning, there is less emphasis on describing relationships between objects and may contain concepts that are inferred from the image. it additionally needs to ensure that: 2) the story must be topically coherent, similar to how a human would tell a story in a social setting. sentences should not sound disjointed e.g. 'we went to the park. i grew up in sydney'. and finally 3) avoids repetition which appears to be a common issue in current vst models. for instance, candidate story 1 in figure 1 exhibits inter-sentence repetition between the first sentence and last sentence. we also find that some output stories may contain repetition within sentences (i.e. intra-sentence repetition) e.g. 'we had a good time and had a great time!'. moreover, it is noted that open-ended text generation tasks usually suffer from the one-to-many issue, whereby there are multiple plausible outputs for the same input which are not fully reflected in the reference sentences @xcite . this issue is even more prominent in the vst task as different individuals may tell significantly different stories and have diverse interpretations given the same image sequence. all these issues suggest that we require evaluation metrics that do not simply rely on comparison with reference sentences. in addition, given that the vst task requires several aspects, one single metric is not sufficient to evaluate a story and there is a need to design multiple interpretable metrics that each target a specific vst criteria. hence, in this paper, we propose several unreferenced metrics for the vst task based on the three aforementioned criteria: 1) visual grounding, 2) coherence, and 3) non-redundancy. to address criteria 1), we propose a learned metric to calculate relevance scores between nouns in the vst sentences with the bounding box regions in the images. we decide to focus on nouns as they provide the most visual information. other words like adjectives and adverbs are difficult to ground and such words may differ significantly depending on the person writing the story. the second criteria which is story coherence requires that consecutive sentences flow and that each sentence is not just an isolated description of the image. existing methods for measuring coherence have used next sentence prediction (nsp) to find the probability that a sentence comes after a preceding sentence @xcite . inspired by this method, we fine-tune the albert @xcite model on story sentences and build a sentence-order prediction (sop) model. finally, to address criteria 3), we propose an additional metric to explicitly measure intersentence and intra-sentence repetition. the contributions are summarized as follows: 1) we propose an interpretable and reference-free metric that addresses 3 criteria required for vstvisual grounding, coherence and non-redundancy. 2) we conduct human evaluation studies to assess a sample of machine generated stories obtained from 4 state-of-the arts vst models. 3) we test the effectiveness of our proposed metrics by analyzing its correlation with human scores and show that our metrics outperform other existing metrics that are commonly used for vst and nlg tasks. visual grounding metrics past studies have proposed examining the images in addition to human written references. @xcite trained a binary classifier to discriminate between human and machine captions using image and text representations obtained from a cnn and rnn. tiger @xcite employs the pretrained scan model @xcite to calculate the text-toimage grounding scores and compares the relevance ranking and grounding weights distribution among image regions between the references and the candidate. @xcite later introduced vilbertscore which uses the same approach as bertscore but utilizes the vilbert model @xcite to retrieve image-conditioned token embeddings. however, we note that these methods are initially designed for evaluating image captioning systems. hence, while they do consider the text-to-image similarity aspect, they do not explicitly address the extra criteria required for vst such as story coherence. moreover, such metrics still rely on reference sentences to some extent. story generation metrics language models like bert @xcite trained with nsp and masked language modelling tasks can identify appropriate use of words and sentences and hence, may show promising results when applied to evaluating open-ended text generation. @xcite proposed union, an unreferenced metric for scoring machine generated stories. they leverage a bert model trained with negative samples created by perturbing ground truth stories and predicts a score representing how human-like a story is. they showed the effectiveness of bert in identifying stories with conflicting logic, repeated plots and incoherence. however, union purely evaluates the output text and cannot be applied to analyse the text-to-image relatedness required for the vst multimodal task. additionally, a single score is outputted which is not informative enough to gauge what specific errors were made by the model. moving to vst, @xcite designed reward functions to capture story quality for vst models that use a reinforcement learning framework based on 3 criteria: image relevance, coherence and expressiveness. image relevance is measured by ngram precision of entities between candidate and reference sentences, coherence through bert's nsp task, and word diversity by computing bleu scores between generated sentences. inspired by this, we also analyze story quality from 3 similar perspectives 1) visual grounding, 2) coherence, and 3) non-redundancy. we attempt to extend the methods of @xcite , provide a reference-free approach and conduct a comprehensive analysis with human evaluation. ## method we describe our proposed metric in detail. given a machine story, we aim to output 3 scores that explicitly evaluates the story based on 1) visual grounding, 2) coherence, and 3) non-redundancy. ## evaluation setup 2 evaluation metrics to assess the performance for rovist, we analyze its correlation with reliable human judgements by recruiting many responders (26) whereas related works @xcite have used 3-7 annotators. in total, the 26 responders analysed 400 machine generated sentences across 80 stories and 4 models, including arel, glacnet, kg-story and mcsm+bart. a likert scale was used to score 3 different criteria for each story based on what we believe defines a good story -1) the story is visually grounded, 2) sentences are natural sounding and topically coherent, and 3) there is no repeating plots within the story. annotators were additionally asked to vote for which of the 4 models produced the best story relating to the visual prompt based on no particular criteria. we follow existing literature and report the spearman's correlation ρ, pearson's correlation r and kendall's correlation τ . baseline we select 11 baseline metrics to compare with our metric: bleu-1,2,3,4 @xcite , rouge-l @xcite , meteor 2 the implementation details can be found in the appendix figure 3 : average human scores for an example story across 3 criteria for 4 different vst models. 'proportion of votes' refers to the percentage of voters who voted that model's story as the best out of the 4. blue highlighted words visually relate to the image. @xcite , cider @xcite , spice @xcite , wmd @xcite , f bert (f1-measure version of bertscore) @xcite and tiger @xcite . ## conclusion we propose rovist, a metric for evaluating vst tasks on 3 aspects: visual grounding, coherence and non-redundancy. rovist correlates well with human judgement, outperforming other metrics when comparing the coherence and non-redundancy criteria as well as when combining all 3 criteria. while some existing metrics slightly outperform our method on visual grounding, we note that image-to-text similarity is just one aspect of vst and this aspect alone is insufficient in defining a good story. unlike other metrics, rovist is reference-free and hence, robust to the number of references which are costly to obtain for vst. it is also interpretable and can be used to gauge out where the model is underperforming. we hope that rovist provides preliminary insight into future work on developing vst models and evaluations.