SCO - VIST : Social Interaction Commonsense Knowledge-based Visual Storytelling (2024)

Abstract: Visual storytelling aims to automatically generate a coherent story based on a given image sequence. Unlike tasks like image captioning, visual stories should contain factual descriptions, worldviews, and human social commonsense to put disjointed elements together to form a coherent and engaging human-writeable story. However, most models mainly focus on applying factual information and using taxonomic/lexical external knowledge when attempting to create stories. This paper introduces SCO-VIST, a framework representing the image sequence as a graph with objects and relations that includes human action motivation and its social interaction commonsense knowledge. SCO-VIST then takes this graph representing plot points and creates bridges between plot points with semantic and occurrence-based edge weights. This weighted story graph produces the storyline in a sequence of events using Floyd-Warshall’s algorithm. Our proposed framework produces stories superior across multiple metrics in terms of visual grounding, coherence, diversity, and humanness, per both automatic and human evaluations.

## introduction beyond interpreting the factual content of scenes with expressions, like image captioning, visual storytelling (vst) aims to conduct a human-like understanding of the idea of a sequence of images and generate more complicated visual scenarios with human-like textual expressions @xcite . in order to achieve this aim, the ai agent is required to model relationships between the images while remaining visually grounded, identify concepts that are implied (but not explicitly shown) in the images, as well as generate coherent, conversational language resembling how a human would tell a story in a social setting. numerous past studies have employed encoderdecoder frameworks that first utilise a computer vision algorithm to extract image-specific features, which are then fed into a language generation model to decode the story @xcite @xcite . although these methods can yield reasonable stories to some extent, they often lack common sense reasoning, thus producing stories that are "generic" sounding with limited vocabulary, and irrelevant to the images. to alleviate these issues, more recent approaches adopt content planning methods that try to explicitly predict textual concepts from the images via detecting objects in the image by using external knowledge data sources to identify implicitly related concepts @xcite @xcite . those external knowledge data sources mainly comprise taxonomic, lexical and physical relations, whereas human-like storytelling tends to use the social-aspect relations of everyday human experiences. social-interaction relations comment on socially-triggered states and behaviours. it is crucial to gauge people's intentions and purpose and predict situationally-relevant human reactions and behaviours, which is directly aligned with the aim of human-like storytelling. this paper proposes a new social-interaction commonsense-enhanced vst framework, sco-vist, for producing human-like stories by interpreting socially-triggered situations and reactions. we introduce a three-stage commonsense enhanced framework that attempts to construct a reasonable plot of story events from the given image stream for story decoding. stage 1 focuses on constructing a story graph representing causal and logical relationships between social interactions and events. motivated by the idea that captions may already have embedded social commonsense within them, we first generate a caption for each image to literally capture the event depicted in the photo. ad-ditionally, we further extract commonsense from external data related to social situations, interactions and behavioural responses (i.e. character's intentions, desires or needs). each extracted caption and commonsense is thus considered a different event or plot point, and we connect the plot points (nodes) with causal ordering. in stage 2, we convert the story graph to be weighted by conducting a comprehensive analysis on different edge weight assignment methods based on semantic similarity between nodes and graph learning. intuitively, this weighted story graph reflects the branching space of plausible event continuations where the edge weights indicate the likelihood of transition between connected plot points. given the weighted story graph, the optimal storyline is the path of nodes that yields the largest sum of weights from the left to the right-most nodes in the graph. therefore, stage 3 negates the edge weights and employs floyd-warshall's shortest path search algorithm to extract the optimal sequence of story events which is later fed into a transformer for story generation. the main contributions of this research are: ## related work earliest works on vst consist of an encoderdecoder structure incorporated in an end-to-end model @xcite @xcite . recently, there has been increasing interest in reinforcement learning architectures which include a reward model to evaluate the generated stories @xcite . however, the training process of such methods are inherently unstable. other approaches first translate images to semantic scene graphs to capture image features and then employ graph convolutional networks (gcn) to enrich regions and object representations @xcite @xcite . instead, we use literal text descriptions of images which can better explicitly represent the image contents. to promote more diverse stories, newer works have also used knowledge graphs to assist the storytelling process, allowing for richer stories capable of expressing imaginative concepts that are not explicitly shown in the image scene. most of these methods involve querying conceptnet @xcite with detected image objects or predicted key image concepts to find a set of related candidate concepts @xcite @xcite . while these methods show promising improvements in outputs, conceptnet mainly comprises of taxonomic and physical relations, whereas our framework leverages commonsense that are more social-interaction focused and event-centred. finally, most related to our work, recent studies try to form the story plot by first using external knowledge to connect concepts between images to reason about potential temporal relationships @xcite @xcite . however, these methods often employ complex network architectures to iteratively predict subsequent events. we alleviate these complexities and present a simple yet effective approach for storyline construction. ## evaluation setup 2 data the visual storytelling dataset (vist) @xcite consists of 210,819 unique images obtained from flickr albums. the dataset is split into training/validation/testing with 8,031/998/1,011 albums where each album contains a set of similar image sequences with each sequence made up of 5 photos. each album also has 5 human written stories where each story is usually comprised of one sentence per image. the unique number of stories in the training, validation and testing set is 40,155, 4,990 and 5,055 respectively. ## conclusion in this paper, we presented sco-vist, a multistage novel framework for visual storytelling that utilises social-interaction knowledge for enhancing commonsense reasoning in stories. we design a heterogeneous story graph with causal ordering that connects captions and commonsense extracted from external sources and employ shortest path algorithms to find the optimal storyline for story generation. extensive experiments on the benchmark dataset, analysis of automatic metrics and human evaluations demonstrate that sco-vist outperforms existing baselines and is capable of generating diverse stories that are highly coherent with strong visual grounding.