Visual Coherence Loss for Coherent and Visually Grounded Story Generation (2023)

Abstract: Local coherence is essential for long-form text generation models. We identify two important aspects of local coherence within the visual storytelling task: (1) the model needs to represent re-occurrences of characters within the image sequence in order to mention them correctly in the story; (2) character representations should enable us to find instances of the same characters and distinguish different characters. In this paper, we propose a loss function inspired by a linguistic theory of coherence for self-supervised learning for image sequence representations. We further propose combining features from an object and a face detector to construct stronger character features. To evaluate input-output relevance that current reference-based metrics don’t measure, we propose a character matching metric to check whether the models generate referring expressions correctly for characters in input image sequences. Experiments on a visual story generation dataset show that our proposed features and loss function are effective for generating more coherent and visually grounded stories.

## introduction stories play an important role in natural language understanding and generation because they are the key to the human understanding of the world @xcite . automatically generating a coherent and interesting story is a complex task requiring various capabilities such as language processing, event comprehension, and world knowledge. in this paper, we focus on the visual storytelling task of generating stories from visual narratives, i.e., a sequence of images with a plot @xcite . stories, unlike image captions, contain several characters and events involving recurrent characters and their interactions with each other and the environment. especially, characters are among the most important aspects of story writing @xcite . however, the current state-of-the-art (sota) visual storytelling models often fail to generate correct referring expressions for characters @xcite . to confirm this, we generate 50 stories using a sota model (tapm; @xcite and annotate different types of errors foot_0 . the most prevalent error that appears in 60% of the generated stories is lacking local coherence which manifests as the characters appearing in the image not being mentioned correctly in the stories. for instance, in the first two rows of figure 1 , the character jeremy appears across all images. the sota model tapm first mentions him as jeremy in the first sentence but misses him in the second sentence and inconsistently misnames him as adam in the last sentence. we identify two main causes of this problem: 1) most previous models don't represent the recurrence of characters in the image sequence explicitly. in particular, it is not clear whether the parameters in the model can capture character recurrence such that the visual coherence, local coherence in image sequences, can be reflected in textual coherence by repeated mentions of the recurrent characters in the story. we argue that a model that captures and represents the visual coherence better would also lead to more coherent stories. our contributions for tackling these limitations: ## related work visually-grounded story generation. most of the previous work on visual story generation is based on the visual storytelling dataset (vist, @xcite . they only use global features, full image features extracted with a general vision backbone model trained on the image classification task @xcite @xcite . some recent researches use local features, which are features of a specific part of the image such as objects, to generate visually grounded stories @xcite @xcite @xcite . only a few works make use of humanrelated features like emotions @xcite , sentiments @xcite or persona @xcite . one work that focuses on recurrences of characters is @xcite , which employs a bi-lstm to encode conference chains in texts. however, their model requires textual features from the corresponding conference chains as input in the test time. on the contrary, our model only requires visual features from the images. parallel to our research, @xcite annotate recurring characters in the vist dataset which enable further exploration of grounding human characters. ## image-text relevance metric in previous visually-grounded story generation work, models are evaluated with reference-based metrics like meteor. however, the referencebased metrics only measure similarities between reference texts and generated stories such that the actual relevance of input images to output text is not evaluated. specifically, we design a charactermatching metric (cm) to determine whether expressions referring to human characters in image sequences are generated correctly. the computation process of cm is in figure 2 . for each pair of image sequence and story, we first construct the appearance matrices @xmath0, -1} m×nv and @xmath1, -1} m×nt , which indicate for each character whether it is present (1) vs. absent (-1) in an image or a sentence, where n v is the number of characters in the image sequence, n t is the number of referring expressions in generated stories and m is the number of images/texts. we then compute the matrix of matching scores of each pair as: @xmath2. we normalize the matching scores to the domain [0, 1] and obtain the matrix m ′ = (m/m + 1)/2. to obtain the optimal match between the images and sentences by the occurrences of characters, we compute the maximum weighted bipartite matching by applying the hopcroft-karp algorithm (hopcroft and karp, 1973) implemented in scipy @xcite . the final overall character matching score is obtained by averaging the scores of the maximum matches across all characters (e.g. 0.9 in figure 2 ). ## experiment and evaluation in this section, we experiment with the visual writing prompt (vwp; anonymous, 2023) dataset to demonstrate the effectiveness of our loss function (see section 3.1). we follow their settings to separate the data into train, validation and test split. we extract global features for all images using the swin transformer @xcite , a state-of-the-art supervised vision backbone model. we use the base model pre-trained on the imagenet-21k dataset released on hugging face models. for local features, object features are obtained using a cascade mask r-cnn object detector @xcite with the same swin transformer backbone. we crop the bounding boxes of the top 20 objects that the detector predicts for each image. then we extract the object features (obj) similarly as how we extract the global features. we also construct the oracle (oracle) features by passing the ids of the characters to the language decoder. we encode the character ids via the positions of the feature vectors. for example, character #1 takes the 1st position of the character feature matrix. because the positional encodings in the language decoder are sensitive to the positions, the character ids are also encoded in the input. we use gpt-2 @xcite , a transformer-based language model pre-trained on large-scale text as the language decoder. we use the small version which is widely used in previous works of story generation. we compare baseline models using the visual coherence loss combined with different character features against the two baseline models defined in section 3.1. ## character representation another possible cause of local incoherence is that previous models mostly consider only the global or local visual features without a specific focus on human characters. the only local features that are related to human characters are object features, i.e., the bounding boxes or masks around the person objects from the object detector. object features are effective and widely used in visual story generation. however, these features are either extracted using a general vision backbone model trained on the imagenet dataset or directly from the output of the last hidden layer of the object detector. these features do not represent human age, gender, face, or body shape. these features can thus not distinguish between different instances of the same human character, which introduces wrong mentions of characters in generated stories. ## conclusions we identified a major important problem of current vision-based story generation models, lack of local coherence (characters are not mentioned correctly). we dissect this problem into two aspects: 1) whether the latent visual representations fed to the language decoder are sensitive to the coherent structure in the visual narratives; 2) whether the visual features can distinguish different characters or identify instances of the same character. we then propose a visual coherence loss to constrain the latent visual representations together with the parameters in the language decoder such that they can represent recurrences of characters. our model trained with visual coherence loss generates more coherent stories and obtains superior performance on both automatic metrics and human judgments. we also find that using the visual features of characters allows us to produce stories with better identification of recurring characters, which are as a result more locally coherent.