Context-aware Visual Storytelling with Visual Prefix Tuning and Contrastive Learning (2024)

Abstract: Visual storytelling systems generate multi-sentence stories from image sequences. In this task, capturing contextual information and bridging visual variation bring additional challenges. We propose a simple yet effective framework that leverages the generalization capabilities of pretrained foundation models, only training a lightweight vision-language mapping network to connect modalities, while incorporating context to enhance coherence. We introduce a multimodal contrastive objective that also improves visual relevance and story informativeness. Extensive experimental results, across both automatic metrics and human evaluations, demonstrate that the stories generated by our framework are diverse, coherent, informative, and interesting.

## introduction visual storytelling (vist; @xcite aims at crafting a narrative from a sequence of ordered images. this task involves a number of key challenges, some of which are well-studied problems in computational narrative generation, while others arise from the visually grounded nature of the task: vist image sequences exhibit semantic and temporal gaps, so that (i) a successful vist system needs to balance textual coherence @xcite with (ii) visual grounding @xcite . at the same time, (iii) generated narratives should capture the reader's attention, necessitating a degree of creativity and interestingness @xcite , but should also (iv) be informative @xcite , that is, incorporate relevant details of the entities and activities in the visual content. existing models usually include a vision encoder and language decoder either trained from scratch or finetuned @xcite @xcite @xcite @xcite on the vist task. this requires a large amount of computational resources. instead, we propose to benefit from pre-trained models that have already learned meaningful representations from vast amounts of data, following the clipcap approach @xcite that integrates pretrained clip @xcite and gpt2 @xcite via a lightweight mapping network. clipcap trains only the mapping network to construct soft visual prefixes from clip embeddings to guide gpt2 to generate text, while both clip and gpt2 can be kept frozen. although visual prefix tuning has been widely used for image captioning, it has not been adapted for visual storytelling, and its potential here is yet to be explored. our new framework incorporates a contextaware mappping network, while addressing coherence by incorporating previous story sentences. to enhance visual grounding and informativeness, we employ a multimodal training objective. we further compare four common decoding strategies (beam, top-k, nucleus and contrastive search), showing that they have substantial impact on the generation quality, especially as reflected in human evaluation, in contrast to standard metrics. the main contributions of this work are: 1 recent research leverages transformer-based architectures to learn multimodal feature embeddings, integrating image regions with semantic relationships @xcite . several studies have focused on utilizing pre-trained models for visual storytelling, either by fine-tuning pre-trained transformer encoders @xcite , or jointly tuning pre-trained lms with pre-trained image encoders @xcite . other variants consider additional factors such as emotion/sentiment @xcite , personas @xcite @xcite , and writing style @xcite . unlike prior work, our approach efficiently adapts frozen vlms and llms, conditioning on both textual context and visual input to ensure story continuity and coherence. prompt and prefix tuning. prompting means designing "instructions" for pretrained language models (lm) to generate desired outputs, conditioning them on either human-crafted templates or automatically optimized tokens @xcite . much research proposes to automate prompt engineering by learning discrete @xcite @xcite or continuous prompts @xcite . the latter can be updated via backpropagation, making them less constrained than @xcite . with large frozen lms, prompt tuning @xcite simply adds a tunable, real-valued embedding to the input of the decoder, achieving results comparable to full model fine-tuning. on the other hand, prefix tuning @xcite optimizes the inputs of every attention layer in the pretrained lms. constructing soft visual prompts for a frozen llm is an effective way to achieve visionlanguage alignment @xcite . flamingo @xcite adds cross-attention layers to the llm for incorporating visual features, pretrained on billions of image-text pairs. blip-2 @xcite adopts a q-former module to link a frozen image encoder to a frozen llm, learning visual features relevant to text. llava @xcite , trained on multimodal instruction-following, uses a linear layer to map image features from pre-trained clip to the word embedding space of vicuna @xcite . inspired by the widespread application of visual prefix tuning in v&l tasks, we explore its potential in visual storytelling while also considering the context when tuning the prefix. ## method in visual storytelling, the input is a sequence of n images @xmath0, . . . , i n }, where @xmath1. our model aims to generate a multi-sentence story s by predicting the probability p (s|i). in this section, we introduce a visual storytelling pipeline enhanced with prefix tuning ( §3.1), then describe the context-aware components ( §3.2), curriculum training ( §3.3) and finally the contrastive learning loss involved ( §3.4). figure 1 illustrates an overview of our framework. ## conclusion we present a simple yet effective framework for visual storytelling that utilizes pretrained multimodal models with a lightweight vision-language mapping network to construct prefixes for llms. our model enhances the coherence of multisentence stories by integrating contextual information. in addition to teacher-forcing loss, we use a curriculum training scheme and image-text contrastive loss to enhance the concretness and visual grounding of generated stories. extensive evaluation on the vist benchmark using both automatic metrics and human assessment shows that our model obtains strong results compared to sota methods. we empirically confirm that our model demonstrates the ability to generate coherent stories that are closely tied to visual content, and possess more creative and engaging details with minimal degeneration. our study contributes to improved evaluation practices in text generation, recommending a specific human evaluation setup for visual storytelling that assesses four key output qualities. such evaluation enables informative model comparisons and better insight into the relative strengths of different systems. results show that automatic metrics, particularly token overlap measures like bleu, often poorly correspond to human judgments and should not be fully trusted for open-ended tasks like visual storytelling. this echoes similar observations made in other nlg domains @xcite @xcite . limitations. despite having employed diverse automatic metrics and comprehensive human evaluations to assess our models' generated stories, we recognize substantial opportunity for enhancing the evaluation methodology of visual storytelling. as discussed above, correlating with ground-truth text or grounding to the visual content represents just a one-sided view, which downplays the role of diversity and creativity in storytelling. while our proposed human evaluation aims for thorough assessment, human annotation is costly and cannot be continuously applied during model development. future research could explore the balance in visual storytelling between factuality and groundedness on the one hand, and justified deviation from the images in the interest of creativity on the other. additionally, our model exhibits certain biases, such as producing wedding-related stories from images of churches, even though there are no wedding-related elements in the images. this may stem from the biases in vist dataset or the pretraining data of clip and gpt2. lastly, this study primarily investigates the utility and performance of two specific pre-trained models, clip and gpt-2. while these models have demonstrated broad applicability and strong performance across various tasks, they represent only a subset of the rapidly evolving landscape of pre-trained vision an language models. future work could benefit from incorporating a wider array of models, such as blip-2 @xcite , llava @xcite , llama 3 (meta ai, 2024) and mistral @xcite , to provide a more comprehensive understanding of the strengths and limitations inherent to different foundation models.