Visual Storytelling with Question-Answer Plans (2023)

Abstract: Visual storytelling aims to generate compelling narratives from image sequences. Existing models often focus on enhancing the representation of the image sequence, e.g., with external knowledge sources or advanced graph structures. Despite recent progress, the stories are often repetitive, illogical, and lacking in detail. To mitigate these issues, we present a novel framework which integrates visual representations with pretrained language models and planning. Our model translates the image sequence into a visual prefix, a sequence of continuous embeddings which language models can interpret. It also leverages a sequence of question-answer pairs as a blueprint plan for selecting salient visual concepts and determining how they should be assembled into a narrative. Automatic and human evaluation on the VIST benchmark demonstrates that blueprint-based models generate stories that are more coherent, interesting, and natural compared to competitive baselines and state-of-the-art systems.

## introduction visual storytelling involves narrating an engaging and logically coherent story based on a sequence of images (see the example in figure 1 ). the task lies at the intersection of natural language processing and computer vision and has recently attracted increasing interest from both communities @xcite @xcite @xcite @xcite . visual storytelling differs from image captioning, which typically focuses on generating descriptive text, e.g., by identifying and depicting objects within an image. it requires a deeper understanding of how images and the events they illustrate relate to each other in order to create a convincing narrative. visual storytelling is commonly modeled as a two-stage process. the image sequence is first a small indie rock group performed for a music store in our city. the girls doing harmonies in the songs were phenomenal. the band had three guitarists. one of the guitarists resembled james morrison. more independent bands played for the grand opening of the music store. q 0 : what group performed for a music store in our city? a 0 : a small indie rock group q 1 : what kind of music did the girls do? a 1 : harmonies q 2 : who did the harmonies in the songs? a 2 : the girls q 3 : how many guitarists did the group have? a 3 : three guitarists q 4 : who resembled james morrison? a 4 : one of the guitarists q 5 : who played for the grand opening of the music store? a 5 : more independent bands encoded into a representation which typically includes image embeddings and detected objects. subsequently, a decoder generates a story token by token based on the encoding of the image sequence. recent work has mainly focused on enhancing the first stage of the generation process e.g., by leveraging external knowledge sources @xcite @xcite . advanced representations for image sequences have also been explored, such as scene graphs @xcite and story graphs @xcite . despite recent progress, these methods struggle to produce meaningful narratives, are prone to hallucination and repetition, often generate vague sentences, and have difficulty identifying salient visual concepts. we attribute the lack of story quality to at least two reasons. previous work on text-based genera-tion has demonstrated that planning can improve story coherence, allowing to control the trajectory of events, the characters described and their actions @xcite @xcite @xcite . however, planning has not been considered in the context of visual storytelling, existing models adopt black-box architectures which are not particularly controllable or interpretable. another limitation concerns the nature of current models which are essentially trained from scratch, and as a result have limited language modelling and generalization capabilities (they only see multimodal training samples; see top of figure 1 ). although pretrained language models @xcite @xcite have been widely adopted for generalpurpose story generation, their potential for visual storytelling remains unexplored. in this work we propose an approach to visual storytelling which integrates pretrained language models with visual representations and incorporates an intermediate planning step before generating the full story. our encoder translates the image sequence into a visual prefix, a sequence of continuous embeddings which language models can interpret. following @xcite , we represent plans as a sequence of question-answer pairs, called blueprints, which serve as a proxy for content selection (i.e., what to say) and planning (i.e., in what order). blueprints are loosely related to the question-under-discussion (qud) theory of discourse @xcite @xcite , which posits that text structure can be analyzed by identifying implicit questions raised and answered by subsequent spans of text. we augment visual storytelling training data with story blueprints (see figure 1 ), which we obtain automatically thanks to state-of-the-art question generation technology. we fine-tune pretrained language models to generate blueprints from image sequences and the stories based on them. we showcase two types of storytelling models, which vary in how the planning mechanism is implemented. a top-down model generates the blueprint first and then continues to generate the corresponding story in one go, whereas an integrated model interleaves planning with text generation rather than determining a plan in advance; generation is iteratively conditioned on the image input, the blueprint and the story gener-ated so far. experiments on the vist benchmark @xcite show that blueprint-based models generate more coherent, interesting, and humanlike stories compared to the state of the art and large language models (llms) like gpt-3.5, according to automatic and human evaluation. most existing approaches @xcite @xcite train transformer models from scratch, with the exception of @xcite , who employ a vanilla bart model as a baseline without task-specific adaptation. in contrast, our work leverages the language modeling and generalization capabilities of pretrained language models for visual storytelling. planning and generation in the domain of automatic story generation, planning has been effective at capturing the content and structure of stories. the generation process is often decomposed into two stages, namely planning an outline and then elaborating on it, e.g., by filling in specific details of a story. plans have been represented as a sequence of event or phrase keywords @xcite @xcite , character actions @xcite , plot structures @xcite , and more elaborate descriptions including details about the setting of the story, its characters, and main plot points @xcite . the idea of having a separate planning stage has also been explored for other text generation tasks including summarization @xcite ) and data-to-text generation @xcite . our work is closest to @xcite who propose the use of question-answer pairs as intermediate plans for summarization. however, their approach is designed for descriptive text. our work extends their framework to a multimodal setting, where the input consists of image sequences, and the output are narratives characterized by more abstract and figurative language. ## blueprint-based visual storytelling let i represent a sequence of k images, denoted as {v 1 , v 2 ..., v k }. given this input, our goal is to generate a blueprint plan b (i.e., an ordered set of question-answer pairs) and a story s based on it. most generation datasets do not include blueprint annotations, and visual storytelling is no exception. we first describe how we automatically obtain ## results our results are summarized in ## human evaluation we conducted a judgment elicitation study to further evaluate the stories generated by our models. specifically, we compared the best performing blueprint model (iterative) and three other systems: (a) pr-vist, which represents the current planning-based state of the art; (b) vp-bart, our proposed model without blueprints; and (c) ground truth stories written by humans. raters were shown an image sequence, alongside two stories and were asked to provide pairwise references along the following dimensions: relevance, fluency, coherence, interestingness, and overall. the full instructions are given in appendix a. we recruited nine native english speakers and elicited preferences for 100 stories (three judgments per story). our human evaluation results are summarized in the blueprint model excels in terms of interestingness and grounding, indicating its effectiveness in creating engaging and memorable stories. our model's superior grounding performance aligns with our hypothesis that blueprints serve not only as a planning strategy but also as a visual concept selector. this is due to the way blueprints are structured (as answer-question pairs), which explicitly forces the model to first identify salient visual concepts and then generate questions based on them. figure 4 shows example stories created by the models used in our human evaluation study and gpt-3.5+bp. the story generated by the iterative model is coherent, rich in detail, and fluent. pr-vist: i bought a cart to the market with some food. it looked great in the market. there were so many of their enthusiasm. they even had a sale with their bumper. the market was sold out of fresh vegetables. the [location] car is very nice and beautiful. ## controllable generation in this section we showcase how the blueprint plan allows us to control the content and length of model output without additional mechanisms or training. for example, in cases where the generated story contains entities which do not appear in the image sequence, it is possible to refine the story generation process, mitigating hallucinations. specif-ically, we apply a filtering step which removes non-grounded entities (and corresponding qa pair) from the blueprint before generating the story. we consider as non-grounded any blueprint entity which is not included in the output of the concept detector (see section 3.2). figure 5 shows how this refinement approach can be used to adjust the model's output. in the first example, we observe that the story generated with a refined blueprint effectively avoids hallucinations (highlighted in blue) and is overall more faithful. however, it is important to note that imagination plays a crucial role in crafting an engaging story, especially when the image sequence provides limited information. therefore, employing the refinement method may result in shorter and less detailed stories, as illustrated in the second example. while the refined blueprint successfully eliminates all hal-iterative + grounded-ent: we were so excited to see where our vacation was going. we arrived at the hotel and it was quite busy. we couldn't believe how crowded it was getting. some of the architecture was very cool. this store was our favorite since it was close to the city center. iterative: we took a trip to brazil. we went to the big city where we stay in, and took a lots of pictures. when we arrived, there were oompa pompa loompa hompa, and the people were walking around. we were at the altar, and it was standing. we ended the day getting hungry, so we decided to eat at a little elegant restaurant. iterative + grounded-ent: there was a question and answer session. everyone was listening. the speaker was very interesting. everyone was taking notes. the speaker talked very long. iterative: i went to my colleagues the other day. my boss was the one that invited me to attend the meeting. he talked about how to improve the company for the future. after the presentation, i looked and listened to my coworkers on how the company would be doing. my boss was pleased that i invited them to talk for hours. we talked for hours and we were very pleased when my boss asked. ## conclusion in this work, we have introduced a novel approach to visual storytelling which integrates visual representations with pretrained language models and a blueprint-based planning method for story gen-we took a car out onto the street that day. here we are looking at the car that the winner of the race was in. the car with the winner in blue was very close throughout the whole race. it was neck and neck as the one in blue held the lead throughout the entire race. finally, we crossed the lap and the winner pulled in for first place at the landing line. we were very close to winning the race and finishing in the final place line. it was a very very, very close finish! last night, the band played an awesome concert. the guitarist had an incredible performance. the keyboard player was incredible as well. the guitarist was superb. he played his guitar very well. the next song he sang was very good. they had such a good performance, and i'm glad they went. the audience loved them so much. the night was over when the band wrote and put together their music. it was a fantastic way to finish the night. eration. blueprint models leverage a sequence of question-answer pairs as intermediate plans, enabling better selection of salient concepts from the image sequence and guiding the construction of the final narrative. specifically, we have showcased two model variants: a top-down model which relies on a global plan, and an iterative model, which interleaves planning with sentence generation. our experiments have shown that blueprint models excel in concept grounding and their ability to create human-like stories. additionally, they are controllable: blueprints can be made shorter or longer and their details can be refined (e.g., by emphasising specific entities or characters), thus enabling human-in-the-loop and personalized storytelling.