Every picture tells a story: Image-grounded controllable stylistic story generation (2022)

Abstract: Generating a short story out of an image is arduous. Unlike image captioning, story generation from an image poses multiple challenges: preserving the story coherence, appropriately assessing the quality of the story, steering the generated story into a certain style, and addressing the scarcity of image-story pair reference datasets limiting supervision during training. In this work, we introduce Plug-and-Play Story Teller (PPST) and improve image-to-story generation by: 1) alleviating the data scarcity problem by incorporating large pre-trained models, namely CLIP and GPT-2, to facilitate a fluent image-to-text generation with minimal supervision, and 2) enabling a more style-relevant generation by incorporating stylistic adapters to control the story generation. We conduct image-to-story generation experiments with non-styled, romance-styled, and action-styled PPST approaches and compare our generated stories with those of previous work over three aspects, i.e., story coherence, image-story relevance, and style fitness, using both automatic and human evaluation. The results show that PPST improves story coherence and has better image-story relevance, but has yet to be adequately stylistic.

## introduction enabling machine-generated stories based on visual cues opens up promising directions, and leads language models (lms) to be viewed as an interface, allowing its involvement in artistic tasks such as advertisement creation and ai-generated movie scripting @xcite @xcite . in that direction, vision-language understanding and generation works succeed in leveraging image as well as text as cross-modal knowledge to solve various tasks @xcite @xcite . one fundamental task, image captioning, which involves the model to generate an informative textual caption according to a given image, opens up a venue for creativity to be explored. humans can compose concise descriptions of pictures by focusing on what they find important. @xcite ; @xcite ; @xcite ; @xcite lay a solid foundation on the current capability of machine learning models to relay cross-modal knowledge for the language models to do generation out of images. beyond generating captions, creating stories-which utilize linguistics to compose and narrate an interrelated series of events @xcite @xcite -according to a single input image offers even possibilities for creativitybased tasks @xcite @xcite . from the recent advancement on the image-tostory task, it is evident that multiple challenges still remain to be properly solved. one of the main challenges is that model-generated stories tend to lose their coherence as their length increases. further-more, the generated text needs to go beyond the pure description of an image as captioning does. data scarcity, in this context the lack of readyto-use datasets of image associated with a short story, is also a challenge. lastly, to the best of our knowledge, there is still limited control over generated stories aside from their relevance to the corresponding image, especially with regards to style @xcite . style has a role to convey a message or story through certain variations of diction and ways of delivery appropriate for a specific context @xcite @xcite . in this work, we introduce plug-and-play story teller (ppst). we take a step towards generating a stylistic story from an image while alleviating the data scarcity issue by leveraging large pre-trained models such as clip @xcite and gpt-2 @xcite , and to add the possibility to control the rendered style through plug-and-play adapters, explored in @xcite and @xcite . ppst yields improved natural and on-topic stories, and the resulting stylistic stories also have a strong image-story relevance. our results highlight the performance of ppst, especially in story coherence and imagestory relevance, improving the previous state-ofthe-art performance. lastly, we provide an analysis on the generated stories, including the occurring issues such as repetition and lack of common sense. we present an example of our generated stories using ppst in figure 1 . ## plug-and-play story teller (ppst) we present the overview of our approach: plug-and-play story teller (ppst) during inference in figure 2 . to generate stories out of an image, ppst involves two main components: visual input encoder (enc) and plug-and-play stylistic story decoder (dec). we use two datasets: an image captioning dataset @xmath0, c d i )} n @xmath1, where v d denotes image as the visual content and c d denotes the caption with the textual description of the respective image, and a book passage collection @xmath2, g b i )} n @xmath3, where p b denotes the passage chunk and g b denotes its style (genre). ## discussions our works have moved image-grounded story generation forward by improving the generated story coherence and image-story relevance, and by adding a layer of style control on top of it. however, as explained in §5, the current progress still leaves room for improvement. story coherence taking inspiration from recent works, a few strategies to refine story coherence can be implemented as the next step, for instance an unsupervised hierarchical story infilling @xcite , a semantic dependency skeleton generation to extract key information @xcite or storyline @xcite , a deeper understanding of causal and temporal relations of events through commonsense knowledge @xcite , the utilization of both sentence-level and discourse-level prefix information for decoding @xcite , and making use of a story dataset with rich and fine-grained annotations @xcite . image-story relevance for the relevance, rather than simple embedding concatenation, other ways to incorporate visual information to textual @xcite @xcite can be further investigated. ## limitations we discuss here about the limitations of our work, specifically concerning the chosen heuristic to align style and passages from bookcorpus, the limited amount of data in choice of style for the adapters, and possible biases. as explained in section 4.1, we choose to split book texts based on paragraphs in order to retain a certain degree of logical fluency throughout each story samples, as a paragraph usually deals with a single theme or idea. while this helps keeping the passages relatively short, one limitation of this approach is that some passages might not fully reflect the style of narrative that it is classified as. for example, not every paragraph taken out of context from a romance book will exhibit its genre. there could be a sizable amount of passages that focus on world-building and laying the groundwork for the book's main plot to progress. we decide to focus on romance and action because these styles are the most represented in the dataset used, as well as being more straightforward to capture in terms of style compared to other genres that rely on an underlying plot throughout the book such as historical or adventure. generaliz-ing ppst to these styles with a lower amount of resources might require further experiments. lastly, previous works have shown that captioning models can exhibit harmful biases, such as gender bias @xcite and racial bias @xcite . since we pair those image captions data with written stories from a wide variety of books, those biases can be further amplified. thus, such generative processes must be used with caution. while tackling unwanted biases in images or captions is a must, the bias exhibited in stories is sometimes justified by the context and the surrounding narrative. not all stories should be completely neutral, and this balance should be considered carefully in future directions. ## conclusion by leveraging text-image alignment representations to describe the visual content of a given image in words, we can use the resulting semantic embeddings as prior knowledge to generate a short story out of a given picture through a plug-and-play controllable language model approach. it also allows us to tackle the data scarcity issue in this task. the results show that our plug-and-play story teller (ppst) generates more consistent and ontopic stories according to the visual information, as well as performing better in relevance and imagestory relationship than the previous state-of-the-art. we also found that ppst without style adapters (non-styled) generates more coherent stories, and ppst utilizing style adapters (romance and action) have a similar, if not a slightly better, imagestory relationship than the other approaches.