Improved Visual Story Generation with Adaptive Context Modeling (2023)

Abstract: Diffusion models developed on top of powerful text-to-image generation models like Stable Diffusion achieve remarkable success in visual story generation. However, the best-performing approach considers historically generated results as flattened memory cells, ignoring the fact that not all preceding images contribute equally to the generation of the characters and scenes at the current stage. To address this, we present a simple method that improves the leading system with adaptive context modeling, which is not only incorporated in the encoder but also adopted as additional guidance in the sampling stage to boost the global consistency of the generated story. We evaluate our model on PororoSV and FlintstonesSV datasets and show that our approach achieves state-of-the-art FID scores on both story visualization and continuation scenarios. We conduct detailed model analysis and show that our model excels at generating semantically consistent images for stories.

## introduction diffusion models trained on broad text-image data @xcite @xcite @xcite @xcite achieved remarkable success in text-to-image generation and showed strong abilities to synthesize photorealistic images of high resolution and great semantic consistency to text prompts. such a huge success drives the extension of modern diffusion text-toimage models into more scenarios like visual story generation, which is to generate a series of images for a story of multiple sentences. a recent work, ar-ldm @xcite , which is built upon open-sourced stable diffusion, achieves the state-of-the-art fid on the benchmark datasets for visual story generation. ar-ldm encodes previous text-image context as a sequence of additional conditions, which is then attended by 4 poby, loopy, pororo and crong are clapping. 5 eddy is standing in front of the eight glasses containing different colors of fluids. the unet decoder for image generation. despite its remarkable success, one limitation is that previous text-image pairs of the same story are all flattened as conditioning memories. this is different from the fact that not all the scenes/characters of sentences in the same story are closely related. take figure 1 as an example. the scene of the fourth sentence is not related to either the second or the third sentence. on the contrary, the generation of the fifth image should depend more on the second and third images than others. from this example, we can see that the dependency between images could be largely measured by the semantic relations between sentences. in this work, we present a simple approach 1 that selectively adopts historical text-image data from the same story in the generation of an image. specifically, we freeze the text and image representations produced by off-the-shell encoders, and adaptively compute conditioning vectors of context by considering the semantic relation between the current sentence and all the history. such resulting conditioning vectors will be queried by unet in a traditional way. furthermore, based on the consideration that images should have similar scenes and characters if their corresponding sentences are similar, we further add context-aware guidance like the use of classifier guidance or clip guidance @xcite in standard text-to-image generation. to validate the effectiveness of our approach, we evaluate our model on story visualization and continuation tasks. experimental results on pororosv and flintstonessv datasets show that both adaptive encoder and guidance improve the quality of the generated images as well as the global consistency of the visual story. the contributions of this work are as follows: ## model we introduce our approach in this section. we first present the model architecture of our approach ( §3.1), and then describe three important components: adaptive encoder ( §3.2), conditional diffusion model ( §3.3) and adaptive guidance ( §3.4). ## conclusion in this paper, we explore an effective adaptive context modeling method to improve visual story generation. first, we use an adaptive encoder to select the context closely related to the current image from the historical text-image pairs using the current text. then we fed the context vectors and the text vector to the diffusion model, and use an adaptive guidance to guide the generation of the current image. experimental results verify that adaptive context modeling could help generate higher quality images and more consistent stories. in addition, we analyze the generated images and find potential research directions in the future: (1) focus on the global consistency of the story, (2) pay attention to the action and expression of the character, (3) obtain the exact semantics of long stories.