D iffu VST : Narrating Fictional Scenes with Global-History-Guided Denoising Models (2023)

Abstract: Recent advances in image and video creation, especially AI-based image synthesis, have led to the production of numerous visual scenes that exhibit a high level of abstractness and diversity. Consequently, Visual Storytelling (VST), a task that involves generating meaningful and coherent narratives from a collection of images, has become even more challenging and is increasingly desired beyond real-world imagery. While existing VST techniques, which typically use autoregressive decoders, have made significant progress, they suffer from low inference speed and are not well-suited for synthetic scenes. To this end, we propose a novel diffusion-based system DiffuVST, which models the generation of a series of visual descriptions as a single conditional denoising process. The stochastic and non-autoregressive nature of DiffuVST at inference time allows it to generate highly diverse narratives more efficiently. In addition, DiffuVST features a unique design with bi-directional text history guidance and multimodal adapter modules, which effectively improve inter-sentence coherence and image-to-text fidelity. Extensive experiments on the story generation task covering four fictional visual-story datasets demonstrate the superiority of DiffuVST over traditional autoregressive models in terms of both text quality and inference speed.

## introduction visual storytelling (vst) is the challenging task of generating a series of meaningful and coherent sentences to narrate a set of images. compared to image and video captioning @xcite @xcite , vst extends the requirements of image-to-text generation beyond plain descriptions of images in isolation. a vst system is expected to accurately portray the visual content of each image while also * corresponding author. figure 1 : examples from the vist @xcite and aesop @xcite dataset. so far, prior works on visual storytelling @xcite @xcite typically utilize an autoregressive design that involves an extra sequential model built on top of basic image captioners to capture the relationship among images. more elaborate frameworks with multi-stage generation pipelines have also been proposed that guide storytelling via e.g. storylineplanning @xcite , external knowledge engagement @xcite @xcite , and concept selection @xcite . effective as these existing methods are for describing real-world photo streams (e.g. the upper row in figure 1 ), albeit with their increasingly complex structural designs, their ability to generate stories from fictional scenes (e.g. the lower row in figure 1 ) remains unverified. in addition, because these models are typically trained to infer in an autoregressive way, they produce captions token by token while conditioning on only previous (unidirectional) history, which prohibits these architectures from refining prefix of sentences based on later generated tokens. as a result, existing autoregressive models are restricted to generating less informative narratives at a low inference speed. to tackle the above issues of vst -especially given the increasing demand for narrating fictional visual scenes -we propose a novel visual storytelling framework diffuvst (diffusion for visual storytelling) based on diffusion models utilizing continuous latent embeddings, which have demonstrated promising capabilities for image synthesis @xcite and language modelling @xcite @xcite . our model generates a set of visual descriptions simultaneously in a non-autoregressive manner by jointly denoising multiple random vectors into meaningful word embeddings conditioned on the image features of all panels (i.e., all image-text pairs in a visual-story sequence). specifically, diffuvst leverages a transformer encoder to learn to restore the word embeddings of the ground-truth story texts from sequences of gaussian vectors. to enhance the visual-truthfulness of the generated texts, diffuvst proposes multimodal feature extractor and adapter modules with pretrained backbone that provide in-domain story-image features as condition and story-text features as classifier-free guidance @xcite . since these features represent all panels across the entire picture stream, they serve as the "global history" of all the preceding and following frames and texts, which connects the captioning of the current image to other context panels more closely, thus improving the coherence of the whole visual narrative. we validate the effectiveness of diffuvst by performing the visual story generation task on four visual-story datasets with non-real-world imagery: aesop @xcite , flintstonessv (maharana and bansal, 2021), pororosv @xcite , and didemosv @xcite with the latter three converted from story visualization (sv) to fictional vst datasets. quantitative results show that diffuvst outperforms strong autoregressive baselines across all four datasets. in addition, dif-fuvst manages to reduce the inference time by a large factor thanks to its non-autoregressive nature. our contributions are summarized as follows: ## diffuvst figure 2 presents the overall architecture and training flow of our model diffuvst, which consists of two pretrained multimodal encoders (one for image and one for text) with their separate adapter layers, a feature-fusion module and a transformerencoder as the denoising learner. ## conclusion in this paper, we tackle the challenges posed by visual storytelling, and present a diffusion-based solution, diffuvst that stands out as a novel effort in introducing a non-autoregressive approach to narrating fictional image sequences. underpinned by its key components, including multimodal encoders with adapter layers, a feature-fusion module and a transformer-based diffusion language model, diffuvst models the narration as a single denoising process conditioned on and guided by bidirectional global image-and text-history representations. this leads to generated stories that are not just coherent but also faithfully represent the visual content. extensive experiments attest the outstanding performance of our non-autoregressive diffuvst system both in terms of text generation quality and inference speed compared to the traditional autoregressive visual storytelling models. furthermore, we present comprehensive ablation analyses of our method, which demonstrate that our elaborate designs, based on diffusion-related mechanisms, hold great potential for producing expressive and informative descriptions of visual scenes parallelly with a high inference speed.