Attractive Storyteller: Stylized Visual Storytelling with Unpaired Text (2023)

Abstract: Most research on stylized image captioning aims to generate style-specific captions using unpaired text, and has achieved impressive performance for simple styles like positive and negative. However, unlike previous single-sentence captions whose style is mostly embodied in distinctive words or phrases, real-world styles are likely to be implied at the syntactic and discourse levels. In this work, we introduce a new task of Stylized Visual Storytelling (SVST), which aims to describe a photo stream with stylized stories that are more expressive and attractive. We propose a multitasking memory-augmented framework called StyleVSG, which is jointly trained on factual visual storytelling data and unpaired style corpus, achieving a trade-off between style accuracy and visual relevance. Particularly for unpaired stylized text, StyleVSG learns to reconstruct the stylistic story from roughly parallel visual inputs mined with the CLIP model, avoiding problems caused by random mapping in previous methods. Furthermore, a memory module is designed to preserve the consistency and coherence of generated stories. Experiments show that our method can generate attractive and coherent stories with different styles such as fairy tale, romance, and humor. The overall performance of our StyleVSG surpasses state-of-the-art methods on both automatic and human evaluation metrics.

## introduction over the years, image captioning has made remarkable progress @xcite @xcite . factual image captioning focuses on generating objective and neutral descriptions of image content without considering style characteristics. however, when describing images, stylized captions: … pos:(3) nice man and woman in a boat through water. neg:(3) two stupid people on a lake in a rowboat. … ## method given an image sequence @xmath0, stylized visual storytelling (svst) aims to generate a story @xmath1, where each substory s @xmath2, . . . , w kn } consists of k n words in the word vocabulary. please note that our task is fully unsupervised because there is no paired stylized data (i.e. in the form of (i, s)) for training. to constrain the target style and fully leverage the auxiliary paired data of factual stories, we apply a multi-task training framework (figure 2 (b) ), which attempts to achieve a trade-off between style accuracy and visual relevance. specifically, we utilize the vist dataset {(i (i) , the overall architecture of stylevsg is illustrated in figure 2 (a), which employs stacks of transformers @xcite . we use the pre-trained clip @xcite to extract high-level image features, and sum them with image order embeddings as the model inputs. the image sequence encoder e y /e t is employed to get contextual visual representation @xmath3, which are then fed into the story generator g y /g t to generate each sub-story. as illustrated in figure 2 (c), the story generator consists of: the memory unit m y /m t , which records history information to preserve the coherence and consistency of generated stories; and the style-oriented decoder d y /d t which constrains different styles. ## conclusion in this paper, we propose a new task of stylized visual storytelling, aiming to generate attractive stylized stories for a photo stream. by applying style-dependent components and multi-task training, our proposed stylevsg is able to generate stylistic stories without paired (images, stylized story) corpus. furthermore, our memory unit can preserve the coherence of generated stories. experiments demonstrate that stylevsg achieves better overall performance for complicated styles.