Improving Generation and Evaluation of Visual Stories via Semantic Consistency (2021)

Abstract: Story visualization is an underexplored task that falls at the intersection of many important research directions in both computer vision and natural language processing. In this task, given a series of natural language captions which compose a story, an agent must generate a sequence of images that correspond to the captions. Prior work has introduced recurrent generative models which outperform text-to-image synthesis models on this task. However, there is room for improvement of generated images in terms of visual quality, coherence and relevance. We present a number of improvements to prior modeling approaches, including (1) the addition of a dual learning framework that utilizes video captioning to reinforce the semantic alignment between the story and generated images, (2) a copy-transform mechanism for sequentially-consistent story visualization, and (3) MART-based transformers to model complex interactions between frames. We present ablation studies to demonstrate the effect of each of these techniques on the generative power of the model for both individual images as well as the entire narrative. Furthermore, due to the complexity and generative nature of the task, standard evaluation metrics do not accurately reflect performance. Therefore, we also provide an exploration of evaluation metrics for the model, focused on aspects of the generated frames such as the presence/quality of generated characters, the relevance to captions, and the diversity of the generated images. We also present correlation experiments of our proposed automated metrics with human evaluations.

## introduction while generative adversarial networks (gans) have achieved impressive results on a variety of image generation tasks @xcite , the task of story visualization @xcite is a variation of image generation that is more challenging and underexplored. in this setting, there is a story which consists of a sequence of images along with captions describing the content of the images, e.g., a web comic. the goal of the task is to reproduce the images given the captions (figure 1 ). the benefits of investigating this task are far reaching. it combines two interesting and challenging sub-areas: text-to-image synthesis and narrative understanding, providing an excellent test bed for exploring and developing multimodal modeling techniques. from an application perspective, such a system could be used to enhance existing textual narratives with visual scenes. this tool would be especially useful to comic artists, who are infamously overworked, allowing them to automatically generate initial drawings speeding up their workflow. additionally, such a system would have many applications in an educational setting, allowing educators to cater to a more diverse set of learning styles by automatically generating visualizations for a given topic, such as the water cycle in a science lesson. furthermore, the data in this domain is cartoon-style, meaning the generated im-ages avoid many of the ethical issues associated with real-world data. for a more detailed discussion, see section 9. the challenge of this task extends beyond tasks such as text-to-image or text-to-video synthesis. namely, there is an explicit, narrative component to the data, which must first be accurately extracted from the text, and then consistently reproduced throughout the images. if the setting or a description of a character is provided in the first caption, this must be carried throughout the scene unless modified by a subsequent caption. furthermore, the scenes in a single story can change drastically as the story progresses, requiring models to produce a greater variety of images than in a text-to-video task, which typically consists of short videos displaying a single action. to address these issues, we consider the task as proposed in @xcite , which provides a baseline architecture, storygan, along with datasets for the task. we introduce techniques that build on existing work and are focused on improving consistency across frames, resulting in images of higher visual quality. first, we augment the model with dual learning via video redescription. the output images are fed through a video captioning model, which is trained to reproduce the ground truth story captions. this provides an additional learning signal to the model, forcing it to semantically align with the given narrative. next, we add a copy-transform module that can take generated images from previous timesteps and copy the most relevant features of those images into the next generated frame, thus making the images more consistent in appearance. finally, we propose the use of memory-augmented recurrent transformer (mart) @xcite to model the correlation between word phrases in the input text and corresponding regions in the generated image. the recurrent nature of mart allows for the learning of sophisticated interactions between the image frames, yielding images that are more consistent in terms of character appearances and background imagery. we call the model architecture with the aforementioned additions du(al)-co(py)-storygan or duco-storygan. next, we focus on exploring alternative evaluation methods for story visualization models. while modeling improvements are crucial for progressing in this domain, evaluating these models is a challenge in itself. like many other generative tasks, it is nontrivial to evaluate a story visualization model. human evaluation is the most reliable option, but its monetary and time costs make this ill-suited to be the only evaluation method. most prior work relies upon standard gan evaluation metrics, which may provide some insight into how well the images were reproduced, yet miss out on other aspects of the story visualization task, such as the visual consistency of the setting across frames and global semantic alignment. therefore, we make evaluation another focal point of the paper, exploring a variety of automatic evaluation metrics, which capture various aspects of the task, e.g., evaluating the quality of the images, the relevance to the story, the diversity of the generated frames, and the model's ability to accurately represent the characters. we present results from our model and baseline models on all metrics along with qualitative results, demonstrating the improvements from our proposed techniques. using these metrics, we also provide ablation analyses of our model. our main contributions can be summarized as: ## evaluation of visual story generation as with any task, evaluation is a critical component of story visualization; however, due to the complexity of the task and its generative nature, evaluation is nontrivial. for instance, characters are the focal point of any narrative and similarly should be the focus of a model when producing images for the story. hence, @xcite measure the character classification accuracy within frames of generated visual stories in order to compare models. however, it is also important that the characters and background are consistent in appearance, and together form a cohesive story rather than an independent set of frames. inspired by insights such as this, we explore an additional set of evaluation metrics that capture diverse aspects of a model's performance on visual story generation. character classification. we finetune the pretrained inception-v3 @xcite with a multi-label classification loss to identify characters in the generated image. most earlier work in story visualization report the image-level exact-match (em) character classification accuracy. however, we contend that the exact match accuracy is not sufficient to gauge the performance of generative models, and the micro-averaged f-score of character classification should also be reported. for example, if model a generates one of two characters in a frame with better quality than model b (which generates none), it results in the same em accuracy as model b but an improvement in the recall/f-score of the model, making the latter more reliable as a metric for quality. our conclusion is based on the observation of consistent improvement in character classification scores with increasing training epochs and manual evaluation of image quality (see fig 4 ). ## conclusion in this paper, we investigate the underexplored task of story visualization. we improve upon prior modeling approaches and demonstrate the effectiveness of these new approaches by performing a robust set of ablation experiments. we also present a detailed set of novel evaluation methods, which we validate by demonstrating improvements across various baselines. evaluation for story visualization is a challenging open research question in itself, and we hope that these methods will encourage more work in this domain. from an ethics standpoint, we provide a brief overview of the data that the model is trained on in section 4 and a more detailed discussion in the appendix. we provide some analyses of the data and refer the reader to the original storygan paper, where the dataset was created, for further details. all of the language data consists of simple english sentences. our experimental results are specific to the story visualization task. pororo-sv is the most challenging story visualization task available; therefore, our results would likely generalize to other story visualization datasets. while story visualization is an exciting task with many potential future applications, the generated images still contain many obvious visual artifacts and therefore models trained on this task are still far from being deployed in any real world settings. story visualization minimizes many of the ethical issues associated with image and video generation. deepfakes, which are algorithmically generated fake images, have become increasingly problematic @xcite . oftentimes, these images are indistinguishable from real images, raising privacy concerns and providing a source of misinformation. the images that we generate here are not subject to this same issue, due to the fact that they are cartoons, and are therefore unable to be confused with real images. the focus of the task is not on the realism of the images, but rather on the multimodal narrative. therefore, cartoons are actually better suited for the task as real-world images only add additional visual complexity that is not relevant to the narrative.