Learning to Model Multimodal Semantic Alignment for Story Visualization (2022)

Abstract: Story visualization aims to generate a sequence of images to narrate each sentence in a multi-sentence story, where the images should be realistic and keep global consistency across dynamic scenes and characters. Current works face the problem of semantic misalignment because of their fixed architecture and diversity of input modalities. To address this problem, we explore the semantic alignment between text and image representations by learning to match their semantic levels in the GAN-based generative model. More specifically, we introduce dynamic interactions according to learning to dynamically explore various semantic depths and fuse the different-modal information at a matched semantic level, which thus relieves the text-image semantic misalignment problem. Extensive experiments on different datasets demonstrate the improvements of our approach, neither using segmentation masks nor auxiliary captioning networks, on image quality and story consistency, compared with state-of-the-art methods.

## introduction story visualization is a challenging task, which aims to generate a sequence of story images given a multi-sentence story, and further requires output images to be consistent, e.g., having a consistent background or character appearance. regardless of its difficulties, story visualization has the potential for many applications, including art creation, computer-aided design, and image editing. to address the challenges, current methods @xcite @xcite adopt a fixed storygan-based @xcite architecture, where two gans @xcite ) are adopted, one for single-image quality, and one for story consistency, without considering the semantic alignment between different-modal text and image features involved in the generation process. so, one problem arising is that a fixed network with the involvement of different-modal representations (e.g., text and image) may suffer from a semantic misalignment problem. this is because current methods usually adopt a fixed text encoder and image encoder to extract corresponding features, and then use these features directly in an also fixed gan-based network. however, text representations can be a coarse sentence vector, finegrained word embeddings, or a structured knowledge graph @xcite , while @xcite and @xcite have shown that image features extracted from different layers of a convolutional neural network (e.g., vgg) may contain different-level semantic information. based on this, simply fusing cross-domain representations together using a fixed generative network may cause a considerable adverse impact on the quality of output images. for example, one of the main function of the discriminator in a conditional gan is to evaluate the semantic alignment between input text and output image, and provide the corresponding feedback to the generator, which encourages the generator to generate text-semantic-matched images. however, to evaluate the semantic alignment, the discriminator in current methods only simply concatenates a coarse sentence vector and image features at a small-scale (i.e., 4×4), extracted from a given image using a series of convolutional layers, which may fail to fully match the semantics between text and image features, and thus provide a less precise training feedback to the generator. to address this problem, we explore the semantic alignment between text and image representations by learning to match their semantic levels in the gan-based generative model. more specifically, we introduce dynamic interactions according to learning to dynamically explore various semantic depths and fuse the different-modal information at a matched semantic level. by doing this, the network can learn to dynamically utilize various semantic level information from the given representations, and also learn to selectively fuse them together, which thus mitigates the semantic misalignment problem across different modalities. so, the main contributions are summarized as follows: ## related work story visualization aims to generate a sequence of consistent images corresponding to a multisentence story. storygan @xcite introduced a two-gan-based generation network. cp-csv @xcite , duco @xcite , and vlc @xcite were built on storygan, where cp-csv utilized character segmentation masks to improve the performance, and duco and vlc adopted auxiliary captioning networks to build a text-imagetext circle to ensure the consistency between the input and output. recently, @xcite proposed to utilize fine-grained word information to build a concise single-gan based network, and @xcite proposed to combine both clustering learning and contrastive learning together to ensure better text and image representations in a joint space. however, all these methods were based on a fixed network without considering the semantic alignment between involved text and image representations. text-to-image generation is closely related to our work, which generates one image from one given text description @xcite @xcite @xcite @xcite @xcite @xcite . differently, story visualization is more challenging, as it further requires output story images to be consistent. ## overview differently from current methods that adopt two gans, our network only has a single gan, nei-ther requiring additional segmentation masks nor auxiliary networks for supervision, as we experimentally find that a single-gan-based network can effectively produce high-quality story images with a good consistency. we attribute this improvement to the exploration of semantic alignment between different-modal representations, which enables fine-grained training feedback from the discriminator to the generator. given a story x with n story sentences, a text encoder encodes each story sentence s i into a sentence vector @xmath0, where d is the feature dimension, and l is the number of words in a sentence. then, we feed this n sentence vectors into the generation pipeline using upsampling blocks to produce story images at the required resolution. meanwhile, we further incorporate word embeddings into the generation pipeline, according to our proposed dynamic interactions, allowing the generator to learn to choose semantically aligned inputs from different semantic levels and modalities to achieve a better generation. in the discriminator, we also adopt the proposed dynamic interactions to fuse both text and image features, enabling a better evaluation on the text-image semantic alignment. the complete architecture is shown in fig. 1 . ## experiments to evaluate the performance of our approach, we compare it with storygan @xcite , cp-csv @xcite , duco @xcite , and vlc @xcite . ## conclusion in this paper, we explored the semantic misalignment problem existing in current story visualization methods, and further proposed dynamic interactions via learning to dynamically explore various semantic depths and fuse the different-modal information at a matched semantic level. experiments demonstrate the superior performance of our proposed single-gan based approach, with a fewer number of parameters, neither using segmentation masks nor auxiliary captioning networks.