Reading Between the Lines: Exploring Infilling in Visual Narratives (2020)

Abstract: Generating long form narratives such as stories and procedures from multiple modalities has been a long standing dream for artificial intelligence. In this regard, there is often crucial subtext that is derived from the surrounding contexts. The general seq2seq training methods render the models shorthanded while attempting to bridge the gap between these neighbouring contexts. In this paper, we tackle this problem by using infilling techniques involving prediction of missing steps in a narrative while generating textual descriptions from a sequence of images. We also present a new large scale visual procedure telling (ViPT) dataset with a total of 46,200 procedures and around 340k pairwise images and textual descriptions that is rich in such contextual dependencies. Generating steps using infilling technique demonstrates the effectiveness in visual procedures with more coherent texts. We conclusively show a METEOR score of 27.51 on procedures which is higher than the state-of-the-art on visual storytelling. We also demonstrate the effects of interposing new text with missing images during inference. The code and the dataset will be publicly available at https://visual-narratives.github.io/Visual-Narratives/ .

## introduction humans process information from their surrounding contexts from multiple modalities. these situated contexts are often derived from a modality (source) and expressed in another modality (target). recent advances have seen a surge of interest in vision and language as source and target modalities respectively. one such widely studied task is image captioning @xcite which provides a textual description t given an image i. in contrast, visual storytelling @xcite is the task of generating a sequence of textual ## related work multimodal language: language generation from visual modality has seen a steep rise in interest with the introduction of several large scale tasks such as image captioning @xcite , visual question answering @xcite and visual dialog @xcite @xcite . ## dataset description this section describes our new vipt dataset and highlights the differences with vist. procedures vs stories: narrative properties such as content and structure in these forms are sufficiently contrastive @xcite . content in stories include characters and events while procedures include ingredients, materials and actions. while stories start from setting an era to characterizing participants, procedures start by listing ingredients followed by a step by step description. vipt dataset: though stories have the potential to exhibit the properties listed above, it is challenging to observe them in vist dataset @xcite owing to the shorter sequence lengths. to facilitate multi-domain research with stronger interleaved contexts between surrounding steps, we present a large scale visual procedure telling dataset with 46k procedures comprising of 340k pairwise images and textual descriptions. it is carefully curated from a number of how-to blogging websites. our dataset comprises of pairwise images and textual descriptions of the corresponding images, typiccally describing a step in a procedure. this means that each description of the step is tethered to an image. we categorized the dataset into 10 distinct domains including recipes, crafts, outdoors, lifestyle, technology, styling, fitness, hobbies, pets and miscellaneous. the category wise details of the dataset are presented in ## models description we present infilling based techniques for learning missing visual contexts to generate narrative text from a sequence of images. as the vist and recipes category in vipt are of comparable sizes (both in terms of data size and the average number of steps per instance), we perform comparative experimentation on these two categories. for our vipt category, we use 80% for training, 10% for validation and 10% for testing. an overview of infilling based training is depicted in figure 1 . the underlying encoding and decoding stages are described here. encoding: models 1, 2 and 3 here show different variants of encoding with and without infilling. ## results and discussion in this section, we present the effects of infilling both during both training and inference on vist and vipt datasets. infilling during training: the overall performance of the models is presented in ## conclusions and future work we demonstrate that infilling is a simple yet effective technique and a step towards maximizing the utilization of surrounding contexts in visual narratives. infilling is the strategy of enabling the model to learn surrounding contextual information by masking spans of input while the decoding attempts in generating the entire text. to experimentally support our hypothesis, we collect a new large scale vipt dataset of 46k procedures comprising 10 categories. we compare the performance of our model and conclusively show the higher significance of infilling based techniques in visual procedures compared to visual stories. in future, we plan to explore the following two directions: