Keep it Consistent: Topic-Aware Storytelling from an Image Stream via Iterative Multi-agent Communication (2020)

Abstract: Visual storytelling aims to generate a narrative paragraph from a sequence of images automatically. Existing approaches construct text description independently for each image and roughly concatenate them as a story, which leads to the problem of generating semantically incoherent content. In this paper, we propose a new way for visual storytelling by introducing a topic description task to detect the global semantic context of an image stream. A story is then constructed with the guidance of the topic description. In order to combine the two generation tasks, we propose a multi-agent communication framework that regards the topic description generator and the story generator as two agents and learn them simultaneously via iterative updating mechanism. We validate our approach on VIST dataset, where quantitative results, ablations, and human evaluation demonstrate our method’s good ability in generating stories with higher quality compared to state-of-the-art methods.

## introduction image-to-text generation is an important topic in artificial intelligence (ai) which connects computer vision (cv) and natural language processing (nlp). popular tasks include image captioning @xcite @xcite and question answering @xcite @xcite , aiming at generating a short sentence or a phrase conditioned on certain visual information. with the development of deep learning and reinforcement learning models, recent years witness promising improvement of these tasks for single-image-to-single-sentence generation. visual storytelling moves one step further, extending the input and output dimension to a sequence of images and a sequence of sentences. it requires the model to understand the main idea of an image stream and generate coherent sentences. most of existing methods @xcite @xcite @xcite for visual storytelling extend approaches of image captioning without considering topic information of the image sequence, which causes the problem of generating semantically incoherent content. an example of visual storytelling can be seen in figure 1 . an image stream with five images about a car accident is presented accompanied with two stories. one is constructed by a human annotator and the other is produced by an automatic storytelling approach. there are two problems with the machine generated story. first, the sentiment expressed in the text is inappropriate. in face of a terrible car accident, the model uses some words with positive emotion, i.e., "excited" and "fun". second, some sentence is uninformative. the sentence "we got to see a lot of different things at the event" provides little information about the car accident. this example shows that topic information about the image sequence is important for the story generator to produce an informative and semantically coherent story. in this paper, we introduce a novel task of topic description generation to detect the global semantic context of an image sequence and generate a story with the guidance of such topic information. in (1) ## approach given an image stream @xmath0, ..., x n ), where n is the number of images, we aim to output a topic description y topic and n sub-stories to form a complete story @xmath1, ..., y n ). the proposed framework mainly includes three stages, namely visual encoding, initial stage of generation and iterative updating (iu). the visual encoder ( §2.1) is employed to extract image features as visual context vectors. in the initial stage, we have the initial version of the two generation agents. the initial topic description generator ( §2.2) takes visual context vectors as input and generates a topic vector. the initial story generator ( §2.3) combines the topic vector and visual context vectors via co-attention mechanism and construct the initial version of story. considering that the generated story can also benefit the topic description generator, the two agents communicate with each other in the iu module ( §2.4) via message passing mechanism as fine tuning. the overall architecture of our proposed model is shown in figure 2 . each of these modules will be described in details in the following sections. ## related work this paper is related to the fields of image captioning, visual storytelling and multi-task learning. image captioning in early works @xcite , image captioning is treated as a ranking problem, which is based on retrieval models to identify similar captions from the database. later, the end-to-end frameworks based on the cnn and rnn are adopted by researchers @xcite @xcite . such works focus on the literal description of image content, while the generated texts is limited in a single sentence. visual storytelling visual storytelling is the task of generating a narrative paragraph for an image stream. @xcite introduce the first dataset (vist) for visual storytelling, and establishes some baseline approaches. an attention-based rnn with a skip gated recurrent unit @xcite is designed to maintain longer range information. in addition, @xcite design a hierarchicallyattentive rnn structure. more recently, @xcite and @xcite propose to utilize reinforcement learning frameworks for this task. @xcite propose to translate images to graph-based semantic representations to benefit representing images, while it is not fair to compare with our method because they introduce information from other datasets. the most similar work to ours is from @xcite . they propose to generate a local semantic concept for each image in the sequence and generate a sentence for each image using a semantic compositional network in a fashion of hierarchical reinforcement learning. although both of us consider topic information to facilitate the story generation. our model is different from three aspects. first, the concepts of topic are different. we treat topic as the global semantic context of the album while topic represents local semantic information in their case adhering to each single image. second, our modeling topic is more interpretable. we generate topic description directly instead of producing latent representation and this provides more insights for further improving the performance. third, the communication framework is compatible with any rl based training methods. experiment results also show that with rl, our framework can outperform their model. multi-task learning @xcite first propose a method for processing nlp tasks in a deep learning framework using multi-task learning. @xcite build a multi-task learning framework which jointly performs the prediction of tags and the generation of paragraphs. these multitask learning methods share a certain network structure, and at the output layer design a specific network structure for different tasks, improving the performance of different tasks. however, unlike these multitask learning methods, we use another multi-agent method @xcite . in this work, we define two kinds of agents for two generation tasks which can interact and share useful information. we also notice that in other areas, there are also some works @xcite consider incorporating topic information. ## conclusions and future work in this paper, we introduce a topic-aware visual storytelling task, which identifies the global semantic context of a given image sequence and then generate the story with the help of such topic information. we propose a multi-agent communication framework that combines two generation tasks namely topic description generation and story generation effectively. in future, we will explore to model topic generation as a keyword extraction task.