Stretch- VST : Getting Flexible With Visual Stories (2021)

Abstract: In visual storytelling, a short story is generated based on a given image sequence. Despite years of work, most visual storytelling models remain limited in terms of the generated stories’ fixed length: most models produce stories with exactly five sentences because five-sentence stories dominate the training data. The fix-length stories carry limited details and provide ambiguous textual information to the readers. Therefore, we propose to “stretch” the stories, which create the potential to present in-depth visual details. This paper presents Stretch-VST, a visual storytelling framework that enables the generation of prolonged stories by adding appropriate knowledge, which is selected by the proposed scoring function. We propose a length-controlled Transformer to generate long stories. This model introduces novel positional encoding methods to maintain story quality with lengthy inputs. Experiments confirm that long stories are generated without deteriorating the quality. The human evaluation further shows that Stretch-VST can provide better focus and detail when stories are prolonged compared to state of the art. We create a webpage to demonstrate our prolonged capability.

## introduction visual storytelling (vist) is an interdisciplinary task that takes a sequence of photos as input and produces a corresponding short story as output @xcite . prior work explores either end-to-end or hierarchical methods for visual storytelling, but machine-generated stories still fall far short of human-generated stories. one obvious limitation is the inability to generate stories with diverse length, especially to prolong a story. in real-world applications, when pictures accompany textual stories, the number of sentences is often much greater than the number of images. recent visual storytelling frameworks demonstrate the potential in prolonging visual stories, such as kg-story @xcite , a state-of-the-art framework that uses a knowledge graph to generate one additional sentence and attach it to 5-sentence visual stories for improved coherence. however, current models, including kg-story, are incapable of further "stretching" stories beyond five or six sentences. in short, generating prolonged visual stories faces three main hurdles: first, as vist-the only existing visual storytelling dataset-is mostly constructed as 5-photo sequences paired with 5sentence stories, models trained on it easily overfit to the dominant length. second, in visual storytelling, the quality of the textual story must be maintained when asking the model for more context. third, the model's generation function must generate stories with the desired number of sentences. that is, control of the continuation and termination of natural language generation depends on a given length factor. to meet these challenges, we introduce stretch-vst, a modification of the kg-story framework that greatly increases the number of sentences in visual stories while maintaining the quality thereof. story coherence and detail are improved by using cohesive and relevant information to generate additional sentences. illustrated in fig. 1 , stretch-vst has three main stages: first, it extracts representative terms (e.g., actions or objects) from each image. second, it finds relations between consecutive images using a knowledge graph, after which a scoring model selects the most suitable subset of terms ("term set" hereafter) given its length, term semantics, and cohesion. the length of the term set for the resultant term sequence hence depends on the score. finally, a length-controlled transformer is used to generate the story given the term sequence. the proposed work generates a variable number of sentences, and finds the optimal subset of terms given the story length. the human evaluation shows that stretch-vst generates better stories when prolonging stories, provides more detailed information comparing 5-sentence stories, and is more robust in cohering story context when the images are incoherent. ## related work visual storytelling was proposed by @xcite . two lines of work explore this task: one focuses on model architecture for better story generation @xcite @xcite @xcite , and the other uses adversarial training to generate more diverse stories @xcite @xcite . however, these methods often overfit to the number of sentences in the stories. stretch-vst modifies both the source and generation modules to generate variable-length stories. on the source side, we use knowledge graphs to expand the term set to represent the input image sequence. integrating a knowledge graph into language generation is beneficial @xcite @xcite @xcite @xcite . on the generation side, some explore the use of relative positional encoding @xcite , adding embedding layers, and manipulating the beam search process @xcite . however, these methods control only the number of words and not the number of sentences. ## methodology with variable-length visual sorytelling, stretch-vst brings two major contributions for vist: enriching the ingredients as desired (sect. 3.1) and enabling story generation according to the term sequence length (sect. 3.2). ## conclusion we propose a novel method for generating lengthcontrolled visual stories which includes an enhanced knowledge-graph reasoning module and a length-controlled transformer architecture. using human evaluations, we show that the method tells longer and better stories.