Visual Writing Prompts: Character-Grounded Story Generation with Curated Image Sequences (2023)

Abstract: Current work on image-based story generation suffers from the fact that the existing image sequence collections do not have coherent plots behind them. We improve visual story generation by producing a new image-grounded dataset, Visual Writing Prompts (VWP). VWP contains almost 2K selected sequences of movie shots, each including 5-10 images. The image sequences are aligned with a total of 12K stories which were collected via crowdsourcing given the image sequences and a set of grounded characters from the corresponding image sequence. Our new image sequence collection and filtering process has allowed us to obtain stories that are more coherent, diverse, and visually grounded compared to previous work. We also propose a character-based story generation model driven by coherence as a strong baseline. Evaluations show that our generated stories are more coherent, visually grounded, and diverse than stories generated with the current state-of-the-art model. Our code, image features, annotations and collected stories are available at https://vwprompt.github.io/ .

## introduction in this work, we improve the quality of text stories generated by neural models from image sequences. we do so by improving the curation of the image sequences that form the basis for collecting the story/image pairs used to train the models: we build a dataset in which the images lend themselves better to telling a story. to show the usefulness of our dataset, we train a coherencedriven model where we design a coherence component inspired by entity grid models. experiments show that our model produces more coherent, visually grounded and diverse stories than previous models. stories are essential in natural language understanding and generation because they are the key mechanism for humans to understand the world @xcite . automatically generating good stories is a challenging task requiring various capabilities in language processing @xcite , event understanding @xcite , and world knowledge @xcite to come together. previous approaches to story generation have used different kinds of input to guide the story: some use a textual prompt to start the story @xcite , yet others involve describing a sequence of images to direct the story @xcite . we choose to work inside the latter family of approaches in order to exploit the rich information contained in image sequences and to prevent suffering from the symbol grounding problem @xcite . research on visual narratives shows how it would be possible to construct the sort of dataset we propose: image sequences should consist of a series of coherent events centered around one or more main characters @xcite . in fact, even aristotle points out in poetics that event and character are the most important elements for a good story. to date, several datasets of image sequences for narrative generation exist, such as the visual storytelling (vist; @xcite dataset, which includes sets of images extracted from flickr albums. however, image sequences generated this way have the drawback that they may not lend themselves well to storytelling. consider for instance the image sequence shown in the first column of figure 1 : the people featured across the image sequence are all different, and there is no real development of an event or a plot. so the stories that humans were able to write for these types of image sequences are often quite poor from a narrative point of view and lead to low-quality training data for our story generation algorithms, which in turn, unsurprisingly, generate quite bad stories. we thus argue that image sequences serving as writing prompts should be comprehensible as visual narratives by humans. humans (with reasonable writing proficiency) can then ''translate'' such visual narratives into textual narratives. for an image sequence to qualify as a visual narrative, events and characters must have two properties: coherence, meaning that the events are semantically related and centered around recurring characters; and diversity, meaning that several different events jointly construct a plot. psycholinguistic experiments show that missing either of these properties impedes human comprehension of image sequences as visual narratives @xcite . in addition, the characters should be easily recognized in the image sequences and can be straightforwardly linked to the stories (visual groundedness). image sequences without these properties are hence not effective writing prompts. in this work, we define the term visual tellability to mean the tellability @xcite of image sequences, that is, how likely it is that humans can write a story with an image sequence, which measures whether the image sequences have the three properties described above. we propose a new dataset, visual writing prompts (vwp), containing curated image sequences and matching user-generated stories. our image selection process allows us to choose optimized image sequences that have high visual tellability, and to encourage our crowdsourced storytellers to produce coherent and visually grounded stories with high diversity. to obtain coherent and visually grounded stories, we provide cropped images of characters explicitly with image sequences for storytellers. to improve diversity, we select images from a data source that is already likely to have a plot: image sequences selected from movie scenes with aligned synopses. to further show the importance of coherence and visual groundedness, we propose a story generation model with a representation of visual coherence focused principally on character continuity as a strong baseline. experiments show that our model outperforms the current state-of-the-art model tapm @xcite and generates stories that are more coherent, visually grounded, and diverse. we summarize our contributions in this work as follows: (a) we propose a pipeline to extract image sequences automatically from annotated movies as story writing prompts, which leads to image sequences with higher visual tellability. (b) we collect a new dataset of stories based on curated image sequences with grounded characters, which is more coherent and has better diversity than previous datasets. (c) we propose a character-grounded story generation model driven by visual coherence as a strong baseline for imagebased story generation, which generates more coherent, diverse, and visually grounded stories than the current state-of-the-art model. ## related work story generation. there are several existing datasets for generating a story conditioned on a prompt such as title @xcite , keyword @xcite , cue phrase @xcite , script @xcite , or story plot @xcite . the rocstories corpus @xcite is a collection of short stories with rich causal and temporal relations. in subsequent work, new datasets have also been formed by gathering annotations on subsets of rocstories for specialized story generation tasks such as modeling character psychology @xcite , counterfactual reasoning @xcite , and so forth. the storium dataset @xcite of collaboratively written long stories contains rich annotations such as narrator prompts, character goals, and other attributes to guide story generation. however, all these datasets relying on textual prompts suffer from the symbol grounding problem that the meanings of textual stories are grounded on textual symbols @xcite . in contrast, our dataset contains stories grounded on nonsymbolic prompts from visual perception, that is, characters in image sequences. ## image sequence construction in this section, we describe how we obtain image sequences and design a pipeline to filter and sample images. our objective is to construct image sequences that are visually tellable, that is, are coherent, diverse, and visually grounded. our pipeline for image sequence construction is shown in figure 2 . ## crowdsourcing experiment design in this section, we design a crowdsourcing experiment to collect stories using our collected image sequences as writing prompts. our objective is to obtain coherent stories that have high diversity from crowdsourced storytellers. we design and run all our studies on amazon mechanical turk (amt). 4 the worker user interface is shown in figure 3 . in each assignment, we ask the worker to select a subset of images from the image sequence and write a short story (50 to 300 words) that fits the image sequence. to ensure that the human-written stories are grounded on main characters, we provide names and cropped images of at most five major characters. we retrieve the bounding boxes for each character from the movienet annotations and choose the least blurry appearance of each character in the image sequence. we pose three questions to the workers. the first two questions are used to identify workers who have watched the movie from which the image sequence is taken, as they might exhibit different behaviors during story-writing. the third question is to measure visual tellability on a 5-point likert scale, which is used to show the effectiveness of our image sequence construction pipeline. we also design a review form for story reviewers to judge the quality of collected stories. we ask the reviewers: 1) whether they want to approve the story; 2) if not, which requirement does it break? 3) if yes, judge the statement: this is a good story. on a 5-point likert scale. the first two questions are to assure that the collected stories fulfill the following requirements: the story is grammatical, the story is diverse, and the story is visually grounded. the third question is to get judgments of the quality of the approved stories. ## data collection in this section, we describe how we collect and process the stories in the vwp dataset. our goal is to obtain narratives given the curated image sequences as writing prompts. worker qualification. in order to improve story quality, we apply a qualification process to workers. we first collect 4.5k stories together with visual tellability judgments and obtain 556 candidate workers. each story is reviewed by one of five graduate students from saarland university who are proficient in english. to ensure that the reviewers mutually understand the purpose of the task, we let the reviewers judge 100 stories then check the reviews together to agree on the judgment standards. we then select 58 qualified workers with an acceptance @xmath0, average story @xmath1.1, and accepted @xmath2. we assign a qualification to these workers and invite them to bulk collection. bulk collection. we collect 7.5k stories with the qualified workers in bulk collection. we group about 300 image sequences into a batch and collect 1.5k stories per batch. for each batch, we sample s stories from each worker and review the stories to update the assessment of the worker, text processing. we process the raw text to make it easier for training story generation models. we tokenize all stories with the spacy english tokenizer @xcite . we then recognize all entities using a name entity recognition model @xcite . we change all location names to placeholders and replace all named characters in each story to finally, to mark the alignment between images and story sections, we add a special separator token [sent]. we randomly sample 849 stories as validation split and 586 stories as test split. ## experiment and evaluation in this section, we propose a strong baseline model for character-grounded story generation. we then experiment on our vwp dataset and show the results. our goal is to demonstrate the usefulness of our dataset. we extract features for all images with swin transformer @xcite , a state-of-the-art computer vision backbone model where all parameters are fixed. we use their official model checkpoint, pre-trained on the imagenet-21k dataset, to increase domain generality. we extract three different visual features: we use the following models for visual story generation as baselines: for each baseline, we consider four different variants with different inputs: 1) only global image features; 2) global features and object features; 3) global features and character features; and 4) all three available features. ## conclusions and future work we show that curated image sequences with characters are effective as writing prompts for visual story generation in both data collection and model design. by filtering images without any objects that could be recognized by the object detector and removing highly similar images to boost diversity, we can improve the visual tellability of image sequences. presenting selected characters during the story-writing yields stories with characters grounded in images, which are more coherent and diverse. correspondingly, using character features as input to the story generation model can improve the quality of generated stories. adding the character grid representation can bring further improvements in coherence, grammaticality, visual groundedness, and diversity. future work. one important property of visual narratives not covered in this work is narrativity @xcite , that is, whether an image sequence contains the necessary narrative structures to make a good story. a narrative structure can be achieved by events following a typical order with roles like establisher, initial, initial peak, and release @xcite . we observe that these roles of events emerge in our collected stories. our annotations of different instances of the same character across a story allow us to construct event chains for each character. future work should investigate how to annotate the roles of these events, measure narrativity, and build a model to generate stories with higher narrativity. a major assumption of all previous work in storytelling is that all humans are equally and reasonably proficient in story-writing and can translate visual narratives into textual narratives. however, individual differences in the writing proficiency of humans must have an impact on story quality. exploring this from the perspective of both data selection and model design would be an interesting future direction to take.