Visually Grounded Story Generation Challenge (2023)

Abstract: Recent large pre-trained models have achieved strong performance in multimodal language generation, which requires a joint effort of vision and language modeling. However, most previous generation tasks are based on single image input and produce short text descriptions that are not grounded on the input images. In this work, we propose a shared task on visually grounded story generation. The input is an image sequence, and the output is a story that is conditioned on the input images. This task is particularly challenging because: 1) the protagonists in the generated stories need to be grounded in the images and 2) the output story should be a coherent long-form text. We aim to advance the study of vision-based story generation by accepting submissions that propose new methods as well as new evaluation measures.

## introduction vision-based language generation (vlg) is to generate text from visual input. it is a challenging but interesting task because it requires joint vision and language modeling. recent large pretrained vision-and-language models (vlms) like @xcite or minigpt-4 @xcite have shown great success on several multimodal tasks, such as image captioning @xcite , visual question answering @xcite and visual dialog generation @xcite . despite recent breakthroughs, current tasks only require models to predict a label or generate short texts (i.e., less than 30 words). it is unclear whether the newest vlms can generate coherent texts with multiple sentences from visual input. on the contrary, humans can produce long and localy coherent texts from the same visual input. to investigate machine intelligence, we need a task that is more similar to human behavior @xcite . several previous tasks have been proposed to test the capabilities of vlms to handle longer output, such as visual paragraphs @xcite , localized narratives @xcite , and video captioning @xcite . however, these tasks are designed for literal descriptions where sentences are independent of each other, rather than for coherent text. coherence is a fundamental property of human language. in particular, local coherence, which refers to the relations between entities in context, affects language comprehension and production. local coherence is essential for vision and language (v&l) research because: 1. it has many applications in vision and language tasks. for example, a better model of local coherence can improve the performance of text-to-image retrieval (park and kim, 2015). 2. modeling coherence is a prerequisite for modeling event knowledge as events center around entities. better event modeling improves vision and language pre-training @xcite . story generation is a well-studied task in natural language generation, widely used for testing whether large pretrained models can track entities @xcite and generate locally coherent texts. unlike image captions, stories contain several characters and events involving recurrent characters and their interactions with each other and the environment. in addition, characters and relevant content are among the most critical aspects of story writing @xcite . we argue that story generation is a suitable benchmark for testing whether vlms can generate coherent texts. in this work, we propose a new shared task, visually grounded story generation (vgsg), which requires the vlms to generate stories with protagonists grounded on images. we aim for coherent and visually grounded stories with high diversity. this task is particularly challenging for two reasons: 1. the protagonists in the generated stories need to be grounded in the images, meaning that their actions and descriptions should be consistent with the ## task description we define the vgsg task as follows: given a sequence of images (like the first column of figure 1 ) the system needs to generate a coherent short story conditioned on the image sequence (like the second column of figure 1 ). in addition, the generated story should contain the characters seen in the image sequence. the vgsg shared task focuses on coherent and visually grounded stories with high diversity. ## -1 jack put the phone down after an unsuccessful deal and decided to go get a coffee at the nearby coffee. ## -1 at the coffee shop, he started talking to the waiter will about the unfortunate call. ## 1 will told him he would convince the client to accept the deal if he could work for jack. will then called the client and successfully struck the deal. figure 2 : example a matching matrix between entities in the generated story and the character in the images. ## evaluation we will perform both automatic and human evaluations for the submissions. the scripts for all automatic metrics will be provided after the submission system is open; human evaluation will be conducted after all submissions have been received. we will release the annotator instructions and source code of all metrics after the shared task. ## conclusions this proposal introduces a novel shared task called visually grounded story generation, which necessitates that visual language models formulate narratives with protagonists based on image inputs, ensuring the production of coherent and visually grounded stories with high diversity. the task poses dual challenges: the need for protagonists' actions and descriptions to align with the provided visual information and the requirement for the output story to logically progress with a clear beginning, middle, and end. by initiating this task, the authors aim to foster advancements in visual language generation, inviting global researchers to contribute new methodologies that facilitate the creation of visually consistent, logically structured stories.