Summary of the Visually Grounded Story Generation Challenge (2024)

Abstract: Recent advancements in vision-and-language models have opened new possibilities for natural language generation, particularly in generating creative stories from visual input. We thus host an open-sourced shared task, Visually Grounded Story Generation (VGSG), to explore whether these models can create coherent, diverse, and visually grounded narratives. This task challenges participants to generate coherent stories based on sequences of images, where characters and events must be grounded in the images provided. The task is structured into two tracks: the Closed track with constraints on fixed visual features and the Open track which allows all kinds of models. We propose the first two-stage model using GPT-4o as the baseline for the Open track that first generates descriptions for the images and then creates a story based on those descriptions. Human and automatic evaluations indicate that: 1) Retrieval augmentation helps generate more human-like stories, and 2) Largescale pre-trained LLM improves story quality by a large margin; 3) Traditional automatic metrics can not capture the overall quality.

## introduction vision-based language generation (vlg) is the generation of text from visual input and is an important task in natural language generation and artificial intelligence. recently, large pre-trained vision-and-language models (vlms), such as @xcite and gemini @xcite , have achieved remarkable performance across several multimodal tasks, including image captioning @xcite , visual question answering @xcite , and visual dialogue generation @xcite . although these advancements are notable, most of the current tasks involve predicting labels or generating short pieces of text (typically under 30 words). it remains uncertain whether the latest vlms can create longer, coherent texts consisting of multiple sentences based on visual input. the evaluation of long stories is still challenging @xcite . on the other hand, humans can easily generate extended and logically connected text from visual stimuli. to further assess vlms, a task more aligned with human capabilities is necessary @xcite . previous tasks have been designed to evaluate the ability of vlms to produce more extended outputs, such as visual paragraphs @xcite , localized narratives @xcite , and video captioning @xcite . however, these tasks primarily focus on literal descriptions, where sentences remain independent rather than forming a coherent whole. coherence, especially local coherence-defined as the relationships between entities in a given context-is fundamental to human language comprehension and production. in vision and language research, local coherence is crucial for several reasons: 1. improved models of local coherence can enhance the performance of vision-language tasks, such as textto-image retrieval @xcite . 2. accurately modeling coherence is essential for developing event knowledge, as events revolve around entities. stronger event modeling enhances visionlanguage pre-training @xcite . story generation is a widely researched task in natural language generation and is frequently used to assess whether large pretrained models can track entities @xcite and produce locally coherent texts. unlike image captions, stories involve multiple characters and events, with recurring entities interacting with one another and their surroundings. moreover, the importance of characters and relevant content is central to successful story creation @xcite . we contend that story generation is an appropriate bench- ## task description we define the vgsg task as follows: given a sequence of images (like the first column of figure 1 ) the system needs to generate a coherent short story conditioned on the image sequence (like the second column of figure 1 ). in addition, the generated story should contain the characters seen in the image sequence. the vgsg shared task focuses on coherent and visually grounded stories with high diversity. ## evaluation and results in this section, we describe our designs for both automatic and human evaluations for the submissions. the scripts for all automatic metrics be provided after the submission system is open; human evaluation be conducted after all submissions have been received. we release the annotator instructions and source code of all metrics after the shared task. ## conclusions we organized the visually grounded story generation task (vgsg) this year for the first time. although visual language models have made huge progress in the past couple of years, they are generally not specifically designed with the intention of producing narratively coherent and grounded stories. this task provided further impetus for development in this area. we obtained a couple of submissions, although private communications suggested that other potential participants instead decided to retain their results for publication in other venues. the training data and test platform were mounted on the web and on huggingface in order to enable further progress. consider the automatic measures: in the controlled (closed) experiment, the use of a llava model produced what appear to be modest improvements, although the significance of the result is not clear. however, in the open experiment, it turned out to be difficult to beat gpt-4o with another gpt model, but using a llava-based language model brought noticeable improvements.