Sound of Story: Multi-modal Storytelling with Audio (2023)

Abstract: Storytelling is multi-modal in the real world. When one tells a story, one may use all of the visualizations and sounds along with the story itself. However, prior studies on storytelling datasets and tasks have paid little attention to sound even though sound also conveys meaningful semantics of the story. Therefore, we propose to extend story understanding and telling areas by establishing a new component called background sound which is story context-based audio without any linguistic information. For this purpose, we introduce a new dataset, called Sound of Story (SoS), which has paired image and text sequences with corresponding sound or background music for a story. To the best of our knowledge, this is the largest well-curated dataset for storytelling with sound. Our SoS dataset consists of 27,354 stories with 19.6 images per story and 984 hours of speech-decoupled audio such as background music and other sounds. As benchmark tasks for storytelling with sound and the dataset, we propose retrieval tasks between modalities, and audio generation tasks from image-text sequences, introducing strong baselines for them. We believe the proposed dataset and tasks may shed light on the multi-modal understanding of storytelling in terms of sound.

## introduction story understanding and telling is one of the important but challenging problems in natural language processing and machine learning, and there have been extensive studies on this problem for decades @xcite . many prior researches have focused on building models that understand stories such as utilizing pre-trained model @xcite * these authors contributed equally to this work. 2022; @xcite , eventcentric understanding @xcite and using multiway-sampler @xcite . also, lots of approaches are proposed that aim to understand the stories within the provided text @xcite @xcite , image @xcite @xcite and video @xcite @xcite @xcite . despite there being many benchmarks for story understanding and telling, most of them less focus on audio even if it is one of the key elements for humans to understand stories @xcite . therefore, we propose to extend story understanding and telling areas by establishing a new component called background sound which is story context-based audio without any linguistic information. for this purpose, we introduce a novel dataset called the sound of story (sos), which aims to leverage audio information for story understanding, distinguishing it from other existing story or audio datasets. to create the sos dataset, all the modalities are extracted from condensed movie dataset (cmd) @xcite and large scale movie description challenge datasets (lsmdc) @xcite . image and text data extracted from the same movie clip have their contexts and both of them have the possibility to infer the story from the same movie clip. and in the case of audio such as background sound and music, it has the ability to enrich storytelling by representing and recognizing the context of the situations @xcite . due to these characteristics, audio, image, and text extracted from the same movie clip become correlated and complementary. so, our sos dataset allows three modalities generated from the same movie clip to be used for story understanding as a pair. in addition, we propose new story underimage sequence . . . . . . <image @xmath0: overall data generating process flow for our sound of story (sos) dataset. all the modalities are extracted from the same movie clips. we exclude the speech from audio to make it contain only pure audio data, which does not contain linguistic information. these background sound tasks can be utilized in various applications. for example, for movies and cartoons, background sound is often produced according to a designated scene using various props or objects for a sense of reality. in the case of the retrieval task, among the sound sources made, it is possible to recommend the sound source which fits the best to the scene. furthermore, background sound generation can produce more realistic audio while reducing human resource costs by directly generating the right audio for a given scene. the overall data generating process can be shown in figure 1 . in summary, our contributions are as follows: for this purpose, we introduce a new dataset named sound of story (sos) consisting of text and image sequences and audio for a story. to the best of our knowledge, this is the largest well-curated dataset for storytelling with sound. in addition to text-based story plots, there exist numerous story datasets that incorporate images or videos. visual storytelling @xcite introduces the first sequential vision-to-language dataset and explores its applicability in various tasks. it goes beyond simple object understanding or captioning for concrete scenes by considering input that encompasses temporal events. similarly, imagecode dataset @xcite also contains stories but is designed to enable models to the football coach, the assistants, and alabama players cheer for forrest.forrest runs across the field.he speeds past the defending players.forrest runs past the opposite players.the football coach runs along the sidelines as he yells. forrest … jackie breaks into tears.they share a helpless half-laugh --then frank appears in the doorway.she starts to hand him a large mirror from her purse --then thinks better of it.but he holds out his hand and she gives him the mirror.simon holds … … … dora parties with her friends over the end credits. po's biological father li shows up at his village. learn detailed situations from finer visual differences within similar sets of images. movie data inherently encompasses storytelling. various datasets such as movieqa @xcite and movienet @xcite have been collected by using movie data and have been used in various multimodal domains. additionally, datasets like cmd @xcite and lsmdc @xcite not only provide movie videos but also include descriptions for the corresponding clips. this offers researchers an opportunity to explore models from a multimodal perspective. however, there are few datasets for story with sound. existing benchmarks for audio mostly consist of music @xcite @xcite or animal/environmental sounds @xcite @xcite , and are designed to evaluate simple matching performance between objects (or sentiment) and audio input. multi-modal retrieval tasks multi-modal retrieval models are trained to find similar features across different modalities by fusing. these models are tasked with understanding both the context of a given input and the interplay between features provided by other modalities. as a result, retrieval becomes a highly challenging and crucial task in order to comprehend the distinctive characteristics of multiple modalities. the multi-modal retrieval task has been studied using various modalities such as image-text retrieval @xcite @xcite @xcite , video-text @xcite @xcite , audio-image @xcite @xcite , video-audio @xcite @xcite and audio-text @xcite . particularly, clip4clip @xcite , which performs well in the videotext retrieval task by calculating the similarities between the features of each modality obtained from the encoder, and x-clip @xcite expands clip4clip and proposes a multi-grained regulation function to improve performance. furthermore, wav2clip @xcite conducts retrieval for audio-image modalities based on clip architecture @xcite , while clap @xcite focuses on audio-text retrieval. in this paper, we propose a story-based audio retrieval task for retrieving proper modal features like image sequences or descriptions. ## dataset we propose a novel dataset named the sos dataset with 27,354 audio-image-text pairs. 22,330 pairs are created from cmd @xcite and 5,024 from lsmdc dataset @xcite . we extract the image and audio from the same video and pair them with the description accordingly so that the images and text share the same storyline. in the case of audio, it is related to the other modalities due to being extracted from the same clip so can be used in complementary. the examples of the sos dataset are shown in figure 2 . ## conclusion in this paper, we emphasize that sound also conveys meaningful semantics of the story. thus, we propose a novel dataset named sound of story (sos) dataset, audio-image-text pair dataset. one main feature of our dataset is that all the linguistic information inside the audio is eliminated. we named the audio without any linguistic information as "background sound". background sounds can be utilized in various applications and we propose two related tasks, retrieval task and background sound generation task. to give an example of using it from videos such as movies and cartoons, background sound is often produced according to a designated scene using various props or objects for a sense of reality. in the case of the retrieval task, among the different sound sources made, it is possible to recommend the sound source which fits the best to the scene. furthermore, background sound generation can produce more realistic audio while reducing human resource costs by directly generating the right audio for a given scene. to the best of our knowledge, it is the first attempt to do multimodal tasks with the speech-decoupled background sound relevant to the storyline.