VSTAR : A Video-grounded Dialogue Dataset for Situated Semantic Understanding with Scene and Topic Transitions (2023)

Abstract: Video-grounded dialogue understanding is a challenging problem that requires machine to perceive, parse and reason over situated semantics extracted from weakly aligned video and dialogues. Most existing benchmarks treat both modalities the same as a frame-independent visual understanding task, while neglecting the intrinsic attributes in multimodal dialogues, such as scene and topic transitions. In this paper, we present Video-grounded Scene&Topic AwaRe dialogue (VSTAR) dataset, a large scale video-grounded dialogue understanding dataset based on 395 TV series. Based on VSTAR, we propose two benchmarks for video-grounded dialogue understanding: scene segmentation and topic segmentation, and one benchmark for video-grounded dialogue generation. Comprehensive experiments are performed on these benchmarks to demonstrate the importance of multimodal information and segments in video-grounded dialogue understanding and generation.

## introduction however, these abilities have barely been investigated in modern literature of multimodal conversational understanding. over the recent years, we have witnessed a growing trend of modeling longer and more diverse video-grounded dialogues due to growing computing capacity and hardware acceleration. however, most tasks are based on multi-turn visual question answering (vqa) for its simplicity in answer evaluation @xcite , which yields huge differences compared with realistic conversations. to fill the gap between the current multimodal dialogue systems and realistic video-grounded dialogue understanding, in this work, we introduce a new challenge, video-grounded scene&topic aware dialogue (vstar) dataset, aiming to address the frequent scene and topic transition within open-domain video-grounded dialogues. we start by constructing a vstar dataset by collecting a total of 395 tv series and 8,159 tv episodes together with their corresponding story-line and keywords; figure 1 shows a typical example. to test the machine's abilities on scene and topic transition identifications, we annotate video scene boundary and dialogue topic boundary through determining the semantic transition among short video-dialogue turn pairs. complicated video understanding. we carefully selected story-based tv series as the main data source. compared with movies or homemade short videos, tv series containing more complicated plots: there are many remarkable scene transitions and topic shifts within each video clip, which brings in extra challenge in video understanding. multimodal scene&topic transition identification. it is worth noting that identifying scene and topic boundary in a video clip is non-trivial. besides the complexity of long videos, both identifications require reasoning over multi-modalities. for example in figure 1 , it is hard to separate topics or scenes solely based on visual cues or dialogue text. ## related work recently, multimodal dialogue systems have attracted interest of many researchers with a number of benchmarks and datasets proposed. visualdialog @xcite @xcite @xcite treated the problem as a multi-turn vqa that aims to develop a dialogue agent that can answer questions given dialogue histories and corresponding images. empowered by today's high computational capacity, similar dialogue systems have been extended to the video domain @xcite : questions about a given video are positioned in a multi-turn qa-pairs. in each qapair turn, a question usually exhibits different types of cross-turn relations to other questions in prior turns. to mimic the situations where common dialogues happen. twitch-fifa @xcite and monologue. one of the closest work to us is the recently proposed movienet @xcite , a large scale dataset for movie understanding. in this work, shot is viewed as the minimal visual unit of a movie. however, shot boundary is annotated automatically by existing tool @xcite , which leads to inaccurate predictions. besides, few methods take the multi-modal information as input which make current models learn little high-level semantic information. ## the vstar dataset the vstar is collected from 395 tv series (8,159 episodes, 185k 90-second clips) with carefully cleaned dialogues and metadata information. the collection and data cleaning details are as follow: data source we carefully selected and purchased video sources from the copies of blueray disks and online reddit open directories. specifically, we filtered out animation and documentary series because the former differ a lot in terms of visual input styles and are unrealistic, while the latter mostly contain monologues. to ensure the quality of selected series, we selected the tv shows rated by over 1,000 imdb users. the ultimate genre distribution is shown in figure 2a . as seen, there are 19 genres covering almost all common genres of tv series. for research purposes, we segmented each tv episode into 90-second video clips. in the end, we got 185k multi-modal dialogue clips. metadata we crawled metadata for each episode from imdb foot_0 as complementary information. each tv episode is paired with genres, keywords, and storylines. compared to daily chat, the conversations in tv series are much longer and contain richer background knowledge. therefore the metadata will be helpful for further dialogue understanding. we show the wordcloud of the story-lines in figure 2b . it is interesting to observe that most salient words are relevant with "work and life". ## benchmarks and models we set three benchmarks based on vstar for video-grounded dialogue understanding and gener-ation. in sec.4.1, we first introduce the task formulation along with their evaluation metrics. then in sec.4.2 we present our proposed transformer-based video dialogue frameworks to benchmark our tasks. ## experiments we split the whole dataset into train, val, and test with the ratio 17:1:1 on utterance level. for each task, we demonstrate the main results compared with baseline methods along with ablation studies on each module. detailed information about baseline methods and implementation details are introduced in ?? and ??. ## limitations and ethics concerns we point out potential limitations and ethical concerns of this work. limitation: data and modeling the dialogues in our dataset are made by playwright, which are slightly different from daily chat. second, the automatic evaluation metrics for the response generation task can not perfectly reflect the interactiveness of dialogue system. lastly, our autoregressive generative model simply add the segment embedding to the inputs. similar to the position encoding in transformer, our coarse method does not make good use of the segmentation, and lacks interpretability. ethics: copyright and licensing the data source are publicly available tv series. its collection and annotation procedure is designed for videogrouned dialogue understanding and generation purpose, and does not involve privacy issues. following lsmdc @xcite and movienet @xcite , we will polish an agreement and release tv shows content under very strict conditions but will open-source all the scrawling code, pretrained features and sampled images. ## conclusion in this paper, we introduce vstar, a scene and topic-aware video-grounded dialogue understanding and generation dataset. the main purpose of our dataset is to improve the situated multimodal semantic perception capability of the dialogue system, so that the dialogue system can generate the response that are both semantically and logically consistent with the dialogue scene in common situation. we introduce three challenging benchmarks on different aspects of video-grounded dialogue understanding and generation, i.e., discovering scene transition and topic transition on video-grounded dialogue, and getting proper response. furthermore, we propose three new baselines for corresponding benchmarks. experiment results shows the multi-modal information can benefit dialogue understanding. besides, we find scene boundary and topic scene boundary contribute to generate more relevant and coherent responses. by introducing vstar, we hope to shed light on future research towards building conversational agents that can comprehend complicated realistic multimodal signals.