Event-Content-Oriented Dialogue Generation in Short Video (2024)

Abstract: Understanding complex events from different modalities, associating to external knowledge and generating response in a clear point of view are still unexplored in today’s multi-modal dialogue research. The great challenges include 1) lack of event-based multi-modal dialogue dataset; 2) understanding of complex events and 3) heterogeneity gap between different modalities. To overcome these challenges, we firstly introduce a novel event-oriented video-dialogue dataset called SportsVD (Sports-domain Video-dialogue Dataset). To our best knowledge, SportsVD is the first dataset that consists of complex events videos and opinion-based conversations with regards to contents in these events. Meanwhile, we present multi-modal dialogue generation method VCD (Video Commentary Dialogue) to generate human-like response according to event contents in the video and related external knowledge. In contrast to previous video-based dialogue generation, we focus on opinion-based response and the understanding of longer and more complex event contents. We evaluate VCD’s performance on SportsVD and other baselines under several automatic metrics. Experiments demonstrate VCD can outperform among other state-of-the-art baselines. Our work is available at https://github.com/Cheng-Fenghua/SportsVD.

## introduction despite great success achieved in large language models @xcite @xcite , most of them are still limited to incapacity of reading visual information and multi-modal interactions. to build conversational agents' abilities to interact with human and specific environment by not only reading natural language but also perceiving the physical world using all senses is one of long-term goals of artificial intelligence (ai). communicating in a multimodal way can greatly enhance the engagement of user into the communication between agents. therefore, there has been increasing interest on multi-modal dialogue generation. recent works have @xcite achieved a significant progress in multi-modal dialogue generation. however, there are still research gaps. first, much research interest are focused on image-grounded dialogue. video-grounded dialogue generation still needs to be explored. state-of-the-art works on video-grounded chatbot are still unsatisfying due to difficulty in extracting information from video and integrating external knowledge. second, existing video-grounded dialogue works mainly focus on entity movement but more complex event content is neglected. however, this differs largely from real-world scenarios. a longer and more complex event is more likely to be the topic of a conversation. to this end, constructing a dialogue generating model that can understand more complex scenarios and events is necessary for ai to interact with the real world. here we give formal definition of entity movement and event contents to distinguish them easily. event-contents based dialogue generation is still unexplored. third, much attention is concentrated on question-answer pairs in multi-modal dialogue generation. containing only question and answer in a dialogue is also inconsistent with reality as simple q&a is largely based on facts whereas opinions are much more involved in real dialogue. hence, opinion-based response generation still needs to be explored. here we give a formal definition of fact-based dialogues and opinion-based dialogues: to this end, we propose a new dataset sportsvd which contains dialogue-video pairs on sportsdomain events together with vcd, a novel multimodal dialogue generation method based on understanding of event contents and external knowledge. sportsvd contains 5114 sports-events videos and 39k corresponding dialogues. we collect videos and multi-turn comments as dialogues from youtube. comments from real users based on their own knowledge help form high-quality opinionbased conversations. figure 1 demonstrates some examples in sportsvd. we choose sports-domain events to form our dataset because we believe people are highly likely to express diverse opinions when watching a sports event. vcd utilizes video encoding from pre-trained video-language model internvideo @xcite and unified transformer structure and input representation to reduce heterogeneity gap. additionally, we utilize prompt to integrate external knowledge into dialogue generation. we evaluate vcd and other baselines on sportsvd under both automatic metrics and human evaluation. experiments demonstrates the efficiency and effective of our work. our model can outperform other baselines. in summary, the contributions of this work include: to our best knowledge, sportsvd is the first video-dialogue dataset based on complex events. 2. we present a novel dialogue generation method vcd which generate response based on event contents shown in a short video and external knowledge. 3. we provide data analysis of the new dataset sportsvd and compares it with other multimodal dataset. 4. we evaluate vcd on sportsvd and compare it to other state-of-the-art baselines. ## related works we conclude recent advances achieved in two related areas which are related to this work. ## sportsvd dataset to create a dataset specially for the event-content dialogue generation is an inevitable task in this work due to lack of training corpus. to this end, we present sportsvd which contains 5114 sportsevent based videos and 39k opinion-based dialogues regarding to them. we desire these videos to contain at least one event and dialogues to be not question-answering but opinion-based and highly related to events in the video. in this section, we firstly introduce the data collection method and then detail the analysis of the new dataset. ## video commentary dialogue we propose vcd (video commentary dialogue), a new multi-modal dialogue generation method that understand event contents from multiple modalities in a video and generate opinion-based response. to understand longer and more complex event contents in a video, we samples several frames from the video and utilizes videointern @xcite to encode them. meanwhile, in order to reduce modality gaps, unified transformer structure and input representation are applied. besides, we integrate external knowledge from atomic @xcite and google knowledge graph foot_0 by natural language prompt. ## experiments we conduct experiments among vcd and other state-of-the-art baselines on sportsvd under several automatic metrics. in this section, we elaborate the experiments and results. ## conclusion in this paper, we introduce a new multi-modal dialogue dataset sportsvd. to our best knowledge, it is the first events-content-based videodialogue dataset. we collect videos and dialogues in sports domains on youtube. real users' comments ensure the quality of dialogue and clarity of views towards the event. meanwhile, we present vcd, a novel video-dialogue generation method which can understand event contents, integrate external knowledge and generate reasonable response. experiments between vcd and other baselines on sportsvd demonstrate the outstanding performance of our method.