Collaborative Reasoning on Multi-Modal Semantic Graphs for Video-Grounded Dialogue Generation (2022)

Abstract: We study video-grounded dialogue generation, where a response is generated based on the dialogue context and the associated video. The primary challenges of this task lie in (1) the difficulty of integrating video data into pre-trained language models (PLMs) which presents obstacles to exploiting the power of large-scale pre-training; and (2) the necessity of taking into account the complementarity of various modalities throughout the reasoning process. Although having made remarkable progress in video-grounded dialogue generation, existing methods still fall short when it comes to integrating with PLMs in a way that allows information from different modalities to complement each other. To alleviate these issues, we first propose extracting pertinent information from videos and turning it into reasoning paths that are acceptable to PLMs. Additionally, we propose a multi-agent reinforcement learning method to collaboratively perform reasoning on different modalities (i.e., video and dialogue context). Empirical experiment results on two public datasets indicate that the proposed model can significantly outperform state-of-the-art models by large margins on both automatic and human evaluations.

## introduction conversing with computers has become a crucial step toward general artificial intelligence, and it has attracted increasing attention from ai and nlp researchers. multi-turn dialogue response generation and multi-modal question answering are two high-profile initiatives made toward this goal. the task of multi-turn dialogue response generation necessitates the agent comprehending the key information in the dialogue context in order to provide a cohesive, fluent and informative response @xcite . multi-modal question answering, on the other hand, necessitates the * equal contribution. although having made notable progress in videogrounded dialogue, existing approaches still fail to recognize the aforementioned challenges. on one hand, existing approaches cannot be effectively combined with plms, which presents obstacles to exploiting the power of state-of-the-art pre-training technology. the reasons can be summarized into two categories: (1) simply appending the video features to the text embeddings presents a challenge for the model to obtain an in-depth understanding of the video @xcite @xcite . to investigate this problem further, we compare the performance of these models before and after removing the video from the input. as demonstrated in existing approaches fall short when it comes to reasoning jointly on multi-modalities, since they either separate the reasoning of different modalities @xcite or employ a cross-modal attention mechanism which is difficult to train without direct supervision @xcite @xcite . to address the aforementioned issues, we propose extracting relevant information from videos and converting it into reasoning paths, which are in the form of natural language and can be fed directly into plms. besides, we propose a multiagent reasoning framework that is based on the multi-agent reinforcement learning (marl) theory. specifically, we design a video agent and a context agent which learn to find the chains of reasoning on the multi-modal semantic graphs. we further design a central communicator to make the two agents work in a collaborative manner. our framework has the following advantages: (1) the multi-modal reasoning paths are compatible with the input of plms; (2) the reasoning process can be "supervised" by designing appropriate reward functions; and (3) the communication mechanism allows the information from different modalities better complement each other. we conduct extensive experiments on two benchmark datasets for video-grounded dialogue generation, including avsd@dstc7 @xcite and twitch-fifa @xcite . experiment results show that, thanks to the multi-agent reasoning framework, our model can significantly outperform state-of-the-art methods in terms of both automatic and human evaluations. our contributions in the paper are three-fold: (1) identifying the issue that current plms-based approaches are unable to fully comprehend the video content although showing promising results in automatic evaluation metrics. (2) proposal of a multiagent reasoning framework upon plms that can let information from different modalities reinforce each other and discover multi-modal reasoning paths. (3) empirical verification of the effectiveness of the proposed model on two benchmarks of video-grounded dialogue generation. ## related work the majority of early works on dialogue generation use hand-crafted rules or templates to construct dialogue systems @xcite . a number of initiatives have been made to develop end-to-end open-domain dialogue generation models @xcite @xcite , which have been inspired by the developments in the field of machine translation. following that, the vanilla encoder-decoder architecture is frequently utilized to enhance response quality, and numerous modifications to this architecture have been made to enhance response diversity @xcite , model the structure of conversation contexts @xcite , introduce external knowledge @xcite and control response attributes @xcite @xcite . the research on generating dialogue from video was started by @xcite . after that, @xcite present an lstm-based encoderdecoder architecture with multi-modal attention that merely combines textual and visual data via a projection matrix. a multi-modal transformer network is introduced in @xcite to encode videos and incorporate data from several modalities. @xcite uses a joint student-teacher learning approach to make up for a missing video description in which the student network is trained to mimic the teacher's response. vgd-gpt @xcite ) is based on a pre-trained gpt-2 model and formulates the video-grounded dialogue generation as a sequence-to-sequence task. on a pre-trained gpt-2 model, rlm @xcite provides a multi-task learning strategy. additionally, bist @xcite ) models the dependencies between text and visual in two directions: spatial to temporal and temporal to spatial. with visual attention, pdc-gpt @xcite learns to anticipate the reasoning process on turn-level semantic graphs. for further reasoning, scga @xcite constructs a structured graph based on a multi-modal coreference technique, while stsgr @xcite introduce a shuffled transformer reasoning framework on semantic scene graph. in contrast to previous approaches, this paper focuses on how to build a multi-modal reasoning approach that can cooperate with plms in a way that facilitates the complementary nature of information from various modalities. the study of reasoning on various types of graph structures for dialogue generation is related to our work. @xcite create a kg walk path for each entity retrieved in an effort to explain conversational reasoning in a natural way. jung et al. (2020) develop a dialogue-conditioned path traversal model with attention flows and improve the comprehension of the path reasoning process. @xcite propose to represent dialogue transitions as graphs. previous approaches typically concentrate on textual graphs, but video-grounded dialogue contains multi-modal contexts, which makes it difficult to conduct reasoning. ## conclusion we propose a multi-modal reasoning framework that can be used in conjunction with plms to enable the complementation of information from various modalities. specifically, we devise a video agent and a context agent to extract reasoning paths on video and dialogue contexts respectively. a central communicator is also designed to transport information between the two agents and enables their cooperative operation. the general framework is optimized through multi-agent reinforcement learning. evaluation results on two benchmarks indicate that our model can significantly outperform stateof-the-art methods.