Video-Grounded Dialogues with Pretrained Generation Language Models (2020)

Abstract: Pre-trained language models have shown remarkable success in improving various downstream NLP tasks due to their ability to capture dependencies in textual data and generate natural responses. In this paper, we leverage the power of pre-trained language models for improving video-grounded dialogue, which is very challenging and involves complex features of different dynamics: (1) Video features which can extend across both spatial and temporal dimensions; and (2) Dialogue features which involve semantic dependencies over multiple dialogue turns. We propose a framework by extending GPT-2 models to tackle these challenges by formulating video-grounded dialogue tasks as a sequence-to-sequence task, combining both visual and textual representation into a structured sequence, and fine-tuning a large pre-trained GPT-2 network. Our framework allows fine-tuning language models to capture dependencies across multiple modalities over different levels of information: spatio-temporal level in video and token-sentence level in dialogue context. We achieve promising improvement on the Audio-Visual Scene-Aware Dialogues (AVSD) benchmark from DSTC7, which supports a potential direction in this line of research.

## introduction recent work in large-scale pre-training transformerbased neural networks @xcite @xcite has boosted the performance in various nlp tasks. the transformer-based architecture of these models allows them to capture various dependencies when trained on very large datasets. the pre-trained models are adapted into downstream tasks to generate text that is more natural, fluent, and richer than models not initialized with pre-trained weights. similar to pre-trained cnn-based neural networks developed in computer vision research @xcite which can learn highresolution features in images, pre-trained language models (lms) are capable of capturing fine-grain textual dependencies in text data of rich semantics. while the benefits of pre-trained language models present in many downstream nlp tasks such as machine translation and question answering (qa) @xcite , they are particularly suitable to adapt to dialogue response generation tasks for two major reasons: (1) dialogue response generation usually involves more complex dynamics between input and output text sequences. the input typically involves dialogue history, including conversational exchanges between users and dialogue agents. a dialogue agent needs to capture relevant dependencies along each dialogue turns to generate a sensible response. (2) compared to other nlp tasks, it is very challenging to collect and create large-scale dialogue datasets. adopting pre-training approaches could ameliorate the limited dialogue datasets by leveraging rich linguistic dependencies learned from other available text data. we are motivated by these observations to adapt pre-trained language models into a dialogue task and improve the quality of generated responses. along the line of research that combines both vision and language @xcite , transformer-based neural networks can also be applied to capture various dependencies across different types of input modalities (text and image) with appropriate objective loss functions @xcite @xcite . the multi-head attention mechanism of these models can detect long-range dependencies between each token in the input text and each image patch or spatial objects in the input image. we extend this framework to a video-dialogue task and fully lever- video how man y peo ple are in the vide o sep sep ther e pers on one just is pre-trained transformer usr usr usr usr usr usr usr sys usr sys sys sys sys sys dialogue history response 8 9 10 11 12 13 14 7 40 word level modality level position level ther e is pers on one just 5 5 5 5 5 5 5 5 5 5 5 5 5 5 turn level sep sys 2 3 4 5 6 1 1 a man is stan ding sep cap cap cap cap cap 0 0 0 0 0 42 43 44 45 41 caption 52 vis vis vis 47 48 49 50 51 46 -1 -1 -1 vis vis vis vis vis -2 -2 -2 -1 -2 53 spatial level modality level position level temporal level mask ... ... ... ... mask eos ... mask mask mask mask mask mask mask mask mask mask mask mask mask mask mask mask mask mask mask mask @xmath0. specifically, we tackle the audio-visual scene-aware dialogues (avsd) task @xcite which aims to generate dialogue responses grounded on both visual and audio features of the video. the dialogue agent needs to create responses that not only match the dialogue flow but also address user questions about a given video over multiple dialogue turns. first, we detail how to formulate input components of a video-grounded dialogue as a downstream task of pre-trained language models. we follow the general sequence-to-sequence framework, whereby the input components are combined to a structured sequence of multiple modalities and output is a system response. we then apply pre-trained models @xcite to leverage the deep attention neural networks to capture text and video dependencies with fine granularity. specifically, we propose to capture dependencies between each token in text data and each spatial feature along the temporal dimension of the input video. lastly, we present a multi-task learning framework that includes additional learning objectives in addition to dialogue response generation objective. our promising results on the avsd benchmark demonstrate the efficacy of our proposed framework. ## related work we briefly describe related work in two major lines of research: dialogues and vision-text modeling. vulić (2019) assumes access to ground-truth dialogue states and generates responses in task-oriented dialogues by combining input components into a single sequence. as dialogue states and database states are used as raw text input, the models can be fine-tuned from a deep pre-trained language model such as gpt. @xcite and lai et al. (2020) use pre-trained lms to track dialogue states in taskoriented dialogues by utilizing the output representations to predict slot values. in this work, we aim to address video-grounded dialogue tasks and generate natural responses in an end-to-end manner. ## method our model architecture can be seen in figure 1 . we are inspired by transformer-based lm approaches that leverage different levels of features in text, such as word, character, and position levels. we apply this principle and technique to overcome the challenge in avsd which involves multi-turn dialogue input combined with video input with spatialtemporal variations. we propose to decompose videos into patches but maintain a structured temporal sequence. this sequence is then directly combined with text inputs of dialogue which are also arranged in a temporally ordered manner. this kind of feature reformulation is simple yet powerful as it allows explicit dependency learning across all pairs of text tokens and video patches. therefore, it can facilitate stronger signals to answer human queries in greater granularities. ## conclusions in this work, we leverage pre-trained language models for a video-grounded dialogue task. we propose a sequence-to-sequence framework and a multitask fine-tuning approach to adapt the pre-trained models to the video dialogue domain. despite using gpt-2 models, our framework can be extended with other language models and similarly adopted to improve other multi-modal dialogues. our early fusion strategy effectively unifies different levels of features in both dialogues and video without complicating the network architecture