B i ST : Bi-directional Spatio-Temporal Reasoning for Video-Grounded Dialogues (2020)

Abstract: Video-grounded dialogues are very challenging due to (i) the complexity of videos which contain both spatial and temporal variations, and (ii) the complexity of user utterances which query different segments and/or different objects in videos over multiple dialogue turns. However, existing approaches to video-grounded dialogues often focus on superficial temporal-level visual cues, but neglect more fine-grained spatial signals from videos. To address this drawback, we proposed Bi-directional Spatio-Temporal Learning (BiST), a vision-language neural framework for high-resolution queries in videos based on textual cues. Specifically, our approach not only exploits both spatial and temporal-level information, but also learns dynamic information diffusion between the two feature spaces through spatial-to-temporal and temporal-to-spatial reasoning. The bidirectional strategy aims to tackle the evolving semantics of user queries in the dialogue setting. The retrieved visual cues are used as contextual information to construct relevant responses to the users. Our empirical results and comprehensive qualitative analysis show that BiST achieves competitive performance and generates reasonable responses on a large-scale AVSD benchmark. We also adapt our BiST models to the Video QA setting, and substantially outperform prior approaches on the TGIF-QA benchmark.

## introduction a video-grounded dialogue agent aims to converse with humans not only based on signals from natural language but also from other modalities such as sound and vision of the input video. recent efforts @xcite @xcite consider video-grounded dialogues as an extension of video question-answering (qa) figure 1 : examples of video-grounded dialogues from the benchmark datasets of audio-visual scene aware dialogues (avsd) challenge @xcite . h: human, a: the dialogue agent. @xcite @xcite whereby the agent answers questions from humans over multiple turns rather than a single turn (see figure 1 ). this is a very complex task as the dialogue agent needs to possess not only strong language understanding to generate natural responses but also sophisticated reasoning over video information, including the related objects, their positions and motions, etc. compared to image-based nlp tasks such as image qa and captioning @xcite @xcite , video-grounded dialogues are more challenging as the feature representation of a video involves both spatial and temporal dimensions. ideally, a dialogue agent has to process information of both dimensions to address the two major questions: "where to look" (spatial reasoning) and "when to look" (temporal reasoning) in the video. however, current approaches in video-grounded dialogues @xcite @xcite often overlook spatial features and assume each spatial region is equally important to the current task (each spatial region is assigned with a uniform weight). such approach is appropriate for cases where the video involves just few objects and spatial positions can be treated similarly. however, in many scenarios (e.g. examples in figure 1 ), each video frame often contains multiple distinct objects and not all of them are relevant to the given question. related tasks to video-grounded dialogues are video qa and video captioning. previous efforts in these research areas such as @xcite explicitly consider both spatial and temporal features of input video. these models learn to summarize spatial features based on their importance to question rather than considering each region equally. we are motivated by these approaches and propose to extend spatiotemporal reasoning to dialogues. however, rather than fixing on processing spatial inputs then learning temporal inputs, we note that in some cases, e.g. extended videos over a long period, it is more practical to first identify the relevant video segments before pinpointing the specific subjects of interest. considering questions in a dialogue setting, it is appropriate to assume the questions are relevant to varying temporal locations of the video rather than just a small fixed segment. we, thus, propose to explore a bidirectional vision-language reasoning approach to fully exploit both spatial and temporallevel features through two reasoning directions. our approach includes two parallel networks to learn relevant visual signals from the input video based on the language signals from user utterances. each network projects the language-based features to a three-dimensional tensor which is then used to independently learn video signals following a reasoning direction either as @xmath0. the output from each network is dynamically combined by importance scores computed based on language and visual features. the weighted output is recurrently used as input to the reasoning modules to allow the models to progressively derive relevant video signals over multiple steps. intuitively, @xmath1. @xmath2. we name our proposed approach bidirectional spatio-temporal learning (bist), with the following contributions: (1) rather than exploit-ing temporal-level information only, our approach equally emphasizes both spatial and temporal features of videos for higher-resolution queries of visual cues. (2) to tackle the diverse queried information from conversational queries, we propose a bidirectional strategy, denoted spatial↔temporal, to enable comprehensive information diffusion between the two visual feature spaces. (3) our models achieve competitive performance on the "avsd" (audio-visual scene aware dialogues) benchmark from the 7 th dialogue system technology challenge (dstc7) @xcite . we adapt our models to a video qa task "tgif-qa" @xcite and achieve significant performance gains. (4) we conduct a comprehensive ablation and qualitative analysis and demonstrate the efficacy of our bidirectional reasoning approach. ## related work our work is related to two research topics: videogrounded dialogues and spatio-temporal learning. video-grounded dialogues. following recent efforts that combine nlp and computer vision research @xcite @xcite , video-grounded dialogues are extended from the two major research fields: video action recognition and detection @xcite @xcite and dialogues/qa @xcite @xcite . approaches to video-grounded dialogues @xcite @xcite typically use pretrained video models, such as 2d cnn models on video frames @xcite , and 3d cnn models on video clips @xcite , to extract visual features. however, these approaches mostly exploit the superficial information from the temporal dimension and neglect spatial-level signals. these approaches integrate spatial-level features simply through sum pooling with equal weights to obtain a global representation at the temporal level. they are, thus, not ideal for complex questions that investigate entity-level or spatial-level information @xcite . the dialogue setting exacerbates this limitation as it allows users to explore various aspects of the video contents, including both low-level (spatial) and high-level (temporal) information, over multiple dialogue turns. our approach aims to address this challenge in video-grounded dialogues by retrieving fine-grained information from video through a bidirectional reasoning framework. spatio-temporal learning. most efforts in spatiotemporal learning focus on action recognition or detection tasks. @xcite proposes to progressively refine coarse-scale information through temporal extension and spatial displacement for action detection. @xcite uses a shared network of 2d cnns over three orthogonal views of video to obtain spatial and temporal signals for action recognition. @xcite adopts a twopath network architecture that integrates global and local information of both temporal and spatial dimensions for video classification. other research areas that investigate spatio-temporal learning include video captioning @xcite , video super-resolution @xcite , and video object segmentation @xcite . in general, spatio-temporal learning approaches aim to process higher-resolution information from complex videos that involve multiple objects in each video frame or motions over video segments @xcite . we are motivated by a similar reason observed in video-grounded dialogues and explore a vision-language bidirectional reasoning approach to obtain more fine-grained visual features. ## bist model the input includes a video v , dialogue history of (t -1) turns (where t is the current turn), each including a pair of (human utterance h, dialogue agent response a) (h 1 , a 1 , ..., h t-1 , a t-1 ), and current human utterance h t . the output is a system response a t that can address current human utterance. the input video can contain features in different modalities, including vision, audio, and text (such as video caption or subtitle). without loss of generalization, we can denote each text input as a sequence of tokens, each represented by a unique token index from a vocabulary set v : dialogue history x his , user utterance x que , text input of video x cap , and output response y . we also denote l s as the length of a sequence s. for instance, l que is the length of x que . our model is composed of four parts: (1) the encoders encode text sequences and video inputs, including visual, audio, and text features, into continuous representations. for non-text features such as vision and sound, we follow previous work @xcite and assume access to pre-trained models. (2) several neural reasoning components learn dependencies between user utterances/queries and video features of multiple modalities. for video visual features, we propose to learn dependencies at both spatial and temporal levels in two directions (see figure 2 ). specifically, we allow interaction between each token in user query and each spatial position or temporal step of the video. the outputs from spatial-based or temporal-based reasoning are sequentially incorporated in two directions, @xmath3. the bidirectional strategy enables information being fused dynamically and captures complex dependencies between textual signals from dialogues and visual signals from videos. ## conclusion we proposed bist, a novel deep neural network approach for video-grounded dialogues and video qa, which exploits the complex visual nuances of videos through a bidirectional reasoning framework in both spatial and temporal dimensions. our experimental results show that bist can extract relevant, high-resolution visual cues from videos and generate quality dialogue responses/answers.