VGNMN : Video-grounded Neural Module Networks for Video-Grounded Dialogue Systems (2022)

Abstract: Neural module networks (NMN) have achieved success in image-grounded tasks such as Visual Question Answering (VQA) on synthetic images. However, very limited work on NMN has been studied in the video-grounded dialogue tasks. These tasks extend the complexity of traditional visual tasks with the additional visual temporal variance and language cross-turn dependencies. Motivated by recent NMN approaches on image-grounded tasks, we introduce Video-grounded Neural Module Network (VGNMN) to model the information retrieval process in video-grounded language tasks as a pipeline of neural modules. VGNMN first decomposes all language components in dialogues to explicitly resolve any entity references and detect corresponding action-based inputs from the question. The detected entities and actions are used as parameters to instantiate neural module networks and extract visual cues from the video. Our experiments show that VGNMN can achieve promising performance on a challenging video-grounded dialogue benchmark as well as a video QA benchmark.

## introduction vision-language tasks have been studied to build intelligent systems that can perceive information from multiple modalities, such as images, videos, and text. extended from image-grounded tasks, e.g. @xcite , recently @xcite ; @xcite propose to use video as the grounding features. this modification poses a significant challenge to previous image-based models with the additional temporal variance through video frames. recently @xcite further develop videogrounded language research into the dialogue domain. in the proposed task, video-grounded dialogues, the dialogue agent is required to answer questions about a video over multiple dialogue turns. using figure 1 as an example, to answer video caption: a boy and a man walk to the room. the boy carries his backpack while the man… visual: ... audio: ... question: what is he doing while carrying it? find(he) find(it) summarize() where (a backpack) where (a boy) when( while carryi ng it) describe (what) dialog history question: how many people are in the video? answer: there are a boy and a man question: what is the boy doing? answer: the boy walks downstairs and carries a backpack predicted answer: he is cleaning a mirror . ... ... ## method in this section, we present the design of our model. an overview of the model can be seen in figure 2 . ## experiments datasets. we use the avsd benchmark from the dialogue system technology challenge 7 (dstc7) @xcite . the benchmark consists of dialogues grounded on the charades videos @xcite . each dialogue contains up to 10 dialogue turns, each turn consists of a question and expected response about a given video. for visual features, we use the 3d cnn-based features from a pretrained i3d model @xcite and object-level features from a pretrained fasterrnn model @xcite . the audio features are obtained from a pretrained vggish model @xcite . in the experiments with avsd, we consider two settings: one with video summary and one without video summary as input. in the setting with video summary, the summary is concatenated to the dialogue history before the first dialogue turn. we also adapt vgnmn to the video qa benchmark tgif-qa @xcite . different from avsd, tgif-qa contains a diverse set of qa tasks: frame: an open-ended qa about visual contents of one of the video frames for the tgif-qa benchmark, we use the extracted features from a pretrained resnet model @xcite . training details. we follow prior approaches @xcite @xcite by obtaining the annotations of the programs through a language parser @xcite and a reference resolution model @xcite . during training, we directly use these as groundtruth labels of programs to train our models. the ground-truth responses are augmented with label smoothing technique @xcite . during inference time, we generate all programs and responses from given dialogues and videos. we run beam search to enumerate programs for dialogue and video understanding and dialogue responses. we use a training batch size of 32 and embedding dimension @xmath0. where transformer attention is used, we fix the number of attention heads to 8 in all attention layers. in neural modules with mlp layers, the mlp network is fixed to 2 linear layers with a relu activation in between. in neural modules with cnn, we adopt a vanilla cnn architecture for text classification (without the last mlp layer) where the number of input channels is 1, the kernel sizes are {3, 4, 5}, and the number of output channels is d. we initialize models with uniform distribution @xcite . during training, we adopt the adam optimizer (kingma and ba, 2015) and a decaying learning rate @xcite where we fix the warm-up steps to 15k training steps. we employ dropout @xcite of 0.2 at all networks except the last linear layers of question parsers and response decoder. we train models up to 50 epochs and select the best models based on the average loss per epoch in the validation set. all models are trained in a v100 gpu with a capacity of 16gb. we approximated each training epoch took about 20 minutes to run. for each model experiment with vgnmn, we obtained at least 2 runs and reported the average results. we implemented models in pytorch and released the code and model checkpoints 1 . optimization. we optimize models by joint training to minimize the cross-entropy losses to generate responses and functional programs. ## conclusion in this work, we introduce video-grounded neural module network (vgnmn). vgnmn consists of dialogue and video understanding neural modules, each of which performs entity and action-level operations on language and video components. our comprehensive experiments on avsd and tgif-qa benchmarks show that our models can achieve competitive performance while promoting a compositional and interpretable learning approach.