C3: Compositional Counterfactual Contrastive Learning for Video-grounded Dialogues (2023)

Abstract: Video-grounded dialogue systems aim to integrate video understanding and dialogue understanding to generate responses that are relevant to both the dialogue and video context. Most existing approaches employ deep learning models and have achieved remarkable performance, given the relatively small datasets available. However, the results are partially accomplished by exploiting biases in the datasets rather than developing multimodal reasoning, resulting in limited generalization. In this paper, we propose a novel approach of Compositional Counterfactual Contrastive Learning (C3) to develop contrastive training between factual and counterfactual samples in video-grounded dialogues. Specifically, we design factual/counterfactual samples based on the temporal steps in videos and tokens in dialogues and propose contrastive loss functions that exploit object-level or action-level variance. Different from prior approaches, we focus on contrastive hidden state representations among compositional output tokens to optimize the representation space in a generation setting. We achieved promising performance gains on the Audio-Visual Scene-Aware Dialogues (AVSD) benchmark and showed the benefits of our approach in grounding video and dialogue context.

## introduction visual dialogue research @xcite @xcite @xcite aims to develop intelligent systems that can reason and answer questions about visual content in a multi-turn setting. compared to traditional visual question answering (vqa) @xcite @xcite , visual dialogues bridge the gap between research and practical applications by allowing turn-based human-machine interactions. recently, many deep learning approaches have been proposed to develop system: he walks towards the back of the living room human: does he walk back and forth in the video? visual dialogue systems and achieved remarkable performance @xcite @xcite . however, as these methods are heavily trained on relatively small datasets @xcite , they are subject to inherent bias from the datasets and limited generalization into real-world applications @xcite . while training on large-scale data can alleviate this problem, visual dialogues are expensive to procure and require manual annotations. this challenge becomes more obvious in highly complex visual dialogue tasks such as video-grounded dialogues @xcite (figure 1 ). in recent years, we have seen increasing research efforts in contrastive learning to improve deep learning performance @xcite @xcite . the common strategy of these methods is an objective function that pulls together representations of an anchor and "positive" samples while pushing the representations of the anchor from "negative" samples. these methods are specifically beneficial in selfsupervised image representation learning. specifically, these methods often do not require additional annotations by augmenting data of existing samples to create "positive" and "negative" samples. we are motivated by this line of research to improve visual dialogue systems and propose a framework of compositional counterfactual contrastive learning (c 3 ). c 3 includes loss functions that exploit contrastive training samples of factual and counterfactual data that are augmented to be object-variant or action-variant. compared to traditional deep learning tasks, a major challenge of applying contrastive learning @xcite @xcite in video-grounded dialogues lies in the complexity of the task. specifically, in a discrimination task of image classification, given an image, positive samples are created based on non-adversarial transformations on this image e.g. by cropping inessential parts without changing the labels, and negative samples are randomly sampled from other image instances. however, such transformations are not straightforward to apply on visual dialogues, each of which consists of a video of spatio-temporal dimensions, a dialogue of multiple turns, and an output label in the form of natural language at the sentence level. in visual dialogues, the random sampling method, in which negative samples are created by swapping the input video and/or dialogue context with random components from other training samples, becomes too naive. in domains with high data variance like dialogues or videos, a system can easily discriminate between such positive and negative instances derived using previous approaches. to mitigate the limitations of conventional contrastive learning in video-grounded dialogues, we propose a principled approach to generate and control negative and positive pairs by incorporating compositionality and causality (an overview of our approach can be seen in figure 2 and 3 ). specifically, we develop a structural causal model for visual dialogues by decomposing model components by object and action-based aspects. we then create hard negative samples of grounding videos by masking temporal steps that are relevant to actions mentioned in target output responses. hard negative dialogue samples are created by masking tokens that are referenced to the entity mentioned in target output responses. positive samples of videos and dialogues are developed similarly by masking irrelevant temporal steps or tokens for them to remain factual. finally, based on an object or actionbased variance between factual and counterfactual pairs, we only select specific hidden state representations of the target dialogue response sequence, to apply contrastive loss functions. compared to existing approaches, our method has better control of data contrast at the granularity of object and action variance. we conducted experiments with comprehensive ablation analysis using the audio-visual scene-aware dialogues (avsd) benchmark @xcite and showed that our method can achieve promising performance gains. more related to our work are counterfactual augmentation methods in generative tasks. @xcite introduced a new benchmark for counterfactual story rewriting. @xcite explored augmented counterfactual dialogue goals to evaluate dialogue state tracking models. @xcite proposed a synthetic 3d environment for learning the physical dynamics of objects in counterfactual scenarios. different from prior tasks, in the task of video-grounded dialogue, a target response is not easy to be flipped/negated, and hence, supervised learning is not straightforward. we propose to automatically develop counterfactual and factual samples and improve representation learning via unsupervised learning. contrastive learning. our work is related to the research of contrastive learning in deep learning models. the research is particularly popular in self-supervised learning of image representations @xcite @xcite @xcite . these methods do not require additional annotations but aim to improve representations through loss functions. the loss functions are often inspired by noise contrastive estimation (nce) @xcite and applied in lower-dimensional representation space. in the language domain, similar loss functions have been introduced to improve word embeddings @xcite and sentence embeddings @xcite . more related to our work is @xcite @xcite , introducing positive and negative pairs of sentences for contrastive learning in generative tasks such as language modelling, word alignment, and machine translation. in the multimodal research domains, our work is related to contrastive learning methods introduced by @xcite @xcite . specifically, our work complements @xcite by incorporating causality into contrastive learning. however, we focus on a very different task of video-grounded dialogues that involves turn-based question-answering. the task requires multimodal reasoning performed on both dialogue context and video context. moreover, we improve models by tightly controlling data variance by adopting compositionality and our loss functions optimize hidden state representations of decoding tokens by their object or action-based semantics. ## discussion and conclusion in this work, we proposed compositional counterfactual contrastive learning (c 3 ), a contrastive learning framework to address the limitation of data in video-grounded dialogue systems. we introduced a factorized object-action structural causal model, described a temporal-based and tokenbased augmentation process, and formulated contrastive learning losses that exploit object-level and action-level variance between factual and counterfactual training samples. in our proposed approach, we train models to minimize the distance between compositional hidden state representations of factual samples and maximize the distance between counterfactual samples. we noted our proposed c 3 still entails some limitations. we describe these limitations and suggest potential ways to overcome them for future extension. first, in our approach, we made the assumption of independence between c obj and c act to mask tokens/video segments as a way to generate counterfactual data samples. however, in many cases, this assumption might be too strong. therefore, our approach might disrupt the natural data distribution and create negative noise in model training. a more advanced counterfactual data generation should be able to better capture the nature of counterfactual scenarios, avoiding the above assumption and generalizing the model better. secondly, in our approach, we require external text-processing tools to decompose the input components. more sophisticated tools could be used to improve data quality of counterfactual/factual examples. finally, after this work was completed, there have been several more advanced approaches following mtn @xcite . as our approach is model-agnostic, we encourage readers to review and adapt our work to these more advanced models.