Debiasing Multimodal Models via Causal Information Minimization (2023)

Abstract: Most existing debiasing methods for multimodal models, including causal intervention and inference methods, utilize approximate heuristics to represent the biases, such as shallow features from early stages of training or unimodal features for multimodal tasks like VQA, etc., which may not be accurate. In this paper, we study bias arising from confounders in a causal graph for multimodal data, and examine a novel approach that leverages causally-motivated information minimization to learn the confounder representations. Robust predictive features contain diverse information that helps a model generalize to out-of-distribution data. Hence, minimizing the information content of features obtained from a pretrained biased model helps learn the simplest predictive features that capture the underlying data distribution. We treat these features as confounder representations and use them via methods motivated by causal theory to remove bias from models. We find that the learned confounder representations indeed capture dataset biases and the proposed debiasing methods improve out-of-distribution (OOD) performance on multiple multimodal datasets without sacrificing in-distribution performance. Additionally, we introduce a novel metric to quantify the sufficiency of spurious features in models’ predictions that further demonstrates the effectiveness of our proposed methods.

## introduction the success of multimodal models in various tasks has been attributed to their ability to rely on spurious correlations (or biases) present in the training data @xcite @xcite . an example of image bias in vqa is when the model tends to look at prominent objects in the image rather than focusing on the object about which the question figure 1 : multimodal models tend to rely on spurious correlations in the dataset to answer a question. existing methods remove unimodal biases whereas our method removes biases arising from cross-modal interactions as well and is more invariant to irrelevant features (e.g., the coffee mug) in this example. existing methods attempt to diminish a model's reliance on these shortcuts by taking one or both of two primary strategies: (a) by balancing the sample groups with and without spurious correlation, e.g. via data augmentation @xcite or sample synthesis @xcite @xcite , and (b) by explicitly eliminating the impact of spurious correlations during model training or inference @xcite @xcite . in the former approach, the identification of the unique set of spurious correlations in each sample becomes essential to curate augmented samples for achieving balance. consequently, approaches that alleviate biases in features or predictions, independent of the availability of non-spurious data, are more desirable. such meth-ods also offer the additional advantage of being agnostic to the specific dataset and task at hand. recent research on debiasing models has emphasized the significance of causal theory @xcite @xcite i.e., many spurious correlations originate from confounding variables that induce noncausal dependencies between inputs and labels @xcite . however, effectively identifying and representing biases that undermine prediction accuracy remains a challenging task. previous studies on multimodal models have utilized image features from early training stages as contextual biases for multi-label image classification @xcite , or introduced unimodal training branches to mitigate spurious correlations in visual question answering (vqa) @xcite . moreover, these approaches overlook biases stemming from multimodal interactions within their causal graphs. hence, in this work, we represent the bias as confounder variables that have a direct causal effect on multimodal features and the corresponding predictions (see fig. 2(a) ). spurious correlations represent the simplest predictive features that explain biased datasets @xcite , thereby making them easily learnable by machine learning models under limited representation capacity @xcite . we capitalize on this notion to study a novel framework that combines information theory and causal graphs to learn confounder representations capable of capturing spurious features. we examine two approaches to learn the confounder representations by imposing information loss on biased multimodal features i.e., (a) latent variable modeling using a generative model and (b) rate-distortion minimization @xcite . subsequently, we utilize these confounders in our proposed debiasing methods, namely ate-d and te-d, leveraging the concepts of average treatment effect @xcite and total effect @xcite causal mechanisms, respectively. in ate-d, we employ an autoencoder to reconstruct the biased features. the autoencoder projects these features into a lower-dimensional latent space, capturing latent features that act as substitutes for unobserved confounders @xcite . by clustering the learned confounder representations across the dataset, we construct a dictionary of confounders. we subsequently perform backdoor adjustment based on the average treatment effect, utilizing feature reweighting @xcite . in te-d, we leverage the rate-distortion function which controls the number of bits required to encode a set of vector representations @xcite . we minimize the ratedistortion function for a non-linear projection of the features extracted from a biased pretrained model, while simultaneously minimizing the cross-entropy loss of predicting from these projected features. this results in the loss of diverse information from the features and the retention of simple features that are also maximally predictive of the biased dataset. we treat these features as the confounder representations that stem from spurious correlations in the dataset and compute the (unbiased) total effect of the input by taking the difference between the biased feature and its respective confounder. we evaluate the proposed methods on several multimodal tasks and along multiple dimensions i.e., in-distribution and out-of-distribution performance, efficiency, and robustness. results show that these methods not only outperform baseline models with lower training overhead but also yield additional gains on top of unimodal debiasing methods. in this work, we demonstrate the presence of multimodal biases and the need for multimodal debiasing along with the potential of confounder modeling via information loss in causal multimodal debiasing. our contributions are as follows: human explanations can be used as additional training signals to promote reasoning @xcite @xcite . we debias models using existing biased data. inductive bias in model architecture. agrawal et al. (2018a) explicitly design inductive biases to prevent the model from relying on training priors. clark et al. (2019); cadene et al. (2019); ramakrishnan et al. (2018) rely on a separate qa branch to weaken the language prior in vqa models via adversarial or multi-task learning. wen et al. (2021) use contrastive loss to remove unimodal biases for vqa. peyrard et al. (2022) discover invariant correlations in data across different training distributions to enable generalization. inductive bias for modeling confounders. kallus et al. (2018) recover latent confounders via low-rank matrix factorization and sen et al. (2017) utilize low-dimensional variables for encoding confounders. we use low-dimensional features to limit representational capacity for encoding confounders in multimodal data. causal perspective. lin et al. (2022) use causal intervention through backdoor adjustment (glymour et al., 2016) to disentangle the biases for unsupervised salient object detection. huang et al. (2022) use ate to debias referring expression models. niu et al. (2021) compute the total indirect effect (tie) of the multimodal branch to omit the influence of unimodal branches. veitch et al. (2021) formalize counterfactual invariance and its relation to ood performance. liu et al. (2022) use features from early training as confounders and compute the total direct effect (tde) for multi-label image classification. we combine information theory and causal theory to learn confounders from biased representations and use them via ate and te causal mechanisms to debias a model. ## causal theory preliminaries in this section, we discuss our proposed causal graph for multimodal tasks and the two causal mechanisms relevant to our debiasing methods. ## measuring sufficiency & necessity of spurious features in multimodal tasks ood generalization accuracies indicate the model's ability to learn causal relationships between inputs and labels @xcite ). another approach to assess causal learning is by examining the models' invariance to spurious features in the dataset. necessity. to assess the robustness of models to type 1 features, we compare their performance on samples with and without a specific type 1 feature. in an unbiased model, the absence of this feature should have no impact on performance. however, a biased model tends to rely on it due to spurious correlations that confound the features and labels. an effective debiasing method should render the model invariant to such features. type 1 features predominantly arise from the image in multimodal tasks, as depicted in fig. 5 . therefore, we evaluate the necessity of these features using counterfactual images @xcite (refer to sec.6). ## results & discussion in this section, we discuss the results from the evaluation of our methods for generalization, robustness, effectiveness, and efficiency, and analysis of the learned confounder representations. ## conclusion we propose ate-d and te-d to mitigate biases in models by imposing causally-driven information loss on biased features to learn confounders. experimental results across various multimodal tasks, datasets, and backbones demonstrate that the learned confounders capture biases successfully, and our methods effectively eliminate biases from both unimodal and multimodal interactions.