Causal Reasoning through Two Cognition Layers for Improving Generalization in Visual Question Answering (2023)

Abstract: Generalization in Visual Question Answering (VQA) requires models to answer questions about images with contexts beyond the training distribution. Existing attempts primarily refine unimodal aspects, overlooking enhancements in multimodal aspects. Besides, diverse interpretations of the input lead to various modes of answer generation, highlighting the role of causal reasoning between interpreting and answering steps in VQA. Through this lens, we propose Cognitive pathways VQA (CopVQA) improving the multimodal predictions by emphasizing causal reasoning factors. CopVQA first operates a pool of pathways that capture diverse causal reasoning flows through interpreting and answering stages. Mirroring human cognition, we decompose the responsibility of each stage into distinct experts and a cognition-enabled component (CC). The two CCs strategically execute one expert for each stage at a time. Finally, we prioritize answer predictions governed by pathways involving both CCs while disregarding answers produced by either CC, thereby emphasizing causal reasoning and supporting generalization. Our experiments on real-life and medical data consistently verify that CopVQA improves VQA performance and generalization across baselines and domains. Notably, CopVQA achieves a new state-of-the-art (SOTA) on the PathVQA dataset and comparable accuracy to the current SOTA on VQA-CPv2, VQAv2, and VQA- RAD, with one-fourth of the model size.

## introduction the visual question answering (vqa) task involves answering questions about images, requiring multimodal processing and common sense understanding @xcite . vqa research has various applications, including autonomous systems @xcite , healthcare @xcite , and education @xcite . however, real-life multimodal data diversity poses a challenge for vqa models to achieve out-of-distribution (ood) generalization, which involves performing well on data beyond the training distribution instead of relying on independent and identically distributed (iid) data @xcite @xcite . recent studies have highlighted a risk of ood generalization in vqa, where models may respond solely based on the question and ignore the input image due to correlations between the question and answer distribution @xcite that exist in human knowledge. for example, questions starting with "is this...?" are typically expected to be yes/no questions (e.g. , "is this a cat?") rather than multiple-choice questions (e.g. , "is this a cat or dog?"), leading to possible correct answers with a simple "yes" or "no". there are many attempts to solve this issue, such as (1) reducing the linguistic correlation @xcite by avoiding answers generated from only the question used as input, (2) strengthening the visual processing @xcite , and (3) balancing the answer distribution by generating new image-question pairs @xcite @xcite . however, these approaches tend to overlook the critical aspect of enhancing multimodal predictions, instead focusing on unimodal aspects (either language or visual) or the data itself. our assumption is that enhancing the quality of multimodal predictions would be a potential route for improving generalization in vqa. in fact, solving the vqa task requires the integration of multimodal processing and common sense knowledge. consequently, vqa can be conceptualized as a two-stage process: input interpreting and answering, which involves generating an interpretation of the multimodal input and answering the question by querying the knowledge space. besides, similarly to how humans tackle the vqa task, the input misunderstanding can harm the answering stage, posing a risk to the overall performance of vqa. therefore, comprehending the causal reasoning behind this two-stage process becomes crucial to improving the vqa task. in this work, we propose cognitive pathways vqa (copvqa) to boost the causal reasoning in vqa to enhance the ood generalization. copvqa derives from the findings of knowledge modularity and cognitive pathways from cognitive neuroscience @xcite @xcite , which mentions (1) the human brain organizes independent modules to handle distinct knowledge pieces and (2) the communication efficacy among these modules supports the generalization. we decompose each interpreting and answering stage into a set of experts and a cognition-enabled component (cc) strategically activates one expert for each stage at a time. by this approach, copvqa disentangles the vqa task into specialized experts connected by pathways through interpreting-answering process. subsequently, we extend the biases in vqa that are also from the monolithic procedure (instead of expert selection) besides the linguistic correlation. finally, we emphasize answers governed by the disentangled stages with multimodal input and disregard other answers, including ones from the monolithic procedure and from unimodal input. the contributions of this work are summarized as follows: (1) we propose copvqa to improve ood generalization by enhancing causal reasoning that is compatible with diverse vqa baselines and domains; (2) to our best knowledge, copvqa is the first work that formulates vqa as two layers of cognitive pathways, further facilitating research on causal reasoning in vqa; and (3) we achieve the new sota for the pathvqa dataset and mark the comparable results to the current sotas of vqa-cpv2, vqav2, and vqa-rad datasets with only one-four of the model sizes. ## related work the generalization restriction in vqa arises from biases between questions and answers in human knowledge, where certain question types exhibit strong correlations with predictable answers based on common knowledge. these biases are also reflected in vqa datasets. for example, in the vqav1 dataset @xcite , a vqa model can quickly achieve an accuracy of around 40% on sport-related questions by simply providing the answer "tennis." to further challenge the generalization ability of the vqa models, the vqa changing priors (vqa-cpv2) dataset @xcite is introduced. this dataset is deliberately designed to feature different answer distributions between its train and test sets. therefore, the emergence of the vqa-cp dataset has brought about a significant shift in the vqa landscape, demanding that vqa models go beyond exploiting correlations and overcoming biases from the given training data. among attempts of capturing language biases, rubi @xcite introduces a noteworthy approach that computes the answer probability using only the input question, thereby capturing linguistic biases. then, the question-only branch carrying the biases is used to compute a mask that aims to mitigate biased prediction of unimodal input from the multimodal prediction. building upon this line of research, cfvqa @xcite pioneers a comprehensive causal-effect view of vqa that subtracts the impact of the question-only branch, represented as the answer logit, from the overall answer logit (further discussed in section 3). similarly, dvqa @xcite proposes subtracting the question-only branch's impact from the multimodal branch in the hidden layer. by doing so, dvqa aims to mitigate biased predictions before passing the output through the classifier model for final answer prediction. in this study, we delve deeper into capturing biases using separate branches and subsequently eliminate them. however, besides the linguistic correlations, we assume another potential restriction in vqa generalization is the monolithic approach across the diverse scenarios of the multimodal input. therefore, we attempt to eliminate answers by monolithic procedures involved in interpreting and answering stages parallelly emphasize answers by disentanglement approach. ## preliminaries: causality view in vqa introduced in cfvqa @xcite , the causal graph of vqa is depicted in figure 1a in which the inputs v and q cause an answer a, and a mediator foot_0 k represents the knowledge space. we have the direct paths, which are @xmath0, representing the answers based solely on unimodal input (either question or image). in contrast, the indirect path, (v, q) → @xmath1, represents the answer by considering the multimodal input and the interaction in the knowledge space. with a single mediator k, the idea of cfvqa is to capture the language prior by the @xmath2, which involves only the question and produces an answer logit z q . likewise, the answer produced through k with multimodal input is z k . subsequently, they produce the final answer logit as follows, where c is a parameter for normalization: regarding optimization, they utilize cross-entropy loss (celoss) and define @xmath3, a) + celoss(logσ(z q ), a). ## cognitive pathways vqa -copvqa in this section, we present copvqa from multiple viewpoints, which are the overview in section 4.1, the causal-effect view of copvqa in section 4.2, and the copvqa implementation in section 4.3. ## experiment setup we conducted experiments to validate: h 1 -causal reasoning in multimodal prediction benefits vqa performance (section 6.1), h 2 -causal reasoning in multimodal prediction enhances ood generalization (section 6.1), and h 3 -the disentangled architecture is crucial for reasoning (section 6.2). ## ablation studies individual effects in copvqa we conduct ablations to observe the effect of components in equations 7. the analysis confirms the dominance of copvqa over the modified versions. figure 6 compares the overvall score of z f inal to ones from: ## conclusion we proposed copvqa, a novel framework to improve ood generalization by leveraging causal reasoning in vqa. copvqa emphasizes the answer with a full reasoning flow governed by disentangled knowledge space and cognition-enabled component in both interpreting and answering stages while eliminating answers in incompleted reasoning flows, which involve unimodal input or monolithic procedures. copvqa outperforms baselines across domains in iid and ood. notably, copvqa achieves new sota on pathvqa and comparable results with the current sotas of other datasets with significantly fewer parameters.