Debiasing Event Understanding for Visual Commonsense Tasks (2022)

Abstract: We study event understanding as a critical step towards visual commonsense tasks. Meanwhile, we argue that current object-based event understanding is purely likelihood-based, leading to incorrect event prediction, due to biased correlation between events and objects. We propose to mitigate such biases with do -calculus, proposed in causality research, but overcoming its limited robustness, by an optimized aggregation with association-based prediction.We show the effectiveness of our approach, intrinsically by comparing our generated events with ground-truth event annotation, and extrinsically by downstream commonsense tasks.

## introduction recently, commonsense reasoning tasks on visiolinguistic input have been actively studied in both vision and language communities, with the goal of commonsense reasoning. for example, in visual comet @xcite , given an image x and event e as input, we are tasked to predict intent (or, events before/after), known as intent prediction. similarly, visual commonsense reasoning (vcr; @xcite , given image and question q, requires to provide an answer a or provide a rationale for a, known as justification task. for such tasks, understanding event e plays a crucial role, either required as input or expected as output-for example, 41% of rationales from vcr are related to e. however, assuming the availability of e for new images at test time is considered impractical, for which @xcite consider two baselines: 1) setting input e as null, and 2) generating e by training generator gen. however, according to @xcite , empirical results reported gen performs even worse than null, or, shows a negative transfer, which is counterintuitive. our goal is robustifying event understanding, or, event generator gen with a higher transferability to the downstream tasks of visual commonsense reasoning. our hypothesis is that object-event bias in the dataset hinders transferability: in figure 1 , while the image and human-annotated e, denoted as gold, are not related to an event "sitting", existing gen, denoted as b-gen for biased generation, generates an event "sitting", even when no one is sitting on the chair. we argue that the generation is biased by the frequent association of chair-sitting in the dataset. ## methodology figure 3 overviews our framework in three steps: (1) probability estimation, (2) probability optimization, and (3) test-time debiasing. ## experimental results we now proceed to empirically validate the effectiveness of our approach, in three dimensions: 1) capturing rare-but-causal correlations 2) transferability and 3) out-of-domain generalization. capturing rare-but-causal: to investigate the effectiveness of our approach in capturing rare-butcausal correlations, we perform the event understanding task on the visualcomet dataset. for the experiment, we split the validation samples into a frequent set and a rare set, where the latter requires identifying rarely observed causal correlations. among the images with the rarely observed object-verb pairs, i.e., @xmath0, we collect the images with an object-verb, where the verb exists in the ground-truth event description e gold , as rare-but-causal set. a desirable model should generalize well to the challenging set, or, the gap between the original and challenging sets should be small. we report our results on event understanding in transferability: we evaluate how our approach contributes to transferability, which we define as the gap between event description as null and gen, on intent inference task. we report our results on the intent inference task in ## related work inspired by the success of large-scale pre-trained language models @xcite , large-scale pre-training on transformers have shown that it can also benefit visio-linguistic tasks, showing better transferability on various downstream tasks @xcite . however, it is reportedly bad when the transformer is trained on out-of-domain datasets, that are not aligned with its pre-training corpus @xcite , as they are purely likelihood-based, leading to spurious correlations and hurting the generalization ability. to this end, recent approaches adopt a traditional docalculus @xcite , encouraging causal intervention-based estimation to remove spurious correlations, that can be exploited in vision-only dataset @xcite or visio-linguistic dataset @xcite . different from these works, we focus on identifying spurious correlation comparing association-based and intervention-based knowledge. ## conclusion we studied the problem of robustifying event understanding, to overcome dataset bias, by combining observational and interventional estimations. our experiments suggest that this extension improves event understanding, and eventually visual commonsense tasks.