Dataset Bias Mitigation in Multiple-Choice Visual Question Answering and Beyond (2023)

Abstract: Vision-language (VL) understanding tasks evaluate models’ comprehension of complex visual scenes through multiple-choice questions. However, we have identified two dataset biases that models can exploit as shortcuts to resolve various VL tasks correctly without proper understanding. The first type of dataset bias is Unbalanced Matching bias, where the correct answer overlaps the question and image more than the incorrect answers. The second type of dataset bias is Distractor Similarity bias, where incorrect answers are overly dissimilar to the correct answer but significantly similar to other incorrect answers within the same sample. To address these dataset biases, we first propose Adversarial Data Synthesis (ADS) to generate synthetic training and debiased evaluation data. We then introduce Intra-sample Counterfactual Training (ICT) to assist models in utilizing the synthesized training data, particularly the counterfactual data, via focusing on intra-sample differentiation. Extensive experiments demonstrate the effectiveness of ADS and ICT in consistently improving model performance across different benchmarks, even in domain-shifted scenarios.

## introduction visual question answering (vqa) is a challenging vision-language task that requires reasoning with integrated information from visual and text modalities @xcite @xcite @xcite @xcite . vqa benchmarks @xcite @xcite @xcite present complex scenarios with multiple entities in a multiple-choice question format, where the model selects the correct answer from multiple long, context-dependent candidate answers. previous studies have examined dataset bias in vqa benchmarks with short-phrase answers multiple-choice question with the long answers (vcr, vlep, etc.) bottle dining table person1 person2 why is person2 looking down? what color is the truck? ## bias analysis in vqa-long in this section, we identify and analyze two distinct types of biases that commonly occur in vqa-long and other vlu benchmarks. ## time consumption our coarse-to-fine region removal method is flexible and generalizable as the number of runs of the image inpainting process can be adjusted depending on the scenarios to decrease the time consumption. after selecting the region to be removed, our proposed coarse-to-fine region removal will be conducted by two main steps: 1) initial one-pass full region removal/inpainting by sp l p ; 2) triplepass autoregression region removal/inpainting by sp l f . the triple-pass autoregression strategy ensures fine-grained images and avoids superficial artifacts, as addressed in prior works. after the 1st run of full region inpainting by sp l f , we split the region into smaller m regions evenly and run the inpainting process for each smaller region, respectively. a similar procedure applies to the third pass of n runs. triple-pass autoregression is a flexible solution, as both m and n can be set to any arbitrary positive integers (2 ⩽ m ⩽ n ) depending on the situation to decrease the overhead time consumption. hence, the total runs of the inpainting process in the tr iple-pass strategy is 1 + m + n . based on ## conclusion this paper analyzed dataset biases and their underlying causes in vqa-long. our findings shed light on how these biases can impact the evaluation of vl models and the importance of mitigation. we hope our work will inspire future research in developing rigorous data annotation processes and strategies to mitigate the influence of dataset biases.