Compressing and Debiasing Vision-Language Pre-Trained Models for Visual Question Answering (2023)

Abstract: Despite the excellent performance of vision-language pre-trained models (VLPs) on conventional VQA task, they still suffer from two problems: First, VLPs tend to rely on language biases in datasets and fail to generalize to out-of-distribution (OOD) data. Second, they are inefficient in terms of memory footprint and computation. Although promising progress has been made in both problems, most existing works tackle them independently. To facilitate the application of VLP to VQA tasks, it is imperative to jointly study VLP compression and OOD robustness, which, however, has not yet been explored. This paper investigates whether a VLP can be compressed and debiased simultaneously by searching sparse and robust subnetworks. To this end, we systematically study the design of a training and compression pipeline to search the subnetworks, as well as the assignment of sparsity to different modality-specific modules. Our experiments involve 2 VLPs, 2 compression methods, 4 training methods, 2 datasets and a range of sparsity levels. Our results show that there indeed exist sparse and robust subnetworks, which are competitive with the debiased full VLP and clearly outperform the debiasing SoTAs with fewer parameters on OOD datasets VQA-CP v2 and VQA-VS. The codes can be found at https://github.com/PhoebusSi/Compress-Robust-VQA.

## introduction visual question answering (vqa) @xcite is an important task at the intersection of cv and nlp. in the last decade, deep neural networks have made promising progress in vqa. however, recent studies @xcite have found that vqa models are prone to dataset biases. as a result, they always suffer from sharp performance drops when faced with outof-distribution (ood) test datasets, whose answer distributions are different from the training set. although large-scale vision-language pretrained models (vlps) achieve further improvements in the in-distribution (id) vqa benchmark @xcite , they also fail to address the dataset-bias problem @xcite , e.g., lxmert @xcite suffers a 23.26% drop between id and ood accuracy. at the same time, the improvement brought by vlps is partly due to their large model size, which increases the computational cost of deploying vqa models. to facilitate the application of vlps to vqa tasks, the two problems should be addressed simultaneously. however, existing researches mostly focus on each of them separately. the dataset-bias problem in vqa is well studied by numerous debiasing methods based on conventional small-scale models @xcite . their main solution @xcite @xcite is to regularize the loss according to the bias degree of training samples. in terms of the increased computational cost, a line of recent efforts have been made to compress pre-trained language models (plms) in the nlp field @xcite @xcite @xcite and vlps for visual-linguistic tasks @xcite . they show that large-scale plms and vlps can be compressed into lightweight models without degrading performance. refer to app. a for more related work. this paper jointly studies the compression and debiasing problems of vlp for the vqa task. to this end, we combine the existing debiasing and pruning methods to establish a training and compression pipeline, and conduct extensive experiments with the pre-trained lxmert, which is the most popular vlp in vqa, under different ood settings. we show that there exist sparse lxmert subnetworks that are more robust than the full model, which suggests that the goal of ood robustness and computational efficiency can be achieved simultaneously. we also present a comprehensive study on the design of the training and compression pipeline, as well as the assignment of sparsity to different model modules, to identify subnetworks with better ood generalization. our findings highlight the importance of 1) employing a two-stage training and compression pipeline and integrating the debiasing objective throughout the entire process. 2) if there are two debiasing methods working well with the full model, training the full model with the relatively poor-performing one and compressing it with the better one. 3) assigning modality-specific sparsity to different modules of vlp. our main contributions are as follows: (1) we present the first (to our knowledge) systematic study on sparsity and ood robustness for vlps. ## experiments in this section, we mainly investigate three questions: (1) how does compression affect lxmert's ood generalization ability? (2) how to design the training and pruning pipeline to achieve a good sparsity-performance trade-off? (3) how to assign sparsity to different modality-specific modules? ## conclusion to facilitate the application of vlp-based vqa systems, this paper presents the first joint study on the compression and debiasing problems of vlp for the vqa task. through extensive experiments with three vlps (i.e., lxmert, visual-bert and mplug), we analyze the impact of compression on the ood generalization ability. we present a comprehensive study on the design of the training and compression pipeline for a good sparsity-performance trade-off, and provide some valuable findings about the assignment of sparsity to different modality-specific modules. the compressed lxmert subnetworks in this paper outperform the sota debiasing methods with fewer or similar model parameter counts.