Towards Efficient Visual-Language Alignment of the Q -Former for Visual Reasoning Tasks (2024)

Abstract: Recent advancements in large language models have demonstrated enhanced capabilities in visual reasoning tasks by employing additional encoders for aligning different modalities. While the Q-Former has been widely used as a general encoder for aligning several modalities including image, video, audio, and 3D with large language models, previous works on its efficient training and the analysis of its individual components have been limited. In this work, we investigate the effectiveness of parameter efficient fine-tuning (PEFT) the Q-Former using InstructBLIP with visual reasoning benchmarks ScienceQA and IconQA. We observe that applying PEFT to the Q-Former achieves comparable performance to full fine-tuning using under 2% of the trainable parameters. Additionally, we employ AdaLoRA for dynamic parameter budget reallocation to examine the relative importance of the Q-Former’s sublayers with 4 different benchmarks. Our findings reveal that the self-attention layers are noticeably more important in perceptual visual-language reasoning tasks, and relative importance of FFN layers depends on the complexity of visual-language patterns involved in tasks. The code is available at https://github.com/AttentionX/InstructBLIP_PEFT.

## introduction pre-trained large language models (llms) can be fine-tuned with instruction tuning to align the model responses with human intentions @xcite . recently, model alignment with instruction tuning has been extended to the image domain by using an external encoder to align visual-language modalities and enhance the model's capabilities for visual reasoning. llava @xcite uses a projection layer to project clip @xcite image embeddings to the text embedding space of language models. however using a projection layer to convert every clip embedding from an image can take up a lot of context tokens and increase inference time. blip-2 @xcite and in-structblip @xcite use a q-former for visual-language alignment that transfers visual features into a fixed number of learnable embeddings (32 in blip-2), which is similar to perceiver io @xcite and flamingo @xcite . the q-former architecture is especially significant for its generalizability in aligning several modalities. this architecture of using crossattention to transfer features to a small number of learnable embeddings has been used in recent studies for aligning many different modalities including image @xcite video @xcite , and 3d @xcite . however, despite the increased usage and significance of the q-former, prior research on its sublayers and their importance in different tasks has been limited. elucidating the importance of each sublayer for different visual reasoning tasks can assist in designing more efficient training methods with effective parameter allocation. moreover, although peft methods have been successfully applied to efficiently train language models @xcite @xcite @xcite evaluating the effectiveness of peft on the q-former and visual language models also remains under-explored. these two areas are critical for advancing the efficiency of training multimodal language models. in this work, we evaluate the performance of peft on instructblip with two benchmarks, scienceqa @xcite and iconqa @xcite , that respectively evaluate knowledgegrounded visual reasoning and perceptual visual reasoning. we apply lora to the q-former and base llms, flan-t5-xl (chung et al., 2022) and vicuna-7b @xcite , and comprehensively test the performance of lora applied to different sublayers in the q-former with different ranks. we also examine the importance of each sublayer in the q-former for 4 different visual reasoning benchmarks using adalora @xcite , which dynamically allocates parameter budgets to improve performance. to the best of our knowledge, we are the first to inspect the effectiveness of peft methods on the q-former and analyze its sublayers for visual reasoning tasks. our contributions can be summarized as follows: (1) we demonstrate that applying peft to the q-former can reduce the trainable parameters to less than 2% while maintaining comparable performances. (2) we show that in contrast to full fine-tuning the q-former and freezing the llm, applying peft to both components can achieve superior results and reduces the total trainable parameters to less than 12%. (3) we examine the significance of the different sublayers in the q-former using adalora, and find that the self-attention layers are relatively more important for tasks that require stronger visual-language alignment, and more intrinsic ranks on ffn layers are needed to train on complex visual-language patterns. ## method in this work, we apply the peft method lora to the q-former and the llm in instructblip, and evaluate the performance on two visual reasoning benchmarks scienceqa and iconqa. additionally, we apply adalora to analyze the significance of each sublayer of the q-former on visual reasoning. lora reduces trainable parameters by decomposing the weight update matrix ∆@xmath0. after fine-tuning, the weight matrix can be reparametrized by adding the weight update to the original pre-trained model weights: w + ∆@xmath1, where @xmath2, @xmath3, @xmath4, r ≪ min(d, k). unlike the original lora implementation, which confines its application to only the self-attention layers (hu et al., 2021), we extend the use of lora to multiple transformer sublayers in both the q-former and the llm. specifically, we apply lora to the q, v layers in self-attention layers, the q, k, v, o layers in cross-attention layers, and the ffn layers in the q-former. adalora decomposes the intrinsic weight update matrix ∆@xmath5. during training, the less significant singular values are adaptively pruned based on their importance scores, adjusting the rank of the weight update matrices. the importance score of the ith singular value is calculated as follows: as a result, applying adalora leads to appropriate rank allocation across modules for better performance. since high-rank updates learn more complex signals, we use adalora for examining which sublayers in the q-former are critical for each visual reasoning task and which sublayers should be prioritized in parameter budget allocation for efficient fine-tuning. we apply adalora to the self-attention(q, v), cross-attention(q, k, v, o) layers, and ffn layers altogether for overall comparison. (figure 1 ) base models and benchmarks. we employ in-structblip as the base model for its pioneering use of the q-former and its strong performance on several downstream tasks @xcite including scienceqa (img) @xcite , ocr-vqa @xcite , and a-okvqa @xcite . we use the instructblip implementation of lavis @xcite 2021) tasks. these benchmarks were held-out datasets for instructblip, and were not involved in training the baseline instructblip model. for analyzing the q-former with adalora we use two additional benchmarks, vizwiz @xcite and flickr30k @xcite . knowledge grounded visual reasoning is a task of answering questions with a provided image related to the knowledge in diverse academic areas including physics, biology, and math. we use the scienceqa dataset which covers a variety of science topics with corresponding extensive explanations. we only use the questions with image context (img). scienceqa (img) has 6,218/2,097/2,017 samples for train/validation/test set. perceptual visual reasoning is a task of answering questions after comprehending the abstract meanings from an image. we use iconqa (multi-text-choice) which contains question-answer pairs for natural images that require comprehensive reasoning abilities to understand abstract diagrams. iconqa (multitext-choice) has 18,946/6,316/6,316 samples for train/validation/test set. ## conclusion in this work, we systematically evaluate the effectiveness of applying peft to the q-former and visual language models. our results show that applying peft to the q-former achieves comparable performance to full fine-tuning while only utilizing less than 2% of the trainable parameters. additionally, we employ dynamic parameter budget allocation with adalora to analyze the significance of the q-former's sublayers for different visual reasoning tasks. our findings reveal that the importance of ffn layers increases when visual-language pattern becomes more complex, and importance of self-attention layers increases as significance of visual-language alignment in task increases.