M a PPER : Multimodal Prior-guided Parameter Efficient Tuning for Referring Expression Comprehension (2024)

Abstract: Referring Expression Comprehension (REC), which aims to ground a local visual region via natural language, is a task that heavily relies on multimodal alignment. Most existing methods utilize powerful pre-trained models to transfer visual/linguistic knowledge by full fine-tuning. However, full fine-tuning the entire backbone not only breaks the rich prior knowledge embedded in the pre-training, but also incurs significant computational costs. Motivated by the recent emergence of Parameter-Efficient Transfer Learning (PETL) methods, we aim to solve the REC task in an effective and efficient manner. Directly applying these PETL methods to the REC task is inappropriate, as they lack the specific-domain abilities for precise local visual perception and visual-language alignment. Therefore, we propose a novel framework of Multimodal Prior-guided Parameter Efficient Tuning, namely MaPPER. Specifically, MaPPER comprises Dynamic Prior Adapters guided by a aligned prior, and Local Convolution Adapters to extract precise local semantics for better visual perception. Moreover, the Prior-Guided Text module is proposed to further utilize the prior for facilitating the cross-modal alignment. Experimental results on three widely-used benchmarks demonstrate that MaPPER achieves the best accuracy compared to the full fine-tuning and other PETL methods with only 1.41% tunable backbone parameters.

## introduction referring expression comprehension (rec) @xcite @xcite is a crucial and challenging task within the multimodal fields, which needs to localize the local image region according to the language expression semantics. rec is fundamental * equal contribution † corresponding author for visual language understanding, with broad applications in fields such as visual-language navigation @xcite and human-machine interaction @xcite . different from vanilla object detection task, rec needs to extract not only global and local spatial information from images, but also relies on the alignment of multimodal features. existing approaches @xcite @xcite transfer the language and vision knowledge from pre-trained models by fully fine-tuning. however, such a fine-tuning strategy is sub-optimal for rec, as reflected in the following aspects: 1) fine-tuning the entire backbone might suffer catastrophic forgetting and undermine the extensive prior knowledge learned from pre-training. to address these issues, we shift our focus to parameter-efficient transfer learning (petl) @xcite . petl methods like adapter tuning and prompt tuning provide efficient ways to utilize pretrained models by adjusting a small set of parameters instead of fine-tuning the entire network @xcite . this approach saves computational resources while still competitive performance improvements. by integrating petl techniques, we can enhance our models' flexibility and efficiency in adapting to rec. however, we empirically find that directly using these petl methods cannot achieve satisfactory results in rec (see figure 1 ). we argue the main reasons are twofold: 1) the target objects that require attention in rec often occupy local regions of uncertain size in images, and most existing petl methods lack the crucial ability to extract multi-scale local semantics for visual perception. 2) rec is a task that strongly relies on multimodal alignment, and language-oriented adapters are obviously deficient in aligning with visual information. recently, petl methods have also been introduced into rec tasks @xcite . hivg @xcite adopts lora to fine-tune the frozen clip model, but it is not an efficient enough approach due to the heavy alignment design using cross-attention module. in contrast, dara @xcite is a lightweight method in petl paradigm. however, dara does not fully address the need for local visual adaptation in the referring expression comprehension, potentially compromising the model's ability to capture fine-grained visual details. considering the aforementioned issues, in this paper, we propose a novel framework of multimodal prior-guided parameter efficient tuning for rec (mapper) that improves text understanding with the aligned prior and enhances vision perception by combining local visual semantics with global perception. as shown in figure 2 , we introduce the vision-aligned text module to generate the aligned prior, which works for the alignment of vision and language feature. moreover, we insert the local convolution adapter (loca) into vision blocks for enhancing visual perception. specifically, we propose the dynamic prior adapter (dypa) presented in figure 3 , dypa can dynamically adjust each token by considering the significance score guided by the aligned prior. in order to promote the interaction of text and vision features, we further propose the prior-guided text module (pgt) for fusing the prior and text feature. for the visual branch, most pre-trained visual models are powerful transformer architectures. unfor-tunately, vision transformers are observed ignoring local feature details @xcite , which decreases the discriminability between backgrounds and foregrounds. motivated by this, we introduce the local convolution adapter (loca), which integrates multi-scale local knowledge, thereby enhancing the representational power for pre-trained vision transformers. extensive experiments on re-fcoco @xcite , refcoco+ @xcite , and refcocog @xcite demonstrate the effectiveness and efficiency of our framework. our main contributions are as follows: • we perform an in-depth exploration of parameterefficient transfer learning (petl) methods for rec tasks. we introduce mapper aimed at improving both the effectiveness and efficiency of visual-text alignment, as well as enhancing visual perception by incorporating local visual semantics. • we propose the novel dynamic prior adapter (dypa) and local convolution adapter (loca). the former employs aligned prior to dynamically adjust the language encoder, while the latter introduces local visual features for enhancing the visual encoder. • extensive experiments demonstrate that our method can outperform the state-of-the-art (sota) methods in rec tasks, with only 1.41% tunable parameters within pre-trained backbones. ## conclusion in this study, we present an innovative parameter-efficient transfer learning (petl) approach designed for multi-modal language grounding tasks, especially in referring expression comprehension. mapper enhances the adapters with multi-modal prior through the implementation of a simple yet effective fine-tuning strategy. we aims at improving both the effectiveness and efficiency of visualtext alignment, as well as enhancing visual perception by incorporating local visual semantics. the dynamic prior adapter (dypa) employs aligned prior to dynamically adjust the language encoder, while the local convolution adapter (loca) introduces local visual features for enhancing the visual encoder. mapper not only surpasses the performance of fully fine-tuned models but also does more efficiently.