Preserving Multi-Modal Capabilities of Pre-trained VLM s for Improving Vision-Linguistic Compositionality (2024)

Abstract: In this paper, we propose a new method to enhance compositional understanding in pre-trained vision and language models (VLMs) without sacrificing performance in zero-shot multi-modal tasks. Traditional fine-tuning approaches often improve compositional reasoning at the cost of degrading multi-modal capabilities, primarily due to the use of global hard negative (HN) loss, which contrasts global representations of images and texts. This global HN loss pushes HN texts that are highly similar to the original ones, damaging the model’s multi-modal representations. To overcome this limitation, we propose Fine-grained Selective Calibrated CLIP (FSC-CLIP), which integrates local hard negative loss and selective calibrated regularization. These innovations provide fine-grained negative supervision while preserving the model’s representational integrity. Our extensive evaluations across diverse benchmarks for both compositionality and multi-modal tasks show that FSC-CLIP not only achieves compositionality on par with state-of-the-art models but also retains strong multi-modal capabilities. Code is available at: https://github.com/ytaek-oh/fsc-clip.

## introduction humans naturally excel at multi-modal understanding, effortlessly perceiving and interpreting different modalities, such as images and text, and forming associations between them. this capability is evident in recognizing novel concepts @xcite , cross-modal retrieval @xcite , and compositional reasoning @xcite . to achieve this ability in artificial intelligence, foundational vision and language models (vlms) have been trained on large-scale imagetext datasets @xcite , significantly bridging the gap between human and ma-figure 1 : a holistic comparison of fine-tuning methods for vision-language compositionality. enhancing compositionality often compromises multi-modal task performance in previous approaches. our fsc-clip bridges this gap, minimizing these trade-offs. full experimental results are provided in tab. 1. despite these advances, vlms still face challenges in compositional reasoning @xcite . humans intuitively understand complex compositional language in combination with images, engaging in spatial reasoning @xcite , recognizing attributes and relationships in objects @xcite , and perceiving equivariance between image and text @xcite . in contrast, vlms often fail to understand these nuanced relationships @xcite . this shortfall is attributed to their reliance on global, single vector representations @xcite and limited ability to match compositional knowledge @xcite . to improve compositionality in vlms, both pretraining @xcite and fine-tuning @xcite methods have been proposed. in particular, finetuning, which leverages pre-trained knowledge and is cost-effective, is widely adopted in academia. typically, this involves incorporating hard negative texts @xcite @xcite into training. however, as shown in fig. 1 , this approach can result in a trade-off, where gains in compositionality come at the expense of performance in the multi-modal tasks: zero-shot classification (zs) and image to text retrieval (i2t ret). previously, hard negative (hn) losses are applied to global image and text representations. since hn texts are encoded too similarly to the original ones @xcite , pushing them away with the hn loss can disrupt the multi-modal representations. to address this, we propose a new fine-tuning framework for vlms that enhances compositional reasoning while preserving performance in multi-modal tasks. our approach mitigates the degradation caused by global hard negative loss on single vector representations, which struggles to capture subtle informational differences between hard negative texts and the original text. our framework introduces two key innovations: (1) local hard negative (lhn) loss. we utilize dense alignments between image patches and text tokens to calculate the hard negative loss. this approach, inspired by the dense alignment for visionlanguage representation @xcite , aggregates local similarity scores to enhance compositional understanding without undermining multi-modal representations. the whole framework, dubbed fine-grained selective calibrated clip, offers fine-grained supervision of hard negatives while preserving the integrity of multi-modal representations. as shown in fig. 1 , fsc-clip not only improves compositionality but also maintains high performance in multi-modal tasks. it outperforms dac-llm in zs and i2t ret scores, while achieving similar compositionality (comp) across a wide range of tasks. we summarize our contributions as follows: a bridge traveling over a train. a timber traveling over a bridge. ## methodology we first outline the fine-tuning setup for clip in sec. and each text t i into a sequence of textual tokens t @xmath0. these sequences are represented in a shared multi-modal space, with v @xmath1,i } p @xmath2,i } w @xmath3. the global representations of image and text v i and t @xmath4: v @xmath5, respectively. for example, pool(•) corresponds to avgpool and argmax for images and texts in @xcite . clip aligns the corresponding images and texts by measuring the global-level similarity: and the text to image loss l t2i is the reciprocal of l i2t which aligns the matching image per text. the final clip loss is l @xmath6. incorporating hard negative texts. to enhance the compositional reasoning of clip, hard negative (hn) texts are commonly incorporated into training, whether they are rule-based @xcite or generated by language models @xcite . consider a set of k different hn texts @xmath7. we introduce a separate hard negative loss added to l clip , similar to @xcite . first, we compute a similarity prediction probability p g i , assigned to the original caption t i as follows: here, g represents the global representation, and the hard negative (hn) loss applied to this similarity assignment is formulated as cross entropy: however, incorporating such global hn loss can inadvertently harm the multi-modal representations due to the similarly encoded global text representations between original and hn texts. ## conclusion in this paper, we introduce fine-grained selective calibrated clip (fsc-clip), a new fine-tuning framework for vision-language compositionality. it aims to preserve multi-modal capabilities and address the limitations of existing methods relying on global representations. we achieve this by employing dense representations between images and texts and regularizing the hard negative losses to prevent degradation, thereby facilitating the introduction of local hard negative loss and selective calibrated regularization. our extensive validation shows improved compositional reasoning and promising performance in standard multi-modal tasks. limitations. our methodology, including all the prior approaches, relies on short captions for both training and evaluation benchmarks. this practice constrains the models' exposure to and understanding of longer contexts, which are essential for achieving a genuine vision-language compositional understanding. longer and detailed captions involve more complex associations and contextual nuances @xcite that are essential for advanced compositionality in vision and language models. moving forward, there is a compelling need within the community to develop training and evaluation protocols that incorporate longer captions, better addressing the challenges of compositionality.