H omo G raph A dapter: A Homogeneous Graph Neural Network as an Effective Adapter for Vision-Language Models (2025)

Abstract: Vision-Language Models (VLMs), such as CLIP, have exhibited significant advancements in recognizing visual concepts through natural language guidance. However, adapting these models to downstream tasks remains challenging. Existing adaptation methods either overlook the structural knowledge between the text and image modalities or create overly complex graphs containing redundant information for alignment, leading to suboptimal classification performance and increased computational overhead. This paper proposes a novel adapter-tuning methodology named Homogeneous Graph Adapter (HomoGraphAdapter), which transforms diverse textual and visual descriptions into a unified set of node representations and establishes edges between nodes for inter-modal and cross-modal semantic alignment. We leverage a straightforward homogeneous Graph Neural Network (GNN) to adapt positive and negative classifiers across text and image modalities. The classifiers comprehensively enhance the performance for few-shot classification and OOD generalization. Compared with the SOTA approach HeGraphAdapter, HomoGraphAdapter improves classification accuracy by an average of 1.51% for 1-shot and 0.74% for 16-shot on 11 datasets, while also reducing both precomputation time and training time.

## introduction pre-trained vision-language models (vlms), especially clip @xcite and its variants @xcite , have opened a new chapter for various computer vision tasks. to efficiently adapt the clip model to downstream tasks, researchers typically employ one of two main strategies: prompt-tuning or adaptertuning. ## related work textual classifier and prompt-tuning methods for clip. the clip model @xcite computes the cosine similarity between image embeddings and textual embeddings, assigning the predicted class to the text with the highest similarity score. in this setup, textual embeddings for all classes, generated by inputting positive prompts like "a photo of a ..." into clip's text encoder, can be interpreted as a positive textual classifier. conversely, as suggested by prior studies @xcite @xcite , negative prompts such as "a photo of no ...", "a photo without ...", and "a photo not containing ..." yield a negative classifier, where the class corresponding to the text with the lowest similarity score is identified as the predicted class. in a similar vein, prompt-tuning methods @xcite enhance hard prompts by integrating a learnable vector as a soft prompt to clip's text encoder. these methods adapt the textual embeddings of clip to enhance alignment with visual embeddings, enabling more flexible and task-specific adaptations. ## experiments in this section, we evaluate homographadapter on few-shot classification tasks across 11 benchmark datasets, including imagenet @xcite , standfordcars @xcite , ucf101 @xcite , caltech101 @xcite , flowers102 @xcite , sun397 @xcite , dtd @xcite , @xcite , fgvcaircraft @xcite , and food101 @xcite . these datasets include vision tasks such as remote sensing classification, action recognition, texture classification, and fine-grained classification. we compare homographadapter with two state-of-theart graph adapter methods: graphadapter @xcite and hegraphadapter @xcite , as well as three typical adapter methods implementation details. resnet-50 @xcite is used as the default backbone of the clip model. we fine-tune homographadapter with fewshot labeled data sampled from the downstream dataset. we use adamw optimizer (kingma, 2014) with a cosine scheduler. the training epochs are set to 20 for most datasets, with the following exceptions: eurosat is set to 100 epochs, fgvcaircraft to 30 epochs, and oxfordflowers to 70 epochs. for building the visual cache and training, the batch size is set to 256; for testing, it is set to 64. the learning rate is set to 5×10 -4 . because tasks differ in complexity, certain hyperparameters vary across datasets. further details on the hyperparameters are provided in the appendix. all experiments are conducted on a single nvidia a100 gpu. ## conclusions this paper proposes a homogeneous graph learning approach to tune clip in the data-limited conditions. we introduce a unified, single-layer homogeneous gnn that encapsulates diverse natural language and visual embeddings within a homogeneous graph comprising only one node type and one edge type. this design jointly enhances positive learning and negative learning across both modalities. extensive experiments validate the effectiveness and efficiency of our method. in the future, we plan to explore how graph learning can be extended to fine-tune other pre-trained multimodality models.