LLM -empowered Dynamic Prompt Routing for Vision-Language Models Tuning under Long-Tailed Distributions (2025)

Abstract: Pre-trained vision-language models (VLMs), such as CLIP, have demonstrated impressive capability in visual tasks, but their fine-tuning often suffers from bias in class-imbalanced scenes. Recent works have introduced large language models (LLMs) to enhance VLM fine-tuning withsupplementaryy semantic information. However, they often overlook inherent class imbalance in VLMs’ pre-training, which may lead to bias accumulation in downstream tasks. To address this problem, this paper proposes a Multi-dimensional Dynamic Prompt Routing (MDPR) framework. MDPR constructs a comprehensive knowledge base for classes, spanning multiple visual-semantic dimensions. During fine-tuning, the dynamic routing mechanism aligns global visual classes, retrieves optimal prompts, and balances fine-grained semantics, yielding stable predictions through logits fusion. Extensive experiments on long-tailed benchmarks, including CIFAR-LT, ImageNet-LT, and Places-LT, demonstrate that MDPR achieves comparable results with current SOTA methods. Ablation studies further confirm the effectiveness of our semantic library for tail classes and show that our dynamic routing operates with a slight increase in computational overhead, making MDPR a flexible and efficient enhancement for VLM fine-tuning under data imbalance. The codes are available in https://github.com/Sha843/MDPR.

## introduction pretrained vision-language models (vlms), such as clip @xcite , have demonstrated remarkable capabilities in visual tasks by leveraging cross-modal knowledge and tuning @xcite . however, fine-tuning of vlm under imbalanced downstream † corresponding author. llm [horse] data exhibits significant bias @xcite , i.e., models favor many-sampled class optimization while under-performing on few-sampled classes, as shown in figure 1(b) and (c ). this challenge of learning from imbalanced data is not unique to vlm fine-tuning and has been extensively studied in various contexts. for instance, some works explore causal inference to disentangle robust features from spurious correlations @xcite , while others in federated learning tackle data heterogeneity across distributed clients through techniques like feature space alignment @xcite and prototypical calibration @xcite .lately, large language models (llms) are introduced to enhance vlm tuning, which faces two fundamental questions: (1) what semantic information from llms is effective in alleviating distributional bias? (2) how to leverage augmented information during fine-tuning process? to address vlms' bottlenecks in data-scarce scenarios, prior works leverage llms for classlevel semantic enhancement, sample synthesis, and open-world concept expansion. for semantic enhancement, llms generate discriminative prompts to improve inter-class separability @xcite . for sample synthesis, ltgc @xcite guides diffusion models to synthesize tailclass samples. for concept expansion, pervl @xcite and custom diffusion @xcite enable open-set generalization via text descriptions. however, these methods often overlook intrinsic vlms' biases, leading to cumulative bias during fine-tuning, and rely on static prompts or costly generative models, limiting adaptability. to enhance the effectiveness of llm knowledge in fine-tuning vlms under imbalanced distributions, we propose multi-dimensional dynamic prompt routing (mdpr), as illustrated in figure 1( @xcite . specifically, to address the implicit imbalance present during the vlm pre-training phase, mdpr firstly introduces a multi-dimensional prompt construction strategy. during training, it leverages zero-shot vlms to extract and construct a prompt pool for each class, capturing multiple distinct dimensions: general appearance, fine-grained appearance, functionality, contextual information, and differential features. this multi-faceted prompt design helps mitigate prior biases for classes. subsequently, in the dynamic prompt routing stage, it further alleviates the impact of imbalanced data by implementing global visual-class alignment, dynamic routing-based visual-prompt matching, and fine-grained semantic balancing. this process generates predictions from multiple perspectives, and robust results are achieved through a logit fusion mechanism. as an effective enhancement architecture, the proposed mdpr can be flexibly integrated with various vlm fine-tuning methods. to evaluate the effectiveness of the proposed mdpr, we conducted extensive experiments on three long-tailed visual recognition benchmarks, namely cifar-100-lt, imagenet-lt, and places-lt. the experimental results demonstrate that mdpr, through the comprehensive prompt construction and dynamic routing mechanisms, effectively mitigates class imbalance biases in both pretrained models and downstream data, achieving robust performance improvements across head and tail classes while maintaining high compatibility with existing fine-tuning frameworks. the primary contributions of this work are: ## experiments to comprehensively evaluate the efficacy of mdpr, we conduct extensive experiments on three longtailed image recognition benchmarks. ## conclusion addressing the class bias in fine-tuning visionlanguage models under long-tailed distributions, we propose the multi-dimensional dynamic prompt routing (mdpr) framework. unlike traditional static prompt or high-cost sample generation methods, mdpr leverages a structured multi-dimensional semantic knowledge base and an image-driven dynamic routing mechanism to efficiently mitigate biases from pre-training and downstream data. first, mdpr constructs a multi-dimensional prompt pool, providing comprehensive class understanding to counter prior biases. second, an image-guided dynamic routing module, combined with regularization, generates instance-adaptive class representations by optimizing routing and representation stability. experiments on cifar-100-lt, imagenet-lt, and places-lt demonstrate that mdpr significantly enhances tail-class performance while balancing head and medium-class robustness, achieving sota or highly competitive results. as a lightweight plug-and-play module, mdpr offers an effective paradigm for open-world long-tailed recognition.