Spotlighter: Revisiting Prompt Tuning from a Representative Mining View (2025)

Abstract: CLIP’s success has demonstrated that prompt tuning can achieve robust cross-modal semantic alignment for tasks ranging from open-domain recognition to fine-grained classification. However, redundant or weakly relevant feature components introduce noise and incur unnecessary computational costs. In this work, we propose Spotlighter, a lightweight token-selection framework that simultaneously enhances accuracy and efficiency in prompt tuning. Spotlighter evaluates each visual token’s activation from both sample-wise and semantic-wise perspectives and retains only the top-scoring tokens for downstream prediction. A class-specific semantic memory bank of learned prototypes refines this selection, ensuring semantic representativeness and compensating for discarded features. To further prioritize informative signals, we introduce a two-level ranking mechanism that dynamically weights token–prototype interactions. Across 11 few-shot benchmarks, Spotlighter outperforms CLIP by up to 11.19% in harmonic mean accuracy and achieves up to 0.8K additional FPS, with only 21 extra parameters. These results establish Spotlighter as an effective and scalable baseline for prompt tuning.

## introduction recent advances in vision-language models have demonstrated remarkable capabilities in prompt tuning, particularly through approaches like clip @xcite ) that achieve robust cross-modal semantic alignment. these methods have demonstrated impressive performance in tasks such as open-domain recognition @xcite , fine-grained categorization @xcite , and long-tailed distribution scenarios (liu et al., 2022), leading to breakthroughs in practical applications, including intelligent surveillance and medical image analysis. the superior performance of vision-language models primarily originates from their ability to learn discriminative joint embeddings that enable precise cross-modal alignment, a fundamental driver of continual model enhancement. the alignment between visual and textual feature spaces enables effective classification, with ongoing research continuously enhancing representation quality through techniques like prompt learning @xcite and feature enhancement @xcite . however, existing methods face two primary challenges: (1) feature noise interference: redundant or weakly relevant components within the aligned features introduce noise, undermining the contribution of semantically critical information @xcite . (2) computational efficiency bottleneck: full-scale feature interactions across the entire representation space result in unnecessary computational burdens and higher costs in practical applications @xcite . to address these challenges, prior works @xcite have proved that during clip's encoding process for effective imagetext alignment, the model inherently captures a mixture of semantic signals. since the image and text encoders operate independently, they are designed to cover a wide range of possible semantics. this results in varied importance across different parts of the feature representation concerning specific classification goals. crucially, only a subset of these features contributes meaningfully to crossmodal alignment, while the rest may introduce redundancy. therefore, performing the sample-level evaluation of feature importance enables us to selectively emphasize critical features and suppress irrelevant ones, enhancing accuracy and efficiency. existing approaches @xcite @xcite employ global learnable text-image prompts or lightweight adapters in frozen layers to capture semantic information, as shown in fig. 1(a ), yet have not thoroughly explored the synergistic optimization between features representations and computational efficiency, leaving this as an open area for further research. based on the above analysis, we revisit crossmodal feature alignment in few-shot image classification and propose a simple yet effective model, spotlighter, which achieves a favorable balance between accuracy and computational efficiency. the key idea is to identify and retain sparse but highly representative feature tokens while discarding redundant ones. specifically, we evaluate each token's cross-modal semantic relevance from both sample-wise and semantic-wise perspectives, quantified as an activation score. only a few highly activated tokens are retained for prediction, while the rest are discarded as redundant. unlike standard methods @xcite that enhance typical tokens while keeping all, we strengthen representative tokens while discarding non-informative ones for greater efficiency. to guide this selection, we introduce a semantic memory bank that stores a set of class-specific semantic prototypes. these prototypes help refine class boundaries during token activation, ensuring that the selected features are both semantically representative and capable of compensating for potentially missing information from discarded regions. furthermore, recognizing the varying contributions of activated features to classification, we introduce a two-level ranking mechanism over the prototypes. this mechanism dynamically adapts to the activation distribution of each sample, allowing the model to prioritize more informative features. the final representative features for prediction are then formed by fusing features with their corresponding prototypes according to their activation levels. extensive experiments conducted across 11 benchmark datasets demonstrate the effectiveness of our proposed method. compared to clip @xcite and clipfit @xcite , our approach achieves consistent improvements in both harmonic mean accuracy (hm) and computational speed, with an improvement of 11.19% / 3.86% in hm score and 0.8k/3.8k more fps, respectively. remarkably, these gains come at the cost of only 21 additional parameters, highlighting the efficiency and scalability of our design. our main contributions are lies in: ## conclusion we introduce spotlighter, a plug-and-play framework that revisits few-shot image classification from the perspective of representative token mining. by progressively selecting and categorizing informative tokens, spotlighter effectively filters noise and reduces redundant feature interactions. leveraging both activated tokens and representative tokens, the model enhances fine-grained cross-modal alignment with minimal parameter overhead. extensive experiments across 11 benchmarks and diverse generalization settings show that spotlighter consistently improves accuracy and efficiency over strong baselines. our work highlights the importance of token-level selection and structured refinement for efficient and robust few-shot learning with vision-language models.