LMPT : Prompt Tuning with Class-Specific Embedding Loss for Long-Tailed Multi-Label Visual Recognition (2024)

Abstract: Long-tailed multi-label visual recognition (LTML) task is a highly challenging task due to the label co-occurrence and imbalanced data distribution. In this work, we propose a unified framework for LTML, namely prompt tuning with class-specific embedding loss (LMPT), capturing the semantic feature interactions between categories by combining text and image modality data and improving the performance synchronously on both head and tail classes. Specifically, LMPT introduces the embedding loss function with class-aware soft margin and re-weighting to learn class-specific contexts with the benefit of textual descriptions (captions), which could help establish semantic relationships between classes, especially between the head and tail classes. Furthermore, taking into account the class imbalance, the distribution-balanced loss is adopted as the classification loss function to further improve the performance on the tail classes without compromising head classes. Extensive experiments are conducted on VOC-LT and COCO-LT datasets, which demonstrates that our method significantly surpasses the previous state-of-the-art methods and zero-shot CLIP in LTML. Our codes are fully public at https://github.com/richard-peng-xia/LMPT.

## introduction long-tailed multi-label visual recognition (ltml) @xcite is a common and practical task owing to the highly imbalanced data distribution @xcite and diverse objects of real-world images @xcite . compared with longtailed recognition and multi-label recognition tasks, ltml is more complex and challenging, because it requires capturing multiple categories and the label co-occurrence in individual images @xcite , which needs to compensate for the negative impacts caused by the long-tailed distribution (i.e., low performance on the tail classes). several approaches have been proposed to address the ltml problem from different perspectives, such as re-sampling @xcite @xcite , re-weighting @xcite and modeling more powerful structures @xcite @xcite . despite their great contributions, these works neglect to take into account two crucial aspects. first of all, the importance of semantic feature interaction between classes to capture label co-occurrence. however, these methods are limited to balancing the distribution of categories from the perspective of samples, without considering the feature correlation between different classes. second, synchronous improvements in head-to-tail category performance, while some of these works improve the performance of tail classes at the expense of the head classes. recently, graphic models have been introduced to model the semantic label correlation in a few works @xcite , whereas these works are complex and are modeling label dependencies mainly based on the image modality without additional semantic information from other modal data. vision-language models (vlms) @xcite @xcite @xcite demonstrate the huge potential of text modality on semantic context feature for downstream visual tasks, especially for the prompt tuning methods @xcite @xcite , which provide an efficient way to transfer pre-trained vlms to downstream tasks by learning the task-specific prompts rather than finetuning the entire model. nonetheless, the existing prompt tuning methods @xcite for visual recognition simply minimize prediction errors using the classification loss (e.g., cross-entropy loss) with respect to the learnable prompts, which may lead to learning general embeddings or inaccurate classrelated embeddings. for instance, when presented with an image (fig. 1a ) that contains both a head class [person] and a tail class [horse], the zero-shot method (fig. 1b ) relies solely on the rich knowledge of the pre-trained vlms to assess the similarity between the image and the word embeddings of the class names, while the existing prompt tuning method (fig. 1c ) further learns more generalized prompt tokens to improve model performance. however, these methods do not consider the interclass relationships, particularly between head and tail classes, which is a critical factor for ltml. this underscores the need for approaches that incorporate such relationships to improve performance in such scenarios. therefore, to address these issues, we present the class-specific embedding loss for prompt tuning on long-tailed multi-label visual recognition, called lmpt. the abundance of image-caption data facilitates prompt learning that encompasses more nuanced and specific textual descriptions, as well as the semantic inter-dependencies between categories (fig. 1d ) that share information, such as similar features or common descriptions. this attribute is particularly critical in the identification of both head and tail classes. more specifically, we propose the class-specific embedding loss to enhance the inclusivity of class-related embeddings within prompts. by gradually approaching the embeddings of the corresponding caption, our proposed approach enables prompt tokens to effectively judge the association between different classes with the aid of textual modality. aiming for class imbalance and consistency improvements between head classes and tail classes, we integrate class-aware soft margin and re-weighting into the class-specific embedding loss, which serves to assign larger margins and more weights to tail classes. notably, for images containing both head and tail classes, our approach outperforms visual models and current prompt tuning methods. moreover, we adopt the distribution-balanced loss @xcite as the classification loss. to sum up, the main contributions of this work include: ## conclusion in this work, we propose a new view of prompt tuning for long-tailed multi-label visual recognition by learning class-specific contexts from the alignment of prompts and textual description (caption), which complements more fine-grained features and builds semantic relationships across head and tail classes. considering the class imbalance, a novel class-specific embedding loss with the class-aware soft margin and re-weighting strategy is introduced to promote increased generalization among the tail classes. furthermore, we integrate a distributionbalanced loss as the classification loss function in consideration of its empirical efficacy compared to alternative loss functions. our method exhibits significant improvement over the previous state-of-theart (sota) and zero-shot clip on voc-lt and coco-lt. additionally, we hope our approach will inspire future work in this field.