Not All Negatives are Equal: L abel-Aware Contrastive Loss for Fine-grained Text Classification (2021)

Abstract: Fine-grained classification involves dealing with datasets with larger number of classes with subtle differences between them. Guiding the model to focus on differentiating dimensions between these commonly confusable classes is key to improving performance on fine-grained tasks. In this work, we analyse the contrastive fine-tuning of pre-trained language models on two fine-grained text classification tasks, emotion classification and sentiment analysis. We adaptively embed class relationships into a contrastive objective function to help differently weigh the positives and negatives, and in particular, weighting closely confusable negatives more than less similar negative examples. We find that Label-aware Contrastive Loss outperforms previous contrastive methods, in the presence of larger number and/or more confusable classes, and helps models to produce output distributions that are more differentiated.

## introduction fine-grained classification involves distinguishing between classes that have subtle variations among them. for example, in image classification, we can classify birds from non-birds, or attempt a more fine-grained classification of bird species @xcite . in nlp, one example is sentiment analysis, where we could have a coarse positive/negative classification, or a fine-grained set of categories that differentiate "positive" and "very positive" (i.e., an ordinal scale), such as in @xcite . similarly, for emotion classification, we could try to classify a text into 4 to 6 emotions, or into much finer classifications of 27 @xcite or 32 @xcite emotion categories. this involves distinguishing between some closely confusable pairs of emotions, such as "sad" and "devastated", or "furious" and "annoyed". fine-grained classification tasks are challenging precisely due to the presence of class interference amongst closely confusable classes @xcite . the standard approach today to task classification involves using a pre-trained language model (e.g., bert) which is fine-tuned on downstream tasks using a standard cross-entropy loss. however, this standard loss may not be the optimal manner in which to train fine-grained classification models. a simple counterexample is that cross-entropy loss treats misclassifications as nominal, not ordinal, so misclassifying a "positive" as a "very positive" is no worse (in terms of the loss) as "very negative". but even within nominal categories, misclassifying "annoyed" as "furious" is quite different from a misclassification of "joyful", as there are varying degrees of semantic similarity between nominal categories. intuitively, we can try to improve model performance by modifying the loss to reflect the contrast between pairs of examples of the same or different classes. such contrastive approaches are widely used in computer vision tasks for label-noise reduction, semi-supervised and selfsupervised learning tasks @xcite . more recently in nlp, @xcite used a supervised contrastive loss to improve fine-tuning performance of pre-trained language models in several few-shot learning scenarios. in this work, we incorporate inter-class relationships into a label-aware contrastive loss (lcl), which helps the model to differentiate the weights between different negative samples. at a high level, the model adaptively learns which pairs of classes are more similar, and which are more different. we use a dual-model approach where a weighting model learns the inter-label relationships that are used in the main embedding model's contrastive objective. we evaluate our approach on two popular tasks in nlp: emotion recognition (4 datasets to span both coarse-and fine-grained classification), and sentiment analysis (with a coarse and fine-grained version of the same dataset). we find that lcl outperforms existing contrastive learning losses, and performs comparably with the stateof-the-art. we supplement our findings with targeted experiments to provide evidence for boundary conditions-situations in which lcl should work best-and for how lcl affects model prediction confidence. ## conclusion in this paper we introduced a label-aware contrastive loss that weights (negative) classes based on how closely confusable they are with the target class. fine-tuning with lcl showed increased classification performance, especially in situations with (i) larger number of classes, and (ii) more confusable classes. lcl also seems to encourage the model to be more confident in its decisions. we view our approach as just one way to instantiate the general idea of adaptively weighting different classes, and future work could explore other methods such as incorporating external knowledge about the class labels, or incorporating different distance metrics between different classes. we feel that this class of approaches are promising, as they exemplify the idea that not all negative classes are or should be treated equally.