Logits Reranking via Semantic Labels for Hard Samples in Text Classification (2024)

Abstract: Pre-trained Language Models (PLMs) have achieved significant success in text classification. However, they still face challenges with hard samples, which refer to instances where the model exhibits diminished confidence in distinguishing new samples. Existing research has addressed related issues, but often overlooks the semantic information inherent in the labels, treating them merely as one-hot vectors. In this paper, we propose Logits Reranking via Semantic Labels (LRSL), a model-agnostic post-processing method that leverages label semantics and auto detection of hard samples to improve classification accuracy. LRSL automatically identifies hard samples, which are then jointly processed by MLP-based and Similarity-based approaches. Applied only during inference, LRSL operates solely on classification logits, reranking them based on semantic similarities without interfering with the model’s training process. The experiments demonstrate the effectiveness of our method, showing significant improvements across different PLMs. Our codes are publicly available at https://github.com/SIGSDSscau/LRSL.

## methodology mlp-based plms have achieved remarkable success in text classification, but the effect is greatly reduced when facing hard samples. as shown in figure 1(b) , compared with the banking-77, the model's classification accuracy tends to decrease more significantly for the other two long-tailed datasets, both of which have more hard samples, as the range of top-k accuracy is narrowed. tran-sitioning from top-3 to top-1 reveals a sharp decline in accuracy, indicating that mlp-based models struggle to handle hard samples effectively. thus, we let the model detect hard samples automatically and post-process these samples. utilizing an additional text embedding, we use its semantic distance between each sample and labels as the weights to rerank the logits of hard samples to enhance classification capabilities. this approach presents three challenges: how to make good use of text embedding? how to detect the hard samples? how to rerank the logits? the overview of our approach is shown in figure 2 . we fine-tune an additional text embedding with triplets for semantic mapping ( §3.1), allowing text embedding to provide a rich and accurate semantic space where texts and labels can be directly associated. we then propose an auto detection mechanism ( §3.2), enabling the model to identify its own hard samples. for these hard samples, we use semantic distance as an alternative view of the logits, combining the two for reranking ( §3.3). ## conclusion we propose lrsl, a model-agnostic postprocessing method that leverages label semantics and auto detection of hard samples to improve classification accuracy. our method demonstrates that label semantics serve as an effective reranker for hard samples in mlp-based classification. by reranking classification logits, lrsl provides a plug-and-play solution that does not interfere with the original model's training process. the results demonstrate the effectiveness of our method.