An Effective Deployment of Contrastive Learning in Multi-label Text Classification (2023)

Abstract: The effectiveness of contrastive learning technology in natural language processing tasks is yet to be explored and analyzed. How to construct positive and negative samples correctly and reasonably is the core challenge of contrastive learning. It is even harder to discover contrastive objects in multi-label text classification tasks. There are very few contrastive losses proposed previously. In this paper, we investigate the problem from a different angle by proposing five novel contrastive losses for multi-label text classification tasks. These are Strict Contrastive Loss (SCL), Intra-label Contrastive Loss (ICL), Jaccard Similarity Contrastive Loss (JSCL), Jaccard Similarity Probability Contrastive Loss (JSPCL), and Stepwise Label Contrastive Loss (SLCL). We explore the effectiveness of contrastive learning for multi-label text classification tasks by the employment of these novel losses and provide a set of baseline models for deploying contrastive learning techniques on specific tasks. We further perform an interpretable analysis of our approach to show how different components of contrastive learning losses play their roles. The experimental results show that our proposed contrastive losses can bring improvement to multi-label text classification tasks. Our work also explores how contrastive learning should be adapted for multi-label text classification tasks.

## introduction multi-label text classification is an important branch of text classification technology @xcite . different from binary classification tasks or multi-class classification tasks, multi-label classification tasks need to assign at least one label to a piece of text. since the number of labels the text belongs to is not fixed, it greatly increases the difficulty of the model prediction. specifically, the uncertainty in the number of labels poses two challenges to the training of multi-label text classification models: the output logic of the model and the semantic representation space of the model. in recent years, most multi-label text classification research has focused on designing better output logic to solve the uncertainty of the number of labels, such as transforming the multi-label text classification problem into a multi-task problem @xcite . however, for another challenge, how to construct a better semantic representation space for multi-label text classification models, little research attention has been paid. the existence of multi-label samples can easily confound the semantic representation space, thereby posing a challenge in data analysis and modeling. when confronted with multi-label samples, the semantic representation space becomes susceptible to distractions, where the boundaries between different classes become blurred. this blurring effect stems from the inherent ambiguity that arises when multiple labels coexist within a single sample, causing uncertainty in the multi-label classification tasks. take the multi-label emotion classification task as an example (shown in figure 1 ), in which the "happy" sample (assumed to be sample a) shares a label with the "happy, surprise" sample (assumed to be sample b), and at the same time, the "surprise" sample (assumed to be sample c) also shares a label with the sample b (shown in figure 1 (a) ). therefore, in the ideal state, the multi-label classification model assumes that sample a and sample b are located in a similar semantic space, and that sample b and sample c are located in another similar semantic space (shown in figure 1 (b) ). figure 1 (c ) shows how the samples of the "happy" category and the samples of the "surprise" category are confounded in the semantic space. this will cause sample a and sample c to be brought closer indirectly, even if their labels are completely different. as far as we know, the semantic representation of multi-label samples is still an open issue in the multi-label text classification task. therefore, this paper focuses on using contrastive learning to improve the semantic representation of multi-label text classification models. as an emerging technology, contrastive learning has achieved good performance in various fields of natural language processing @xcite . how to construct positive and negative samples correctly and reasonably is the core challenge of contrastive learning. in multilabel text classification tasks, it is a great challenge to incorporate the contrastive learning module. it is more difficult for contrastive learning to perform well in multi-label text classification tasks than in other text classification tasks because implicit information representation of multi-label text is richer in the semantic space, which makes it more difficult to define positive and negative samples. existing studies have proposed unsupervised contrastive learning methods to improve the performance of the model on multi-label text classification tasks @xcite , and there are also working to improve supervised contrastive learning @xcite . however, the exploration of contrastive learning in multi-label text classification tasks is still very limited. as the typical task in multi-label text classification, multi-label emotion classification task @xcite @xcite and multi-label news classification task @xcite have received extensive attention. in this paper, we propose five contrastive losses for multilabel text classification tasks and verify the performance of our method with the multi-label emotion classification task and multi-label news classification task as the representative tasks. more specifically, they are strict contrastive loss (scl), intra-label contrastive loss (icl), jaccard similarity contrastive loss (jscl), jaccard similarity probability contrastive loss (jspcl), and stepwise label contrastive loss (slcl). these five different strategies define the positive samples and negative samples of contrastive learning from different perspectives to pull the distance among different types of samples into the semantic space. to compare the effects of the five strategies, we further conduct an interpretable analysis to investigate how the different contrastive learning methods play their roles. the experimental results show that our proposed contrastive losses can bring improvement for multi-label text classification tasks. in addition, our methods could be considered as a set of baseline models of viable contrastive learning techniques for multi-label text classification tasks. this series of contrastive learning methods are plug-andplay losses, which can be applied to any multi-label text classification model, and to a certain extent, bring effective improvements to the multi-label text classification model. the major contributions of this paper can be summarized as follows: ## contrastive loss for multi-label text classification in this section, we describe in detail the application of our proposed different contrastive learning methods on multi-label text classification tasks. we take the multi-label emotion classification task as an example to describe our method. it is worth noting that our proposed method can not only be applied to multi-label emotion classification tasks, but also can be applied to other multi-label text classification tasks. suppose a minibatch which contains k samples @xmath0, y 1 ), (x 2 , y 2 ), . . . , (x k , y k )} and @xmath1, . . . , k} is the sample index set. given a sample index i, x i is the text sequence of samples i and its label set is denoted as y i . after encoding by the multi-label text classification model m , we could obtain the sentence representation vector e t i and the emotion representation matrix e e i of x i , where e e @xmath2, e i2 , . . . , e il } and l represents the total number of emotion labels. it is worth noting that the model m here can be any deep learning multi-label language model. y i is the one-hot encoding of the label, i.e. y @xmath3, y 2 , ..., y l }. for a given i-th emotion y @xmath4, 1}, y @xmath5, and y @xmath6. we further define the label prediction probability distribution of the model m output as p i . contrastive learning aims to change the semantic representation space of the model. since the multilabel classification tasks are more complex than the single-label classification tasks, the main exploration of our paper is how one can construct positive and negative samples for contrastive learning. when contrastive learning is applied to multilabel text classification, for an anchor, the definition of its positive samples can be diversified. for example, when a strict standard is implemented, positive samples are defined as samples with exactly the same label set (shown in figure 2 (a) ), when a loose standard is implemented, positive samples are defined as samples with partly the same label set (shown in figure 2 (b) ). for different positive samples, contrastive learning pulls different samples closer in semantic space for a given anchor. in the strict standard, we could find that for an anchor point, there are fewer positive samples, and samples containing some similar features cannot be pulled closer. in the loose standard, there are more positive samples for an anchor point, which may indirectly bring samples of different labels closer. therefore, different positive and negative sample construction methods affect the optimization goal of the model. what's more, there are two different types of contrastive learning, feature-based contrastive learning (feacl) @xcite and probability-based contrastive learning (procl) @xcite . feacl uses semantic representations of sentences as the basic component to build the contrastive objective function. procl constructs the contrastive objective function from the perspective of probability distributions instead of semantic representations. using different features for contrastive learning will also affect the optimization of the model. in order to explore how contrastive learning can be better applied to multi-label text classification tasks, we introduce five different contrastive learning methods scl, icl, jscl, jspcl, and slcl, as below. ## conclusion to investigate the efficacy of contrastive learning using various methodologies, we offer five effective contrastive losses for multi-label text classification tasks. the experimental results of this paper show that contrastive loss can improve the performance of multi-label text classification tasks. furthermore, we find that strict criteria contrastive learning and feature-based contrastive learning outperform other contrastive learning methods on multi-label text classification tasks. in the future, based on these two methods, we will further explore the contrastive loss that is more suitable for multi-label text classification tasks.