Towards Better Representations for Multi-Label Text Classification with Multi-granularity Information (2023)

Abstract: Multi-label text classification (MLTC) aims to assign multiple labels to a given text. Previous works have focused on text representation learning and label correlations modeling using pre-trained language models (PLMs). However, studies have shown that PLMs generate word frequency-oriented text representations, causing texts with different labels to be closely distributed in a narrow region, which is difficult to classify. To address this, we present a novel framework \textbf{CL} ( \underline{C} ontrastive \underline{L} earning)- \textbf{MIL} ( \underline{M} ulti-granularity \underline{I} nformation \underline{L} earning) to refine the text representation for MLTC task. We first use contrastive learning to generate uniform initial text representation and incorporate label frequency implicitly. Then, we design a multi-task learning module to integrate multi-granularity (diverse text-labels correlations, label-label relations and label frequency) information into text representations, enhancing their discriminative ability. Experimental results demonstrate the complementarity of the modules in CL-MIL, improving the quality of text representations and yielding stable and competitive improvements for MLTC.

## introduction multi-label text classification (mltc) is a fundamental task in the natural language processing (nlp) field which refers to assigning multiple relevant labels to each text. mltc has been widely applied to various real-world scenarios like recommendation system @xcite , sentiment analysis @xcite and information retrieval @xcite and others @xcite @xcite . in mltc, there are diverse text-labels correlations. as shown in figure 1 , labels l 1 , l 2 are directly correlated to specific type of keywords and phrases (fine-grained) t 2 , t 4 , whereas latent correlations are determined by multiple information * *corresponding author (coarse-grained). for example, l 4 is jointly correlated with t 2 , t n-2 and t n . previous methods solve mltc task in diverse perspectives. among them, some methods focus on fine-grained (local semantic features) @xcite and coarse-grained (global semantic features) textual information @xcite , respectively. besides, @xcite utilize joint embedding and label-wise attention to enhance the text-labels correlations, and other methods learn the correlations between labels @xcite . these methods separately utilize information from different granularities to improve the classification performance. however, learning information on single granularity is insufficient to derive effective text representations compared to keeping a proper balance of multi-granularity information. recently, plms @xcite have been proved powerful to generate high-quality semantic representations. competitive and strong plms based methods @xcite are proposed to explore labels correlations and knowledge between samples. yet, @xcite @xcite ) reveal that plms appear anisotropy problem after fine-tuned, leading text representations collapse into a narrow cone space. texts consisting of high-frequency words are gathered closely, even correlated to extremely low-frequency labels. when samples with low-frequency labels concentrate densely, it will become more difficult to classify and also aggravate the long-tail problem @xcite due to the lack of samples. to address these, we propose a multi-granularity information enhancement framework to improve the quality of representations and relieve the anisotropy problem. we first perform contrastive learning to implicitly introduce label frequency into the initial representations and refine the uniformity of it. it only focuses on target samples (correlated to low-frequency labels), and push away source samples that share low label co-occurrence with labels in target sample according to the label distribution. moreover, we adopt the multi-granularity information learning (mil) module via multi-task learning to enhance the expressiveness of representations. specifically, mil module explores direct and latent text-labels correlations through dualgrained interactive learning (dil) task and models relations (strong relate, weak relate and contradict) between labels by constraint label relations learning (clrl) task. both tasks share the same encoder that jointly embeds text and labels to directly introduce label frequency into representations, which provides mil an access to transform the orientation of anisotropy problem. in dil, we use two different interactive learning tasks to obtain both direct and latent text-labels correlations. in clrl, we utilize label frequency and labels co-occurrence to explore the constraint relations, since the correlations between labels exists not only in single instance, predicting the label correlations from one instance may contradict with the the others. we illustrate that our method outperforms a series of competitive baseline on aapd and rcv1-v2 datasets by experimental results. we summarize our contributions as follows: multi-label text classification in mltc, early works like cnn @xcite @xcite and rnn @xcite label correlations, @xcite and @xcite used unique label combinations and classification chain. seq2seq-based methods @xcite transform mltc into label sequence generation tasks. these methods tackle mltc task via different granularities information, which strikes us the feasibility of balancing multi-granularity information to improve the expressiveness of text representations. anisotropy problem recently, pre-trained language models (plms) @xcite have become paradigms for various nlp tasks. plms based methods @xcite @xcite further achieve remarkable performance on mltc. however, ethayarajh (2019b); @xcite have demonstrated the anisotropy problem limits the expressiveness of text representations. the representations generated by plms are constrained within a narrow cone-shaped space after fine-tuned. @xcite reveals that texts with high-frequency words concentrate densely in the narrowest part of the cone, while texts with low-frequency words distribute sparsely. the anisotropy problem in mltc, owing to high word frequency, leads samples with low-frequency labels to be closely gathered with other samples of different label frequencies, which can corrupt semantic information of representation and hinder the classification performance of mltc. therefore, we use contrastive learning to improve the uniformity of representations and introduce label frequency information into representations to alleviate the anisotropy problem. ## proposed method our framework, illustrated in figure 2 , consists of two modules: a contrastive learning (cl) module and a multi-granularity information learning (mil) module. the cl module refines initial rep- target sample source sample initial enhancement batch s 1 s 2 s 3 s i s n s n-1 t 1 pull together push apart text reps labels reps dual-grained interactive learning coarse-grained information extraction layer c l d fine-grained information extraction layer y 3 y 2 y 1 y l y i + y i * y i - refer @xmath0,t1) low co-occurrence e [cls] h [cls] ## experimental settings in this section, we evaluate the main experimental results of the baseline models and our method on two datasets. implementation details of our method can be found in appendix a. ## conclusion this paper proposed a multi-granularity information enhanced mltc framework. through extensive experiments, we demonstrated the competitiveness and stability of our method and confirmed the effectiveness of incorporating multi-granularity information to enhance representations in mltc. additionally, our extended analysis highlighted the complementarity of the modules in our framework and emphasizes the necessity of each module.