From Text Segmentation to Enhanced Representation Learning: A Novel Approach to Multi-Label Classification for Long Texts (2024)

Abstract: Multi-label text classification (MLTC) is an important task in the field of natural language processing. Most existing models rely on high-quality text representations provided by pre-trained language models (PLMs). They hence face the challenge of input length limitation caused by PLMs, when dealing with long texts. In light of this, we introduce a comprehensive approach to multi-label long text classification. We propose a text segmentation algorithm, which guarantees to produce the optimal segmentation, to address the issue of input length limitation caused by PLMs. We incorporate external knowledge, labels’ co-occurrence relations, and attention mechanisms in representation learning to enhance both text and label representations. Our method’s effectiveness is validated through extensive experiments on various MLTC datasets, unraveling the intricate correlations between texts and labels.

## introduction multi-label text classification (mltc) aims to assign multiple relevant labels to a given text. it plays a crucial role in various applications such as content recommendation @xcite , sentiment analysis @xcite , and information retrieval @xcite . the pre-trained language models (plms), e.g., bert @xcite and roberta @xcite significantly improve the performance of various natural language processing tasks including mltc. however, plms have an inherent limitation in handling long texts due to their maximum input sequence length of 512 tokens. this limitation could lead to information missing when dealing with long texts. more than this, text representations directly obtained from plms should be enhanced further, by incorporating e.g., domain knowledge @xcite . another * corresponding author. difficulty for mltc arises from intricate labellabel and word-label relations. these complicated relations, if effectively captured, are expected to improve prediction performance. to overcome the input length limitation of plms, investigators have proposed a series of techniques that are based on e.g., text truncation @xcite , sparse attention mechanism @xcite , and text segmentation @xcite . @xcite proposes two methods bert+rank and bert+random. bert+rank concatenates the first 512 tokens of a text representation with the representations of the top-ranked sentences (up to 512 tokens). unlike bert+rank, bert+random concatenates the first 512 tokens of a text representation with at most 512 tokens randomly selected from the text corpus. these methods often perform poorly due to the loss of text information. the reason why plms limit the input length is that they incorporate a self-attention mechanism, whose computational complexity is in quadratic time of the input length. therefore, bigbird @xcite and longformer @xcite utilize the sparse self-attention mechanism to handle longer input sequences. nevertheless, they haven't fundamentally solved the issue and can not process sequences of arbitrary length. @xcite proposes a dynamic programming method, named cogltx, for text segmentation. @xcite splits a long text into fixed-length chunks, encodes these chunks by bert, and concatenates the representations of chunks as the representation of the long text. however, the methods above do not take into consideration the semantic change before and after segmentation, hence may damage the semantic coherence of a text. the above methods mainly focus on the content of texts alone, ignore the integration of external knowledge. as pointed out by @xcite , the utilization of high-quality external knowledge can enhance text representations, and then improve prediction accuracy. however, to incorporate external knowledge, two crucial questions have to be answered: (a) how to establish the connection between texts and external knowledge? (b) how to develop a model to produce text-related knowledge representations? if the above issues were not effectively resolved, external knowledge would become noise and may worsen prediction performance. label correlations have been explored in mltc. a typical model is @xcite . the model first captures labels' co-occurrence relation and then employs the graph convolutional network (gcn) to update label representations based on the co-occurrence relation. however, ldgn initializes label representations randomly without considering the labels' semantic information. indeed, when the semantic information is incorporated, not only label representations can be improved, but also the importance of words to labels can be captured. to tackle the issues above, we propose a model (skfrl) that integrates dynamic segmentation, knowledge fusion, and representation learning for multi-label long text classification. skfrl is equipped with a text segmentation method to partition a long text into sub-texts. it also fuses text representations with external knowledge as enhanced text representations. it further enhances both text and label representations by capturing intricate label-label and word-label relations. in summary, our contributions are as follows: extensions on plms. pre-trained language models like bert @xcite , roberta @xcite significantly advance the field of natural language process. however, one limitation of plms is their fixed input sequence length, typically capped at 512 tokens. this limitation causes a significant challenge when dealing with long texts. @xcite directly truncates a text and discards the remaining part. pappagari et al. is the label-weighted text representation obtained through (d), h (cls) i is the representation of the special token [cls]. (2019); jin et al. (2021) simply split a long text into small segments to meet the input length of plms. @xcite proposes an objective function, that measures the goodness of a segmentation purely by cut points, and cuts a long text into sub-texts via optimizing the objective function. @xcite splits a long text into fixed-length chunks and processes these chunks by bert independently. however, the segmentation of the above methods does not consider the semantic information and the coherence of a text, hence may damage the semantic information of the original text. @xcite proposes a digest algorithm to extract more important sentences at the beginning and end of a text. recently, extended transformer architectures enable plms to handle longer sequences by introducing sparse attention mechanisms. typical works include transformer-xl @xcite , reformer @xcite , longformer @xcite , and bigbird @xcite . specifically, @xcite develops a segment-level recurrence mechanism along with a novel positional encoding scheme. @xcite replaces the dot-product attention by one that uses locality-sensitive hashing, to reduce computational complexity. longformer @xcite combines local windowed attention with a task motivated global attention. bigbird @xcite designs a sparse attention mechanism to reduce the quadratic dependency to linear. though they attempt to reduce complexity by introducing various attention mechanisms, they still face challenges in balancing computational efficiency and model performance. our goal is threefold. (1) we develop a text segmentation method that segments a long text by considering semantic coherence, to address the input length constraints of plms. (2) we incorporate external knowledge to enrich both text and label representations. (3) we leverage label-label and wordlabel relations to further enhance text and label representations. by doing so, we aim to obtain a robust model for multi-label long text classification. ## method in this section, we introduce an approach to addressing the issue of mltc on long texts. as shown in figure 1 , our approach consists of four parts, i.e., long text segmentation, external knowledge fusion, enhanced label representation learning, and representation enhancing with attention (maxatt for words and labels). overall, given a long text, skfrl first cuts it into several sub-texts with the text segmentation algorithm txtseg. after segmentation, all the sub-texts can be encoded via bert directly, thereby addressing the issue of input length limit. skfrl then introduces external knowledge to enhance the representation of each sub-text. meanwhile, skfrl processes the labels along the similar line as that on texts and gains the fused label representations. it then updates the fused label representations through gcn and obtains the enhanced label representations. finally, to capture the importance of words to different labels, skfrl utilizes the max attention mechanism (maxatt) to obtain the label-weight text representations. we next elaborate on each of them, starting from the problem formulation. problem formulation: ## conclusion in this work, we propose a novel model for multilabel long text classification. the model achieves superior text segmentation by following dynamic programming, thereby addressing the issue of input length limitation caused by plms. the use of external knowledge enriches the text representation encoded by plms effectively. to further enhance the text representation, gcn and max-attention mechanisms are utilized to obtain the label-weighted text representation. experimental results show that our model outperforms baselines, which confirms the effectiveness of our model in text segmentation and incorporation of external knowledge.