HILL : Hierarchy-aware Information Lossless Contrastive Learning for Hierarchical Text Classification (2024)

Abstract: Existing self-supervised methods in natural language processing (NLP), especially hierarchical text classification (HTC), mainly focus on self-supervised contrastive learning, extremely relying on human-designed augmentation rules to generate contrastive samples, which can potentially corrupt or distort the original information. In this paper, we tend to investigate the feasibility of a contrastive learning scheme in which the semantic and syntactic information inherent in the input sample is adequately reserved in the contrastive samples and fused during the learning process. Specifically, we propose an information lossless contrastive learning strategy for HTC, namely \textbf{H} ierarchy-aware \textbf{I} nformation \textbf{L} ossless contrastive \textbf{L} earning (HILL), which consists of a text encoder representing the input document, and a structure encoder directly generating the positive sample. The structure encoder takes the document embedding as input, extracts the essential syntactic information inherent in the label hierarchy with the principle of structural entropy minimization, and injects the syntactic information into the text representation via hierarchical representation learning. Experiments on three common datasets are conducted to verify the superiority of HILL.

## introduction self-supervised learning (ssl) has exhibited remarkable success across various domains in deep learning, empowering models with potent representation capabilities. based on these achievements, researchers have incorporated contrastive learning into hierarchical text classification @xcite , a challenging sub-task within the realm of text multi-label classification. beyond processing text samples, hierarchical text classification methods should handle a predefined directed acyclic graph in the corpus, referred to as the label hierarchy. while language models such as bert @xcite are pretrained on textual data, their efficacy in handling structural input is limited. consequently, researchers have introduced a graph neural network (gnn)-based encoder to establish a dual-encoder framework for htc @xcite @xcite . although the structure encoder contributes to representing the label hierarchy, the dual-encoder framework simply blends the outputs of encoders. in an effort to integrate the label hierarchy into bert, @xcite propose a contrastive learning framework, in which bert functions as a siamese network @xcite accepting both the raw text and the masked text, where the mask is generated by the structure encoder. however, their contrastive learning process essentially relies on data augmentation, even with the involvement of the structure encoder in the masking process, as depicted in figure 1 @xcite . according to the data processing inequality @xcite , applying data augmentation to the raw text may potentially erase sufficient semantic information relevant to downstream prediction. to maximally preserve the semantic information in the text and effectively leverage the structural encoder in the contrastive learning process for htc, we tend to inject the essential information inherent in the label hierarchy into text representations rather than augmenting the input document. as shown in figure 1 (b), our structure encoder directly generates feature vectors by fusing textual and structural information, in contrast to masking the text as illustrated in figure 1 @xcite . following the insights of @xcite , where structural entropy encodes and decodes the essential structure of the original system to support its semantic analysis. since hierarchical text classification holds both semantic and syntactic information, we aim to design our model with the guidance of structural entropy. specifically, we implement a suite of algorithms to minimize the structural entropy of label hierarchies by constructing their coding trees. subsequently, we design a structure encoder to perform representation learning on coding trees, in which the leaf-node embeddings are initialized by the text encoder while the non-leaf node embeddings are iteratively obtained from bottom to top. afterward, the structure encoder generates an overall representation of the coding tree, which serves as a contrastive sample for the text encoder. additionally, we provide a definition of information lossless learning and prove that the information retained by our approach is the upper bound of any augmented data. in comparison with other contrastive and supervised learning methods, our model achieves significant performance gains on three common datasets. overall, the contributions of our work can be summarized as follows: given the success of pretrained language models (plms), researchers attempt to utilize their powerful abilities in htc. @xcite propose a contrastive learning framework for htc to make bert learn from the structure-encoder-controlled text augmentation. @xcite introduce prompt tuning and construct dynamic templates for htc. @xcite encode the global hierarchies with bert, extract the local hierarchies, and feed them into bert in a prompt-tuning-like schema. despite their success, neither existing contrastive learning nor prompt tuning methods try to improve the structure encoder. contrastive learning. inspired by the pretext tasks in gpt @xcite and bert @xcite , researchers originally proposed contrastive learning in computer vision @xcite , addressing the limitations of previous methods in training with massive unlabeled visual data. numerous studies text encoder structure encoder classifier projector contrastive loss classification loss label hierarchy input tokens semantic information label hierarchy optimal coding tree document embeddings structural entropy minimization projection initial coding tree hierarchical representation learning data flow information iossless positive pair syntactic information layer-wise feature pooling node representations on coding tree we first feed the document d into the text encoder to extract the semantic information. next, the structure encoder takes label hierarchy g l as input and constructs the optimal coding tree t l with algorithm 1 under the guidance of structural entropy. in the hierarchical representation learning module, the leaf node embeddings are initialized by the document embeddings, and the representations of non-leaf nodes are learned from bottom to top. the structure encoder finally generates an information lossless positive view for the text encoder, which is formulated in section 3.4 and proved in appendix c. ## methodology in this section, we first give a problem definition of hierarchical text classification. next, we elaborate on the working process of the text encoder (in section 3.2) and the structure encoder (in section 3.3) of the proposed hill. theoretical analysis is further given to reveal the information lossless property of hill for htc in section 3.4. overall, the framework of hill is shown in figure 2 . ## conclusion in this paper, we design a suite of methods to address the limitations of existing contrastive learning models for htc. in particular, we propose hill, in which the syntactic information is sufficiently fused with the semantic information after structural entropy minimization and hierarchical representation learning. theoretically, we give the definition on information lossless learning for htc and the information extracted by hill is proved to be the upper bound of other contrastive learning based methods. experimental results demonstrate the effectiveness and efficiency of our model against state-of-the-arts.