H i TIN : Hierarchy-aware Tree Isomorphism Network for Hierarchical Text Classification (2023)

Abstract: Hierarchical text classification (HTC) is a challenging subtask of multi-label classification as the labels form a complex hierarchical structure. Existing dual-encoder methods in HTC achieve weak performance gains with huge memory overheads and their structure encoders heavily rely on domain knowledge. Under such observation, we tend to investigate the feasibility of a memory-friendly model with strong generalization capability that could boost the performance of HTC without prior statistics or label semantics. In this paper, we propose Hierarchy-aware Tree Isomorphism Network (HiTIN) to enhance the text representations with only syntactic information of the label hierarchy. Specifically, we convert the label hierarchy into an unweighted tree structure, termed coding tree, with the guidance of structural entropy. Then we design a structure encoder to incorporate hierarchy-aware information in the coding tree into text representations. Besides the text encoder, HiTIN only contains a few multi-layer perceptions and linear transformations, which greatly saves memory. We conduct experiments on three commonly used datasets and the results demonstrate that HiTIN could achieve better test performance and less memory consumption than state-of-the-art (SOTA) methods.

## introduction hierarchical text classification is a sub-task of text multi-label classification, which is commonly applied in scenarios such as news document classification @xcite , academic paper classification @xcite , and so on. unlike traditional classification tasks, the labels of htc have parent-child relationships forming a hierarchical structure. due to the complex structure of label hierarchy and the * equal contribution. † correspondence to: junran wu. imbalanced frequency of labels, htc becomes a challenging task in natural language processing. recent studies in htc typically utilize a dualencoder framework @xcite , which consists of a text encoder for text representations and a structure encoder to inject the information of labels into text. the text encoder could be a traditional backbone for text classification, for instance, tex-trcnn @xcite or bert @xcite . the structure encoder is a graph neural network (gnn) that treats the label hierarchy as a directed acyclic graph (dag) and propagates the information among labels. to maximize the propagation ability of the structure encoder, zhou et al. (2020) learn textual features of labels and count the prior probabilities between parent and child labels. based on the dual-encoder framework, researchers further complicated the model by adding complementary networks and loss functions from different aspects, such as treating htc as a matching problem @xcite , introducing mutual information maximization @xcite . however, more complementary components result in more memory consumption, as shown in figure 1 . on the other hand, their structure encoders still rely on the prior statistics @xcite or the representation of labels @xcite . that is, their models require a mass of domain knowledge, which greatly reduces the generalization ability. to this end, we intend to design a more effective structure encoder with fewer parameters for htc. instead of introducing domain knowledge, we try to take full advantage of the structural information embedded in label hierarchies. inspired by @xcite , we decode the essential structure of label hierarchies into coding trees with the guidance of structural entropy, which aims to measure the structural complexity of a graph. the coding tree is unweighted and could reflect the hierarchical organization of the original graph, which provides us with another view of the label hierarchy. to construct coding trees, we design an algorithm, termed coding tree construction algorithm (circa) by minimizing the structural entropy of label hierarchies. based on the hierarchical structure of coding trees, we propose hierarchical-aware tree isomorphism network (hitin). the document representations fetched by the text encoder are fed into a structure encoder, in which we iteratively update the node embeddings of the coding tree with a few multi-layer perceptions. finally, we produce a feature vector of the entire coding tree as the final representation of the document. compared with sota methods of dual encoders on htc tasks @xcite @xcite , hitin shows superior performance gains with less memory consumption. overall, the contributions of our work can be summarized as follows: global studies in htc try to improve flat multilabel classification by introducing various information from the hierarchy. @xcite propose a recursive regularization function to make the parameters of adjacent categories have similar values. @xcite propose a regularized graph-cnn model to capture the nonconsecutive semantics from texts. besides, various deep learning techniques, such as sequenceto-sequence model @xcite , attention mechanism @xcite , capsule network @xcite , reinforcement learning @xcite , and meta-learning @xcite are also applied in global htc. recently, zhou et al. (2020) specially design an encoder for label hierarchies which could significantly improve performance. chen et al. (2020) learn the word and label embeddings jointly in the hyperbolic space. @xcite formulate the text-label relationship as a semantic matching problem. @xcite introduce information maximization which can model the interaction between text and label while filtering out irrelevant information. with the development of pretrained language model (plm), bert @xcite based contrastive learning @xcite , prompt tuning @xcite , and other methods @xcite have brought huge performance boost to htc. ## problem definition given a document @xmath0, w 2 , . . . , w n }, where w i is a word and n denotes the document length, hierarchical text classification aims to predict a subset y of the holistic label set y . besides, every label in y corresponds to a unique node on a directed acyclic graph, i.e. the label hierarchy. the label hierarchy is predefined and usually simplified as a tree structure. in the groud-truth label set, a non-root label y i always co-occurs with its parent nodes, that is, for any y @xmath1, the parent node of y i is also in y. ## methodology following the dual-encoder scheme in htc, the architecture of hitin that consists of a text encoder and a structure encoder is shown in figure 2 . the text encoder aims to capture textual information from the input document while the structure encoder could model the label correlations in the hierarchy and inject the information from labels into text representations. ## conclusion in this paper, we propose a suite of methods to address the limitations of existing approaches regarding htc. in particular, tending to minimize structural entropy, we design circa to construct coding trees for the label hierarchy. to further extract textual information, we propose hitin to update node embeddings of the coding tree iteratively. experimental results demonstrate that hitin could enhance text representations with only structural information of the label hierarchy. our model outperforms existing methods while greatly reducing memory increments.