HTCI nfo M ax: A Global Model for Hierarchical Text Classification via Information Maximization (2021)

Abstract: The current state-of-the-art model HiAGM for hierarchical text classification has two limitations. First, it correlates each text sample with all labels in the dataset which contains irrelevant information. Second, it does not consider any statistical constraint on the label representations learned by the structure encoder, while constraints for representation learning are proved to be helpful in previous work. In this paper, we propose HTCInfoMax to address these issues by introducing information maximization which includes two modules: text-label mutual information maximization and label prior matching. The first module can model the interaction between each text sample and its ground truth labels explicitly which filters out irrelevant information. The second one encourages the structure encoder to learn better representations with desired characteristics for all labels which can better handle label imbalance in hierarchical text classification. Experimental results on two benchmark datasets demonstrate the effectiveness of the proposed HTCInfoMax.

## introduction hierarchical text classification (htc) is a particular subtask of multi-label text classification @xcite . many datasets have been proposed to study htc for decades, such as rcv1 @xcite and nytimes @xcite , which categorize a news into several categories/labels. and all the labels in each dataset are usually organized as a tree or a directed acyclic graph. thus, there is a label taxonomic hierarchy existing in each dataset. the goal of htc is to predict multiple labels in a given label hierarchy for a given text. there are two groups of existing methods for htc: local approaches and global approaches. local approaches usually build a classifier for each label/node @xcite , or for each parent node, or for each level of the label hierarchy @xcite @xcite . global approaches just build one classifier to simultaneously predict multiple labels of a given text. the earlier global approaches ignore the hierarchical structure of labels and assume there is no dependency among labels which leads to flat models such as @xcite . later on, more and more works try to make use of the label taxonomic hierarchy to improve the performance by employing different strategies such as recursively regularized graph-cnn @xcite , reinforcement learning @xcite , attentional capsule network @xcite , meta-learning @xcite and structure encoder @xcite . many attention-based models are also proposed to learn more refined text features for text classification tasks such as @xcite . among these methods, hiagm proposed by @xcite is the state-of-the-art model for htc which designs a structure encoder that integrates the label prior hierarchy knowledge to learn label representations, and then proposes a model hiagm with two variants (one is hiagm-la, the other is hiagm-tp) based on the structure encoder to capture the interactions between text features and label representations. however, there are some limitations of hiagm. firstly, it utilizes the same label hierarchy information for every text sample which cannot distinguish the relevant and irrelevant labels to a specific text sample. although hiagm-la can implicitly relate each text to its corresponding labels by soft attention weights, there are still irrelevant and noisy information. secondly, for hiagm-la, there is no statistical constraint on the label embeddings generated by the structure encoder, while statistical constrains for representation learning are proved to be helpful by @xcite . to address the two limitations of hiagm-la, we propose htcinfomax which introduces information maximization consisting of two new modules which are text-label mutual information maximization and label prior matching on top of hiagm-la. specifically, the first new module makes a connection between each text sample and its corresponding labels explicitly by maximizing the mutual information between them, and thus can filter out irrelevant label information for a specific text sample. the label prior matching module can impose some constraints on the learned representation of each label to force the structure encoder to learn better representations with desirable properties for all labels and thus also improve the quality of representations for low-frequency labels, which helps handle label imbalance issue better. in summary, our main contributions are: 1) we propose a novel global model htcinfomax for htc by introducing information maximization which includes two modules: text-label mutual information maximization and label prior matching. 2) to our best knowledge, this is the first work to utilize text-label mutual information maximization for htc which enables each text to capture its corresponding labels' information in an effective way. 3) also, to our best knowledge, this is the first work to introduce label prior matching for htc which encourages the structure encoder to learn desired label representations for all labels which can better handle inherent label imbalance issue in htc. 4) experimental results demonstrate the effectiveness of our proposed model for htc. 5) we release our code to enable replication, available at https://github. com/ringbdstack/htcinfomax . ## conclusion we propose htcinfomax to address the limitations of hiagm by introducing information maximization which includes two modules: text-label mutual information maximization and label prior matching. the label prior matching can drive the model to learn better representations for all labels, while the other module further fuses such learned label representations into text to learn better text representations containing effective label information for prediction. the experimental results demonstrate the effectiveness of htcinfomax.