Incorporating Hierarchy into Text Encoder: a Contrastive Learning Approach for Hierarchical Text Classification (2022)

Abstract: Hierarchical text classification is a challenging subtask of multi-label classification due to its complex label hierarchy. Existing methods encode text and label hierarchy separately and mix their representations for classification, where the hierarchy remains unchanged for all input text. Instead of modeling them separately, in this work, we propose Hierarchy-guided Contrastive Learning (HGCLR) to directly embed the hierarchy into a text encoder. During training, HGCLR constructs positive samples for input text under the guidance of the label hierarchy. By pulling together the input text and its positive sample, the text encoder can learn to generate the hierarchy-aware text representation independently. Therefore, after training, the HGCLR enhanced text encoder can dispense with the redundant hierarchy. Extensive experiments on three benchmark datasets verify the effectiveness of HGCLR.

## introduction hierarchical text classification (htc) aims to categorize text into a set of labels that are organized in a structured hierarchy @xcite . the taxonomic hierarchy is commonly modeled as a tree or a directed acyclic graph, in which each node is a label to be classified. as a subtask of multi-label classification, the key challenge of htc is how to model the large-scale, imbalanced, and structured label hierarchy @xcite . the existing methods of htc have variously introduced hierarchical information. among recent researches, the state-of-the-art models encode text and label hierarchy separately and aggregate two representations before being classified by a mixed feature @xcite . as denoted in the left part of figure 1 , their main goal is to sufficiently interact between text and structure to * corresponding author. text encoder graph encoder taxonomic hierarchy joseph bruno, the state senate majority leader, ... text mixed text-label representation (a) text feature label feature text encoder graph encoder taxonomic hierarchy joseph bruno, the state senate majority leader, ... text mixed text-label representation (a) text feature label feature hierarchy-aware text representation text encoder graph encoder taxonomic hierarchy joseph bruno, the state senate majority leader, ... achieve a mixed representation @xcite , which is highly useful for classification @xcite . however, since the label hierarchy remains unchanged for all text inputs, the graph encoder provides exactly the same representation regardless of the input. therefore, the text representation interacts with constant hierarchy representation and thus the interaction seems redundant and less effective. alternatively, we attempt to inject the constant hierarchy representation into the text encoder. so that after being fully trained, a hierarchy-aware text representation can be acquired without the constant label feature. as in the right part of figure 1 , instead of modeling text and labels separately, migrating label hierarchy into text encoding may benefit htc by a proper representation learning method. to this end, we adopt contrastive learning for the hierarchy-aware representation. contrastive learning, which aims to concentrate positive samples and push apart negative samples, has been considered as effective in constructing meaningful representations @xcite . previous work on contrastive learning illustrates that it is critical to building challenging samples @xcite @xcite . for multi-label classification, we attempt to construct high-quality positive examples. existing methods for positive example generation includes data augmentation @xcite , dropout @xcite , and adversarial attack @xcite . these techniques are either unsupervised or taskunspecific: the generation of positive samples has no relation with the htc task and thus are incompetent to acquire hierarchy-aware representations. as mentioned, we argue that both the ground-truth label as well as the taxonomic hierarchy should be considered for the htc task. to construct positive samples which are both label-guided and hierarchy-involved, our approach is motivated by a preliminary observation. notice that when we classify text into a certain category, most words or tokens are not important. for instance, when a paragraph of news report about a lately sports match is classified as "basketball", few keywords like "nba" or "backboard" have large impacts while the game result has less influence. so, given a sequence and its labels, a shorten sequence that only keeps few keywords should maintain the labels. in fact, this idea is similar to adversarial attack, which aims to find "important tokens" which affect classification most @xcite . the difference is that adversarial attack tries to modify "important tokens" to fool the model, whereas our approach modifies "unimportant tokens" to keep the classification result unchanged. under such observation, we construct positive samples as pairs of input sequences and theirs shorten counterparts, and propose hierarchy-guided contrastive learning (hgclr) for htc. in order to locate keywords under given labels, we directly calculate the attention weight of each token embedding on each label, and tokens with weight above a threshold are considered important to according label. we use a graph encoder to encode label hierarchy and output label features. unlike previous studies with gcn or gat, we modify a graphormer @xcite as our graph encoder. graphormer encodes graphs by transformer blocks and outperforms other graph encoders on several graph-related tasks. it models the graph from multiple dimensions, which can be customized easily for htc task. the main contribution of our work can be summarized as follows: ## problem definition given a input text @xmath0, x 2 , ..., x n }, hierarchical text classification (htc) aims to predict a subset y of label set y , where n is the length of the input sequence and k is the size of set y . the candidate labels y @xmath1, e), where node set y are labels and edge set e denotes their hierarchy. for simplicity, we do not distinguish a label with its node in the hierarchy so that y i is both a label and a node. since a non-root label of htc has one and only one father, the taxonomic hierarchy can be converted to a tree-like hierarchy. the subset y corresponds to one or more paths in g: for any non-root label y @xmath2, a father node (label) of y j is in the subset y. ## methodology in this section, we will describe the proposed hg-clr in detail. figure 2 shows the overall architecture of the model. ## conclusion in this paper, we present hierarchy-guided contrastive learning (hgclr) for hierarchy text classification. we adopt contrastive learning for migrating taxonomy hierarchy information into bert encoding. to this end, we construct positive examples for contrastive learning under the guidance of a graph encoder, which learns label features from taxonomy hierarchy. we modify graphormer, a state-of-the-art graph encoder, for better graph understanding. comparing to previous approaches, our approach empirically achieves consistent improvements on two distinct datasets and comparable results on another one. all of the components we designed are proven to be effective.