Hierarchy-aware Biased Bound Margin Loss Function for Hierarchical Text Classification (2024)

Abstract: Hierarchical text classification (HTC) is a challenging problem with two key issues: utilizing structural information and mitigating label imbalance. Recently, the unit-based approach generating unit-based feature representations has outperformed the global approach focusing on a global feature representation. Nevertheless, unit-based models using BCE and ZLPR losses still face static thresholding and label imbalance challenges. Those challenges become more critical in large-scale hierarchies. This paper introduces a novel hierarchy-aware loss function for unit-based HTC models: Hierarchy-aware Biased Bound Margin (HBM) loss. HBM integrates learnable bounds, biases, and a margin to address static thresholding and mitigate label imbalance adaptively. Experimental results on benchmark datasets demonstrate the superior performance of HBM compared to competitive HTC models.

## introduction hierarchical text classification (htc) aims to classify text into a predefined label hierarchy. htc currently faces two fundamental challenges: utilizing structural information and mitigating label imbalance. as shown in figure 1 , recent research can be categorized into global and unit-based approaches based on exploiting feature representations combined with text and structural information. the global approach, hiagm @xcite , hi-match @xcite , hgclr @xcite , k-htc @xcite , hitin @xcite , and hjcl @xcite , generates a holistic feature representation of text that encompasses an entire hierarchy and use it to compute label scores comprehensively. in contrast, the unit-based approach, hpt @xcite and hidec @xcite , generates feature representations of text at the unit level, where a unit refers to a subset of a hierarchy partitioned by specific strategies, and classification is performed on labels within these units. recently, the unit-based approach has achieved significant improvements over the global approach. however, there are two significant limitations in existing research: static thresholding and label imbalance. static thresholding is problematic because most htc models utilizing binary cross entropy (bce) loss predict positive labels using a fixed threshold, typically set at 0.5, when the output probability exceeds this threshold. determining optimal thresholds for target labels is computationally intensive, particularly when considering various units. label imbalance can lead to overtraining on frequent labels and undertraining on infrequent ones and degrade performance due to the dominance of numerous high-confidence labels in a loss. previous htc methods address this issue by leveraging hierarchical structures through the exploitation of auxiliary loss functions @xcite @xcite or per-unit classification @xcite @xcite @xcite . recently, hpt and hjcl presented zero-bounded log-sum-exp pairwise rank-based (zlpr) loss @xcite , but the static thresholding problem still remains. to tackle these limitations, this paper introduces a novel hierarchy-aware loss function for unit-based htc models: hierarchy-aware biased bound margin (hbm) loss. our key innovations in hbm are summarized as follows. first, we introduce learnable bounds for all units within a hierarchy to address the static thresholding problem. these bounds are optimized for various units during training and serve as dynamic unit thresholds during inference. second, we introduce biases and a margin to mitigate the label imbalance. the biases promote low-confidence labels by adjusting the bounds of positive and negative label sets. these adjustments are dynamically determined based on the label logits and amplify the importance of undertrained labels. in contrast, the margin aims to diminish the domination of overtrained labels in a loss by excluding high-confidence labels. through a series of experiments, we demonstrate the effectiveness of our loss function applied to recent unit-based htc models, hpt and hidec, using three benchmark datasets: rcv1-v2 @xcite , nyt @xcite , and eu-rlex57k @xcite . notably, our loss function outperforms competitive htc models on all three benchmark datasets. we comprehensively analyze how the bounds address the issue of static thresholding, and how the biases and margin address the problem of label imbalance in htc. our contributions are summarized as follows: ## related work recent htc research based on deep learning can be categorized into global and unit-based approaches, each with its unique way of creating feature representations that incorporate both text and hierarchy structure. the unit-based approach generates feature representations at the unit level by partitioning the entire hierarchy into units using specific strategies. each unit corresponds to a subset of labels within a hierarchy. various models employ diverse unit construction strategies, including "for-eachclass" @xcite , "for-each-parent" @xcite , "for-eachlevel" @xcite , and "for-each-sub-hierarchy" @xcite . hdltex @xcite introduces htc models using dnn, cnn, and rnn architectures. htrans @xcite enhances hdltex by employing transfer learning to preserve path information. hr-dgcnn @xcite utilizes recursive hierarchical segmentation to divide a hierarchy into sub-hierarchies and construct local unit models. however, the unit-based approach often suffers from a lack of hierarchical information. in contrast, the global approach generates a holistic feature representation encompassing the entire label hierarchy. hiagm @xcite ) merges text and structural representations through text propagation, while hgclr @xcite propagates structural representation through a text encoder and employs contrastive learning. hi-match @xcite applies a hierarchyaware matching loss to hiagm and adjusts feature representations based on hierarchy information. k-htc @xcite tries to incorporate a knowledge graph into htc using knowledgeaware hierarchical label attention and contrastive learning. hitin @xcite reduces the complexity of the existing global models by reconstructing a hierarchy to minimize structural entropy. hjcl @xcite proposes a unified loss function integrating instance and label-wise contrastive learning losses, along with zlpr loss @xcite . the global models effectively leverage hierarchical information through structure encoders in bce, a loss is computed for each label and then averaged. in zlpr and hbm, a loss is calculated separately for positive and negative target sets and combined. the green and red lines are positive and negative biased bounds, respectively, during training. in hbm, a bound is optimized for each unit and used as dynamic thresholds during inference. the yellow lines represent a margin. labels exceeding the margin are excluded in computing a loss. @xcite , outperforming unit-based models. despite their achievements, they face challenges of label imbalances and hierarchy-dependent model parameters. to address these challenges, hpt @xcite and hidec @xcite incorporate a structure encoder @xcite and attention mechanism @xcite into their unit-based htc models. hidec utilizes an encoder-decoder architecture to generate a subhierarchy sequence based on the target labels of each document using a parent-level unit construction strategy. by dividing a hierarchy based on levels, hpt integrates level-specific feature representations from a structure encoder into a text encoder and proceeds with unit-wise prediction. furthermore, hpt incorporates zlpr loss by introducing a zero-bound to mlce loss @xcite . with zlpr loss, hpt transforms htc into a multi-label masked language modeling task. however, these methods still encounter label imbalance in large-scale hierarchies and suffer from static thresholding. ## conclusion this paper introduces a hierarchy-aware biased bound margin (hbm) loss function, offering two key innovations to address the challenges of static thresholding and label imbalance in htc. first, hbm introduces learnable bounds for all units within a hierarchy to address static thresholding. these bounds are optimized for various units during training and are used as dynamic thresholds during inference. second, hbm introduces biases and a margin to mitigate label imbalance. the biases promote low-confidence label training, while the margin excludes high-confidence labels from the loss. third, extensive experiments on benchmark htc datasets demonstrate the superiority of hbm loss based on unit-based htc models by comparing competitive htc models and comprehensive analysis. we plan to extend hbm to extremely large-scale hierarchies and improve imbalance relations among units.