H i G en: Hierarchy-Aware Sequence Generation for Hierarchical Text Classification (2024)

Abstract: Hierarchical text classification (HTC) is a complex subtask under multi-label text classification, characterized by a hierarchical label taxonomy and data imbalance. The best-performing models aim to learn a static representation by combining document and hierarchical label information. However, the relevance of document sections can vary based on the hierarchy level, necessitating a dynamic document representation. To address this, we propose HiGen, a text-generation-based framework utilizing language models to encode dynamic text representations. We introduce a level-guided loss function to capture the relationship between text and label name semantics. Our approach incorporates a task-specific pretraining strategy, adapting the language model to in-domain knowledge and significantly enhancing performance for classes with limited examples. Furthermore, we present a new and valuable dataset called ENZYME, designed for HTC, which comprises articles from PubMed with the goal of predicting Enzyme Commission (EC) numbers. Through extensive experiments on the ENZYME dataset and the widely recognized WOS and NYT datasets, our methodology demonstrates superior performance, surpassing existing approaches while efficiently handling data and mitigating class imbalance. We release our code and dataset here: https://github.com/viditjain99/HiGen.

## introduction hierarchical text classification (htc) is a task that involves categorizing text data into predefined categories organized in a hierarchical structure @xcite @xcite . it holds great importance in various text mining applications, including scientific paper recommendation @xcite ), * equal contribution † work done by the author while at georgia institute of technology semantic indexing @xcite , and online advertising @xcite . htc poses unique challenges when compared to traditional text classification, as it deals with imbalanced data distributions and complex dependencies between multiple levels of categories within the hierarchy. the hierarchical structure is represented as a directed acyclic graph @xcite , which must be encoded in the predictive model along with the text to generate the final hierarchal label. however, the data imbalance becomes more pronounced as we move down the levels, presenting a significant challenge for htc. existing methods for hierarchical text classification (htc) can be categorized into three groups: global @xcite , local @xcite @xcite and generative @xcite @xcite . local approaches predict each hierarchical level using independent classifiers. global models, on the other hand, use a single classifier and incorporate hierarchical information into the loss function. finally, generative approaches use text generation frameworks to model the hierarchical structure. some prior works flatten the label structure, leading to an exponential increase in the number of classes and the loss of hierarchical dependency. additionally, some models fail to capture the correlation between text and label name semantics. as a result, these approaches struggle to perform well on long-tailed classes with limited training data. harnessing the power of large pretrained language models (plms) to capture text-label correlation, we employ a transformer-based sequence-tosequence (seq2seq) framework @xcite . these models are pretrained on extensive text data, enabling them to encode transferable linguistic features across tasks @xcite . our model transforms htc into a text generation problem, generating labels conditioned on the input text and previously generated labels while utilizing a hier-archical level-guided semantics framework. while @xcite introduces a matching loss for hierarchy-aware text-label relationships, it neglects the level-dependency between the transformed text and label representations. our approach incorporates a margin loss, aligning text semantics with positive labels at each taxonomic level while pushing negative semantics apart. furthermore, we propose a token constraint loss during training to discourage undesired token generation. inspired by the success of pretraining tasks @xcite , we propose a task-specific pretraining strategy for our generative model. our approach presents a notably simplified design in contrast to previous works which use complex encoder architectures or training paradigms @xcite @xcite . we propose a pretraining step that takes advantage of weak supervision to jointly model text representations and hierarchical label information. the encoder encodes the input document and masked hierarchical label, while the decoder regenerates the original label. the pretraining step leverages weakly labeled data from the same domain, generated using an llm. this domain-specific knowledge, discussed in @xcite ; @xcite , proves advantageous for addressing the data imbalance challenge commonly encountered in htc datasets. the major contributions of this work are: ## related work htc is a multi-label classification problem where the classification task is performed in a hierarchical manner. most prior work in htc can be categorized into three broad categories: local, global, and generative. in the local approach, each hierarchical level has its own classifier tailored to the unique classes at that level. conversely, the global approach employs a unified classifier that encompasses all classes across every hierarchy level. the generative approach, a recent advancement in hierarchical text classification, capitalizes on the generative capabilities of language models to predict text labels. more details on each of the categories are explained below. local htc local approaches in htc use local classifiers at each level or class @xcite @xcite . @xcite al. (2019) global htc global approaches to htc utilize a single classifier @xcite @xcite to predict labels at different hierarchy levels. early works consider parent-child dependencies @xcite @xcite , while recent approaches focus on global label structure @xcite , disjoint features @xcite , label imbalance @xcite , prior hierarchy knowledge @xcite , and semantic matching @xcite . these works learn text and hierarchy semantics separately, fusing them later. however, @xcite proposes a global approach using contrastive learning to learn a shared representation. similarly, @xcite also aims for common representations but incorporates both local and global hierarchies. lastly, @xcite uses prompt-tuning and multilabel mlm to learn shared semantics. generative htc early works on text generation for htc @xcite use rnn and transformer-based seq2seq models, with dynamic document representations outperforming static encoder methods. recent approaches @xcite propose t5-based models @xcite . the former addresses label inconsistency with dfs-based linearization and constrained decoding, while the latter captures dependencies using bfs-based linearization and hierarchical path-based attention. these methods overlook the dependencies between the label name from different levels and the document text. we overcome this by capturing the document-labelname dependency across different levels using the proposed level-guided semantic loss. in this work, we explore htc under the umbrella of a sequence generation framework using a bart-based model @xcite . we harness the benefits of denoising autoencoder pretraining to imbue the model with a strong understanding of the hierarchical structure. we also employ a well-designed objective function during supervised training for improved learning. ## problem definition given an input text x @xmath0, x 2 , ..., x n }, htc aims to classify the text into a subset y i of label set y . the label set y is arranged as a directed acyclic graph (dag), denoted by @xmath1, e). l represents the set of nodes and e represents the edges indicating the nodes' parent-child relations. all the labels in y constitute the nodes in the above graph i.e. l. we use bfs @xcite to flatten the hierarchical labels into a multi-level sequential string of label nodes. we define a special set of symbols @xmath2, <@xmath3. to classify x i , the proposed sequence generation model generates y @xmath4, y 2 , ...., y k }, where y @xmath5. bfsbased linearization helps each y i to correspond to one or more paths in root to leaf paths in h. ## methodology this section presents the proposed generative framework based on the bart model for htc. we first describe the pretraining regime aimed at learning robust joint text-label representations (section 4.1). following this, supervised training is performed to generate hierarchical label sequences for input documents (section 4.2). we also explain the proposed objective function used (section 4.3) for this phase that aims to capture text and label semantics jointly. ## conclusion in this paper, we propose a sequence generation framework for htc that captures the hierarchical nature of labels. the pretraining strategy enhances the model's performance by adapting it to the specific domain and task. moreover, we demonstrate the effectiveness of utilizing synthetic data generated through powerful language models like chatgpt. the objective function during supervised training provides the model with additional contextual information about the hierarchical structure. our proposed approach outperforms baseline models on three datasets, as measured by standard evaluation metrics. leveraging the knowledge embedded in pretrained language models, our model performs exceptionally well on classes with limited examples and is data-efficient as well.