Label Augmentation for Zero-Shot Hierarchical Text Classification (2024)

Abstract: Hierarchical Text Classification poses the difficult challenge of classifying documents into multiple labels organized in a hierarchy. The vast majority of works aimed to address this problem relies on supervised methods which are difficult to implement due to the scarcity of labeled data in many real world applications. This paper focuses on strict Zero-Shot Classification, the setting in which the system lacks both labeled instances and training data.We propose a novel approach that uses a Large Language Model to augment the deepest layer of the labels hierarchy in order to enhance its specificity. We achieve this by generating semantically relevant labels as children connected to the existing branches, creating a deeper taxonomy that better overlaps with the input texts. We leverage the enriched hierarchy to perform Zero-Shot Hierarchical Classification by using the Upward score Propagation technique. We test our method on four public datasets, obtaining new state-of-the art results on three of them. We introduce two cosine similarity-based metrics to quantify the density and granularity of a label taxonomy and we show a strong correlation between the metric values and the classification performance of our method on the datasets.

## introduction hierarchical text classification (htc) @xcite is a machine learning problem that consists in classifying documents into multiple labels which are organized in the form of a hierarchical taxonomy. in recent times, this problem has increasingly gathered the interest of both academia and industry due to its relevance in realistic scenarios @xcite . in fact, real-world challenges such as the organization of products in e-commerce categories or the classification of documents such as papers or news in a hierarchical structure can be tackled by htc @xcite . the greatest difficulty found in this practice is the lack of labelled data and the cost, especially in an industrial framework, of manually annotating data samples. moreover, the structure of a hierarchy can change in time and gain or lose classes, which, potentially, can result in additional costs necessary to reorganize existing data and retrain models. for these reasons, researchers have turned their attention to few-shot @xcite and zero-shot classification (zsc) @xcite settings in which only few or no annotated documents are given at training time. in this paper we will focus on strict zsc: a highly constrained scenario where not even the unlabeled training instances are given to the system. in the context of textual classification the standard approach used to tackle a strict zsc problem is to transform it into a textual entailment task solved by a llm such as as bart-mnli @xcite . in this approach, the llm is asked to determine if a premise sentence (the text to be classified) entails semantically a hypothesis sentence (the class to be predicted). even without fine-tuning, these models are able to classify documents into unseen classes with a high degree of success. another approach to the zsc task is to use labels embeddings as prototypes or centroids for a 1-nearest neighbor classification problem @xcite . input texts are vectorized in the same embedding space as the labels and the corresponding class is determined via some distance or similarity metric such as the cosine similarity. while this is a natural approach, it has been criticized @xcite on the basis that it looks for similarities between few or single words and long and complex texts. furthermore, documents with a very high level of detail may pose a challenge for models to accurately classify them. for instance, a product review categorized under "strollers" might solely discuss the instability experienced when using the three wheels on sidewalks, making it challenging for the model to identify the appropriate label. while there exists a growing body of research focusing on zsc, only few works deal with hierarchical data. a recent work @xcite proposes a zero shot htc (zs htc) model that exploits the labels' taxonomy to improve classification results. the authors compute the similarity between the text to be classified and all the labels in the hierarchy, then they define a technique called upward score propagation (up) that propagates similarity scores upward in the hierarchy and exploits the propagated information to improve the classification of the upper levels of the hierarchy. although this technique takes advantage of the taxonomy, it cannot improve the classification results for the deepest level of labels (i.e., the leaves of the hierarchy) which, we argue, are often the most important to be classified correctly in practical applications. this paper introduces hierarchical label augmentation (hila), a novel technique aimed at enhancing a provided label hierarchy by leveraging a large language model (llm) to introduce meaningful branches to the existing taxonomy. namely, we augment the deepest layer of the hierarchy by generating terms that are connected as children of the existing leaves (see figure 1 ). the idea behind this process is to augment the taxonomy specificity so that the new layer of labels gets semantically closer to input texts. we then apply up as in @xcite to the deepened hierarchy to perform zero-shot hierarchical classification. we test our method on four public datasets and obtain new state-of-the-art results for three of them. moreover, we define a set of cosine similarity-based metrics to quantify the granularity of a taxonomy of labels. we conjecture that the accuracy of our ap-proach highly depends on how granular the leaves of the taxonomy are. indeed we show that the metrics values for the four datasets taxonomies are strongly correlated with the results of our method. empirical results show that the metrics can be used as a prior test to measure the goodness of a label hierarchy and to check if our proposed method is going to improve the final classification results. the main contributions of our paper are: the rest of the paper is organized as follows: in section 2 we point out relevant related works. in section 3 we summarize the up procedure, we present our novel method and two metrics to examine taxonomy structures. we then comment the experiments we performed and their results in sections 4 and 5. finally, we draw brief conclusions in section 6. ## related works many past works have studied the problem of htc @xcite and have found solutions based on machine learning models such as decision trees @xcite or support vector machines @xcite . since the advent of transformers @xcite more recent studies approached the problem using advanced deep learning (dl) techniques and language models (lm). in @xcite the authors take into account the hierarchical structure of the taxonomy by training a different deep neural network on each node of the taxonomy. in this way, they employs stacks of deep architectures to provide specialized understanding at each level of the document hierarchy. in @xcite , the researchers develop a dl methodology to capture both local and global information across various levels of the taxonomy. they first learn representations for both the document and the taxonomy and then employ an attention mechanism to model dependencies in a top-down manner. finally, a classifier is used to decide whether a document merits labeling with a specific node. all the mentioned works leverage the hierarchical structure of the labels but specifically rely on labeled data. the challenge of zero-shot classification has been the focus of attention in recent years and has produced many works proposing appealing solutions. in @xcite the authors address the zsc problem with a self-training based approach. they first compute the similarity of a document with all the labels. secondly, they select the highest scoring documents and confidently treat them as labeled data. they use then the self-labeled documents as data to fine-tune a lm. @xcite ; @xcite ; @xcite ; @xcite propose to deal with zsc as a textual entailment problem. they convert the labels into the hypothesis @xmath0. all the cited methods either do not address zsc in a strict sense or are not able to leverage the labels taxonomy structure. when handling contexts with limited data it is a common practice to resort to data augmentation (da) strategies @xcite . several successful techniques have been presented in this field, most of them perform da in the feature space of vectorized texts @xcite or at text level via generative models @xcite . all these methods are applicable in a few shot context only. in the domain of few or zsc, it is a common strategy to employ the class taxonomy to mitigate the absence of data. this approach has been widely adopted across various fields, including computer vision @xcite but, to the best of our knowledge, there are no works trying to perform data augmentation over the label hierarchy. a work strongly aligned with ours, and which we deeply rely on, is @xcite , where zs htc is performed taking advantage of hierarchical information. the authors present a new methodology for text classification based on a custom hierarchical taxonomy, achieved without relying on labeled data. their approach initially involves leveraging semantic information rooted inside pre-trained deep language models to assign a preliminary relevance score to each label of the taxonomy through zero-shot techniques. next, they leverage the hierarchical structure to reinforce the initial scores, thereby improving the overall classification process. while they do not update the relevance scores for the last level of the taxonomy, our work is strongly directed towards improving them. ## method in this section, we provide an overview of the up method as outlined in @xcite . subsequently, we present our proposed label augmentation method designed to enhance a given label hierarchy by adding an additional level of labels. afterwards, we illustrate the up application to the novel hierarchy generated by our approach. finally, we introduce a set of metrics aimed at quantifying the granularity of label hierarchies. we will use a notation largely inspired by @xcite @xcite , extending and customizing it to align with the specific requirements of our study. specifically, we will use an upward arrow to denote the parent of a node in the hierarchy, for instance ↑ c l j is the parent node of the j-th label at level l of the hierarchy. a fat upward arrow will denote the set of all ancestors of a given node (as in, e.g., ⇑ c l j ). similarly, we will use a downward arrow to denote the set of children of a given category (e.g., ↓ c l j ) and a fat downward arrow to denote the set of all descendants categories of a given category (e.g., ⇓ c l j ). a summary of the notation is presented in ## conclusions in this paper we proposed an llm-based label augmentation technique to deepen a given hierarchy of labels. we also defined a set of metrics to measure the granularity of a taxonomy. by applying our method to four public hierarchical datasets, we obtained new sets of coherent and meaningful labels. we then used the deepened hierarchies to perform zero-shot hierarchical tc using the up technique. we obtained sota results on three out of four datasets. the classification results are strongly correlated with the metric values, that can therefore be used to study the behaviour of the hila approach.