Retrieval-style In-context Learning for Few-shot Hierarchical Text Classification (2024)

Abstract: Hierarchical text classification (HTC) is an important task with broad applications, and few-shot HTC has gained increasing interest recently. While in-context learning (ICL) with large language models (LLMs) has achieved significant success in few-shot learning, it is not as effective for HTC because of the expansive hierarchical label sets and extremely ambiguous labels. In this work, we introduce the first ICL-based framework with LLM for few-shot HTC. We exploit a retrieval database to identify relevant demonstrations, and an iterative policy to manage multi-layer hierarchical labels. Particularly, we equip the retrieval database with HTC label-aware representations for the input texts, which is achieved by continual training on a pretrained language model with masked language modeling (MLM), layer-wise classification (CLS, specifically for HTC), and a novel divergent contrastive learning (DCL, mainly for adjacent semantically similar labels) objective. Experimental results on three benchmark datasets demonstrate superior performance of our method, and we can achieve state-of-the-art results in few-shot HTC.

## introduction hierarchical text classification (htc), a specialized branch of multi-label text classification, involves the systematic arrangement and categorization of textual data throughout a tiered label structure. the output labels are organized in a parent-child hierarchy, with the higher-level labels encompassing broader concepts, and the child labels delineating more specific subtopics or attributes. in recent years, htc has gained significant attention, due to its applicability across a variety of fields, including recommendation systems @xcite , document categorization @xcite , and information retrieval @xcite . in standard supervised htc, there is an underlying assumption of abundant training samples @xcite @xcite , which is often unattainable and expensive to construct manually. moreover, htc datasets are characterized by a complex hierarchical label structure, with leaf labels typically following a zipfian distribution, resulting in very few data instances for these labels. as a result, the few-shot setting is more realistic, and has gained increasing interest recently @xcite @xcite . nevertheless, existing works often struggle with unsatisfactory performance in this setting. for example, bert with the vanilla fine-tuning strategy performs extremely poorly in few-shot htc. recently, large language models (llms) have achieved notable success on various nlp tasks @xcite @xcite , which have significantly enhanced the efficacy of in-context learning (icl) with relevant demonstrations in the few-shot setting @xcite @xcite . however, the application of icl on htc faces unique challenges, diverging from traditional text classification scenarios. these challenges are primarily due to two distinct characteristics of htc, as delineated in figure 1 . firstly, htc features a deep hierarchical labeling structure and expansive label sets, resulting in large label sets in icl, which adversely impacts its performance. secondly, as the hierarchy deepens, the semantic similarity between adjacent labels increases @xcite , making it very challenging to select relevant demonstrations that guide the learning process efficiently. in this work, we introduce the first icl-based framework for few-shot htc. specifically, we use a llm as the foundation model for inference, and provide demonstrations to guide htc label generation through icl. our success depends on finding suitable demonstrations for a given input. in order to achieve this, we build a retrieval database that can find the most-relevant demonstrations for the input. further, in order to avoid providing an enormous set of multi-layer contextual htc labels all at once, as is required for icl, we suggest an iterative policy to infer the labels layer-by-layer, reducing the number of candidate labels greatly. the quality of our retrieval database is highly critical. the key idea is to obtain the htc label-aware representations for input texts, which are then used for subsequent retrieving. given an input, we define prompt templates for each-layer htc, concatenating them with the raw input to form a new text. the hidden vectors of prompts are exploited as label-aware representations. we perform continual training to learn our representations using three types of objectives: the masked language model (mlm), the multi-label classification (mlc) for htc particularly and a well-designed divergent contrastive learning (dcl) objective. the dcl is especially useful for the semantically closed htc labels (e.g., adjacent labels from the same parent). in addition, we can incorporate label descriptions naturally by dcl, which can benefit icl-based htc much. we conduct experiments on two classic english hierarchical classification datasets, wos @xcite and dbpedia @xcite , as well as a chinese patent dataset, foot_0 and measure model performance by both micro-f1 and macro-f1 metrics consistent with previous work. results show that our method is highly effective, giving improved results compared with a series of baselines for the few-shot htc with different shots. we can also achieve state-of-the-art (sota) results on these htc datasets. further, we perform thorough qualitative analysis in order to gain a comprehensive understanding of our methodology. in summary, our contributions can be summarized as follows: our code is publicly available at https:// github.com/dreamh1gh/tacl2024 to facilitate future research. ## related work htc is initially presented by @xcite , and neural network models have achieved great advances in this task. previous approaches treat htc as multi-label classification and adopt traditional text classification methods for htc problems @xcite . the majority of recent htc studies pay the focus on finding ways to insert the hierarchical label knowledge into the model @xcite @xcite . several studies attempt to solve htc problems by modeling the hierarchical labels as a graph or a tree structure @xcite while other researchers try to apply meta-learning @xcite or reinforcement learning @xcite to leverage htc label structure. however, existing methods mainly concentrate on encoding the holistic label structure, ignoring the classification of nuanced long-tail terminal labels. there have been efforts prove that retrieval augmented methods could help classification task with only few-shot samples @xcite @xcite , which could be a solution of the long-tail challenge. given this insight, we make an attempt to convert the htc task to a retrieval form. moreover, with the development of llms, recent work explores solutions that tackle traditional nlp tasks with the icl paradigm and achieve surprising effectiveness @xcite @xcite . but icl strongly relies on demonstration selection @xcite @xcite @xcite . many studies explore adjusting demonstrations for better performance through instruction formatting @xcite , example ordering @xcite , and demonstration filtering @xcite . in our work, we combine the icl-based framework with retrieval for htc, selecting demonstrations that involve both the language knowledge of llm and advantages of retrieval. proposed framework figure 2 illustrates our icl-based framework for htc. we first train a plm-based indexer and build a retrieval database containing reference samples (the training data). after that, we perform a similarity search in the retrieval database with the text to be inferred. finally, we construct an icl prompt with highly similar demonstrations for prediction. we introduce our icl prompt policy for htc ( § 3.1), and then detail the retrieval database construction ( § 3.2) and demonstration retrieval methods ( § 3.3). ## conclusion in this paper, we proposed a retrieval-style icl framework for few-shot htc. we uniquely identify the most relevant demonstrations from a retrieval database to improve icl performance and meanwhile designed an iterative policy to inference hierarchical labels sequentially, significantly reducing the number of candidate labels. the retrieval database is achieved by using a htc label-aware representation for any given input, enabling the differentiation of semantically closed labels (especially the leaf adjacent labels). the representation learning is implemented by continual training on a plm with three carefully designed objectives including mlm, layer-wise classification, and a novel dcl objective. we conducted experiments on three benchmark datasets to evaluate our method. the results show that our method is highly effective, which is able to gain large improvements among a serious of baselines. finally, our method can bring the state-of-the-art results in few-shot htc on the three datasets. further, we performed comprehensive analysis for deep understanding of our method, spreading various important factors. this work still includes several unresolved problems, which might be addressed in the future. first, llms are currently confined to expanding text via label descriptions and their application to full training set expansion has not been effective. in order to fully utilize llms in text expansion, we need further optimization. second, the performance gap between supervised methods and our icl-based approach appears to diminish with increasing training dataset size, suggesting the need for further analysis.