A Simple and Effective Framework for Strict Zero-Shot Hierarchical Classification (2023)

Abstract: In recent years, large language models (LLMs) have achieved strong performance on benchmark tasks, especially in zero or few-shot settings. However, these benchmarks often do not adequately address the challenges posed in the real-world, such as that of hierarchical classification. In order to address this challenge, we propose refactoring conventional tasks on hierarchical datasets into a more indicative long-tail prediction task. We observe LLMs are more prone to failure in these cases. To address these limitations, we propose the use of entailment-contradiction prediction in conjunction with LLMs, which allows for strong performance in a strict zero-shot setting. Importantly, our method does not require any parameter updates, a resource-intensive process and achieves strong performance across multiple datasets.

## introduction large language models (llms) with parameters in the order of billions @xcite have gained significant attention in recent years due to their strong performance on a wide range of natural language processing tasks. these models have achieved impressive results on benchmarks @xcite , particularly in zero or few-shot settings, where they are able to generalize to new tasks and languages with little to no training data. there is, however a difficulty in tuning parameters of these large-scale models due to resource limitations. additionally, the focus on benchmarks has led to the neglect of real-world challenges, such as that of hierarchical classification. as a result, the long-tail problem @xcite has been overlooked. this occurs when a vast number of rare classes occur in the presence of frequent classes for many natural language problems. in many industrial real-world applications, a strong performing method for hierarchical classification can be of direct utility. new product categories are emerging in e-commerce platforms. existing categories, on the other hand, may not be very intuitive for customers. for example, upon browsing categories such as night creams, we may be unable to find a product in a sibling-node category of creams. this is further highlighted by platforms in which a systematic structure is not created for users; parent nodes may be in place of child nodes, and vice versa @xcite . manually categorizing product categories can be a costly redesigning endeavour. to tackle this problem, we suggest refactoring traditional hierarchical flatlabeled prediction tasks @xcite to a more indicative long-tail prediction task. this involves structuring the classification task to closely reflect the real-world long-tail distribution of classes. in doing so, we are enabled to leverage llms for long-tail prediction tasks in a strict zero-shot classification setting. through a series of experiments, results in this work show that our proposed method is able to significantly improve the performance over the baseline in several datasets, and holds promise for addressing the long-tail problem in real-world applications. the contributions of this work can be summarized as follows: our method obtains strong performance on longtail classification tasks. ## methodology in this paper, we investigate the limitations of llms in three overlooked settings, when-(i) the model is not provided with sufficient examples due to the input text length, (ii) the label space includes tokens largely unobserved in the model's pretrained vocabulary, and (iii) there are a large number of distractors in the label space @xcite @xcite . these scenarios are common in real-world tasks, but are often overlooked in the development and evaluation of llms. to address these challenges, we propose the use of entailment-contradiction prediction @xcite , the task of determining whether a premise logically entails or contradicts a hypothesis. through our method, we are able to leverage strong reasoning from @xcite with the retrieval abilities of llms @xcite to improve overall performance in a strict zero-shot setting, where the model must classify samples from a new task without any fine-tuning or additional examples used for training from the same distribution as the inference dataset. importantly, our proposed combined method does not require parameter updates to the llm, a resource-intensive process that is not always feasible with increasingly large model size @xcite . our simple framework is shown in figure 1 . considering the label space, @xmath0, c 2 , ...c n } as the set of classes for any given dataset, and a text input, x, we can utilize the entailment predictor, e to make a contradiction, or entailment prediction on each label. this is done by using x as the premise, and "this text is related to c i ." by combining these two components, we can create a template which utilizes the entailment relation explicitly, and the contradiction relation implicitly by constructing l(e(x)) to deseminate combined information into an entailment predictor for classification. the template we use is task-dependent, and is generally robust given an understanding of the domain. on web of sciences we use: "here is some text that entails e(x): x. what area is this text related to?". for amazon beauty, we use "here is a review that entails e(x): x. what product category is this review related to?". in this setting, our method still meets a barrier due to limitations of llms. by constructing a composite function, e(l(e(x)), we are able to leverage our llm in producing tokens which may steer the entailment predictor to correct its prediction. the template used for this composite function is "here is some text that entails l(e(x)): x." across all datasets. . our observations show this setting having comparable, or marginal improvements with our dataset. however, this may prove to be beneficial in other tasks. we will investigate, and urge other researchers to explore this direction in future work. ## conclusion in this work, we utilize an llm in the form of a noisy knowledge graph to enhance the capabilties of an entailment predictor. in doing so, we achieve strong performance in a strict zero-shot setting on several hierarchical prediction tasks. we also show the necessity of refactoring existing hierarchical tasks into long-tail problems that may be more representative of the underlying task itself. the utility in practical industry settings is also highlighted through this setting.