X -Shot: A Unified System to Handle Frequent, Few-shot and Zero-shot Learning Simultaneously in Classification (2024)

Abstract: In recent years, few-shot and zero-shot learning, which learn to predict labels with limited annotated instances, have garnered significant attention. Traditional approaches often treat frequent-shot (freq-shot; labels with abundant instances), few-shot, and zero-shot learning as distinct challenges, optimizing systems for just one of these scenarios. Yet, in real-world settings, label occurrences vary greatly. Some of them might appear thousands of times, while others might only appear sporadically or not at all. For practical deployment, it is crucial that a system can adapt to any label occurrence. We introduce a novel classification challenge: **X-shot**, reflecting a real-world context where freq-shot, few-shot, and zero-shot labels co-occur without predefined limits. Here, **X** can span from 0 to positive infinity. The crux of **X-shot** centers on open-domain generalization and devising a system versatile enough to manage various label scenarios. To solve **X-shot**, we propose **BinBin** (**B**inary **IN**ference **B**ased on **IN**struction following) that leverages the Indirect Supervision from a large collection of NLP tasks via instruction following, bolstered by Weak Supervision provided by large language models. **BinBin** surpasses previous state-of-the-art techniques on three benchmark datasets across multiple domains. To our knowledge, this is the first work addressing **X-shot** learning, where **X** remains variable.

## introduction for classification problems, the distribution of label occurrences in real-world scenarios often varies widely, with some labels appearing frequently (frequent-shot), others infrequently (few-shot), and some not at all (zero-shot). given this variability, it becomes imperative to craft learning systems adept at managing labels across the full frequency spectrum. regrettably, current few-shot systems often fall short when confronted with zero-shot challenges @xcite @xcite . in contrast, zero-shot systems, while adept in their domain, cannot fully benefit from the potential advantages of annotations when available @xcite @xcite . thus, developing the skill to manage all possible label occurrences simultaneously is crucial for systems that are intended for practical use. in this work, we introduce a more challenging and practically useful task: x-shot learning. this task mirrors real-world environments where label occurrence spans a continuum, seamlessly incorporating frequent-shot, few-shot, and zero-shot instances, all without a priori constraints. in this paradigm, variable x, the number of times each label is seen during the training, is unbounded, ranging freely within the interval [0, +∞). at the heart of x-shot lies the objective of attaining open-domain generalization and architecting a system resilient across a plethora of label scenarios. tackling x-shot spawns two core technical conundrums: (q 1 ) how can one identify suitable sources of indirect supervision @xcite in few-shot and zero-shot settings, given the notable scarcity of annotations. (q 2 ) traditional multiclass classifiers struggle with the diversity in label sizes across tasks, frequently requiring customized classification heads for each variation. here, the challenge is formulating a cohesive system capable of effectively adapting to labels of diverse sizes. to address q 1 , we identify the most effective source of indirect supervision as being from instruction tuning datasets, such as super-naturalinstruction @xcite . these datasets primarily contain various nlp tasks enriched with textual instructions. our method trains the model on these datasets, aiming for robust generalization to the unseen x-shot task when supple-mented with pertinent instructions, especially for the low-shot (few-shot and zero-shot) labels. for q 2 , we advocate a triplet-oriented binary classifier. this classifier functions by accepting a triplet of (instruction, input, label), anticipating a binary response ("yes" or "no") that confirms the suitability of the label for the specified input under the given instruction. such a triplet-oriented classifier acts as a cohesive architecture that manages text classification tasks with labels of varied sizes. by combining solutions for both q 1 and q 2 , we forge a holistic framework, binbin (binary inference based on instruction following). there are, however, no existing datasets that explicitly cater to this challenge. to evaluate our system, we turn to three representative classification tasks: relation classification, event detection, and argument role identification. we recompile their associated datasets: fewrel @xcite , maven @xcite , and rams @xcite and simultaneously include frequentshot, few-shot, and zero-shot instances. sourced from diverse domains (wikipedia, news articles, etc.), and featuring vast label counts (ranging from 30 to 78), these datasets pose a formidable challenge to contemporary text classification systems. moreover, the maven dataset uniquely integrates a "none" label, further amplifying the realistic nature of the task. experiments on multiple model scales and architectures reveal our system's resilience across datasets, consistently outperforming leading baselines, including gpt-3.5. our contributions can be summarized as follows: (i) we introduce x-shot, a hitherto underexplored, open-domain open-shot text classification problem that mirrors real-world complexities. (ii) we innovate a unique problem setting that reframes any text classification challenge into a binary classification task, adaptable to any number of label sizes and occurrences. (iii) our binbin, harnessing the potential of instruction-following datasets, excels past existing approaches, demonstrating versatility across various domains, label magnitudes, and classification paradigms. indirect supervision there has been a burgeoning interest in indirect supervision @xcite in recent years. here, easily available signals from relevant tasks (source tasks) are used to aid in learning the target task. using the entailment task for indirect supervision in zero-shot classification was first proposed by @xcite and has since been adapted for a variety of nlp tasks, including few-shot intent identification @xcite , event argument extraction @xcite and relation extraction @xcite . beyond entailment, knowledge from areas like question answering @xcite , summarization @xcite and dense retrievers @xcite has been incorporated. however, previous indirect supervision is collected from a single source task. in contrast, our work is inspired by recent studies in instruction learning observing the efficacy of nlp models when given task instructions and their ability to generalize knowledge across tasks @xcite @xcite . their approach cannot be adapted to diverse tasks, and therefore, is less flexible than ours. most importantly, this system benchmarks its performance against generative models, rather than comparing it with state-of-the-art (sota) systems specifically designed for target classification tasks. ## problem statement each x-shot target task has the following components: the task of x-shot is to figure out label l @xmath0, where |l s | might be zero (i.e., "none"). research questions of x-shot: i) given that the above formulation encompasses various text classification problems, how can we move away from constructing individual models for each problem, and instead develop a single classifier adept at handling diverse classification settings? ii) beyond frequently-encountered labels, low-shot labels necessitate additional supervision for effective reasoning. where can we find such supervision? in the following section, we delve deeper into our approach concerning the universal system and the provided supervisions. ## methodology this section first explains how binbin adapts to different classification problems, then introduces the supervision to train it. ## conclusion this work introduces x-shot, a text classification setting characterized by diverse label occurrences: freq-shot, few-shot, and zero-shot. our approach, binbin, leverages indirect supervision and llms' weak supervision to consistently outperform stateof-the-art methods across three benchmark datasets in various domains.