Semantic matching for text classification with complex class descriptions (2023)

Abstract: Text classifiers are an indispensable tool for machine learning practitioners, but adapting them to new classes is expensive. To reduce the cost of new classes, previous work exploits class descriptions and/or labels from existing classes. However, these approaches leave a gap in the model development cycle as they support either zero- or few-shot learning, but not both. Existing classifiers either do not work on zero-shot problems, or fail to improve much with few-shot labels. Further, prior work is aimed at concise class descriptions, which may be insufficient for complex classes. We overcome these shortcomings by casting text classification as a matching problem, where a model matches examples with relevant class descriptions. This formulation lets us leverage labels and complex class descriptions to perform zero- and few-shot learning on new classes. We compare this approach with numerous baselines on text classification tasks with complex class descriptions and find that it achieves strong zero-shot performance and scales well with few-shot samples, beating strong baselines by 22.48% (average precision) in the 10-shot setting. Furthermore, we extend the popular Model-Agnostic Meta-Learning algorithm to the zero-shot matching setting and show it improves zero-shot performance by 4.29%. Our results show that expressing text classification as a matching problem is a cost-effective way to address new classes. This strategy enables zero-shot learning for cold-start scenarios and few-shot learning so the model can improve until it is capable enough to deploy.

## introduction advances in supervised text classification have enabled the automation of countless classification pipelines, but such methods struggle when new classes are introduced. in the typical setting, labels are collected for a fixed number of classes, and a model is trained to assign examples to their respective classes. however, incorporating new classes into the model incurs substantial costs. the process often involves labeling new examples, relabeling existing examples, and retraining the model. to reduce these costs, we seek a method which rapidly adapts from seen classes to new/unseen ones, using few to no labeled examples. we study a common scenario wherein we have labeled data for a multi-class or multi-label text classification problem and then new classes are introduced. crucially, we assume access to class descriptions. such information is not uncommon in applications with frequent class changes. for example, in product compliance, new classes are often accompanied by documentation describing the products covered by the new regulations. concise descriptions may suffice for simple classes like books, but detailed descriptions are needed for more specific or complicated classes like books from genre 1 or genre 2 or . . . or genre 10. given labeled data and descriptions for existing classes, and only descriptions for new classes, our goal is to build a model to classify examples into new classes with minimal additional labeling, making it deployable in read-world scenarios. we focus on modest sized language models rather than large ones since they more readily scale to the large problems common in real-world applications. this paper addresses three problems in this setting: how to (1) leverage both class descriptions and existing labeled data to generalize to new classes (in zero-and few-shot settings); (2) effectively use long, complex class descriptions; and (3) align training with zero-shot evaluation. first, to generalize from seen classes to unseen ones, an ideal approach should exploit all available data sources and be applicable in both zero-and few-shot settings. intuitively, a model which incorporates more information should be more accurate. labels from existing classes help language models adapt from pre-training objectives to the problem domain, while class descriptions aid in generaliz-matching model @xcite example text x i @xcite [sep] c j section 1 c j section 2 [sep] ing to new classes with fewer labels. flexibility between zero-and few-shot learning is important for ensuring the model works when no or few labels are available. additionally, zero-shot performance is often insufficient to meet real-world performance requirements, making it desirable for the model to significantly improve with more collected labels. existing techniques for zero-and few-shot learning tend to satisfy only some of these desiderata. they either specialize in only zero-or few-shot settings, or they exploit only one of the aforementioned data sources. to address this, we follow yin et al. 2019; @xcite and reformulate multi-class or multi-label text classification as a binary matching problem, where we train a model to match example texts with class descriptions based on semantic similarity. the model assigns each (example text, class description) pair a score, representing the likelihood of the example belonging to the respective class as in figure 1 . we use seen class labels to teach the model to match examples with applicable class descriptions. for the zero-shot setting we simply pass in descriptions for new classes, and for few-shot setting we continue to fine-tune the model on labels and descriptions from new classes. therefore this approach uses (a) class descriptions and existing labels to perform zero-shot classification and (b) additional labels from new classes to accomplish few-shot learning. second, we seek a model which is effective for long, complex class descriptions, but this area is relatively unexplored. previous work @xcite @xcite @xcite @xcite has primarily focused on short inputs such as class names or brief sentences. they work when classes can be concisely summarized in a few words (e.g. bicycle or socks) but may be inadequate when class descriptions are necessarily long or specific (e.g. category i equipment). to fully leverage the in-formation in long, complex class descriptions, we use a cross-encoder architecture with a pre-trained transformer backbone. this approach facilitates rich interaction between example text and class descriptions by ingesting them together as a single concatenated input. third, the matching formulation described above unifies zero-and few-shot learning, but it still exhibits a discrepancy between training and evaluation. matching model training follows the standard supervised learning paradigm: training loss is computed on seen classes and descriptions in the hope that the model will generalize to unseen descriptions. we propose a novel extension of modelagnostic meta-learning (maml) @xcite to the matching setting, specifically targeting zeroshot learning. similarly to how maml was designed to align model training to few-shot evaluation, our extension augments the training process to better mimic zero-shot evaluation. in this paper we investigate the efficacy of the matching approach using a cross-encoder architecture for zero-and few-shot text classification with long, complex class descriptions. our contributions are as follows: the remainder of this paper is structured as follows. we review related work in section 2. section 3 formalizes our problem statement, introduces the matching model and baselines, and details our extension of maml to zero-shot learning. in section 4 we discuss our experiment setup then present and interpret the results of our experiments. we give closing thoughts in section 5. ## related work few-shot learning few-shot learning was originally explored in the domain of image classification @xcite @xcite where a subset of classes-train classes-have sufficient examples, while the remaining classes-test classes-have relatively few examples. such methods have since been repurposed for nlp tasks @xcite , and new techniques, such as in-context and prompt-based learning, have shown great promise for few-shot text classification. large language models like gpt-3 and 4 @xcite ) can solve nlp tasks by including few-shot examples incontext. however, their large parameter sizes and maximum token limitations hinder their ability to efficiently utilize more labels. prompt-based methods like pattern exploiting training (pet) @xcite address these challenges by prompt-tuning language models and using them to annotate large unlabeled datasets for downstream tasks. however, pet's dependence on domainspecific data and handcrafted patterns limits its applicability. methods like lm-bff seek to automate prompt selection, but they are limited to cases when classes can be described in a few tokens @xcite . @xcite addresses the shortcomings of prompt-based methods by finetuning a sentence transformer in contrastive manner with a prompt-free framework. finally, adapters @xcite have been shown to perform well in few-shot settings because of their ability to preserve the generalizability of pre-trained transformers while avoiding catastrophic forgetting and overfitting. in this work we test methods which do not require tailored prompts: prototypical networks, adapters, and setfit. zero-shot learning zero-shot learning for text classification involves classifying texts into classes which were not seen during the learning stage (unseen classes). to this end, zero-shot techniques often rely on transferring knowledge from seen to unseen classes by leveraging semantic attributes or descriptions of classes @xcite , interclass correlations @xcite , or joint embeddings of classes and examples @xcite . given the complexity of class descriptions in our datasets, we focus on methods that use semantic attributes. matching models, one such approach, are of particular interest. these models take pairs of texts as inputin our case a class description and example textand output a score. they differ from "matching networks" @xcite , which are instead aimed at few-shot learning. matching models can be categorized into a taxonomy @xcite based on the amount of interaction they allow between input pairs. in increasing order of interaction, performance, and computational complexity, we have (1) representation-based models (or bi-encoders) @xcite , which embed the inputs separately before computing their matching score; (2) late interaction models @xcite , which allow more interplay between input pairs, but still have separate encoding stacks; and label-aware classification label-aware classification techniques enhance classifiers by incorporating additional label information alongside standard input features. these methods use various types of label information, including label hierarchy @xcite , label dependency or correlation @xcite @xcite , or label semantic information. the datasets in this paper lack label hierarchies, and label correlation is not practical in zero-and few-shot settings due to the need for data to learn the correlations. however, label semantic information is useful for zero-and few-shot applications. in the absence of labeled examples, models require some sort of information to scale to new classes. early approaches gravitated toward representation-based or late interaction models and usually assumed short label @xcite . recent work @xcite @xcite @xcite has shown the promise of using textual entailment as a format for data-poor nlp tasks. these methods convert nlp tasks into entailment problems then leverage trained entailment models. this approach has yet to be explored for datasets with long class descriptions. other labelaware classification methods use combinations of different types of label information, such as commonsense knowledge graphs for labels (nayak and bach, 2020); heterogeneous label graphs @xcite ; and word embeddings, class descriptions, class hierarchy, and knowledge graphs @xcite . ## models matching differs from standard supervised learning because it uses class descriptions as model inputs, in addition to other features. we formalize the matching problem in section 3.1. next we describe our proposed solution in section 3.2: a matching model that measures semantic similarity between class descriptions and examples. finally, we discuss the baselines against which we compare the matching model in section 3.3. ## experiments we conduct two experiments to test the models from section 3 in zero-and few-shot scenarios. our primary experiment compares the ability of each model to adapt to new classes. we then explore whether we can improve the zero-or few-shot performance of the matching model by incorporating meta-learning during training. both experiments involve four text classification datasets: two public and two internal real-world e-commerce datasets. the experiment setup is similar in each case. we first select 20% of classes to be unseen, simulating new classes. next we partition the data into training, few-shot, and test sets consisting of examples from seen, unseen, and seen and unseen classes, respectively. we train the models to predict seen classes on the train sets, update them to predict unseen classes using the few-shot sets, and evaluate them on the test sets. in order to measure the speed at which the models adapt to unseen classes we vary the number of labels available in the few-shot sets as 10, 50, 100, 500, and 1000. the few-shot samples are nested, e.g. a 100-shot dataset is a subset of the 500-and 1000-shot sets. section 4.1 describes the four datasets. sections 4.2 and 3.2.1 present the results of our model comparison and meta-learning experiments, respectively, and discuss their implications. appendix d gives details on how we constructed d train , d few-shot , d test and their triplet counterparts. ## conclusion this paper shows that casting text classification as a matching problem is an effective way to address new classes. matching models have multiple benefits relative to other methods in this space. namely, they (1) leverage both class descriptions and standard labels; and (2) can perform zero-and few-shot classification-both of which are often needed in practical scenarios. we show that they are able to exploit long, complex class descriptions, achieving 13.30% macro ap with zero-shots (relative to 10shot bert) and surpassing few-shot baselines by 22.48% with 10-shots. we also generalize maml to the zero-shot setting and show that our proposed method improves zero-shot performance by a further 4.29%. this demonstrates that, when class descriptions are available, the proposed matching models are capable of handling new classes from the beginning (zero-shot, cold-start) to the end of the model development cycle (few-to many-shot).