The Devil is in the Details: On Models and Training Regimes for Few-Shot Intent Classification (2023)

Abstract: In task-oriented dialog (ToD) new intents emerge on regular basis, with a handful of available utterances at best. This renders effective Few-Shot Intent Classification (FSIC) a central challenge for modular ToD systems. Recent FSIC methods appear to be similar: they use pretrained language models (PLMs) to encode utterances and predominantly resort to nearest-neighbor-based inference. However, they also differ in major components: they start from different PLMs, use different encoding architectures and utterance similarity functions, and adopt different training regimes. Coupling of these vital components together with the lack of informative ablations prevents the identification of factors that drive the (reported) FSIC performance. We propose a unified framework to evaluate these components along the following key dimensions:(1) Encoding architectures: Cross-Encoder vs Bi-Encoders;(2) Similarity function: Parameterized (i.e., trainable) vs non-parameterized; (3) Training regimes: Episodic meta-learning vs conventional (i.e., non-episodic) training. Our experimental results on seven FSIC benchmarks reveal three new important findings. First, the unexplored combination of cross-encoder architecture and episodic meta-learning consistently yields the best FSIC performance. Second, episodic training substantially outperforms its non-episodic counterpart. Finally, we show that splitting episodes into support and query sets has a limited and inconsistent effect on performance. Our findings show the importance of ablations and fair comparisons in FSIC. We publicly release our code and data.

## introduction intent classification deals with assigning one label from a predefined set of classes or intents to user utterances. this task is vital for task-oriented dialog (tod) systems since the predicted intent of an utterance is an essential input to other modules (i.e., dialog management) in these systems @xcite @xcite . although intent classification has been widely studied, it still represents a challenge in settings where dialogue systems, including their intent classifiers, need to have the ability to be quickly adjusted to new domains and intent classes. the main challenges in training intent classifiers in such settings lies in the costly labeling of utterances @xcite @xcite @xcite . few-shot intent classification (fsic), which deals with adjusting intent classifiers to new intents given only a handful of labeled instances, is thus of paramount importance for tod systems. various methods ( §2) for fsic have been proposed @xcite @xcite @xcite @xcite @xcite @xcite . these methods are generally similar in that they utilize pretrained language models (plms) to encode utterances and resort to k nearest neighbors (knn) inference: the label of a new instance is determined based on the labels of instances with which it has the highest representational similarity, as encoded by the plm. despite these general similarities, fsic methods differ in design choices across several crucial dimensions, including encoding architectures, utterance similarity scoring, and training regimes. these methods tie together what are, in principle, independent design decisions across these dimensions, hindering ablations and insights into what drives the (reported) fsic performance. in this work, we (1) induce a framework (plmbased utterance encoding, utterance similarity scor-ing, and nearest-neighbor-based inference) that unifies most of existing fsic approaches ( §3); and our extensive experimental results on seven intent classification datasets reveal three new important findings. first, a cross-encoder coupled with episodic training, a previously unexplored fsic combination, consistently yields best performance across all the datasets. second, episodic meta-learning yields robust fsic classifiers across the board: our results demonstrate that it is much more effective for fsic than the conventional nonepisodic training. finally, although episodic metalearning entails splitting utterances of an episode into a support and query set during training, we show, for the first time, that this does not generally have a positive effect on the fsic performance. in sum, our comparative evaluation over various design choices for key components of modern fsic approaches raise the awareness about the importance of ablations and apple-to-apple comparison between complex fsic systems that conflate several key design decisions. we hope that our findings pave the way for more deliberation in research (and in particular evaluation) for this crucial tod task. ## related work we focus on few-shot intent classification (fsic) methods, which perform class inference for utterances based on the labels of nearest neighbor (knn), either directly in the representation space of the plm or according to a trained scorer of utterance pairs. we first describe the existing fsic inference paradigms and explain why we focus on knn-based methods. we then categorize the literature on fsic approaches based on knn-inference along the three key design dimensions. inference algorithms for fsic. classical methods @xcite @xcite for fsic use the maximum likelihood inference, where a vector representation of an utterance is projected by the classifier into a probability distribution over the intent classes. training such probability distribution functions, in particular when they are modeled by neural networks, mostly requires a large number of utterances annotated with intent labels, which are infamously expensive to collect in scenarios where new intents emerge on regular basis. by relying on pretrained language models, more recent fsic methods leverage the language competences they posses (i.e., encode) to alleviate the need for learning to produce probability distributions for a large number intent classes, commonly with a few instances. these recent fsic methods @xcite @xcite @xcite @xcite instead exploit the similarities between utterance embeddings in the representation space of the (finetuned) plm and infer the intents for new utterances from the labels of nearest neighbors (knn-based). since knn-based methods in general report stateof-the-art performance for fsic, our comparative empirical evaluation focuses on the design choices for models that adopt this inference algorithm. model architectures for encoding utterance pairs. a central design decision within the knnbased fsic framework is the choice of the model architecture for encoding utterances. the majority of the approaches @xcite @xcite leverage the bi-encoder architecture @xcite @xcite . the core idea of bi-encoders is that, given a collection of utterances, each utterance is inde-pendently encoded by the plm and mapped into a dense representation space. in such a space, similarities between pairs of utterances can be computed, with a parameterized (i.e., trainable) scoring function or a non-parameterized function such as dot product or cosine similarity. in contrast, some fsic methods @xcite @xcite use the cross-encoder architecture, in which the two utterances are concatenated and encoded jointly by a pretrained text encoder, e.g., bert @xcite . the idea is to represent a pair of utterances together using a plm, where each utterance becomes a context for the other. a cross-encoder thus does not produce an embedding for a single utterance but for a pair of utterances. in general, bi-encoders are more computationally efficient than cross-encoders because of the bi-encoder's ability to cache the representations of the candidates. in return, cross-encoders, by allowing tokens of one utterance to attend over the tokens of the other (and vice versa), capture better the semantic associations between utterances. ## framework we first unify formulations of the components we need for our framework. we then present their alternative configurations along our three central dimensions of comparison: (i) model architecture for encoding utterance pairs, (ii) functions for similarity scoring, and (iii) training regimes. ## experiments we conduct our experiments in two different setups: (i) balabced n -way k-shot and (ii) imbalanced classes in the support sets. the former refers to the typical few-shot learning setup, where the numbers of classes and examples per class are balanced. in contrast, the imbalanced setup randomly defines the numbers of classes and examples, imitating the imbalance nature of some benchmarks for intent classification. while arguably some utterances can be annotated to transform imbalanced episodes into balanced ones, imbalanced few-shot learning is still a huge practical challenge for various expensive domains, e.g., those that require experts for annotation @xcite . datasets, splits, and episodes. for the balanced n -way k-shot setup, we use clinc @xcite , banking @xcite , and hwu @xcite from di-aloglue @xcite as well as liu @xcite . for the sake of fair comparisons, we use the exact splits and episodes as used by @xcite for fsic. for 5 folds, we randomly split intents of each dataset into three sets to construct training, valid and test episodes. we then generate 5-way k-shot episodes for each split in each fold, where @xmath0, 5}. for the imbalanced setup, we use atis @xcite , snips @xcite , and top @xcite . we follow @xcite to construct episodes for these datasets. settings. we use bert-based-uncased and simcse as plms. we fine-tune them using the adamw optimizer @xcite with a learning rate of 2e -5. both batch size and maximum sequence length are set to 64. see the appendix for the full list of hyperparameters. for experiments on each fold of balanced datasets, we train a fsic classifier for a maximum of 10,000 5-way k-shots episodes. we evaluate the classifier on the validation set after every 100 updates, and stop the training if validation performance does not improve over 5 consecutive evaluation steps. to alleviate the impact of random selection of fewshot samples, we report the average performance of a classifier for 600 test episodes, compatible with @xcite . for the experiments on the imbalanced datasets, similar to @xcite , we conduct the experiments over 1 fold due to the limited number of intents. the average number of shots per intent used in episodes of atis, snips, and top is about 4, 5, and 4, respectively (see appendix for details). for both n -way k-shot and imbalanced setups, the number of examples in query sets is identical for all intents in the query sets. so, for all experiments we report the accuracy metric averaged over all runs and folds. ## results and discussion we compare the the configurations described in ( §3) and baselines ( §4) for balanced and imbalanced fsic setups using bert, as the most widely used pretrained language model, and simcse, the state-of-the-art model for encoding the meaning of sentences. our main experimental findings are as follows ## conclusions we shed light on factors that contribute to performance of models for few-shot intent classification (fsic), a crucial task in modular dialogue systems. we categorize fsic approaches across three essential dimensions: (1) the cross-encoder vs. bi-encoder encoder architectures; (2) the parameterized (i.e., trainable) vs non-parameterized utterance similarity scoring; and (3) episodic vs non-episodic training. our extensive evaluation, encompassing seven standard fsic datasets, reveals that the previously unexplored combination of cross-encoder architecture (with parameterized utterance similarity scoring) and episodic training consistently yields the best fsic performance. we additionally find that (i) episodic meta-learning generally outperforms the non-episodic training and (ii) that the widely adopted hypothesis in meta-learning that splitting episodes into support and query sets helps generalization and boost performance may not hold for fsic. we hope that our findings lead to more deliberation on fsic evaluation protocols and more insightful "apple-to-apple" comparisons between competing models and model variants.