A Neural Few-Shot Text Classification Reality Check (2021)

Abstract: Modern classification models tend to struggle when the amount of annotated data is scarce. To overcome this issue, several neural few-shot classification models have emerged, yielding significant progress over time, both in Computer Vision and Natural Language Processing. In the latter, such models used to rely on fixed word embeddings, before the advent of transformers. Additionally, some models used in Computer Vision are yet to be tested in NLP applications. In this paper, we compare all these models, first adapting those made in the field of image processing to NLP, and second providing them access to transformers. We then test these models equipped with the same transformer-based encoder on the intent detection task, known for having a large amount of classes. Our results reveal that while methods perform almost equally on the ARSC dataset, this is not the case for the Intent Detection task, where most recent and supposedly best competitors perform worse than older and simpler ones (while all are are given access to transformers). We also show that a simple baseline is surprisingly strong. All the new developed models as well as the evaluation framework are made publicly available.

## introduction text classification often requires a large number of mappings between texts and target classes, so that it is challenging to build few-shot text classification models @xcite . with the recent advances of transformer-based models @xcite along with their fine-tuning techniques @xcite , text classification has significantly improved. in few-shot settings, methods based on these extracted text representations have been historically made of semi-supervision, especially thanks to pseudo-labeling @xcite @xcite , which aims at propagating known labels to unlabeled data points in the representational space. such methods depend on the number of collected unlabeled data, which can also be costly to obtain @xcite , and also suffer from the infamous pipeline effect in nlp @xcite , as cascade processing tends to make errors accumulate. in order to address the hindrance of collecting unlabeled data, modern approaches include unsupervised data augmentation techniques @xcite . it consists of generating samples through well-established text augmentation techniques in neural machine translation, such as backtranslation @xcite , and then use a consistency loss, training the classifier to assign the same prediction to all variations of the same sample text. while collecting new pseudo-labels can therefore be overcome by manipulating the dataset (especially using data augmentation techniques), the pipelining error accumulation effect instead calls for new neural architectures supporting scarcity of labeled data in an end-to-end fashion. such end-toend few-shot neural architectures for few-shot classification were discovered in image processing -it includes matching networks @xcite , prototypical networks @xcite plus a follow-up known as prototypical networks++ @xcite , and relation networks @xcite . ultimately induction networks @xcite is a meta-learning based method dedicated to few-shot text classification, supposedly the state-ofthe-art. since our contribution considers this family of models, we will further detail them in section 2. nonetheless, it is important to stress that most of these neural architectures were originally devised to integrate image feature extractors. despite both text and image relying on features extractors, a paragraph or sentence of few words hardly convey as much information as a full-fledged three-canals 600 × 400 image (720, 000 numerical values intrinsically). it is therefore of the utmost practical interest to validate and compare if what works best for end-to-end few-shot image classification is the same for end-to-end few-shot text classification. moreover, when applying these end-to-end fewshot models to text, two main system components are into action: the text feature extractor itself and the downstream part of the neural network that provides a learning strategy over few shots. if we want to compare these systems, we need to plug the same feature extractor (hopefully the best one, that is transformer-based currently) into each endto-end model. for the time being, the literature on end-to-end few-shot text classification compare aforementioned techniques using a different text extractor for each system, which is the one available when the technique was discovered -these text encoding varying greatly (section 3.3). from that point-of-view, it is hardly possible to conclude if the improvement over time in few-shot text classification is due to new few-shot learning techniques or plainly to the significant advances made by text feature extractors. the same applies to vectors metrics: one method can use the cosine and another the euclidean distance, and that choice alone can impact conclusions made on the method being the state-of-the-art, although it could well rely only on the metric at work. ultimately, experimental setups are usually restricted to one dataset, and evaluation schemes are heterogeneous among papers @xcite , in a nutshell, we will demonstrate that providing a transformer-based encoder to a previously obsolete few-shot technique makes it the state-ofthe-art again, that standard baselines are surprisingly strong, and that induction networks, while performing well for binary sentiment classification, struggles to perform correctly in the most common setups of few-shot text classification. ## few-shot classification methods in this section, we will describe the few-shot learning methods. in the following section, sentence vectors derived from the sentence encoder are denoted v. v s , v q , and v u represent vectors for support, query, and unlabeled points, respectively. the number of shots is denoted k, and the number of classes per episode is denoted c. the k th support vector of class c is denoted v s c,k . in the equations, s q i,j (resp. s q i,c ) will denote the similarity between the i th query vector and the j th support vector (resp. the c th class). similarly, s u i,j represents the similarity between the i th unlabeled sample and the j th support vector. when needed, the number of unlabeled data is denoted u . for each method relying on a given similarity or distance metric, we devise two experiments, using either the cosine similarity or the euclidean distance. those additional experiments are crucial, as they allow us to compare methods directly, without introducing a metric choice bias. architectures of the different few-shot approaches are illustrated in figure 1 . they are each detailed in section 2.2 and onwards, yet we first introduce the common building blocks among all methods in what follows. ## observations we report results for the arsc dataset in ## conclusion we provided a fair comparison for end-to-end neural few-shot text classification methods discovered over the last few years. when they are all equipped with a transformer-based text encoder, we show that prototypical networks become the state-of-theart again. we also found that a traditional classifier trained on few shots yields very competitive results, especially when given shots are re-sampled at each iteration. ultimately, we also demonstrated the significant impact of the vector metric, illustrated by matching networks strongly improving by only replacing the cosine by the euclidean distance. the complete source code with the re-implementation of all the tested methods and evaluation framework used in this study is publicly available foot_4 -we hope that it will help the community build upon consistent comparative experiments.