Effective Few-Shot Classification with Transfer Learning (2020)

Abstract: Few-shot learning addresses the the problem of learning based on a small amount of training data. Although more well-studied in the domain of computer vision, recent work has adapted the Amazon Review Sentiment Classification (ARSC) text dataset for use in the few-shot setting. In this work, we use the ARSC dataset to study a simple application of transfer learning approaches to few-shot classification. We train a single binary classifier to learn all few-shot classes jointly by prefixing class identifiers to the input text. Given the text and class, the model then makes a binary prediction for that text/class pair. Our results show that this simple approach can outperform most published results on this dataset. Surprisingly, we also show that including domain information as part of the task definition only leads to a modest improvement in model accuracy, and zero-shot classification, without further fine-tuning on few-shot domains, performs equivalently to few-shot classification. These results suggest that the classes in the ARSC few-shot task, which are defined by the intersection of domain and rating, are actually very similar to each other, and that a more suitable dataset is needed for the study of few-shot text classification.

## introduction to deal with emerging topics, events or linguistic phenomena, text classification systems often need to accommodate new classes. recently, there has been increasing interest in addressing this problem through the lens of few-shot learning, which focuses on learning from a small handful of examples rather than a large annotated dataset. few-shot classification research in computer vision and natural language processing studies the problem of generalizing from classes with plentiful training data to previously unseen classes for which labeled data may not be widely available. at the same time, there has been a widespread growth of interest in transfer learning for natural language processing, with transformer-based self-supervised models such as bert @xcite and t5 @xcite achieving strong results on a variety of benchmark problems. these models can be fine-tuned to new tasks with a small number of training examples, suggesting that their generalization capabilities may also be applicable to the few-shot learning setting. in this paper, we study a straightforward transfer learning approach applied to a standard few-shot text classification dataset. specifically, we focus on the well-studied amazon review sentiment classification (arsc) dataset, which was originally developed for domain adaptation research @xcite , and was adapted to the few-shot setting by @xcite . various few-shot learning approaches have been benchmarked on this dataset, including techniques based on metric learning @xcite and meta-learning @xcite . in this work, we consider a simple bert-based classification scheme that first fine-tunes a pretrained model on the full rating classification dataset, and then further fine-tunes on only the held-out few-shot classes. this approach achieves comparable performance to state-of-the-art techniques, suggesting that pretrained models can extend their generalization capabilities to few-shot settings. surprisingly, however, we also find similar performance in zero-shot settings for this dataset, implying that few-shot categories are not sufficiently distinct from the other categories, and consequently motivating the need for new datasets to support future research in this area. the contributions of our work are as follows: ## related work the goal of few-shot learning @xcite @xcite is to adapt a classifier to generalize to new classes using very few training examples. such models typically cannot be trained using conventional methods, as modern classification algorithms require more parameters than there are training examples and this generalize poorly. approaches based on metric learning attempt to relate new classes to those in the training data @xcite @xcite @xcite , while meta-learning techniques modify the optimization strategy to provide a model that can rapidly adapt to related tasks @xcite @xcite @xcite . recent work on both approaches has used the arsc dataset @xcite , which we study in this work. we focus here on a straightforward application of the transfer learning paradigm popularized in natural language processing by models such as bert @xcite . these approaches exploit the language modeling capacity of transformer architectures @xcite by pretraining models on large text corpora, and subsequently adapting these models to downstream tasks. the link between few-shot learning and transfer learning is also suggested by prior work. @xcite demonstrate that pretraining on in-domain unlabeled data can improve performance on downstream tasks, suggesting that the initial training data available to few-shot learners could also be used to improve generalization of pretrained models to few-shot classes. brown et al. (2020) demonstrate that language models with multiple orders of magnitude more parameters than typically used in transfer learning can generalize to a wide variety of tasks with few training samples, implying that even domain mismatch is no barrier to the generalization ability of such models, given sufficient model capacity. ## dataset in this work, we study the amazon review sentiment classification (arsc) dataset used in recent fewshot learning literature @xcite @xcite . this dataset was originally constructed by @xcite as the multi-domain sentiment dataset foot_0 for the purpose of studying domain adaptation-a precursor task for few-shot learning-in sentiment analysis. the dataset consists of the text of reviews submitted to the amazon online retail site for 25 product domains as well as their star ratings (out of 5), although 3-star reviews were excluded due to their ambiguous semantics for binary sentiment classification. later work @xcite proposed alternative labels for these reviews based on the star rating r: @xmath0, @xmath1. we use the version of the dataset adapted by @xcite for few-shot learning. 2 this version of the dataset employs the three rating categories proposed by @xcite , drops stop-words and case from the review text, and partitions the corpus into training, validation and test splits. they propose a few-shot setting in which each domain and rating category is treated as a separate class. four domains used in @xcite -books, dvds, electronics and kitchen & housewares, which are among the largest domains in the dataset -are held out as few-shot domains, leading to a total of 12 few-shot classes (4 domains × 3 rating categories) for which training data is assumed to be scarce. models are therefore able to use all training data from 57 classes (19 remaining domains 3 × 3 rating categories) covering 95,586 training examples, but only 5 positive and 5 negative examples from each of the 12 few-shot classes. trained few-shot classifiers are evaluated only on test sets for these 12 classes. ## model formulation we consider a straightforward application of transfer learning to the few-shot classification scenario described above, based on the original bert model of @xcite , which employs a pretrained transformer encoder supervised by two objectives: masked token prediction and next sentence prediction. although domain labels in the arsc dataset are expected to be disjoint, the rating categories are not, e.g., a review that has a positive label for @xmath2. thus, more than one class label may be positive for a given input example, and so the learning problem is a form of multi-label classification-where examples can be assigned multiple class labels-rather than mutually-exclusive multi-class classification. we address this by formulating the prediction task as a binary classification problem, where the rating category and domain are encoded in the input along with the text to be classified, and the target label is a single binary label corresponding to that rating-domain combination. this is done by adding plain-text prefixes to each review, in training and during inference, to identify the star rating that the model is expected to distinguish and the domain that the data belongs to (e.g., "task two books" for @xmath3. so, instead of training a separate model for each rating-domain pair, we add this class information to the model input, and train a single binary model to learn across all classes. prior work with transformer models has shown that such an approach can be highly effective, and prefix input tokens have been used to inject task information into models by specifying the desired output language or format @xcite , domain @xcite , etc. pretrained bert models are typically fine-tuned on the target task directly; however, the extremely small number of training examples for few-shot classes (120 in total: 5 each for positive/negative examples across 12 rating-domain pairs) renders this approach unviable, as is also shown by @xcite . therefore, in order to adapt the pretrained model to the few-shot learning domain, we conduct two rounds of fine-tuning: first with the 57 classes for which full training data is available, and then a second round of fine-tuning with only the 12 few-shot classes. ## experiments we investigate the usefulness of this simple transfer learning approach through evaluations on the arsc dataset. the bert-base pretrained model released in the transformers library foot_4 is used as a starting point and fine-tuned on the review data with sequence lengths truncated to 256 subword tokens. two kinds of class prefixes are evaluated: the rating category alone (bert-fs r ) and the rating-domain pair (bert-fs r+d ). comparisons are based on the mean per-class accuracy over test data following @xcite and subsequent work. ## conclusions & future work we explore the problem of few-shot text classification from the perspective of transfer learning using the popular approach of fine-tuning a pretrained bert model. using the well-studied arsc dataset, we show that the multi-domain sentiment classification problem can be formulated as a single binary classification task where the rating-domain specification from arsc is encoded within the input. experimental results reveal that this simple approach outperforms most published results on this dataset, and that a zero-shot model achieves comparable accuracy to a five-shot model. since we demonstrate that including domain information, which is used to define the classes for this dataset, only yields minor improvements in training a model, the results also suggest that arsc is no longer an appropriate dataset for studying few-shot learning techniques. we aim to remedy this situation and identify more suitable resources for future research on few-shot and continual text classification.