F ast F it: Fast and Effective Few-Shot Text Classification with a Multitude of Classes (2024)

Abstract: We present FastFit, a Python package designed to provide fast and accurate few-shot classification, especially for scenarios with many semantically similar classes. FastFit utilizes a novel approach integrating batch contrastive learning and token-level similarity score. Compared to existing few-shot learning packages, such as SetFit, Transformers, or few-shot prompting of large language models via API calls, FastFit significantly improves multi-class classification performance in speed and accuracy across various English and Multilingual datasets. FastFit demonstrates a 3-20x improvement in training speed, completing training in just a few seconds. The FastFit package is now available on GitHub, presenting a user-friendly solution for NLP practitioners.

## introduction few-shot classification presents a unique challenge, especially when dealing with a multitude of classes that share similar semantic meanings. expanding the training data can be both time-consuming and costly. to address this challenge, two primary categories of tools have been developed: few-shot prompting of large language models (llms) via api calls, or packages designed for fine-tuning smaller language models using the limited available data. however, we recognize the drawbacks of applying both of these approaches in practice. few-shot prompting of llms leverages their multitasking abilities to tackle data scarcity. however, in the presence of many classes, llms encounter three major challenges: (1) llms struggle to incorporate demonstrations of all classes within their context window. (2) utilization of the long context for the classification task can be challenging @xcite . (3) due to the model size, and prompt length the inference time is slow. 1 fastfit github in contrast, the approach of fine-tuning smaller language models capitalizes on their adaptability to specific tasks, as demonstrated to be effective in recent works. however, these methods can be challenging to deploy as they require architectural adjustments @xcite or, like setfit, may prove less suitable for classification with many classes @xcite . in this work, we present fastfit, a fast and accurate method, and a pip-installable python package designed for fine-tuning small language models in few-shot classification tasks involving many classes. through various experiments, we demonstrate that fastfit training is significantly faster, providing a 3-20x speedup. this enables training within seconds, as illustrated in fig. 1 . fastfit outperforms earlier packages, including setfit, transformer, and multi-task models like flan, or larger llms like llama-70b, in both english and multilingual settings. the core contribution facilitating this speedup and improvement lies in fastfit's use of batch contrastive training, recognized for its efficiency and effectiveness @xcite . this technique brings same-class texts closer while pushing apart all other texts. fastfit also incorporates token-level text similarity measures that leverage fine-grained information @xcite . additionally, we integrate text augmentation techniques to enhance the robustness of the training process @xcite . the fastfit package is easy to install and use, interfacing with standard training apis (see §2). we hope that fastfit will help make text classification easier and faster for the benefit of the whole community. ## the fastfit api the fastfit python package is available on pypi and can be installed with: ## method given a few-shot text classification dataset containing texts and their corresponding classes denoted as {x i , y i } n @xmath0, let @xmath1. our task is to classify each x i into a class y @xmath2. to achieve this goal we aim to encode both texts and class names into a shared embedding space, where they are represented closely, according to a similarity metric s, when they belong to the same class and are represented further apart when they do not. to accomplish this, we optimize the following batch contrastive loss: for each text in the batch, we augment the batch by including its class name as an additional example. additionally, we repeat the texts in the batch r times as a data augmentation technique, following @xcite by treating the dropout as a minimal augmentation at the representation level. this method has demonstrated significant success in generating sentence embeddings, and we leverage it here to enhance representation for text classification. in our data-scarce setting, we employ finegrained token-level similarity metrics, leveraging textual details. this approach, successful in works like bert-score and colbert, defines the similarity metric between texts x i and x j as the sum of cosine similarities between x i and the most similar tokens in x j . specifically, with tokens denoted as x 1 i , . . . , x n i and x 1 j , . . . , x m j respectively, the similarity score is computed as follows: during inference, when provided with a new text, x u we classify it to the most similar class y @xmath3. this method draws inspiration from the way inference is conducted in retrieval systems, eliminating the need for a classification head and aligning the training and inference objectives. ## fastfit ablation & full training to further examine the contribution of some of our method modifications, we compare training with cls and token-level similarity metrics, as well as training with a different number of batch repetitions. we conduct these experiments on three datasets: hwu64, banking77, and clinc150, with 5 random splits, and average their results. we assess the effect of these modifications for both small and large models, with 5 and 10 shots. in ## related work for fine-tuning baselines, we focus on readily available methods. , including setfit with its package, a standard classifier accessible through hf transformers @xcite , or llms through api calls. however, there are various few-shot classifiers, and we will briefly discuss a couple of them. qaid @xcite proposed preand fine-tuning training stages with unsupervised and supervised loss, using colbert architecture, achieving sota results. t-few @xcite , a parameter-efficient fine-tuning method based on t0 @xcite , claims to be better and cheaper than in-context learning. regarding few-shot prompting of llms approaches, a question arises about whether our results will withstand stronger llms or improved prompting techniques. according to @xcite we can deduce that fastfit outperforms gpt4 @xcite with a fraction of the cost. additionally, @xcite demonstrate that retrieval-based few-shot prompts can lead to improved results. however, it's worth noting that currently, these models remain slow and costly. ## conclusions in this paper, we introduce fastfit, a novel fewshot text classification method accompanied by a python package. our results demonstrate that fastfit outperforms large language models (llms) such as flan-xxl and llama-2-chat-70b, as well as fine-tuning methods, including both standard and setfit classifiers, readily available in existing packages. notably, fastfit exhibits fast training and inference. we provide evidence that these results hold for both multilingual and full-data training setups. we hope that fastfit's speed and simplicity will enhance its usability.