Anchoring Fine-tuning of Sentence Transformer with Semantic Label Information for Efficient Truly Few-shot Classification (2023)

Abstract: Few-shot classification is a powerful technique, but training requires substantial computing power and data. We propose an efficient method with small model sizes and less training data with only 2-8 training instances per class. Our proposed method, AncSetFit, targets low data scenarios by anchoring the task and label information through sentence embeddings in fine-tuning a Sentence Transformer model. It uses contrastive learning and a triplet loss to enforce training instances of a class to be closest to its own textual semantic label information in the embedding space - and thereby learning to embed different class instances more distinct. AncSetFit obtains strong performance in data-sparse scenarios compared to existing methods across SST-5, Emotion detection, and AG News data, even with just two examples per class.

## introduction data sparsity remains a problem for training nlp models. pre-trained language models (plms) have been adapted to nlp downstream tasks using only a few ground truth examples. however, many of these methods require substantial computing, reducing their practicality. the setfit method @xcite offers a promising route to fewshot text classification, due to its computational efficiency and relatively low data requirement. the method is prompt-free, which is an advantage in that it skips the variation inherent to prompt-based learning methods since these depend on choosing good/multiple prompts. on the other hand, in a truly few-shot setting, providing the model with information about the labels it should generalise to from the few given training examples is expected to be helpful @xcite . we ask (rq1) if the efficiency of small models in setfit can be retained while also making the method label-aware and if this improves performance, especially in cases with really litle training data. to do this, we propose adding semantic information about the task and labels using fine-tuning of sentence transformers @xcite . through this, we inform the model of what is to be classified by adding textual information. we use contrastive learning and a triplet loss to enforce instances from different classes to be closest to their own class' textual description. additionally, (rq2) we ask to what extent variation in textual information affects model performance. our contributions include: code is publicly available at https://github. com/amaliepauli/ancsetfit . ## ancsetfit: anchoring setfit with semantic label information the intuition in setfit, as discussed in the section above, is to get the embeddings to separate class-wise before inputting them into a classifier. however, if the training data is really small, there might be a lack of information about the classes that would allow the embedding to cluster accordingly and thus to generalise. when only a few samples are available, the samples might have other properties in common than what the task prescribes. we see the benefit of adding task description information, as seen in pet methods. we, therefore, propose to inject information about the task into the method, such that it can help separate the embeddings in the desired directions for a task. intuitively, we want to provide the model with good class-specific starting points in the embedding space for sentences to cluster around. we think of them as starting points since we want the class-specific points to be learnable as well. we introduce anchor statements, which are sentences containing semantic information for each class i.e. a task/class description. as they are textual, they can be embedded with the model and hence updated under fine-tuning. we look to the sentence transformer paper @xcite , which experiments with a triplet loss function for sentence similarity in articles. the input is a triplet of three sentences, two from the same article, and one from a distinct, and then the loss minimises the distance between the same class pair. we propose to modify the input of the loss function to be a class-specific fixed anchor statement, and two samples from distinct classes, one matching the class of the anchor statementthus getting the semantic label information into the loss function. the objective is to ensure that the samples of one class are closer to their own fixed class-specific anchor statement, hence getting the embeddings to separate in the desired direction. more formally, we generate input triplets (a, u, v) by drawing samples u, v from two distinct classes and adding the anchor statement a of one of the classes. we use the loss function: ## results and discussion we compare ancsetfit to setfit on seven different datasets in fig. 1 , with selected details in as ancsetfit only holds an advantage in performances over setfit when training data sizes are low, we compare ancsetfit to adapet and per-fect in the training size range of 2-8 instances per class. (comparing setfit to adapet and per-fect for larger training sizes we refer the reader to @xcite .) ancsetfit outperforms perfect and adapet_base on all six datasets for training sizes 2,4,8 in however, adapet_xxl model size is double that of ancsetfit in terms of parameters raft leaderboard ansetfit achieves higher performance in 4 out of 11 raft tasks than set-fit. setfit shows a higher average accuracy. the 'banking_77' dataset has classes with no training samples, which makes it unsuitable for supervised learning with ancsetfit. further, the majority of the datasets are binary with more labels per class than in the range where our experiments show an acceleration in results. lastly, we note that several tasks are semantically complex, in that the description of the classes is rather long, making it more difficult to capture the information in the anchor statement. for example, the task on 'systematic review inclusion' has a specification list of criteria for the classes. however, ancsetfit gets higher performance on the two twitter tasks of detecting hate speech and complaints. ## ablation study: effect of the anchor statements in the development phase, we study the impact of the anchor statements. we construct the anchor statement in the generic form where a template is specific to each task and w is a word describing the class. we evaluate four ways of constructing w; 1) words of the labels, 2) a single letter representing no actual information, 3) permutated label words from 1 such that the label word does not match the class, and 4) label synonyms (see appendix b ## conclusion this paper proposes ancsetfit, an efficient method for extreme few-shot text classifications on small model sizes. the method injects semantic information about the task and labels to fine-tune a sentence transformer. the method separates samples of different classes in the embedding space by providing anchor points of textual information about the class labels to cluster around. we show in a number of tasks for sentiment and topic classifications that ancsetfit achieves higher performance in the extreme few-shot settings where there are fewer than eight training samples per class.