Few-Shot Learning with S iamese Networks and Label Tuning (2022)

Abstract: We study the problem of building text classifiers with little or no training data, commonly known as zero and few-shot text classification. In recent years, an approach based on neural textual entailment models has been found to give strong results on a diverse range of tasks. In this work, we show that with proper pre-training, Siamese Networks that embed texts and labels offer a competitive alternative. These models allow for a large reduction in inference cost: constant in the number of labels rather than linear. Furthermore, we introduce label tuning, a simple and computationally efficient approach that allows to adapt the models in a few-shot setup by only changing the label embeddings. While giving lower performance than model fine-tuning, this approach has the architectural advantage that a single encoder can be shared by many different tasks.

## introduction few-shot learning is the problem of learning classifiers with only a few training examples. zero-shot learning @xcite , also known as dataless classification @xcite , is the extreme case, in which no labeled data is used. for text data, this is usually accomplished by representing the labels of the task in a textual form, which can either be the name of the label or a concise textual description. in recent years, there has been a surge in zeroshot and few-shot approaches to text classification. one approach @xcite @xcite makes use of entailment models. textual entailment @xcite , also known as natural language inference (nli) @xcite , is the problem of predicting whether a textual premise implies a textual hypothesis in a logical sense. for example, emma loves apples implies that emma likes apples. the entailment approach for text classification sets the input text as the premise and the text repre-senting the label as the hypothesis. a nli model is applied to each input pair and the entailment probability is used to identify the best matching label. in this paper, we investigate an alternative based on siamese networks (sn) @xcite , also known as dual encoders. these models embed both input and label texts into a common vector space. the similarity of the two items can then be computed using a similarity function such as the dot product. the advantage is that input and label text are encoded independently, which means that the label embeddings can be pre-computed. therefore, at inference time, only a single call to the model per input is needed. in contrast, the models typically applied in the entailment approach are cross attention (ca) models which need to be executed for every combination of text and label. on the other hand, they allow for interaction between the tokens of label and input, so that in theory they should be superior in classification accuracy. however, in this work we show that in practice, the difference in quality is small. both ca and sns also support the few-shot learning setup by fine-tuning the models on a small number of labeled examples. this is usually done by updating all parameters of the model, which in turn makes it impossible to share the models between different tasks. in this work, we show that when using a sn, one can decide to only fine-tune the label embeddings. we call this label tuning (lt). with lt the encoder can be shared between different tasks, which greatly eases the deployment of this approach in a production setup. lt comes with a certain drop in quality, but this drop can be compensated by using a variant of knowledge distillation @xcite . our contributions are as follows: we perform a large study on a diverse set of tasks showing that ca models and sn yield similar performance for both zero-shot and few-shot text classification. in contrast to most prior work, we also show that these results can also be achieved for languages other than english. we compare the hypothesis patterns commonly used in the literature and using the plain label name (identity hypothesis) and find that on average there is no significant difference in performance. finally, we present lt as an alternative to full fine-tuning that allows using the same model for many tasks and thus greatly increases the scalability of the method. we will release the code 1 and trained models used in our experiments. ## methodology figure 1 explains the overall system. we follow @xcite and apply symmetric siamese networks that embed both input texts using a single encoder. the encoder consists of a transformer @xcite that produces contextual token embeddings and a mean pooler that combines the token embeddings into a single text embedding. we use the dot product as the similarity function. we experimented with cosine similarity but did not find it to yield significantly better results. as discussed, we can directly apply this model to zero-shot text classification by embedding the input text and a textual representation of the label. for 1 https://tinyurl.com/label-tuning the label representation we experiment with a plain verbalization of the label, or identity hypothesis, as well as the hypotheses or prompts used in the related work. ## related work pre-trained language models (lms) have been proved to encode knowledge that, with taskspecific guidance, can solve natural language understanding tasks @xcite . leveraging that, le scao and rush (2021) quantified a reduction in the need of labeled data of hundreds of instances with respect to traditional fine-tuning approaches @xcite . this has led to quality improvements in zero and few-shot learning. ## experimental setup in this section we introduce the baselines and datasets used throughout experiments. ## results here we present the results of our experiments. the two main questions we want to answer are whether siamese networks (sn) give comparable results as cross attention models (ca) and how well label tuning (lt) compares to regular fine-tuning. ## analysis we analyze the performance of the cross attention (ca) and siamese network-based (sn) models. unless otherwise noted, the analysis was run over all datasets and languages. ## conclusion we have shown that cross attention (ca) and siamese networks (sn) for zero-shot and few-shot text classification give comparable results across a diverse set of tasks and multiple languages. the inference cost of sns is low as label embeddings can be pre-computed and, in contrast to ca, does not scale with the number of labels. we also showed that tuning only these label embeddings (label tuning (lt)) is an interesting alternative to regular fine-tuning (ft). lt gets close to ft performance when combined with knowledge distillation and when the number of training samples is low, i.e., for realistic few-shot learning. this is relevant for production scenarios, as it allows to share the same model among tasks. however, it will require 60 times more memory to add a new task: for a 418 mb mpnet-base model, bitfit affects 470 kb of the parameters. lt applied to a task with 10 labels and using a embedding dimension of 768 requires 7.5 kb. the main disadvantage of bitfit, however, is that the weight sharing it requires is much harder to implement, especially in highly optimized environments such as nvidia triton. therefore we think that lt is an interesting alternative for fast and scalable few-shot learning.