Beyond Fine-tuning: Few-Sample Sentence Embedding Transfer (2020)

Abstract: Fine-tuning (FT) pre-trained sentence embedding models on small datasets has been shown to have limitations. In this paper we show that concatenating the embeddings from the pre-trained model with those from a simple sentence embedding model trained only on the target data, can improve over the performance of FT for few-sample tasks. To this end, a linear classifier is trained on the combined embeddings, either by freezing the embedding model weights or training the classifier and embedding models end-to-end. We perform evaluation on seven small datasets from NLP tasks and show that our approach with end-to-end training outperforms FT with negligible computational overhead. Further, we also show that sophisticated combination techniques like CCA and KCCA do not work as well in practice as concatenation. We provide theoretical analysis to explain this empirical observation.

## introduction fine-tuning (ft) powerful pre-trained sentence embedding models like bert @xcite has recently become the de-facto standard for downstream nlp tasks. typically, ft entails jointly learning a classifier over the pre-trained model while tuning the weights of the latter. while ft has been shown to improve performance on tasks like glue @xcite having large datasets (qqp, mnli, qnli), similar trends have not been observed on small datasets, where one would expect the maximum benefits of using a pre-trained model. several works @xcite @xcite have demonstrated that ft with a few target domain samples is unstable with high variance, thereby often leading to sub-par gains. furthermore, this issue has also been well documented in practice 1 . learning with low resources has recently become an active research area in nlp, and arguably one of the most interesting scenarios for which pre-trained models are useful (e.g., @xcite ). many practical applications have small datasets (e.g., in social science, medical studies, etc), which are different from large-scale academic benchmarks having hundreds of thousands of training samples (e.g, dbpedia @xcite , sogou news @xcite , etc). this necessitates effective transfer learning approaches using pre-trained sentence embedding models for fewsample tasks. in this work, we show that concatenating sentence embeddings from a pre-trained model and those from a smaller model trained solely on the target data, can improve over the performance of ft. specifically, we first learn a simple sentence embedding model on the target data. then we concatenate(cat) the embeddings from this model with those from a pre-trained model, and train a linear classifier on the combined representation. the latter can be done by either freezing the embedding model weights or training the whole network (classifier plus the two embedding models) end-to-end. we evaluate our approach on seven small datasets from nlp tasks. our results show that our approach with end-to-end training can significantly improve the prediction performance of ft, with less than a 10% increase in the run time. furthermore, our approach with frozen embedding models performs better than ft for very small datasets while reducing the run time by 30%-50%, and without the requirement of large memory gpus. we also conduct evaluations of multiple techniques for combining the pre-trained and domainspecific embeddings, comparing concatenation to cca and kcca. we observe that the simplest approach of concatenation works best in practice. moreover, we provide theoretical analysis to explain this empirical observation. finally, our results also have implications on the semantics learning ability of small domainspecific models compared to large pre-trained models. while intuition dictates that a large pre-trained model should capture the entire semantics learned by a small domain-specific model, our results show that there exist semantic features captured solely by the latter and not by the former, in spite of pretraining on billions of words. hence combining the embeddings can improve the performance of directly ft the pre-trained model. related work recently, several pre-trained models have been studied, of which some provide explicit sentence embeddings @xcite , while others provide implicit ones @xcite . @xcite compare the performance of feature extraction (by freezing the pre-trained weights) and ft. there exists other more sophisticated transferring methods, but they are typically much more expensive or complicated. for example, @xcite al. (2019 2019 ) first train a deep network classifier on the fixed pre-trained embedding and then fine-tune it. our focus is to propose alternatives to ft with similar simplicity and computational efficiency, and study conditions where it has significant advantages. while the idea of concatenating multiple embeddings has been previously used @xcite , we use it for transfer learning in a low resource target domain. ## methodology we are given a set of labeled training sentences @xmath0, y i )} m @xmath1. denote the embedding of s from f 1 by v 1s =f 1 (s)∈r d 1 . here f 1 is assumed to be a large and powerful embedding model such as bert. our goal is to transfer f 1 effectively to the target domain using s. we propose to use a second sentence embedding model f 2 , which is different from and typically much smaller than f 1 , which has been trained solely on s. the small size of f 2 is necessary for efficient learning on the small target dataset. let v 2s =f 2 (s)∈r d foot_2 denote the embedding for s obtained from f 2 . our method cat concatenates v 1s and v 2s to get an adaptive representation @xmath2, αv 2s ] for s. here α>0 is a hyper-parameter to modify emphasis on v 1s and v 2s . it then trains a linear classifier c(v s ) using s in the following two ways: (a) frozen embedding models only training the classifier c while fixing the weights of embedding models f 1 and f 2 . this approach is computationally cheaper than ft f 1 since only c is trained. we denote this by cat (locked f 1 , f 2 weights). (b) trainable embedding models jointly training classifier c, and embedding models f 1 , f 2 in an end-to-end fashion. we refer to this as cat . the inspiration for combining embeddings from two different models f 1 , f 2 stems from the impressive empirical gains of ensembling @xcite in machine learning. while typical ensembling techniques like bagging and boosting aggregate predictions from individual models, cat and cat aggregate the embeddings from individual models and train a classifier using s to get the predictions. note that cat keeps the model weights of f 1 , f 2 frozen, while cat initializes the weights of f 2 after initially training on s 2 . one of the benefits of cat and cat is that they treat f 1 as a black box and do not access its internal architecture like other variants of ft @xcite . additionally, we can theoretically guarantee that the concatenated embedding will generalize well to the target domain under assumptions on the loss function and embedding models. ## experiments datasets we evaluate our approach on seven low resource datasets from nlp text classification tasks like sentiment classification, question type classification, opinion polarity detection, subjectivity classification, etc. we group these datasets into 2 categories: the first having a few hundred training samples (which we term as very small datasets for the remainder of the paper), and the second having a few thousand training samples (which we term as small datasets). we consider the following 3 very small datasets: amazon (product reviews), imdb (movie reviews) and yelp (food article reviews); and the following 4 small datasets: mr (movie reviews), mpqa (opinion polarity), trec (question-type classification) and subj (subjectivity classification). we present the statistics of the datasets in ## conclusion in this paper we have proposed a simple method for transferring a pre-trained sentence embedding model for text classification tasks. we empirically show that concatenating pre-trained and domain specific sentence embeddings, learned on the target dataset, with or without fine-tuning can improve the classification performance of pre-trained models like bert on small datasets. we have also provided theoretical analysis identifying the conditions when this method is successful and to explain the experimental results. theorem 1. if the loss function l is λ-lipschitz for the first parameter, and p has full column rank, then there exists a linear classifier f over vs such that l( f ) ≤ l(f * ) + λσ (p † ) w * 2 where p † is the pseudo-inverse of p .