Predictive Model Selection for Transfer Learning in Sequence Labeling Tasks (2020)

Abstract: Transfer learning is a popular technique to learn a task using less training data and fewer compute resources. However, selecting the correct source model for transfer learning is a challenging task. We demonstrate a novel predictive method that determines which existing source model would minimize error for transfer learning to a given target. This technique does not require learning for prediction, and avoids computational costs of trail-and-error. We have evaluated this technique on nine datasets across diverse domains, including newswire, user forums, air flight booking, cybersecurity news, etc. We show that it per-forms better than existing techniques such as fine-tuning over vanilla BERT, or curriculum learning over the largest dataset on top of BERT, resulting in average F1 score gains in excess of 3%. Moreover, our technique consistently selects the best model using fewer tries.

## introduction when deploying deep learning in real-life scenarios, training data is often sparse. transfer learning improves learning of such target tasks by leveraging knowledge from a source task, as shown in figure 1 . the improvement in learning could be measured by either improvement in accuracy (for example, f 1 score), or reduction in the time taken to learn the task. with the increased popularity of the large transformer-based models @xcite , transfer learning in the form of fine-tuning a base model is ubiquitous in nlp. however, the performance of the learned target model depends critically on the chosen source model. simply selecting the largest dataset can lead to sub-optimal performance, and trying all sources is computationally expensive. we demonstrate a prediction technique for the sequence labelling task, which given a target model, selects the "best" source model from among a set of available source models, according to a novel and inexpensive metric. then, only that single selected (trained) source model is further fine-tuned on the target dataset. we show that our selection technique is effective at selecting the source model that most improves f 1 score, over nine different tasks, with no additional training. further, our technique results in an average gain of over 3% in f 1 over selecting a base model randomly, and over 4% in f 1 over training a model without transfer learning from any source model. in the rest of the paper, we chronicle prior work in the area of transfer learning, describe the sequence labeling task, and follow with the description of our predictive model selection methodology. ## prior work the transfer learning literature spans different topics and strategies such as few-shot learning @xcite , domain adaptation @xcite , weight synthesis @xcite , and multitask learning @xcite @xcite . some works propose novel combinations of these approaches, in order to improve transfer performance under conditions of domain transfer with limited or incomplete annotations @xcite . some prior research on optimizing source selection has focused on instance transfer techniques, like @xcite al. (2016) and lin et al. (2013) alternatively, bhattacharjee et al. (2020) focuses on predicting the best source model, in the domains of computer vision and of semantic relations, by measuring the similarity between the source and target datasets, using a mix of metrics like kl divergence and source dataset size. in nlp, this approach works for sentence-level classification tasks such as sentiment classification. but it does not adapt well for sequence labelling tasks, where labels span across tokens rather than sentences, and where the similarity of the label spaces need to be accounted for. additionally, their approach requires either the entire source dataset or its feature vector representation to be available. in contrast, the method we demonstrate here only needs the source model itself. ## task definition a sequence labelling task assigns a label to each member of a sequence of observed values. an example is named entity recognition (ner), which identifies in unstructured text all contiguous typed references to task-specific real-world entities, such as persons, organizations, facilities, locations, etc. an example is shown in figure 2 . we formally define the transfer learning task for this paper as follows: given a set m of n source models trained for sequence labelling, @xmath0, m 2 , . . . , m n }, and one target set t, the task is to find the best source model m k , which when used as a base model for transfer learning, would result in a model with highest performance. we use f 1 as the metric to measure performance, and compute relative gain in f 1 to measure the improvement in performance. ## predictive model selection to select the best transfer learning base model, our method compares the target and source using a novel similarity metric. instead of comparing the source and target datasets, our method compares the target test-set with the output of the source model on the target test-set. this comparison takes label weights into account. to compare a target with each of the source models, we decode the target test-set through the source model. we call this output ŷ, and the original target annotation y. we next compare y and ŷ using the metrics described. the source model with the highest metric score is chosen as the best source model. ## conclusion in this paper we present a simple and effective method to predict the best source model for transfer learning in a sequence labelling task, ner. we show our method outperforms popular baselines such as the selection of the largest source, by an average relative 3 f 1 points. and, it is more than average relative 4 f 1 points better than a method that does not use ner transfer learning. moreover, our method consistently selects the best model with fewer tries, saving computational cycles by roughly 75%.