To Share or not to Share: P redicting Sets of Sources for Model Transfer Learning (2021)

Abstract: In low-resource settings, model transfer can help to overcome a lack of labeled data for many tasks and domains. However, predicting useful transfer sources is a challenging problem, as even the most similar sources might lead to unexpected negative transfer results. Thus, ranking methods based on task and text similarity — as suggested in prior work — may not be sufficient to identify promising sources. To tackle this problem, we propose a new approach to automatically determine which and how many sources should be exploited. For this, we study the effects of model transfer on sequence labeling across various domains and tasks and show that our methods based on model similarity and support vector machines are able to predict promising sources, resulting in performance increases of up to 24 F1 points.

## introduction for many natural language processing applications in non-standard domains, only little labeled data is available. this even holds for high-resource languages like english @xcite . the most popular method to overcome this lack of supervision is transfer learning from high-resource tasks or domains. this includes the usage of resources from similar domains @xcite , domainspecific pretraining on unlabeled text @xcite , and the transfer of trained models to a new domain @xcite . while having the choice among various transfer sources can be advantageous, it becomes more challenging to identify the most valuable ones as many sources might lead to negative transfer results, i.e., actually reduce performance @xcite . current methods to select transfer sources are based on text or task similarity measures @xcite . the underlying assumption is that similar texts and tasks can support each other. an example for similarity based on vocabulary overlap is shown in figure 1 . however, current methods typically consider text and task similarity in isolation, which limits their application in transfer settings where both the task and the text domain change. thus, as a first major contribution, this paper proposes a new model similarity measure that represents text and task similarity jointly. by learning a mapping between two neural models, it captures similarity between domain-specific models across tasks. we perform experiments for different transfer settings, namely zero-shot model transfer, supervised domain adaptation and cross-task transfer across a large set of domains and tasks. our newly proposed similarity measure successfully predicts the best transfer sources and outperforms existing text and task similarity measures. as a second major contribution, we introduce a new method to automatically determine which and how many sources should be used in the transfer process, as the transfer can benefit from multiple sources. our selection method overcomes the limitations of current transfer methods, which solely predict single sources based on rankings. we show the benefits of transfer from sets of sources and demonstrate that support vector machines are able to predict the best sources across domains and tasks. this improves performance with absolute gains of up to 24 f 1 points and effectively prevents negative transfer. the code for our sequence taggers, prediction methods and the results are publicly available. foot_0 2 related work domain adaptation & transfer learning are typically performed by transferring information and knowledge from a high-resource to a low-resource domain or task @xcite @xcite . recent approaches can be divided into two groups: (i) model transfer @xcite by reusing trained task-specific weights @xcite or by first adapting models on the target domain before training the downstream task @xcite and (ii) multi-task training @xcite where multiple tasks are trained jointly by learning shared representations @xcite . we follow the first approach in this paper. for transfer learning, the selection of sources is utterly important. text and task similarity measures @xcite are used to select the best sources for crosstask transfer @xcite , multi-task transfer @xcite , cross-lingual transfer @xcite and language modeling @xcite . alternatively, neural embeddings for corpora can be compared @xcite . in prior work, the set of domains is usually limited and the focus is on the single-best source. in contrast, we exploit sources from a larger set of domains and also explore the prediction of sets of sources, as using multiple sources is likely to be beneficial, as also shown by parvez and chang (2021) contemporaneously to this work. ## similarity measures and predictors in this section, we describe the sequencer tagger model and similarity measures along with metrics for the evaluation. finally, we introduce our new prediction method for sets of transfer sources. ## experimental setup in this section, we introduce the tasks, datasets and transfer settings used in our experiments. ## results this section presents the results of the different transfer settings and analyzes how similarity measures can be used to predict transfer sources. ## conclusion we explored different transfer settings across three sequence-labeling tasks and various domains. our new model similarity measure based on feature mappings outperforms currently used similarity measures as it is able to capture both task and domain similarity at the same time. we further addressed the automatic selection of sets of sources as well as the challenge of potential negative transfer by proposing a selection method based on support vector machines. our method results in performance gains of up to 24 f 1 points.