Evaluating the Values of Sources in Transfer Learning (2021)

Abstract: Transfer learning that adapts a model trained on data-rich sources to low-resource targets has been widely applied in natural language processing (NLP). However, when training a transfer model over multiple sources, not every source is equally useful for the target. To better transfer a model, it is essential to understand the values of the sources. In this paper, we develop , an efficient source valuation framework for quantifying the usefulness of the sources (e.g., ) in transfer learning based on the Shapley value method. Experiments and comprehensive analyses on both cross-domain and cross-lingual transfers demonstrate that our framework is not only effective in choosing useful transfer sources but also the source values match the intuitive source-target similarity.

## introduction transfer learning has been widely used in learning models for low-resource scenarios by leveraging the supervision provided in data-rich source corpora. it has been applied to nlp tasks in various settings including domain adaptation @xcite , cross-lingual transfer @xcite , and task transfer @xcite . a common transfer learning setting is to train a model on a set of sources and then evaluate it on the corresponding target @xcite . 1 however, not every source corpus contributes equally to the transfer model. some of them may even cause a performance drop @xcite . therefore, it is essential to understand the value of each source in the transfer learning not only to achieve 1 in this paper, we focus on two transfer learning scenarios: 1) cross-lingual and 2) cross-domain. we train a model on a set of source corpora and evaluate on a target corpus where each "corpus" refers to the corresponding domain or language. transfer performance transfer performance transfer performance transfer performance transfer performance transfer performance 0.45 0.45 0.95 0.85 0.40 0.80 = 𝟎.𝟒𝟓%𝟎.𝟒𝟓 𝟏 + 𝟎.𝟗𝟓%𝟎.𝟖𝟓 𝟐 + 𝟎.𝟒𝟎%𝟎.𝟖𝟎 𝟐 + 𝟎.𝟒𝟎%𝟎.𝟎𝟓 𝟏 source value small evaluation target corpus source values 𝝓 = {𝝓1, 𝝓2 ,..., 𝝓m} source value (𝝓1) source value (𝝓2) source value (𝝓m) transfer performance transfer performance 0.40 0.05 seal-shap source corpora 𝕯 = {𝑫1, 𝑫2 ,..., 𝑫m} source corpus (𝑫1) source corpus (𝑫2) source corpus (𝑫m) based on source values, select potential subset of the source corpora high transfer performance on test target corpus 0.97 example source corpus for which we elaborate the source value calculation subsampled from the original large source corpus ## source valuation framework we propose seal-shap, a source valuation framework. we start with the setting where we have only one target and multiple sources. we denote the target corpus by v and the corresponding set of source corpora by @xmath0, • • • , d m }. our goal is to quantify the value φ j of each source corpus d j to the transfer performance on v and explain model behaviors. once the source values are measured, we can then develop a method to select either all the sources or a subset of sources (i.e., ⊆ d) that realizes a good transfer accuracy on v . below, we first review the data shapley value and its adaptation for transfer learning. then, we describe how seal-shap efficiently quantifies φ j and how to use it to select a subset of sources for model transfer. ## experimental settings we conduct experiments on zero-shot cross-lingual and cross-domain transfer settings. models are trained only on the source languages/domains and directly applied in target languages/domains. ## results and discussion in the following, we first verify seal-shap is an effective tool for source valuation. then, we evaluate the source values when an evaluation target corpus is unavailable. in sec 4.3, we interpret the relations between sources and targets based on the seal-shap values. finally, we analyze our method with comprehensive ablation studies. ## related work as discussed in section 1, transfer learning has been extensively studied in nlp to improve model performance in low-resource domains and languages. in the litearture, various approaches have been proposed to various tasks, including text classification @xcite , natural language inference @xcite , sequence tagging @xcite @xcite , dependency parsing @xcite . these prior studies mostly focus on bridging the domain gap between sources and targets. in different contexts, methods including influence functions and shapley values have been applied to value the contribution of training data @xcite @xcite . specifically, monte carlo approximation of shapley values has been used in various applications @xcite @xcite @xcite . however they are either task/model specific or not scalable to nlp applications. oppositely, @xcite discuss the problems of using shapley value for model explanation. in contrast, we apply efficient shapley value approximation in nlp transfer learning and analyze the source-target relationships. ## conclusion we propose seal-shap to quantify the value of the source corpora in transfer learning for nlp by computing an approximate shapley value for each corpus. we show that seal-shap can be used to select source corpora for transfer and provide insight on understanding the value of source corpora. in the future, we plan to further improve the runtime of our source valuation approach by limiting the repetition of model training.