How Far Can We Go with Data Selection? A Case Study on Semantic Sequence Tagging Tasks (2020)

Abstract: Although several works have addressed the role of data selection to improve transfer learning for various NLP tasks, there is no consensus about its real benefits and, more generally, there is a lack of shared practices on how it can be best applied. We propose a systematic approach aimed at evaluating data selection in scenarios of increasing complexity. Specifically, we compare the case in which source and target tasks are the same while source and target domains are different, against the more challenging scenario where both tasks and domains are different. We run a number of experiments on semantic sequence tagging tasks, which are relatively less investigated in data selection, and conclude that data selection has more benefit on the scenario when the tasks are the same, while in case of different (although related) tasks from distant domains, a combination of data selection and multi-task learning is ineffective for most cases.

## introduction transfer learning is a common approach for training nlp models that scale across different tasks, domains, and languages. one of the challenges in transfer learning is to deal with the data distribution mismatch between the source (d s ) and the target data (d t ) @xcite . one solution to alleviate the impact of the mismatch is using data selection, a process for selecting relevant training instances from the source data. data selection (ds) has been applied in the context of domain adaptation to address changes in the data distribution for various nlp tasks, such as sentiment analysis and pos tagging @xcite @xcite , machine translation @xcite , dependency parsing @xcite and named entity recognition (ner) @xcite . to our knowledge, all existing previous works apply data selection to different domains, while maintaining the same task. in this work we aim to investigate the benefit of data selection in a more complex setting, where we have not only different domains (d @xmath0, but also different tasks (t @xmath1. intuitively, such setting may bring advantage in situations where large training data are available for a source task t s , and we want to exploit such data for a different (although related) target task t t , where much less training is available. we experiment with the situation where t s is named entity recognition (ner) on a general domain, where several datasets are available, and t t is slot tagging (st) in the context of utterance interpretation for dialogue systems, where much less data is available. both of the tasks are rarely investigated in data selection and there is no consensus about the benefit of data selection for them. we propose an experimental framework where we can compare data selection settings with an increasing level of complexity. first, we consider data selection where ner is both the source and target task, and apply transfer learning from different domains: we call this setting same tasks from different domains (stdd), t @xmath2. in a second, more complex setting, we consider ner as the source task and st as the target: this is called t @xmath3, different tasks from different domains (dtdd). in this scenario, as we have disjoint label space between the source and the target task, we combine the data selection process with multi-task learning (mtl). to our knowledge, this combination has received very little attention in the literature. we base our work on the data selection framework proposed by @xcite , and apply it to our experimental settings. their framework is model-agnostic and has shown significant advantage in sentiment analysis, pos tagging, and parsing. however, it is not obvious to what extent the selection process can actually help on semantic sequence tagging tasks on stdd and dtdd scenarios. the contributions of the paper are the following: (i) we apply previous work to multi-task learning setup to evaluate the effectiveness of data selection in dtdd scenarios; @xcite we systematically compare data selection on settings of increasing complexity, and observe that existing selection metrics do not show clear advantages over baselines in most cases. nevertheless, data selection has more potential in stdd when source and target are more similar, while combining mtl and data selection for dtdd is ineffective for most cases in our experimental settings in which we have different but related tasks (ner and st) from relatively distant domains (news and conversational domains). ## data selection framework in general, the goal of data selection is to select an optimal subset of training instances, x * s , from all the available data x s in t s , to be used for training the model for the target task m t t . given the source data x @xmath4, x s 2 , ..., x s n }, each instance is ranked according to a score s and the top m examples are then used to train m t t . we apply the data selection approach from ruder and @xcite , based on bayesian optimization (bo) @xcite , to evaluate the effectiveness of data selection on both the stdd and dtdd scenarios. specifically, for dtdd we combine data selection and multi-task learning. given x s , the framework performs data selection based on a score s derived from a set of features. the top m examples are then used to train m t t . in case of stdd, the m t t is a single task sequence tagging model, where we use a bilstm-crf model @xcite . as for dtdd, m t t is a hard parameter sharing mtl model, which has been applied to many nlp tasks @xcite @xcite . the performance on the validation set of the target task is then used by the bo optimizer to update the weight of the scoring features. following ruder and @xcite , the selection process is based on a score s computed as the linear combination of weighted features, which include both similarity and diversity features: s θ (x) = θ • φ(x), where θ represents the weight for each feature and φ(x) denotes the feature values of each instance x. the features are calculated between the representation of x s instances and x t . we use term distribution as the representation of the instances. we use the same similarity and diversity measures as ruder and @xcite . the weights θ are learned through bo by taking into account the performance on the validation set when selecting a particular subset of x s . the score s is computed for each x in x s , and then the top m examples are selected for training the m t t model. the loss value l from the m t t in the validation set is used by bo as a feedback to select the next points for θ. ## experiments we systematically investigate how data selection is effective when applied on both the stdd and dtdd scenarios. we address two semantic sequence labeling tasks: named entity recognition (ner) and slot tagging (st). ## stdd scenario: t this is scenario is the same setup as ruder and @xcite , where we use the same tasks both for the source and the target task from different domains, except that we apply the data selection to a semantic sequence tagging task namely ner. in this scenario, we use ner both for the source and the target task. the target domain is one three ontonotes sections namely nw (news), tc (telephone conversation) and bc (mixed of conversation and broadcast) while as source domain (d s ) we use all available sections in ontonotes except the one used as the target domain. we only use 10% of training data for the target domain to simulate limited data settings. at the end of the data selection process, we select the top 50% sentences from d s using the best feature weights learned with the bayesian optimizer. we want to see whether the distance between domains may characterize the performance of the data selection. for this purpose we quantify the domain similarity between each pair d s and d t with jensen shannon divergence (jsd) @xcite . we compute the jsd between the term distribution of d s and d t . the average jsd of each target task with respect to the source tasks are 0.80 (tc), 0.86 (nw), and 0.87 (bc) foot_3 . we observe that the higher the jsd is, the more beneficial is the data selection for the target task. bc, which has the highest jsd average, benefits the most from the data selection. on the other hand, tc with the lowest average similarity, has the largest gap between the baseline and the best ds methods (-1.7 f1 point). based on our experiments, for the stdd scenario we observe that: 1. in most of the cases, ds methods are inferior to the all source baseline. yet, it is clear that each domain has a different selection metric configuration that performs the best. this observation suggests that the hypothesis from ruder and @xcite i.e., different tasks or even different domains demand a different notion of selection metric, is also applicable to semantic sequence tagging tasks such as ner. 2. the gap between the best ds method and the baseline for each d t can be characterized from ## dtdd scenario: t in this scenario we intend to observe whether data selection adds benefit to mtl. as in the stdd case, data selection is performed on the auxiliary task, where data is assumed to be abundant, and we only use a small portion of data for the target task. we use ner as the auxiliary task and st as the target task. prior work from @xcite shows that ner is helpful for st through mtl, although it is not clear whether adding data selection is beneficial. we follow the setup in @xcite , where ontonotes nw is used as the auxiliary task, and the target task is one of the st datasets with only 10% of available training data. observing the results in our findings and lessons learned for dtdd are the following: 1. we observe that mtl performs better than single-task learning (stl) for low-resource slot tagging, confirming the finding from @xcite . however, adding data selection for mtl is ineffective in our dtdd experimental setup. we hypothesize that mtl learns good common feature representations across tasks, this way inherently helping the model to focus on relevant features even from noisy data in t s . in addition to that, due to data sparsity in limited training, using all the training data works better because the model may learn a better text representation (sentence encoder). recent similar work from schröder and biemann (2020) which uses information theoretic based for estimating the usefulness of an auxiliary task for mtl also found that for semantic sequence tagging tasks such as ner and argument mining, it is less clear when a particular dataset is useful as an auxiliary task. 2. data selection typically produces selected sentences with concentrated similarity distribution foot_4 . therefore, it is probably ineffective when the sentence similarity distribution between t s and t t is already concentrated on a very narrow range. ## conclusion in this paper we investigated the benefit of data selection for transfer learning in several scenarios of increasing complexity. we apply an existing model-agnostic state of the art data selection framework, and carried on experiments on two semantic sequence tagging tasks, ner and slot tagging, and two transfer learning scenarios, stdd (same