Research on Task Discovery for Transfer Learning in Deep Neural Networks (2020)

Abstract: Deep neural network based machine learning models are shown to perform poorly on unseen or out-of-domain examples by numerous recent studies. Transfer learning aims to avoid overfitting and to improve generalizability by leveraging the information obtained from multiple tasks. Yet, the benefits of transfer learning depend largely on task selection and finding the right method of sharing. In this thesis, we hypothesize that current deep neural network based transfer learning models do not achieve their fullest potential for various tasks and there are still many task combinations that will benefit from transfer learning that are not considered by the current models. To this end, we started our research by implementing a novel multi-task learner with relaxed annotated data requirements and obtained a performance improvement on two NLP tasks. We will further devise models to tackle tasks from multiple areas of machine learning, such as Bioinformatics and Computer Vision, in addition to NLP.

## introduction deep neural network based machine learning models have shown remarkable progress in the last decades across a wide range of tasks. the typical training regime uses a large amount of labeled data to get a general mapping of the elements in the input space to the label space, which is known as supervised learning. yet, it is shown by numerous studies that these models suffer from overfitting and are sensitive to noise and examples that are not available in the training data @xcite . in addition, these models are usually trained from scratch for each new task where the weights of the models are initialized randomly. this approach does not follow the way humans learn new tasks, i.e. leveraging external world knowledge and information obtained from related tasks when learning a new task @xcite . transfer learning (tl) is a biologically motivated training paradigm that aims to mitigate the above mentioned real-world challenges of conventional supervised learning @xcite . signals in the training set of a source task are used as additional information for a given target task to enable better generalization. it is especially useful when the labeled data is limited for the target task and when the tasks are relatively similar @xcite @xcite . learning the structure among tasks is an essential first step to benefit most from transfer learning, and to this end @xcite proposed a fully-computational framework to learn this structure in the computer vision domain. straightforward application of transfer learning algorithms may lead to catastrophic forgetting where models forget the source task after being exposed to the target task. in addition, there is a lack of theoretical understanding of the task relationships, and as a result, tasks for transfer learning are usually determined with hindsight. multi-task learning (mtl) is a special case of transfer learning where multiple tasks are learned simultaneously. @xcite summarizes multitask learning as leveraging information obtained from the training data of different tasks to improve generalization. it enables better generalization and lowers the annotated data requirements @xcite . current multitask learning systems typically use hard-sharing, where a low layer hidden representation is shared among all tasks to have an inductive bias @xcite . it is recently shown that for dissimilar tasks hard-sharing may degrade the performance, which is also called negativetransfer @xcite . more sophisticated information sharing methodologies must be consid-ered in addition to finding useful task combinations, to make the most out of multi-task learning and to avoid negative-transfer. the above findings and challenges motivate our research on transfer learning in deep neural networks. specifically, we focus our research on investigating the task relations on the currently proposed models and on proposing new task combinations. through our research, we plan to find answers to 'where to transfer from' (task selection), 'what to transfer' (datasets and data selection) and 'how to transfer' (pretraining and model architecture). our main hypothesis is that, 1) neural network based transfer learning models improve over their singletask counterparts both in terms of generalizability and overall performance, 2) currently proposed transfer learning models do not achieve their fullest potential, and 3) there are many task combinations that will benefit from transfer learning. we will focus on the following research questions about transfer learning models throughout this thesis: rq1. how to optimize the model architecture and sharing methodology for a given task combination? rq2. what are some good auxiliary tasks to improve the perfomance of a target task? rq3. how to find useful pretraining schemes? the first question aims to find the most useful architecture and the sharing methodology when the task combination is known/determined. second is a higher-level research question to find useful task combinations and can be considered as the preliminary step for the first one. finally, question three aims to find the right pretraining scheme to make the most ouf of transfer learning for a given set of target tasks. by combining these research questions, we aim to find the most useful multi-task learning setting for a given domain. we started our research by analyzing the limitations of current supervised learning systems and showed the sensitivity of neural network based models to the changes in the domain @xcite . next, we proposed a novel joint learning model that relaxes labeled data requirements for the named entity recognition and dependency parsing tasks and showed improvements over the conventional methods. the results for the model are given in more detail in section 4. we will further devise models to tackle tasks from multiple areas of machine learning, such as bioinformatics and computer vision (cv) in addition to nlp. specifically, we plan to focus on biomedical question answering and object detection tasks from bioinformatics and cv areas, respectively. we motivate the choice of these two domains as follows: transfer learning with imagenet achieved a huge success, and almost all state-of-the-art models for downstream tasks in cv make use of transfer learning. the abundance of transfer learning based models makes cv a good test-domain for evaluating the contributions we will propose for different pretraining schemes for transfer learning. on the contrary, applications of transfer learning is scarce in bioinformatics compared to cv and nlp. hence, there should be various task combinations that can benefit from transfer learning in the bioinformatics domain that were not investigated before. this motivated us to choose bioinformatics as a target domain to find new task combinations. the remainder of this paper is structured as follows. section 2 gives a summary of the related work on transfer learning and multi-task learning. this is followed by the research plan, where we explain the methodology we will use regarding each research question. finally, section 4 describes the evaluation methods and datasets that will be used to assess the significance of our contributions regarding each research question. ## related work our research is related to the works in the subtopics we summarize below. ## research plan in this section, we restate the research questions and explain the approach we are planning to take. rq1: how to optimize the model architecture and sharing methodology for a given task combination? currently proposed multi-task learners mostly use hard sharing, where models share a common low-level layer, and task-specific sharing methods are not analyzed for many task combinations @xcite @xcite . following @xcite , we plan to use learnable parameters to control the information flow between each task-specific model. learning joint label embeddings for disparate label classes @xcite is another promising approach that goes beyond hardsharing. specifically, we will apply this method to leverage our previously proposed joint learner for dependency parsing and named entity recognition. part-of-speech tags strongly correlate with named entities and dependencies @xcite . thus, we argue that learning joint label embeddings of these tasks can help to further capture the relations between them. rq2: what are some good auxiliary tasks to improve the performance of a target task? regarding this research question, we will fix a target task and try to improve the performance by 1) incorporating a transfer learning framework and 2) applying a more sophisticated data selection mechanism. to better understand the task relations (where to transfer from), we will compare the performance on a fixed target task using several auxiliary tasks obtained through different task selection mechanisms. @xcite proposed pretraining the bert model in the biomedical domain and apply the model to make predictions in several different downstream tasks in bioinfomatics such as genedisease relation extraction and biological named entity recognition. we argue that their approach can be combined with multi-task learning to further leverage the information available in the dataset of each task. specifically, we claim that biological named entity recognition can be used as an auxiliary task to improve the performance of biological question answering systems. our preliminary results are given in section 4. the biological named entity dataset consists of several types of entities (genes, chemicals and disease mentions) and each type can be considered as a different task. we will use these set of tasks to compare the performance of the task selection mechanisms. deciding which data are useful (what to transfer), in addition to finding promising task combinations, is another key factor to increase transferability @xcite . however, many of the current multi-task models use all the available data for all tasks @xcite @xcite . to this end, we will apply the previously proposed data selection mechanisms on our new task combinations to find the most useful and relevant examples from each dataset to improve the transferability and to avoid negative-transfer. previous work on data selection successfully showed that using a bayesian suite for deciding which data to use for multi-task learning brings significant improvements @xcite . this motivated us to incorporate similar data selection mechanisms to further improve the performance of transfer learning models. we will compare several data selection mechanisms by fixing the model to be used and the task combination. rq3: how to find useful pretraining schemes? the standard approach in sequential transfer learning is to pretrain a model using an objective that is relevant to and useful for the target task. in nlp, the prevailing method is to train a language model using the next sentence prediction and masked token prediction objectives over huge unlabeled datasets, e.g. the bert model @xcite . the pretrained models are usually fine-tuned on task-specific datasets, yet the characteristics of the downstream task are usually not considered during the pretraining process. regarding this research question, our main goal is to find task-specific pretraining schemes and to compare the performance with fine tuned models that are not pretrained considering the downstream task @xcite . curriculum learning aims to find a good ordering of the training samples to go beyond random sampling @xcite . the training samples are ordered according to their difficulties using prior knowledge. recently, @xcite proposed self paced curriculum learning which tries to learn this ordering dynamically during training to mitigate the drawbacks of defining static difficulties for training samples using external knowledge. following this idea of changing the difficulty of the training samples @xcite @xcite , we propose using 'adaptive masking' for pretraining language models. the standard approach for pretraining with masked language modeling involves predicting the distribution of a randomly masked word using its context @xcite . each masked word can be considered as an instance of a cloze test which is frequently used to assess the linguistic skills in humans. in a cloze test, students are expected to understand the context to fill in the masked word. randomly selecting which words to mask causes the difficulty of each instance to change randomly as well. we propose adaptively changing the difficulty of the next training instance by observing the performance of the model. in this context, we define difficulty as the amount of contextual information necessary to select the most probable word, whereas @xcite defined difficulty as the inverse of the frequency of each masked token regardless of their contexts. ## evaluation in order to evaluate the significance of our contributions, we will do evaluations for each research question separately. below we give the evaluation methodology, together with example tasks and the related datasets that will be used for each research question. ## summary transfer learning is a promising area of research for deep neural network based machine learning models. it helps achieve better generalization and utilization of the training datasets. in this paper, we pointed out the current key challenges and unsolved problems: 1) going beyond the conventional way of hard-sharing in multi-task learning and finding the most useful architecture for a given setting, 2) finding good auxiliary tasks in a multi-task setting for a specific target task, and 3) finding useful pretraining schemes. our research aims to apply the current work on transfer learning to new tasks and also find novel methods to obtain better multitask learning models.