W hat to Pre-Train on? E fficient Intermediate Task Selection (2021)

Abstract: Intermediate task fine-tuning has been shown to culminate in large transfer gains across many NLP tasks. With an abundance of candidate datasets as well as pre-trained language models, it has become infeasible to experiment with all combinations to find the best transfer setting. In this work, we provide a comprehensive comparison of different methods for efficiently identifying beneficial tasks for intermediate transfer learning. We focus on parameter and computationally efficient adapter settings, highlight different data-availability scenarios, and provide expense estimates for each method. We experiment with a diverse set of 42 intermediate and 11 target English classification, multiple choice, question answering, and sequence tagging tasks. Our results demonstrate that efficient embedding based methods, which rely solely on the respective datasets, outperform computational expensive few-shot fine-tuning approaches. Our best methods achieve an average Regret@3 of 1% across all target tasks, demonstrating that we are able to efficiently identify the best datasets for intermediate training.

## introduction large pre-trained language models (lms) are continuously pushing the state of the art across various nlp tasks. the established procedure performs self-supervised pre-training on a large text corpus and subsequently fine-tunes the model on a specific target task @xcite . the same procedure has also been applied to adapter-based training strategies, which achieve on-par task performance to full model fine-tuning while being considerably more parameter efficient @xcite and faster to train @xcite . 2 besides being more efficient, adapters are also highly modular, enabling a wider range of transfer learning techniques @xcite @xcite @xcite . extending upon the established two-step learning procedure, incorporating intermediate stages of knowledge transfer can yield further gains for fully fine-tuned models. for instance, @xcite sequentially fine-tune a pre-trained language model on a compatible intermediate task before target task fine-tuning. it has been shown that this is most effective for low-resource target tasks, however, not all task combinations are beneficial and many yield decreased performances @xcite @xcite . the abundance of diverse labeled datasets as well as the continuous development of new pre-trained lms calls for methods that efficiently identify intermediate dataset that benefit the target task. so far, it is unclear how adapter-based approaches behave with intermediate fine-tuning. in the first part of this work, we thus establish that this setup results in similar gains for adapters, as has been shown for full model fine-tuning @xcite @xcite . focusing on a low-resource target task setup, we find that only a subset of intermediate adapters yield positive gains, while others hurt the performance considerably (see in the second part, we leverage the transfer results from part one to automatically rank and identify beneficial intermediate tasks. with the rise of large publicly accessible repositories for nlp models @xcite , the chances of finding pre-trained models that yield positive transfer gains are high. however, it is infeasible to brute-force the identification of the best intermediate task. existing approaches have focused on beneficial task selection for multi-task learning @xcite , full fine-tuning of intermediate and target transformer-based lms for nlp tasks @xcite , adapter-based models for vision tasks @xcite and unsupervised approaches for zero-shot transfer for community question answering @xcite . each of these works require different types of data, such as intermediate task data and/or intermediate model weights, which, depending on the scenario, are potentially not accessible. 3 in this work we thus aim to address the efficiency aspect of transfer learning in nlp from multiple different angles, resulting in the following contributions: 1) we focus on adapter-based transfer learning which is considerably more parameter @xcite and computationally efficient than full model fine-tuning @xcite , while achieving on-par performance; 2) we evaluate sequential fine-tuning of adapter-based approaches on a diverse set of 42 intermediate and 11 target tasks (i.e. classification, multiple choice, question answering, and sequence tagging); 3) we identify the best intermediate task for transfer learning, without the necessity of computational expensive, explicit training on all potential candidates. we compare different selection techniques, consolidating previously proposed and new methods; 4) we provide a thorough analysis of the different techniques, available data scenarios, and task-, and model types, thus presenting deeper insights into the best approach for each respective setting; 5) we provide computational cost estimates, enabling informed decision making for trade-offs between expense and downstream task performance. while previous work has shown that intermediate task training improves the performance on the target task in full fine-tuning setups, we establish that the same holds true for adapter-based training. ## adapter-based sequential transfer we present a large-scale study on adapter-based sequential fine-tuning, finding that around half of the task combinations yield no positive gains. this demonstrates the importance of finding approaches that efficiently identify suitable intermediate tasks. ## methods for the efficient selection of intermediate tasks we now present different model selection methods, and later in §5, study their effectiveness in our setting outlined above. we group the different methods based on the assumptions they make with regard to the availability of intermediate task data d s and intermediate models m s . access to both can be expensive when considering large pretrained model repositories with hundreds of tasks. ## experimental setup we evaluate the approaches of §4, each having the objective to rank the intermediate adapters @xmath0. we leverage the transfer performance results of our 462 experiments obtained in §3 for our ranking task. ## experimental results access to only d s or m s . these methods typically perform better than our baselines. textemb and semb perform on par in most cases. 11 while fsft outperforms the other approaches in most cases, it comes at the high cost of requiring downloading and fine-tuning all intermediate models for a few steps. this can be prohibitive if we consider many intermediate tasks. if we have access to textemb or semb information of the intermediate task (i.e., individual vectors distributed as part of a model repository), these techniques yield similar performances at a much lower cost. access to both d s and m s . assuming the availability of both intermediate models and intermediate data is the most prohibitive setting. surprisingly, we find bert and roberta to behave considerably differently, especially evident for qa tasks. as shown by @xcite , taskemb performs very well for bert, however we find that the results of this gradient based approach do not translate to roberta. while these approaches perform best or competitively for all task types using bert, they considerably underperform all methods when leveraging pre-trained roberta weights. here, the two much simpler domain embedding methods outperform the taskemb method based on the fim. summary. we find that simple indicators such as domain similarity are suitable for selecting intermediate pre-training tasks for both bert and roberta based models. our evaluated methods are able to efficiently select the best performing intermediate tasks with a regret@3 of 0.0 in many cases. our results, thus, show that the selection methods are able to effectively rank the top tasks with relative certainty, thus considerably reducing the number of necessary experiments. 12 7 analysis computational costs. in addition to the complexity, we calculate the required multiply-accumulate computations (mac) for 42 intermediate tasks and one target task with 1000 training examples, each with an average sequence length of 128. 14 following our experi- method complexity macs metadata 1 0 textemb/ semb f 1.10e+13 taskemb (e + 1)f + eb 3.30e+14 knn/ linear nf 4.61e+14 fsft/ fs-taskemb 2nef + neb 1.38e+15 ## conclusion in this work we have established that intermediate pre-training can yield gains in adapter-based setups, however, around 44% of all transfer combinations result in decreased performances. we have consolidated several existing and new methods for efficiently identifying beneficial intermediate tasks. experimenting with different model types, we find that the previously proposed best performing approaches for bert do not translate to roberta. overall, efficient embedding based methods, such as those relying on pre-computable sentence representations, perform better or often on-par with more expensive approaches. the best methods achieve a regret@3 of less than 1% on average, demonstrating that they are effective at efficiently identifying the best intermediate tasks. the approaches evaluated and proposed in this work, thus, enable the automatic identification of beneficial intermediate tasks, deeming exhaustive experimentation on many task-combinations unnecessary. when applied on a broad scale, these methods can contribute to more sustainable @xcite and more inclusive @xcite natural language processing. xiang zhang, junbo jake zhao, and yann lecun. 2015. character-level convolutional networks for text classification. in advances in neural information processing systems 28: annual conference on neural information processing systems 2015, december 7-12, 2015, montreal, quebec, canada, pages 649-657.