Efficient Multi-Task Auxiliary Learning: Selecting Auxiliary Data by Feature Similarity (2021)

Abstract: Multi-task auxiliary learning utilizes a set of relevant auxiliary tasks to improve the performance of a primary task. A common usage is to manually select multiple auxiliary tasks for multi-task learning on all data, which raises two issues: (1) selecting beneficial auxiliary tasks for a primary task is nontrivial; (2) when the auxiliary datasets are large, training on all data becomes time-expensive and impractical. Therefore, this paper focuses on addressing these problems and proposes a time-efficient sampling method to select the data that is most relevant to the primary task. The proposed method allows us to only train on the most beneficial sub-datasets from the auxiliary tasks, achieving efficient multi-task auxiliary learning. The experiments on three benchmark datasets (RTE, MRPC, STS-B) show that our method significantly outperforms random sampling and ST-DNN. Also, by applying our method, the model can surpass fully-trained MT-DNN on RTE, MRPC, STS-B, using only 50%, 66%, and 1% of data, respectively.

## introduction in recent years, language model pre-training has achieved great success in almost all nlp fields @xcite @xcite @xcite . by learning from large corpus text segments without supervision, the models are able to learn the general representation of word tokens and can be further fine-tuned on downstream tasks. moreover, many downstream tasks have their related tasks, which may benefit from the shared information in the training signal. to better utilize the shared knowledge, multi-task learning (mtl) is a common technique. the recent work @xcite @xcite focused on capturing the shared knowledge by learn-ing multiple tasks simultaneously between the pretraining and fine-tuning stage, in order to benefit the downstream tasks. even though the massive mtl scheme is demonstrated to achieve the improvement in terms of performance, it is either time or computing costly; for example, @xcite used over 4.8 million total labeled examples for mtl. on the other hand, in numerous cases, mtl is applied but we only aim at a single task performance. it is usually addressed as multi-task auxiliary learning, which targets to introduce auxiliary tasks and datasets to boost the performance of the primary task @xcite @xcite . in this scenario, how to wisely select the auxiliary tasks plays the most important role. one straightforward method is to select the auxiliary tasks according to their relatedness to the primary task. however, selecting the "related" tasks is non-trivial. to address to this, @xcite proposed autosem, which learns to automatically select auxiliary tasks and decide the mixing ratio of auxiliary data via a beta-bernoulli multi-armed bandit with thompson sampling and gaussian process, respectively. however, their method cannot decide the specific data samples to use in one auxiliary task, and the sampling approach is extremely time-consuming due to the numerous steps needed to solve the non-stationary multi-armed bandit problem, failing to address the issue of efficiency in mtl. in this paper, we propose a similarity-based sampling method, along with a two-stage mtl pipeline for efficient multi-task auxiliary learning. the experiments on the glue @xcite benchmark show that the proposed method outperforms single-task models, mt-dnn with random sampling, and even fully-trained mt-dnn with full auxiliary data. the analysis also demonstrates the effectiveness and efficiency of our proposed twostage mtl pipeline. this paper has three-fold contributions: ## proposed method this paper proposes a two-stage mtl pipeline for efficient multi-task auxiliary learning. in the first stage, we introduce similarity sampling, a simple and time-efficient sampling strategy to select the most beneficial data samples from the auxiliary tasks to benefit the primary task. in the second stage, we train the mt-dnn model using the selected auxiliary data similar to @xcite . after training, we fine-tune the model on the primary task to further optimize the performance. ## experiments to evaluate the proposed method, we conduct the experiments detailed below. ## discussion to better investigate whether the selected samples are more beneficial than others, we investigate the in-task efficacy of our method. also, considering that multi-task learning often benefits more the lowresource tasks by preventing overfitting through learning other tasks, we analyze the performance of the proposed method in the low-resource scenarios. furthermore, we conduct ablation experiments to ## conclusion this paper introduces a novel two-stage multi-task auxiliary learning framework that utilizes similarity sampling to select the most beneficial auxiliary data for efficiently training an mt-dnn model. our experiments on benchmark glue datasets demonstrate that our proposed method outperforms random sampling and further surpasses the fullytrained mt-dnn with significantly fewer data and time. moreover, we show that our selected samples are the most beneficial data in the auxiliary task and that the proposed method works much better when few-shot scenarios, proving the strong in-task efficacy and the great potential of practical usage.