G rad TS : A Gradient-Based Automatic Auxiliary Task Selection Method Based on Transformer Networks (2021)

Abstract: A key problem in multi-task learning (MTL) research is how to select high-quality auxiliary tasks automatically. This paper presents GradTS, an automatic auxiliary task selection method based on gradient calculation in Transformer-based models. Compared to AUTOSEM, a strong baseline method, GradTS improves the performance of MT-DNN with a bert-base-cased backend model, from 0.33% to 17.93% on 8 natural language understanding (NLU) tasks in the GLUE benchmarks. GradTS is also time-saving since (1) its gradient calculations are based on single-task experiments and (2) the gradients are re-used without additional experiments when the candidate task set changes. On the 8 GLUE classification tasks, for example, GradTS costs on average 21.32% less time than AUTOSEM with comparable GPU consumption. Further, we show the robustness of GradTS across various task settings and model selections, e.g. mixed objectives among candidate tasks. The efficiency and efficacy of GradTS in these case studies illustrate its general applicability in MTL research without requiring manual task filtering or costly parameter tuning.

## introduction mtl @xcite is widely used in nlu research to improve the performance of machine learning (ml) models by enlarging the training data size with datapoints related to the primary tasks. however, its efficacy is largely affected by the selection of auxiliary tasks. the auxiliary task selection problem is addressed mainly under two settings. the first setting treats each task as a whole. for example, @xcite assess task relatedness by exhaustive experiments in all † work done when interning at the minds, machines, and society lab at dartmouth college. task pairs. nonetheless, high pairwise task correlations are often not decisive features for choosing auxiliary tasks. @xcite train a policy for task selection through counterfactual estimation, but their learned policy brings improvements only to one out of nine tasks on glue benchmarks @xcite . the second setting subsamples training instances from auxiliary tasks, e.g. with bayesian optimization @xcite , but these methods are time-and resource-consuming due to their reliance on multitask experiments involving all the candidate tasks. autosem @xcite combines the two settings into one method, selecting candidate tasks with thompson sampling and deciding the ratio with which to draw training instances from the selected tasks via a gaussian process. despite the higher quality of the auxiliary task sets it generates, autosem is still costly, similar to @xcite . to design a better-performing and less costly auxiliary task selection method, we take advantage of the characteristics of transformer networks @xcite . prior research reveals that in a transformer-based model, each attention head attends on specialized linguistic features @xcite @xcite @xcite . since important linguistic features strongly correlate with the goals of tasks, we further hypothesize that a good auxiliary task shares key linguistic features with the primary task. thus, we address the auxiliary task selection problem by maximizing the overlap of important heads in a transformer-based model between primary and auxiliary tasks. as @xcite claim, the importance of attention heads to a task can be approximated by the absolute gradients accumulated at each head. we design our auxiliary task selec-tion method, gradts, accordingly, by ranking the importance of attention heads for each individual task and modeling the correlation between each pair of tasks with their head rankings. by greedily selecting the tasks most closely related to the primary task, gradts constructs auxiliary task sets through trial experiments @xcite . gradts also enables task subsampling to further optimize auxiliary task sets. to achieve this goal, we design another setting of gradts (gradts-fg) that first assesses the correlations between the primary task and each training instance in an auxiliary task selected by gradts-trial and then filters the training instances via thresholding. we assess the strength of gradts via mtl evaluations on 8 glue classification tasks. we use autosem and autosem-p1 foot_0 as our baselines since autosem is among the most advanced auxiliary task selection methods in the nlp field and because it features both task selection and task subsampling. for consistency, we use the bert-basecased model as the backend model of gradts, au-tosem, and the mtl framework. results show that gradts-trial produces better auxiliary task sets than autosem-p1 in all 8 glue tasks while costing on average 6.73% less time. in experiments with task subsampling, gradts-fg again shows superior strength to autosem on all 8 tasks while costing 21.32% less time. these results strongly support the efficacy and efficiency of gradts. in addition to the main experiments, we compare gradts to multiple intuitive auxiliary task selections to show its high performance. we also conduct case studies to show that gradts is effective and robust on difficult tasks or candidate tasks with mixed objectives. these findings reflect the general applicability of gradts in various task settings. in comparison, auxiliary task sets produced by autosem and autosem-p1 are often not optimal in these complicated scenes. further, gradts reuses the head rankings when the candidate task set grows larger, which makes it even more timeand resource-efficient than existing methods. the contributions of this paper are three-fold: • we propose gradts, an automatic auxiliary task selection method based on gradient calculation in pre-trained transformer-based models; • we illustrate the efficacy and efficiency of gradts through comprehensive mtl evalua-tions; and • we show, through case studies, the superior capability and robustness of gradts to complicated candidate task settings compared to both au-tosem and auxiliary task selections based on human intuition. 2 datasets datasets obj lbl training size cola cls 2 8,550 mrpc cls 2 3,667 mnli cls 3 392,701 qnli cls 2 104,742 qqp cls 2 363,845 rte cls 2 2,489 sst-2 cls 2 67,348 wnli cls 2 634 stsb rgr -5,748 pos sl -14,040 ner sl -14,987 sc sl -8,936 meld cls 7 9,988 dyadic-meld cls 7 12,839 following @xcite , we use the 8 classification tasks in glue benchmarks @xcite , namely cola, mrpc, mnli, qnli, qqp, rte, sst-2, and wnli, in our main experiments. we apply the standard split of these datasets as @xcite describe. 2 we also use one regression and three sequence labeling tasks in our case studies about the efficacy of gradts on candidate tasks with mixed training objectives. these tasks include stsb from glue benchmarks, part-of-speech tagging (pos) from universal dependencies 3 , named entity recognition (ner) from conll-2003 challenges @xcite , and syntactic chunking (sc) from conll-2000 shared tasks 2 we report scores on the development set of glue tasks due to the submission quota limit. we sample 10% training instances of each glue task with a random seed of 42 for choosing thresholds and selecting auxiliary tasks. details of the datasets are displayed in ## methodology we design gradts based on the hypothesis that better auxiliary tasks share more important linguistic features with the primary task. since each attention head in a transformer-based model functions similarly as a standalone feature extractor on a specialized set of features, we approximate the important feature set of each task by the heads contributing the most to the task. as the key feature sets are task-specific, gradts does not require multi-task experiments to rank auxiliary tasks given a primary task. this makes gradts a time-and resourceeconomic method especially when the set of candidate auxiliary tasks is large or growing. gradts consists of three successive modules responsible for (1) ranking attention heads for a task based on their contributions, (2) ranking auxiliary tasks based on inter-task correlations, and (3) finalizing the auxiliary task sets, respectively. ## discussions gradts is shown to be effective on 8 classification nlu tasks in our main experiments. in this section, we conduct case studies to ( 1 ) explore whether gradts is effective on tasks that are difficult or have different training objectives, (2) validate that gradts selects better auxiliary task sets than human intuition, and (3) justify our use of bert-basecased as the backend model of gradts and the mtl evaluation framework. ## base model selection for gradts since gradts is built on transformer-based models, we select its backend model from 6 common pre-trained transformer-based models, namely bert-base-uncased, bert-base-cased, bertlarge-uncased, bert-large-cased, roberta-base, and roberta-large. we set up mtl evaluations with mt-dnn on cola, mrpc, sst-2, wnli, qnli, and rte to examine the appropriateness of these backend models in gradts. specifically, we assess the strength and robustness of these models by comparing the mt-dnn performance with auxiliary task sets selected by gradts against that without auxiliary task filtering. the same backend model is used for gradts and mt-dnn in each experiment to eliminate possible discrepancies across models. from ## conclusion and future work this paper presented gradts, an automatic auxiliary task selection method for mtl based on pretrained transformer-based models. on 8 glue classification tasks, gradts produced auxiliary task sets higher in quality than autosem, a strong baseline method, with less time and resource consumption. in our case studies comparing gradts to intuitive task selections, gradts showed greater capability of finding more optimal auxiliary task sets than using trivial heuristics based on statistics of datasets. we additionally demonstrated that gradts was both more effective and more efficient than our baseline on task sets with mixed objectives. gradts was also shown to be robust in task sets containing difficult tasks for its backend model. these findings support the applicability of gradts to a wide range of task and model settings. future work may extend the use of gradts to determine high-quality auxliary datasets for the same task. we show the head importance matrix of the bertbase-cased model on mrpc in figure 3 . the task correlation matrix generated by bert-base-cased on the 8 glue classification tasks is shown in figure 4 .