Task Compass: Scaling Multi-task Pre-training with Task Prefix (2022)

Abstract: Leveraging task-aware annotated data as supervised signals to assist with self-supervised learning on large-scale unlabeled data has become a new trend in pre-training language models. Existing studies show that multi-task learning with large-scale supervised tasks suffers from negative effects across tasks. To tackle the challenge, we propose a task prefix guided multi-task pre-training framework to explore the relationships among tasks. We conduct extensive experiments on 40 datasets, which show that our model can not only serve as the strong foundation backbone for a wide range of tasks but also be feasible as a probing tool for analyzing task relationships. The task relationships reflected by the prefixes align transfer learning performance between tasks. They also suggest directions for data augmentation with complementary tasks, which help our model achieve human-parity results on commonsense reasoning leaderboards. Code is available at https://github.com/cooelf/CompassMTL.

## introduction recent years have witnessed a growing interest in leveraging a unified pre-trained language model (prlm) to solve a wide range of natural language processing tasks @xcite @xcite . the pre-training recipe of a prlm is driving from self-supervised learning @xcite @xcite @xcite to multi-task learning (mtl) with a mixture of standard selfsupervised tasks and various supervised tasks, * work done when zhuosheng zhang and wenhao yu interned at microsoft cognitive services research group. this work was partially supported by key projects of national natural science foundation of china (u1836222 and 61733011). ## experiments 4.1 datasets there are 40 datasets used for training our multitask model, some of which are collected from glue @xcite , superglue @xcite , rainbow @xcite , and lexglue @xcite . figure 3 illustrates the composition of our task families. glue glue (the general language understanding evaluation benchmark) @xcite is a collection of 9 various tasks for sentence-level classification. we only use 8 of them: cola @xcite , sst-2 @xcite , mrpc @xcite , sts-b @xcite , qqp @xcite , qnli @xcite , mnli @xcite and rte @xcite . rainbow rainbow @xcite is a suite of commonsense question answering tasks including αnli @xcite , cosmosqa @xcite , hellaswag @xcite , piqa @xcite , socialiqa @xcite , winogrande @xcite . lexglue lexglue (legal general language understanding evaluation) @xcite is a collection of datasets for evaluating model performance across a diverse set of legal nlu tasks, which contain 7 subtasks, namely ecthr (task a), ecthr (task b), scotus, eur-lex, ledgar, unfair-tos, and casehold. ## conclusions this work presents a task prefix guided multi-task method by making use of task prefix to explore the mutual effects between tasks and improve model performance with complementary tasks. our released model can not only serve as the strong foundation backbone for a wide range of nlu tasks but also be used as a probing tool for analyzing task relationships. our model shows generalizable advances over tasks in diverse formats and establishes human-parity results on commonsense reasoning tasks. based on our pre-trained model, we find that the prefixes necessarily reflect task relationships, which correlate with transfer learning performance between tasks and suggest directions for data augmentation of complementary tasks. in summary, our work has the following prospects for future studies: the recipe of using task prefixes in conjunction with prefix prediction in mlm training has shown effective for large-scale mtl pre-training. the task relationships probed by the prefix embeddings have shown informative in finding the complementary tasks. using complementary tasks helps obtain better performance for a target task, especially for small-scale task datasets. the discovery of task relationships may help determine redundant datasets that assess similar patterns of models. recently, there has been a trend to evaluate the comprehensive skills of deep learning models by using a large number of datasets @xcite , the selection of distinctive datasets can be guided by our relationship discovery criteria to avoid evaluation redundancy and save computation. limitations. we acknowledge the major limitation of this work is that our model may not readily apply to new tasks. it is based on the common assumption of mtl that the set of tasks is known at training time. adaptation to new tasks could be future work.