Coarse-to-Fine: Hierarchical Multi-task Learning for Natural Language Understanding (2022)

Abstract: Generalized text representations are the foundation of many natural language understanding tasks. To fully utilize the different corpus, it is inevitable that models need to understand the relevance among them. However, many methods ignore the relevance and adopt a single-channel model (a coarse paradigm) directly for all tasks, which lacks enough rationality and interpretation. In addition, some existing works learn downstream tasks by stitches skill block (a fine paradigm), which might cause irrational results due to its redundancy and noise. In this work, we first analyze the task correlation through three different perspectives, , data property, manual design, and model-based relevance, based on which the similar tasks are grouped together. Then, we propose a hierarchical framework with a coarse-to-fine paradigm, with the bottom level shared to all the tasks, the mid-level divided to different groups, and the top-level assigned to each of the tasks. This allows our model to learn basic language properties from all tasks, boost performance on relevant tasks, and reduce the negative impact from irrelevant tasks. Our experiments on 13 benchmark datasets across five natural language understanding tasks demonstrate the superiority of our method.

## introduction pre-trained language models have achieved great success on various natural language processing (nlp) tasks. meanwhile, pre-train-then-finetuning has gradually become the mainstream paradigm @xcite @xcite . the pre-training process aims to learn a general language representation from the large-scale corpus. such representations can be further fine-tuned on downstream datasets and perform specific tasks. though great performance † these authors contributed equally to this work. * corresponding author. inspired by human learning, multi-task learning believes that tasks can interact and boost each other. therefore, to obtain a more robust representation that can handle a variety of tasks, a straightforward idea is to fine-tune a pre-trained model on many tasks simultaneously. numerous previous studies have been conducted along this path. for example, mt-dnn @xcite uses a transformerbased model as a shared encoder and trains it on multiple downstream tasks (as shown in the left side of figure 1 ). by this coarse paradigm, the representation model can be more generalized and robust. although it has been observed that some combinations of tasks yield improvements, this is not always the case. researchers @xcite have also found that the impact between tasks is a double-edged sword, meaning that some tasks may also hurt others. indeed, modeling heterogeneous tasks often require distinct representation spaces. for example, naively training natural language inference (nli) tasks with different hypothesis types together can lead to performance declines. some recent work proposes sparsely activating multiple modules for different tasks @xcite to mitigate the negative effects across tasks. these fine paradigms activate distinct modules according to predefined skills for learning downstream tasks (as shown in the middle part of figure 1 ). however, the number of parameters is also multiplied when multiple modules are activated for a task. this problem becomes even more severe when large pre-trained language models are applied. shared layer task 1 task 2 task 3 head 1 head 2 head 3 (1) fully sharing (coarse) (2) partial sharing (fine) (3) hierarchical sharing (coarse-to-fine) shared layer layer layer task 1 task 2 task 3 head 1 head 2 head 3 layer layer layer layer layer layer layer task 1 task 2 task 3 head 1 head 2 head 3 clustering layer specific layer specific layer shared layer specific layer shared layer figure 1 : the illustration of hard sharing structure, partial sharing structure, and our hierarchical sharing structure. the three tasks and the corresponding activated paths are in different colors. in our approach, the first and second are two relevant tasks, so they share the same task-clustering layers. finally, they are fed into different task-specific layers. our main contributions are three-fold: ## related work in this section, we briefly introduce some transformer-based pre-trained models and recent work on multi-task learning. ## conclusion and future work in this work, we explore task correlation and built a hierarchical multi-task learning framework. our framework adopts a coarse-to-fine manner, in which the tasks are gradually separated from the bottom up. by doing so, it can reap the benefits of multi-tasking learning at the lower layers while avoiding its harmful impact on the upper layers. extensive experiments on several challenging nlu datasets showed that our model achieves better performance than existing strategies. further experiments indicated that our methods are flexible with the choice of task relevance metrics, and robust with the hyperparameter selection. as a preliminary study on incorporating task relevance into multi-task learning, there are several potential future directions, such as new backbone models and task relevance metrics. figure 6: per head attention maps' cosine similarity between fine-tuned model and co-trained model using negative task pair(mrpc & rte). darker colors means greater differences. left: mrpc right: rte