A Survey of Multi-task Learning in Natural Language Processing: Regarding Task Relatedness and Training Methods (2023)

Abstract: Multi-task learning (MTL) has become increasingly popular in natural language processing (NLP) because it improves the performance of related tasks by exploiting their commonalities and differences. Nevertheless, it is still not understood very well how multi-task learning can be implemented based on the relatedness of training tasks. In this survey, we review recent advances of multi-task learning methods in NLP, with the aim of summarizing them into two general multi-task training methods based on their task relatedness: (i) joint training and (ii) multi-step training. We present examples in various NLP downstream applications, summarize the task relationships and discuss future directions of this promising topic.

## introduction machine learning generally involves training a model to perform a single task. by focusing on one task, the model ignores knowledge from the training signals of related tasks @xcite . there are a great number of tasks in nlp, from syntax parsing to information extraction, from machine translation to question answering: each requires a model dedicated to learning from data. biologically, humans learn natural languages, from basic grammar to complex semantics in a single brain @xcite . in the field of machine learning, multi-task learning (mtl) aims to leverage useful information shared across multiple related tasks to improve the generalization performance on all tasks @xcite . in deep neural networks, it is generally achieved by sharing part of hidden layers between different tasks, while keeping several task-specific output layers. mtl offers advantages like improved data efficiency, reduced overfitting, and fast learning by leveraging auxiliary information @xcite . there have been relevant surveys that looked into architecture designs and optimization algorithms in mtl. @xcite classified different mtl frameworks into two categories: hard parameter sharing and soft parameter sharing, as well as some earlier mtl examples in both non-neural and neural models; @xcite expanded such two "how to share" categories into five categories, including feature learning approach, low-rank approach, task clustering approach, task relation learning approach, and decomposition approach; @xcite presented more recent models in both single-domain and multi-modal architectures, as well as an overview of optimization methods in mtl. nevertheless, it is still not clearly understood how to design and train a single model to handle a variety of nlp tasks according to task relatedness. especially when faced with a set of tasks that are seldom simultaneously trained previously, it is of crucial importance that researchers find proper auxiliary tasks and assess the feasibility of such multi-task learning attempt. in this paper, we first review recent approaches on multi-task training methods in popular nlp applications. we find that these approaches can be categorized into two multi-task training methods according to the types of task relatedness: (i) joint training methods and (ii) multi-step training meth-multi-task training multi-task frameworks multi-task frameworks related papers methods in our survey defined in @xcite defined in @xcite joint training deep relationship network shared trunk @xcite ) cross-stitch network cross-talk @xcite ) weighting losses with uncertainty @xcite ) sluice network @xcite ) adversarial feature separation @xcite multi-step training prediction distillation @xcite ) low supervision cascaded information @xcite the remainder of this survey is organized as follows. section 2 includes an overview of mtl models in nlp and the rationales of using mtl. section 3 presents a number of joint and multistep training applications in different fields of nlp. section 4 analyzes the task relatedness involved in these mtl approaches. section 5 discusses future 1 we do not specifically distinguish different parameter sharing designs, since this topic is not the focus of our survey. we refer readers to learn details in @xcite . ## nlp task relatedness in this section, we summarize the characteristics of the aforementioned mtl approaches, and look into the task relatedness between the sub-tasks. ## future directions in this section, we will discuss some promising directions regarding either task relatedness or training methods of multi-task training in nlp. ## conclusions in this paper, we reviewed recent work on multitask learning for nlp tasks. according to the types of task relatedness, we categorized multitask nlp approaches into two typical frameworks: joint training and multi-step training. we presented the design of each framework in various nlp applications, and discussed future directions of this interesting topic.