Exploring Logically Dependent Multi-task Learning with Causal Inference (2020)

Abstract: Previous studies have shown that hierarchical multi-task learning (MTL) can utilize task dependencies by stacking encoders and outperform democratic MTL. However, stacking encoders only considers the dependencies of feature representations and ignores the label dependencies in logically dependent tasks. Furthermore, how to properly utilize the labels remains an issue due to the cascading errors between tasks. In this paper, we view logically dependent MTL from the perspective of causal inference and suggest a mediation assumption instead of the confounding assumption in conventional MTL models. We propose a model including two key mechanisms: label transfer (LT) for each task to utilize the labels of all its lower-level tasks, and Gumbel sampling (GS) to deal with cascading errors. In the field of causal inference, GS in our model is essentially a counterfactual reasoning process, trying to estimate the causal effect between tasks and utilize it to improve MTL. We conduct experiments on two English datasets and one Chinese dataset. Experiment results show that our model achieves state-of-the-art on six out of seven subtasks and improves predictions’ consistency.

## introduction multi-task learning (mtl) has received increasing interest with the knowledge transfer potential among related tasks @xcite @xcite . recently, hierarchical mtl models @xcite were proposed for tasks with dependencies and achieved better performance than democratic ones. in their models, the encoders of different tasks were stacked. and the proposition is that complex tasks at top layers require deep processes * corresponding authors to capture semantic richer features, and simple tasks at bottom layers require shallow processes. however, many hierarchical mtl models only consider the dependencies of feature representations and ignore the label dependencies. a direct comparison is shown in figure 1 . let x denote a given input, and y m and y n denote the outputs of two tasks, respectively. the first two mtl schemes @xcite @xcite substantially model the likelihood by p (y m , y n |x)=p (y m |x)p (y n |x) while the third mtl scheme @xcite models it by: from the causal perspective, the first two schemes assume that y m and y n are conditional independent, while the third scheme assumes that y m has a causal effect on y n . in this paper, we suggest that the causal effect is important for logically dependent tasks. and we propose a mechanism referred to label transfer (lt), which lets a task utilize the labels of all its lower-level tasks. when utilizing discrete labels, there remains another issue. for example, the strategy in @xcite used gold labels of low-level tasks during training and used predicted ones during testing. apparently, there was a train-test discrepancy which leads to cascading errors between tasks. and it was similar to the exposure bias problem in the field of text generation @xcite . recently, two approaches have been investigated to overcome the problem, which are gumbel sampling (gs) @xcite and reinforcement learning (rl) @xcite . in this paper, we regard the logically dependent mtl as a task-level label generation problem. and we incorporate gs because it feeds the optimizer with low-variance gradients, improving stability and speed of training over rl. specifically, our model samples a label from the predicted probability distribution for each task and feeds it to its higher-level tasks. and back-propagated gradients will penalize wrong predictions if the causal effect exists. from the perspective of causal inference, the sampling is a counterfactual reasoning process that can estimate the causal effect between different tasks' labels. and we hope a model properly cooperating causality will be more robust and transferable, as argued in @xcite . to verify the effectiveness of our model, we conduct experiments on two english and one chinese mtl datasets. the results show that our model achieves state-of-the-art (sota) on 6 out of 7 subtasks and improves predictions' consistency. and we present the estimated causal effect for several cases, which is consistent with humans' prior knowledge. in conclusion, the contributions of this paper can be summarized as follows: ## related work in natural language processing (nlp), many studies focus on modeling task dependencies to improve mtl. a line of work proposed hierarchical mtl architectures by stacking the encoders of different tasks with simple tasks at lower layers and complex tasks at top layers @xcite @xcite . and zhong et al. (2018) proposed a topological mtl architecture based on the topological hierarchy of tasks. another line of work tried to reencode the predictions of low-level tasks. giannis et al. (2018) re-encoded the predicted labels with the highest probability of low-level tasks during testing, and encoded the gold labels during training. luan et al. (2019) re-encoded the soft predictions during testing and also encoded the gold ones during training. yang et al. (2019) proposed a bidirectional architecture producing initial probability distributions for different tasks and then refine the probability distributions by conditioning on each other during both training and testing. our work is also related to some studies for text generation. the democratic and hierarchical mtl schemes in figure 1 are similar to the nonautoregressive language models like bert @xcite which is possible to generate syntactically incorrect sentences @xcite . the logically dependent mtl scheme is similar to the autoregressive language model but remains a train-test discrepancy. gs or rl learning have been investigated to deal with the discrepancy @xcite @xcite . ## task definition to show that our method's high versatility, we investigate three different mtl scenarios: joint entity and relation extraction (jere), aspect-based sentiment analysis (absa), and legal judgment prediction (ljp). @xcite . ## methodology in this section, we first analyze mtl from the causal perspective in subsection 4.1 and then introduce our models in the following subsections. features: outputs: inputs: figure 2 : two kinds of typical causal assumptions. let h be the feature representation for x, and y m and y n be the outputs for two tasks t m and t n , respectively. the confounding assumption (the left sub-figure) considers y m and y n to be conditionally independent, while the mediation assumption (the right sub-figure) considers the logical dependency from y m to y n . ## conclusion in this paper, we investigated the mtl problem with logically dependent tasks. we first analyze mtl models from the perspective of causal inference and then propose a model cmtl to utilize task dependencies properly. the model achieves sota results on 6 out of 7 subtasks and improves the consistency of predicted results of different subtasks. in the future, we are interested in social science topics, such as modeling the causal effect between mental health and the suicide decisions reflected through social media, which may help predict and stop the final decisions.