Multi-task Learning with Gradient Communication
Authors: Pengfei Liu, Xuanjing Huang
Keywords: Pretend to share, Gradient Communication
TL;DR: We introduce an inductive bias for multi-task learning, allowing different tasks to communicate by gradient passing.
Abstract:   In this paper, we describe a general framework to systematically analyze current neural models for multi-task learning, in which we find that existing models expect to disentangle features into different spaces while features learned in practice are still entangled in shared space,  leaving potential hazards for other training or unseen tasks. We propose to alleviate this problem by incorporating a new inductive bias into the process of multi-task learning, that different tasks can communicate with each other not only by passing hidden variables but gradients explicitly. Experimentally, we evaluate proposed methods on three groups of tasks and two types of settings (\textsc{in-task} and \textsc{out-of-task}). Quantitative and qualitative results show their effectiveness.