Tchebycheff Procedure for Multi-task Text Classification (2020)

Abstract: Multi-task Learning methods have achieved great progress in text classification. However, existing methods assume that multi-task text classification problems are convex multiobjective optimization problems, which is unrealistic in real-world applications. To address this issue, this paper presents a novel Tchebycheff procedure to optimize the multi-task classification problems without convex assumption. The extensive experiments back up our theoretical analysis and validate the superiority of our proposals.

## introduction multi-task learning (mtl) aims to learn multiple related tasks simultaneously, and obtain better performance than learning each task independently by setting inductive bias across tasks. @xcite @xcite . it has achieved great success in various applications ranging from computer vision @xcite to text classification @xcite @xcite . existing mtl methods for text classification, usually set up the inductive bias across tasks by designing a parameterized hypothesis class that shares some parameters across tasks (e.g. shares some hidden layers in a neural network), and cast the multi-task text classification problem as a multiobjective optimization problem. l 1 -metric method is one of the most popular strategies for solving the multi-objective optimization problem. specifically, it learns the parameters by minimizing a weighted linear combination of per-task losses. and this method is able to find an arbitrary pareto optimal solution in the pareto set if the problem is convex. unfortunately, for a non-convex problem, this *corresponding author. method excludes many pareto optimal solutions from its search scope. to illustrate the issue, it is instructive to consider a 2-tasks learning case shown as figure 1 . from figure 1 , we can see that for a non-convex problem, the pareto points located at the concave part of the pareto front are unachievable. according to the uniform convergence properties of mtl @xcite , the exclusion of pareto optimal solutions may degenerate the generalization performance of multi-task text classification. to address the non-convexity problems, this paper proposes a novel tchebycheff procedure to improve the performance of multi-task text classification. to validate the superiority of the proposed method, we conduct the experiments on two classical text classification problems: sentiment analysis on reviews @xcite and topic classification on news @xcite . the results show that our proposed method can converge and outperform several state-of-the-art multi-task text classification methods. ## related works the family of pareto optimality methods, including l 1 -metric methods (weighted sum methods) @xcite @xcite and multiple-gradient descent algorithm (mgda) @xcite , have become one of the most prevalent multi-task learning (mtl) strategies. in multi-task text classification, l 1 -metric methods are widely used @xcite @xcite . however, for non-convex problems, the l 1 -metric methods are likely to exclude the optimal hypothesis from the hypothesis class. to handle the non-convex case, mgda leverages the karush-kuhn-tucker conditions and provides pareto stationary points as solutions. however, the solutions are not sufficient to be pareto optimal. a novel mtl method, which can achieve pareto optimal without any convex assumption, is necessary to compensate for disadvantages in the l 1 -metric and mgda. in this paper, a novel tchebycheff procedure is proposed to achieve pareto optimal without any convex assumption. ## mtl as multi-objective optimization consider a multi-task learning problem with t tasks over an input space x and a collection of task spaces {y t } t @xmath0. there is also a parametric hypoth- correspondingly, the empirical loss of the task t is defined as lt (θ sh , θ t )= 1 nt nt @xmath1, θ sh ), θ t ), y t i ) . we also denote the transpose of the vector/matrix by superscript , the logarithms to base 2 by log. empirical risk of task 1 empirical risk of task 2 metric metric weak pareto optimality ## tchebycheff procedure for multi-task text classification many multi-task neural network models can be used in multi-task text classification, such as hard parameter sharing networks @xcite and soft parameter sharing networks @xcite . this paper adopts a hard parameter sharing network model, because it has the lowest computational cost among the models. ## experiments in this section, firstly, we conduct a synthetic experiment to validate our theory analysis. then, we perform experimental studies on two real-world applications: sentiment analysis and topic classification. the implementation is based on pytorch @xcite . the code can be found in the supplementary materials. ## conclusion most of multi-task text classification problems are non-convex multi-objective optimization problems. however, existing methods ignore the nonconvexity and solve the problems using convex optimization methods. to address this issue, this paper presents an (adversarial) tchebycheff procedure for multi-task text classification without any convex assumption. numerical experiments show that our proposed methods can converge and outperform state-of-the-art methods. in the tchebycheff procedure, we choose the weight for each task according to the empirical risk of learning the corresponding task independently. obtaining the empirical risk is a little laborious. in the future, it would be fruitful to develop a novel weighting strategy for the tchebycheff procedure.