When Does Aggregating Multiple Skills with Multi-Task Learning Work? A Case Study in Financial NLP (2023)

Abstract: Multi-task learning (MTL) aims at achieving a better model by leveraging data and knowledge from multiple tasks. However, MTL does not always work – sometimes negative transfer occurs between tasks, especially when aggregating loosely related skills, leaving it an open question when MTL works. Previous studies show that MTL performance can be improved by algorithmic tricks. However, what tasks and skills should be included is less well explored. In this work, we conduct a case study in Financial NLP where multiple datasets exist for skills relevant to the domain, such as numeric reasoning and sentiment analysis. Due to the task difficulty and data scarcity in the Financial NLP domain, we explore when aggregating such diverse skills from multiple datasets with MTL can work. Our findings suggest that the key to MTL success lies in skill diversity, relatedness between tasks, and choice of aggregation size and shared capacity. Specifically, MTL works well when tasks are diverse but related, and when the size of the task aggregation and the shared capacity of the model are balanced to avoid overwhelming certain tasks.

## introduction multi-task learning (mtl) is a machine learning paradigm where multiple learning tasks are optimized simultaneously, exploiting commonalities and differences across them @xcite . mtl is expected to outperform single-task learning (stl) as it utilizes more training data and enables inter-task knowledge sharing @xcite . however, mtl may also bring about multi-task conflict and negative transfer. empirically, in many mtl systems, only a small portion of tasks benefit from mt joint training while others suffer from negative transfer @xcite @xcite . therefore, it is still an open question when mtl will work. ## background & related work previous work mainly focuses on two categories of mtl practice: mtl as pre-training and mtl as auxiliary training. mtl as pre-training: besides unsupervised pretraining, supervised data can also be utilized for pre-training in an mtl manner (i.e., an intermediate training stage) to improve the model's multiaspect intelligence and generalizability to unseen tasks. such an approach has been shown beneficial for various pre-trained models, including encoderonly models @xcite , encoder-decoder models @xcite , and large language models @xcite @xcite . @xcite show that mtl pre-training does not work with small-scale task aggregation. more recent analysis shows that aggregating related tasks transfers better to a known target task @xcite . mtl as auxiliary training: instead of training a target task alone, we can jointly train it with other auxiliary tasks to improve its performance in an mtl manner (i.e., the final training stage). however, this approach does not work in most cases, especially when multiple skills are aggregated (e.g., glue) (stickland and @xcite @xcite . previous work shows that appropriate algorithmic tricks lead to more success in mtl: (1) mtl architecture: pfeiffer et al. (2021); karimi mahabadi et al. (2021) and ponti et al. (2023) propose mtl architectures that encourage high-level knowledge sharing instead of direct parameter-sharing; and (2) mtl optimization: yu et al. (2020) and wang et al. (2021) geometrically manipulate the gradients to reduce the conflicts, and mao et al. (2022) learn to weight losses of including tasks automatically. in computer vision (cv), fifty et al. (2021) address that task aggregation is also crucial for mtl besides algorithms and proposes an algorithm to select the best task grouping from a task collection. however, rare previous work in nlp analyzes what task aggregation leads to the success of mtl and what qualities of a task aggregation are important. ## findata compilation we compile findata, a task aggregation on financial nlp, to facilitate the case study. we first set the desiderata, and then survey existing financial nlp tasks to select those that meet these criteria. ## multi-task learning systems we consider various mtl systems in the form of shared encoder + one-layer task-specific prediction headers. the mtl problem is formulated as follows: we are given a joint dataset of multiple tasks @xmath0, y t )} @xmath1, y t denotes the training corpus and labels of task t; and t denotes the task collection. we are also given a pre-trained encoder (e.g., finbert) f θ e (•) and task-specific prediction headers h θt (•), which are parameterized by θ = (θ e , {θ t } @xmath2. the training loss for multitask fine-tuning: where l t denotes the loss function for task t, and w t denotes the sampling weight of task t. the generic architecture is illustrated in figure 1 . during training, a task is sampled for each training step, and the corresponding prediction header and the shared encoder are updated (e.g., the tsa example in figure 1 ). ## experimental setup we fine-tune all mtl and stl models on corresponding data for 40 epochs. (our mtl batching scheme is described in appendix c.) for stl, we evaluate the model every 50 steps and save the checkpoint with the best validation score. for mtl, we evaluate every 200 steps, saving and reporting the best checkpoint for each task independently following the setting of @xcite (i.e., each task can be viewed as the target task with others being auxiliary tasks). we follow the evaluation metrics in ## analysis in this section, we analyze the hypotheses about when aggregating multiple skills with mtl works. ## discussion suggestions for mtl practice: based on the results of our case study, we recommend the following practices for future mtl: (1) aggregate not only the target skill but also other related skills; aggregating multiple skills with mtl is a potential way for better financial nlp practice: financial nlp tasks are more complicated than those in the general domain, and many of them suffer from a lack of data. obtaining new financial nlp data is expensive since such annotation usually requires domain expertise. our results show that aggregating financial nlp tasks using mtl can be a practical and relatively cheap way to improve their performance: sc, nc, nad, and cd are improved by up to 1.45, 0.64, 1.09, and 0.68 percentage points accordingly through mtl auxiliary training (contributed by different mtl systems). in appendix g, we also show that mtl pre-training with financial nlp tasks can improve the model's generalizability to unseen tasks. therefore, future research and practice in financial nlp may consider mtl as a potential way to achieve better performance. other possible questions: we address some other possible questions that might be of interest to our readers in appendix a. ## conclusion in this work, we conduct a case study on financial nlp to analyze when aggregating multiple skills with mtl works from a perspective of task relations and skills to be included. we propose a parameter-efficient mtl architecture spal-finbert. our empirical analyses point out potential directions to improve task aggregation for future mtl practice: (1) considering diversified nontarget skills that might be supportive; (2) filtering tasks with their relatedness; and (3) caring whether capacity sharing overwhelms the target task. we also show that aggregating resources through mtl can be a cheap and efficient way to improve financial nlp performance.