The Stem Cell Hypothesis: Dilemma behind Multi-Task Learning with Transformer Encoders (2021)

Abstract: Multi-task learning with transformer encoders (MTL) has emerged as a powerful technique to improve performance on closely-related tasks for both accuracy and efficiency while a question still remains whether or not it would perform as well on tasks that are distinct in nature. We first present MTL results on five NLP tasks, POS, NER, DEP, CON, and SRL, and depict its deficiency over single-task learning. We then conduct an extensive pruning analysis to show that a certain set of attention heads get claimed by most tasks during MTL, who interfere with one another to fine-tune those heads for their own objectives. Based on this finding, we propose the Stem Cell Hypothesis to reveal the existence of attention heads naturally talented for many tasks that cannot be jointly trained to create adequate embeddings for all of those tasks. Finally, we design novel parameter-free probes to justify our hypothesis and demonstrate how attention heads are transformed across the five tasks during MTL through label analysis.

## introduction transformer encoders (tes) have established recent state-of-the-art results on many core nlp tasks @xcite @xcite . however, their architectures can be viewed "over-parameterized" as downstream tasks may not need all those parameters, prone to cause an overhead in computation. one promising approach to mitigate this overhead is multi-task learning (mtl) where a te is shared across multiple tasks; thus, it needs to be run only once to generate final embeddings for all tasks @xcite . despite the success in mtl on closely-related tasks such as language understanding @xcite or relation extraction @xcite , mtl on core nlp tasks (e.g., tagging, parsing, labeling) whose decoders are very distinct has not been well-studied. this work employs the state-of-the-art decoders on five core tasks for mtl and thoroughly analyzes interactions among those tasks to explore a possibility of reducing the computation overhead from tes. surprisingly, our experiments depict that models jointly trained by mtl give lower accuracy than ones trained individually, that is against findings from previous work. in fact, models jointly trained with all five tasks perform the worst among any other combination (section 3). these experimental results urge us to figure out why mtl on core tasks with a shared te leads to worse performance than its single-task counterparts. our exploration begins by detecting essential heads for each task by forcing the te to use as few attention heads as possible while maintaining accuracy similar to a fully-utilized encoder. our experiments reveal that all five tasks rely on almost the same set of attention heads. hence, they compete for those heads during mtl, causing to blur out features extracted by individual tasks. thus, we propose the stem cell hypothesis, likening these talented attention heads to stem cells, which cannot be fine-tuned for multiple tasks that are very distinct (section 4). to validate this hypothesis, many parameter-free probes are designed to observe how every attention head is updated while trained individually or jointly. intriguingly, we find that heads not fine-tuned for any task can still give remarkably high performance to predict certain linguistic structures, confirming the existence of stem cells inherently more talented; it is consistent with previous work stating that tes carry on a good amount of syntactic and semantic knowledge @xcite @xcite . after single-task learning, probing results typically improve along with the task performance, illustrating that the stem cells are developed into more taskspecific experts. on the contrary, mtl often drops both probing and task performance, supporting our hypothesis that attention heads lose expertise when exposed to multiple teaching signals that may conflict to one another (section 5). the stem cell hypothesis is proposed to shed light on a possible direction to mtl research using tes, comprising an unbearable amount of parameters, by wisely assigning attention heads to downstream tasks. although most analysis in this study is based on bert, we also provide extensive experimental results and visualization of other recent tes including roberta @xcite , electra @xcite and @xcite in §a.4 to further demonstrate the generality of our hypothesis. to the best of knowledge, this is the first time that a comprehensive analysis of attention heads is made for mtl on those core tasks by introducing novel parameter-free probing methods. foot_0 ## related work a small portion of our work overlaps with multitask learning. mtl with pre-trained transformers specifically in nlp @xcite @xcite @xcite has been widely studied. most work focus on neural architecture design to encourage beneficial message passing across tasks. our mtl framework adopts conventional architecture and applies tricks of batch sampling @xcite and loss balancing. most of our work falls into the analysis of bert, especially from a linguistic view. since bert was introduced, studies on explaining why bert works have never stopped. the most related studies are those trying to study the linguistic structures learnt by bert. among them, @xcite and @xcite showed part-of-speech, syntactic chunks and roles can be discovered from bert embeddings. using a supervised probe, @xcite successfully discover full dependency parse trees. the encoded dependency structure is also supported by @xcite using probes on embeddings. apart from these parameterized probes, parameter-free approaches @xcite also agree with the existence of rich linguistic knowledge in bert, which is closely related to our probing methods. what remains unclear is the impact of fineturning on tes. using supervised probes, @xcite claim that fine-tuning adapts bert embeddings to downstream tasks, which is later challenged by @xcite since su-pervised probe itself can encode knowledge. then, @xcite propose a methodology to test such encoding of a linguistic phenomenon by comparing the probing performance before and after fine-tuning. our probing methods align with these unsupervised probes while focus more on explaining the impact of multi-task learning. ## multi-task learning our goal of mtl is to build a joint model sharing the same encoder but using a distinct decoder for each task that outperforms its single-task counterparts while being faster and more memory efficient. our model adapts hard parameter sharing @xcite ) such that all decoders take the same hidden states generated by the shared encoder as input and make task-specific predictions in parallel. ## pruning analysis to answer why mtl leads to suboptimal results in section 3, pruning strategies are applied to bert such that only attention heads absolutely necessary to get the best performance are kept for every task. this allows us to see if there exists a common set of heads that multiple tasks want to claim and train for only their objectives, which can cause conflicts for those heads to be shared across all tasks. ## probing analysis this paper hypothesizes the existence of stem cells, which cannot be trained to create adequate embeddings to be shared by multiple tasks that are not so similar. this section provides empirical evidence to this hypothesis by probing what roles each attention head plays once fine-tuned for end tasks. ## conclusion this study analyzes interference on the 5 core tasks by highlighting naturally talented attention heads, whose importance turns out to be invariant for many downstream tasks. the stem cell hypothesis states that these talented heads are like stem cells that can develop into experts but not all-rounders. our hypothesis is validated by several novel parameterfree probes, revealing the interfered representations of stem cells. we will adapt this work to more tasks and languages for broader generality in the future. timothy dozat and christopher d. manning. 2017. deep biaffine attention for neural dependency parsing. in proceedings of the 5th international conference on learning representations, iclr'17. han he and jinho d. choi. 2019. establishing strong baselines for the new decade: sequence tagging, syntactic and semantic parsing with bert. in proceedings of the 33rd international florida artificial intelligence research society conference, flairs'19. luheng he, kenton lee, omer levy, and luke zettlemoyer. 2018. jointly predicting predicates and arguments in neural semantic role labeling. in proceedings of the 56th annual meeting of the association for computational linguistics (volume 2: short papers), pages 364-369, melbourne, australia. association for computational linguistics. pengcheng he, xiaodong liu, jianfeng gao, and weizhu chen. 2020. deberta: decoding-enhanced bert with disentangled attention. john hewitt and percy liang. 2019. designing and interpreting probes with control tasks. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 2733-2743, hong kong, china. association for computational linguistics. john hewitt and christopher d. manning. 2019. a structural probe for finding syntax in word representations. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4129-4138, minneapolis, minnesota. association for computational linguistics. ganesh jawahar, benoît sagot, and djamé seddah. 2019. what does bert learn about the structure of language? in proceedings of the 57th annual meeting of the association for computational linguistics, pages 3651-3657, florence, italy. association for computational linguistics. dan kondratyuk and milan straka. 2019. 75 languages, 1 model: parsing universal dependencies universally. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 2779-2795, hong kong, china. association for computational linguistics. ying lin, heng ji, fei huang, and lingfei wu. 2020. a joint neural model for information extraction with global features. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 7999-8009, online. association for computational linguistics. yongjie lin, yi chern tan, and robert frank. 2019. open sesame: getting inside bert's linguistic knowledge. in proceedings of the 2019 acl workshop blackboxnlp: analyzing and interpreting neural networks for nlp, pages 241-253, florence, italy. association for computational linguistics. nelson f. liu, matt gardner, yonatan belinkov, matthew e. peters, and noah a. smith. 2019a. linguistic knowledge and transferability of contextual representations. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 1073-1094, minneapolis, minnesota. association for computational linguistics. xiaodong liu, pengcheng he, weizhu chen, and jianfeng gao. 2019b. multi-task deep neural networks for natural language understanding. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 4487-4496, florence, italy. association for computational linguistics.