C og T askonomy: Cognitively Inspired Task Taxonomy Is Beneficial to Transfer Learning in NLP (2022)

Abstract: Is there a principle to guide transfer learning across tasks in natural language processing (NLP)? Taxonomy (Zamir et al., 2018) finds that a structure exists among visual tasks, as a principle underlying transfer learning for them. In this paper, we propose a cognitively inspired framework, CogTaskonomy, to learn taxonomy for NLP tasks. The framework consists of Cognitive Representation Analytics (CRA) and Cognitive-Neural Mapping (CNM). The former employs Representational Similarity Analysis, which is commonly used in computational neuroscience to find a correlation between brain-activity measurement and computational modeling, to estimate task similarity with task-specific sentence representations. The latter learns to detect task relations by projecting neural representations from NLP models to cognitive signals (i.e., fMRI voxels). Experiments on 12 NLP tasks, where BERT/TinyBERT are used as the underlying models for transfer learning, demonstrate that the proposed CogTaxonomy is able to guide transfer learning, achieving performance competitive to the Analytic Hierarchy Process (Saaty, 1987) used in visual Taskonomy (Zamir et al., 2018) but without requiring exhaustive pairwise O(m^2) task transferring. Analyses further discover that CNM is capable of learning model-agnostic task taxonomy.

## introduction transfer learning (tl) has attracted extensive research interests in natural language processing with a wide range of forms, e.g., tl from pretrained language models (plm) to downstream tasks @xcite , from a task with rich labeled data to a task with low resource † equal contribution. * corresponding author. @xcite , from highresource languages to low-resource languages @xcite , etc. foot_0 a high-level concept or question on cross-task transfer learning is how these involved tasks are related to each other. is sentiment analysis related to paraphrasing? is textual entailment more related to question answering than named entity recognition? all these sub-questions resolve themselves into whether a structure exists among nlp tasks. such task taxonomy is of notable values to transfer learning in nlp in that it has the potential to guide tl and reduce redundancies across tasks @xcite . in this paper, partially inspired by the task taxonomy in visual tasks @xcite , we study the hierarchical task structure for nlp tasks. but significantly different from the visual taskonomy @xcite , we construct nlp taskonomy from a cognitively inspired perspective. cognitively inspired nlp is the intersection of nlp and cognitive neuroscience that aims at uncovering cognitive processes in the brain, including cognition in language comprehension. with the increasing availability of cognitively annotated data, on the one hand, cognitive processing signals (e.g., eye-tracking, eeg, fmri) have been explored to enhance neural models for a wide range of nlp tasks @xcite @xcite . on the other hand, representations learned in nlp models are used to predict brain activation patterns recorded in cognitive processing data @xcite @xcite . these studies on the bidirectional association between the two areas demonstrate that information underlying cognitive processing data is closely related to tasks and representations in nlp. hence we want to know whether it is feasible to isolate task reprepretrained language models are fine-tuned on downstream tasks to obtain task-specific sentence representations, which are then fed into cra, cnm for estimating task similarity. we can obtain the most similar task for each target task by the corresponding task similarity estimation method and rank it according to the oracle task ranking obtained according to transfer learning performance. r is the task ranking score which is averaged over all target tasks. to examine this hypothesis, we propose cogtaskonomy, a cognitively inspired task taxonomy framework, as illustrated in figure 1 , to learn a task structure for nlp tasks. cogtaskonomy consists of two main cognitively inspired components: cognitive representation analytics (cra) and cognitive-neural mapping (cnm). cra extracts task representations from nlp models and employs representational similarity analysis (rsa) @xcite , which is commonly used to measure the correlation between brain activity and computational model, to estimate nlp task similarity. cnm trains fully connected neural networks to build the mapping from sentence representations of pretrained models finetuned on specific tasks to fmri signals recorded when human subjects read those sentences. it then uses mapping correlation coefficients as task representations to compute task similarity. both methods require sentence representations to compute task representations. we use pretrained language models fine-tuned on specific tasks, particularly bert @xcite and tinybert @xcite , to obtain sentence representations. we compare the proposed cogtaskonomy against the analytic hierarchy process (ahp) used in taskonomy @xcite . we guide tl across tasks with the learned task structure and evaluate the effectiveness of these methods by estimating tl performance from various source to target tasks. ## cogtaskonomy figure 1 illustrates the basic framework of cog-taskonomy. first, we obtain task-specific sentence representations of text stimuli from cognitive data by feeding them into fine-tuned or distilled pretrained language models on 12 downstream tasks (section 3.1). subsequently, task-specific representations are fed into two cognitively inspired components, cognitive representation analytics (section 3.2) and cognitive-neural mapping (section 3.3), for estimating task similarity and inducing task taxonomy. ## conclusions in this paper, we have presented a cognitively inspired framework, termed cogtaxonomy, to learn relation and structure for nlp tasks. experiments demonstrate that the task taxonomy detected by cogtaxonomy can be used to guide transfer learning across 12 different nlp tasks. both cra and cnm, the two essential components of cogtaxonomy, do not require exhaustive transfer learning across all source-target task pairs. the former is robust to different combinations of dissimilarity/similarity measurements. the latter resorts to cognitive signals to learn model-agnostic task relations.