Meta Distant Transfer Learning for Pre-trained Language Models (2021)

Abstract: With the wide availability of Pre-trained Language Models (PLMs), multi-task fine-tuning across domains has been extensively applied. For tasks related to distant domains with different class label sets, PLMs may memorize non-transferable knowledge for the target domain and suffer from negative transfer. Inspired by meta-learning, we propose the Meta Distant Transfer Learning (Meta-DTL) framework to learn the cross-task knowledge for PLM-based methods. Meta-DTL first employs task representation learning to mine implicit relations among multiple tasks and classes. Based on the results, it trains a PLM-based meta-learner to capture the transferable knowledge across tasks. The weighted maximum entropy regularizers are proposed to make meta-learner more task-agnostic and unbiased. Finally, the meta-learner can be fine-tuned to fit each task with better parameter initialization. We evaluate Meta-DTL using both BERT and ALBERT on seven public datasets. Experiment results confirm the superiority of Meta-DTL as it consistently outperforms strong baselines. We find that Meta-DTL is highly effective when very few data is available for the target task.

## introduction owning to the availability of pre-trained language models (plms), the performance of various text classification tasks has been significantly improved. notable plms include bert @xcite , albert @xcite , xlnet @xcite , t5 @xcite , @xcite and many others. it can be safely concluded that plm-based approaches achieve state-of-the-art results for a majority of text classification tasks. among these methods, a key procedure of plms is fine-tuning, which enables parameters of plms to fit specific datasets. hence, the performance of plms on a downstream task may be limited by the availability of the training set. as reported by several benchmarks such as glue @xcite and superglue @xcite , some plms may not perform well in low-resource tasks. a popular solution in nlp is transfer learning @xcite . for plms, these models can be fine-tuned over both source-domain and target-domain datasets by various multi-task training strategies @xcite . unfortunately, several studies reveal that multi-task training of plms across domains does not always guarantee satisfactory results @xcite . as plms usually have large parameter space and strong memorization power, learning from sourcedomain datasets may force plms to memorize nontransferable knowledge of source domains, leading to the negative transfer effect @xcite . besides, a large number of transfer learning algorithms address tasks across similar sub-domains, with the same set of class labels. 1 when there exist large domain gaps and class label differences, these transfer learning solutions are likely to fail. consider a simple, motivation example in figure 1 . de-spite the fact that the two datasets (sst-5 @xcite and amazon reviews @xcite ) are diverse in domains and classification targets, they aim to solve similar review analysis tasks. it would be beneficial for the two task-specific models to learn from each other. a few methods address the distant domain issue in transfer learning @xcite ), but are not designed for plms. a natural question arises: how can we transfer knowledge across distant domains with different classification targets for plm-based text classification? recently, meta-learning has been studied extensively, which learns parameters that can be adapted to a group of similar tasks @xcite . for plms, @xcite suggest that training a meta-learner for plms is highly effective to capture transferable knowledge across sub-domains. however, this method is not designed for tasks across diverse domains and class label sets. additionally, it lacks the mechanism to learn taskagnostic representations and may fit too much to specific targets in certain datasets. to this end, the meta distant transfer learning (meta-dtl) framework is proposed foot_2 . specially, meta-dtl employs a task representation learning procedure to obtain a collection of prototype vectors for each task. to understand how to transfer across these tasks and classes, we construct a meta knowledge graph (meta-kg) to characterize the implicit relations among tasks and classes, based on the representations of multiple tasks. the meta-learner in meta-dtl can be initialized by any plms and trained by multi-task learning with rich meta-knowledge injected from meta-kg. additionally, we design the weighted maximum entropy regularizers to make the model more task-agnostic and unbiased. finally, the metalearner can be fine-tuned to fit each task using its own training set. in this way, the model is able to digest the cross-task, transferable knowledge and alleviates negative transfer. we apply meta-dtl to bert @xcite and albert @xcite for three sets of nlp tasks (seven public datasets in total): i) coarse and fine-grained review analysis across domains; ii) natural language inference (across sentence relation prediction and scientific question answering); and iii) lexical semantics (across hypernymy detection and lexical relation classification). experiments show that meta-dtl consistently outperforms strong baselines. we also show that meta-dtl is highly useful for text classification when very few training samples of the target task are available. ## related work in this section, we summarize the related work on plms, transfer learning and meta-learning. ## meta-dtl: the proposed framework in this section, we first present our task. after that, the technical details of the meta-dtl framework are elaborated. ## experiments in this section, we conduct extensive experiments to evaluate the performance of meta-dtl and compare it against strong baselines. ## conclusion and future work in this paper, we propose the meta-dtl framework for plms, to capture knowledge from tasks with distant domains and class labels. extensive experiments confirm the effectiveness of meta-dtl from various aspects. future work includes: i) applying meta-dtl to other plms and nlp tasks, and ii) exploring how it can benefit other nlp models.