Scientific Keyphrase Identification and Classification by Pre-Trained Language Models Intermediate Task Transfer Learning (2020)

Abstract: Scientific keyphrase identification and classification is the task of detecting and classifying keyphrases from scholarly text with their types from a set of predefined classes. This task has a wide range of benefits, but it is still challenging in performance due to the lack of large amounts of labeled data required for training deep neural models. In order to overcome this challenge, we explore pre-trained language models BERT and SciBERT with intermediate task transfer learning, using 42 data-rich related intermediate-target task combinations. We reveal that intermediate task transfer learning on SciBERT induces a better starting point for target task fine-tuning compared with BERT and achieves competitive performance in scientific keyphrase identification and classification compared to both previous works and strong baselines. Interestingly, we observe that BERT with intermediate task transfer learning fails to improve the performance of scientific keyphrase identification and classification potentially due to significant catastrophic forgetting. This result highlights that scientific knowledge achieved during the pre-training of language models on large scientific collections plays an important role in the target tasks. We also observe that sequence tagging related intermediate tasks, especially syntactic structure learning tasks such as POS Tagging, tend to work best for scientific keyphrase identification and classification.

## related work scientific keyphrase identification and classification (skic) was proposed at semeval 2017 task 10 as task a: keyphrase identification, and task b: keyphrase classification @xcite . the authors of this shared task proposed to employ the bio schema, which refers to the beginning (b), inside (i), or outside (o) of keyphrases, respectively. most of the earlier approaches to skic rely on hand-crafted linguistic features. for example, @xcite and @xcite employ syntactic and semantic features such as part-of-speech tags and word lemmas, as input to conditional random fields (crfs). @xcite formulate skic as a supervised multi-class classification problem. they exploit pre-trained word embeddings and linguistically inspired features, e.g., noun phrase features and orthographic features, which represent character and symbolic features of given tokens, as input to support vector machines. with the success of neural models, recent works try to address skic using neural architectures while exploiting the bio schema. although both tasks, keyphrase identification and keyphrase classification according to their types, are very important, many works focused only on keyphrase extraction/generation or identification/segmentation @xcite @xcite @xcite . the classification task is less explored possibly due to a lack of a large number of gold-label keyphrase classification datasets. precisely, there are only a few publicly available datasets for keyphrase classification @xcite @xcite and these datasets are small in size. to overcome the small dataset size problem, @xcite proposed the multitask learning of skic. according to this work, they reveal that sequence tagging auxiliary tasks such as chunking, super-sense tagging, multi-words expressions identification, and framenet target identification are beneficial to skic within a multitask learning framework (with one auxiliary task at a time). the recent unsupervised pre-trained language models such as bert @xcite achieve stateof-the-art performance in many downstream nlp tasks, such as named entity recognition e.g., 95.5% f1 on the conll-2003 dataset @xcite . @xcite apply bert and use fine-tuning of bert to reduce the vocabulary gap between canonical domains and historical domains, within an unsupervised approach. moreover, the domain-specific bert models such as scibert @xcite , produce better results compared to a general domain-based bert with respect to scientific knowledge required tasks such as scientific term classification e.g., 64.57% f1 on the sciie dataset @xcite . to exploit these pre-trained language models, @xcite propose a data-rich intermediate transfer learning and prove that this method provides a better starting point of target tasks. extended from this, in our work, we propose the transfer of the intermediate tasks fine-tuning to overcome the scarcity of gold-labeled large skic datasets. ## methodology given a document @xmath0, w 2 , . . . w n }, where n denotes the length of the document and w k is a word in d (@xmath1, • • • , n), the objective of skic is to identify and classify keyphrases of d by employing an output sequence {y 1 , y 2 , . . . y n }, where y k follows bio scheme for pre-defined classes, e.g., task, material, process. in the bio scheme, 'b' denotes the beginning word of a keyphrase, 'i' refers to the inside word of a keyphrase, and 'o' indicates all other words that are not part of a keyphrase. an example of input-output pairs of keyphrase identification and classification is shown in figure 1 . ## experiments baseline we compare the intermediate task transfer learning with the following baselines. experimental setup for the semeval 2017 and sciie, we use the published train, validation, and test sets. for the acl rd-tec 2.0 dataset, we perform 60/10/30 split to create the train, validation, and test sets. we estimate models' hyper-parameters via a grid search over combinations. the ranges of parameters we explore are the following: batch size @xcite @xcite @xcite , learning rate [0.01, 0.0001], momentum [0.5, 0.9]. the information about the best hyper-parameters setting is as follows. for each target task, we use the batch size 1. the intermediate tasks of supersense, mwes, framenet use batch size 4, while chunking uses 32, pos tagging uses 64, scitail uses 16, and mnli uses 4. for bert and scibert fine-tuning, we use sgd optimizer with learning rate 5e-3 with momentum 0.9. in intermediate-task transfer learning, we train intermediate tasks with learning rate 5e-3 without applying momentum. we set a higher learning rate as 5e-3 and set a lower learning rate as 5e-4, both with momentum 0.9. we run training for a maximum of 10 epochs with negative log-likelihood loss. aside from these details, we follow the scibert paper for all other training hyper-parameters. all experiments are done in p3.2xlarge settings of amazon web services. all model parameters are estimated on the validation set of each task. we evaluate the performance of each model using phrase-level micro-averaged f1 and use the exact match metric @xcite . ## conclusion, discussion, and future work we investigated the performance of data-rich intermediate task transfer learning for scientific keyphrase identification and classification (skic) using pre-trained language models bert and scibert. we perform experiments on scibert and bert with a total of 42 pairs of intermediate and target tasks, where intermediate tasks are drawn from sequence tagging and natural language inference. we found that employing sequence tagging tasks as intermediate tasks on scibert performs the best on three publicly available keyphrase identification and classification datasets. further, the intermediate task transfer learning using scibert outperforms both the previous work and strong baselines by a large margin. specifically, for the semeval 2017 task 10 dataset, using framenet as an intermediate task on scibert transfer learning yields +7.19 improvement in f1 in keyphrase identification, and using pos tagging yields +8.69 improvement in f1 in keyphrase classification. for the acl rd-tec 2.0 dataset, using pos tagging leads to +6.16 f1 in keyphrase identification, and +4.79 f1 in keyphrase classification. for the sciie dataset, using supersense brings +1.63 f1 in keyphrase identification, and +7.79 f1 in keyphrase classification. according to these results, syntactic structure learning related intermediate tasks such as pos tagging are preferable for skic tasks. further, looking at our intermediate task domain difference that lies in natural language inference tasks, we explore their impact on the pre-trained language models. in particular, we empirically showed that using mnli as an intermediate task on scibert transfer learning returns higher performance than employing scitail as an intermediate task on bert transfer learning. future works in this area will benefit from the improvement of the available intermediate tasks and other related intermediate tasks will be explored. moreover, a better understanding of when and why these intermediate tasks are working is one of the interesting future directions. interestingly, we observe that bert suffers from a serious drop in performance possibly due to catastrophic forgetting. in particular, we observe that almost all of the intermediate tasks fail to provide better starting points for skic pre-trained language models fine-tuning. while semeval 2017 task 10 dataset achieves the best results when transfer from chunking on bert, this result is lower than single scib-ert fine-tuning. on acl rd-tec 2.0 and sciie, no intermediate task produces higher performance on bert intermediate task transfer learning. one potential reason could be a large vocabulary gap between our domain and the collections used to pre-train bert. in the future, we plan to analyze the differences between bert and scibert to better understand the effects of transfer learning for skic.