A Joint Learning Approach based on Self-Distillation for Keyphrase Extraction from Scientific Documents (2020)

Abstract: Keyphrase extraction is the task of extracting a small set of phrases that best describe a document. Most existing benchmark datasets for the task typically have limited numbers of annotated documents, making it challenging to train increasingly complex neural networks. In contrast, digital libraries store millions of scientific articles online, covering a wide range of topics. While a significant portion of these articles contain keyphrases provided by their authors, most other articles lack such kind of annotations. Therefore, to effectively utilize these large amounts of unlabeled articles, we propose a simple and efficient joint learning approach based on the idea of self-distillation. Experimental results show that our approach consistently improves the performance of baseline models for keyphrase extraction. Furthermore, our best models outperform previous methods for the task, achieving new state-of-the-art results on two public benchmarks: Inspec and SemEval-2017.

## introduction keyphrase extraction is the task of automatically extracting a set of representative phrases from a document that concisely describe its content. as keyphrases provide a brief yet precise description of a document, they can be utilized for various downstream applications (d' @xcite @xcite . over the past years, researchers have proposed many methods for the task, which can be divided into two major categories: supervised @xcite @xcite and unsupervised techniques @xcite @xcite . in the presence of sufficient domain-specific labeled data, supervised keyphrase extraction methods are often reported to outperform unsupervised methods @xcite @xcite . recently, many deep learning based methods have achieved promising performance in a wide range of nlp tasks @xcite @xcite @xcite . however, most existing benchmark datasets for keyphrase extraction typically have limited numbers of annotated documents, making it challenging to train an effective deep learning model for the task. in contrast, digital libraries store millions of scientific articles online, covering a wide range of topics. while a significant portion of these articles have author-provided keyphrases, most other articles lack such kind of annotations. for example, major nlp conferences (i.e., acl, emnlp, coling) normally do not require authors to provide keywords of their publications. in this paper, to effectively utilize these large amounts of unlabeled articles available online, we propose a novel joint learning approach based on the idea of self-distillation @xcite . to evaluate the effectiveness of our method, we use the inspec and semeval-2017 datasets. experimental results show that our approach consistently improves the performance of baseline models. furthermore, our best models achieve new state-of-the-art results on the two public benchmarks. in summary, we have made the following contributions: (1) we propose a novel joint learning framework based on self-distillation for improving keyphrase extraction systems (2) experiments on two public datasets, inspec and semeval-2017, demonstrate the effectiveness of our proposed methods. in the following parts, we first describe some preliminaries relating to the formulation of the keyphrase extraction problem and the architecture of our baseline models (section 2). we then go into details at our selfdistillation approach in section 3. after that, we describe the conducted experiments and their results in section 4. finally, we conclude this work in section 5. ## preliminaries problem formulation similar to recent works @xcite , we formulate keyphrase extraction as a sequence labeling task. let @xmath0, t 2 , ..., t n ) be a document consisting of n tokens, where t i represents the i th token of the document. the task is to predict a sequence of labels @xmath1, y 2 , ..., y n ), where y @xmath2, b, o} is the label corresponding to token t i . label b denotes the beginning of a keyphrase, i denotes the continuation of a keyphrase, and o corresponds to tokens that are not part of any keyphrase. an advantage of this formulation is that it completely avoids the candidate generation step required in previous ranking-based approaches @xcite . instead of having to generate a list of candidate phrases and then ranking them, we directly predict the target outputs in one go. this formulation also provides a unified approach to keyphrase extraction, as it has the same format as other sequence labeling tasks. baseline models in this work, we employ the bilstm-crf architecture as the baseline architecture @xcite @xcite . figure 1 shows a high-level overview of our baseline model. given a sequence of input tokens, the model first forms a contextualized representation for each token using a transformer-based encoder. the model then further uses a bidirectional lstm @xcite on top of the transformer-based representations. after that, a dense layer is used to map the output of the bidirectional lstm to the label space. finally, a linear-chain crf is applied to decode the labels. ## joint learning based on self-distillation (jlsd) figure 2 shows a high-level overview of our proposed self-distillation approach. we refer to the labeled dataset as the target dataset and the unlabeled dataset as the source dataset. we first train a teacher model using existing labeled examples. after that, we start training a student model parameterized identically to the teacher model. during each training iteration, we sample a batch of both original labeled examples and unlabeled examples with pseudo-labels generated by the teacher. at any point during the training process, if the student model's performance improves (i.e., achieves better results on the validation set of the target dataset), we re-initialize the teacher model with the current parameters of the student model, and then continue training the student using the same procedure. the intuition is to produce a virtuous cycle in which both the teacher and the student will become better together. a better teacher will generate more accurate pseudo-labels, which in turn helps train a better student. and then the improved student will be used to re-initialize the teacher. algorithm 1 formally describes our proposed approach. note that t denotes the number of training iterations, and r is a hyperparameter that determines how many unlabeled documents to be sampled in each iteration. during an iteration of jlsd, if the teacher happens to generate bad pseudo-labels for an example, the effect will not be detrimental. by design, the performance of the teacher is monotonically non-decreasing with time, therefore, when the teacher comes across the same example again, it is like to generate better pseudo-labels. to the best of our knowledge, the only other work exploring self-learning for keyphrase extraction is that of @xcite . however, during each training epoch, their method needs to label all unlabeled examples and then selects the ones with high confidence scores. this would incur a lot of overhead for every single training epoch as the number of unlabeled examples is typically very large. our approach does not suffer from these issues. our teacher model generates pseudo labels on-the-fly during each training iteration. another highly related work is the paper by @xcite . however, the work focuses on the task of keyphrase generation instead of keyphrase extraction. ## experiments and results data and experiments setup in this work, we experimented with two target datasets: inspec and semeval-2017. the inspec dataset @xcite has 1000/500/500 abstracts of scientific articles for the train/dev/test split. the semeval-2017 dataset @xcite has 350/50/100 scientific articles for the train/dev/test split. in our experiments, we use the kp20k dataset @xcite as the source dataset, because it contains more than 500,000 articles collected from various online digital libraries. even though each article in the dataset has author-provided keyphrases, our proposed method does not require such supervised signals. we implemented two baseline models (section 2) with different pre-trained contextual embeddings: bert (base-cased) foot_0 and scibert (scivocab-cased) foot_1 @xcite . from this point, we will refer to baseline models trained only on labeled data as @xcite and @xcite . we refer to models trained using our joint learning approach as [bert + jlsd] and [scibert + jlsd]. for each variant, two different learning rates are used, one for the lower pretrained transformer encoder and one for the upper layers. the optimal hyperparameter values are variantspecific, and we experimented with the following range of possible values: {4, 8, 16} for batch size, {2e-5, 3e-5, 4e-5, 5e-5} for lower learning rate, {1e-4, 2e-4, 5e-4, 1e-3, 5e-3} for upper learning rate, and {25, 50, 75, 100, 125} for number of training epochs. for jlsd, we experimented with various values for r among {0.25, 0.5, 1, 1.5, 2, 4}. we did hyper-parameter tuning using the provided dev sets. ## conclusion in this work, we propose a novel joint learning approach based on self-distillation. experimental results show that our approach consistently improves the performance of baseline models. our best models even achieve new state-of-the-art results on two public benchmarks @xcite . in future work, we plan to explore how to extend our method to other tasks @xcite and other languages. we will also investigate how our method can be used in few-shot settings.