Multi-Task Knowledge Distillation with Embedding Constraints for Scholarly Keyphrase Boundary Classification (2023)

Abstract: The task of scholarly keyphrase boundary classification aims at identifying keyphrases from scientific papers and classifying them with their types from a set of predefined classes (e.g., task, process, or material). Despite the importance of keyphrases and their types in many downstream applications including indexing, searching, and question answering over scientific documents, scholarly keyphrase boundary classification is still an under-explored task. In this work, we propose a novel embedding constraint on multi-task knowledge distillation which enforces the teachers (single-task models) and the student (multi-task model) similarity in the embedding space. Specifically, we enforce that the student model is trained not only to imitate the teachers’ output distribution over classes, but also to produce language representations that are similar to those produced by the teachers. Our results show that the proposed approach outperforms previous works and strong baselines on three datasets of scientific documents.

## introduction scholarly keyphrase boundary classification is the task of identifying highly summative phrases from scientific papers and classifying them into a set of pre-defined classes @xcite . in a scientific domain, keyphrases and their classes, e.g., task, process, or material, are critical for many downstream applications including effectively mining, searching, and analyzing the scientific literature @xcite ; promoting an efficient understanding of what methods, processes, tasks, or resources are being used or proposed in a given paper and how to track them over time @xcite ; scientific paper summarization @xcite ; keyphrase-based question answering @xcite ; machine comprehension @xcite ; scientific paper rec-ommendation @xcite ; topic classification @xcite @xcite ; and, more broadly, data augmentation for nlp tasks @xcite . the scholarly keyphrase boundary classification (skbc) task shares similarities with the named entity recognition (ner) task. specifically, ner aims to identify entities' boundaries and classify them into pre-defined categories (e.g.,'u.n.' is an 'organization' entity) @xcite . similarly, skbc aims to identify the boundary of keyphrases or entities in scientific text and classify them into a set of pre-defined classes (e.g., 'webvision' is a 'dataset' entity and 'object detection' is a 'task' entity). ner @xcite @xcite is treated as a fundamental task in nlp because the named entities often convey the key information of a text. similar to ner, the key information of the scientific discourse available in research papers revolves around scientific entities (e.g., datasets, tasks, models). despite that much research has been done on ner @xcite @xcite @xcite @xcite @xcite , there are only very few works that exist on skbc @xcite @xcite @xcite , and hence, this task of scholarly keyphrase boundary classification is very much under-explored. one potential reason for this under-exploration is the lack of large annotated datasets for this task and the difficulty of annotations which require human experts in a scientific field. in this paper, we propose a novel framework for skbc, called multi-task knowledge distillation with cosine embedding constraints. @xcite showed the effectiveness of the multitask knowledge distillation framework which dis-tills knowledge from multiple teachers (single-task models) to the student model (multi-task model). however, in the process of learning from multiple teachers corresponding to multiple related auxiliary tasks, the student model may become biased towards a particular auxiliary task (or a subset of these tasks) and may "erase" all the "good" information (linguistic features that "matter") from the other remaining tasks. to overcome this, we propose a way of imposing similarity constraints in the embedding space between the student and all the teachers to ensure that the student's final representations are not biased towards some auxiliary task(s), and hence, the student's representations do not diverge (too much) from those of any of the teachers. consequently, our proposed novel embedding constraint lets the student model be focused on the target task by optimally using the knowledge learned from all auxiliary tasks. notably, we achieve state-of-the-art performance on three datasets with our proposed multi-task knowledge distillation with embedding constraints. to facilitate future research on scholarly keyphrase classification, we release our code. 1 our contributions are summarized as follows: the task of automatically identifying and classifying keyphrases from scientific papers has long been under-explored but has recently started to gain attention in the research community. for example, qasemizadeh and schumann (2016) motivated the importance of this task for many applications such as knowledge acquisition and topic tracking, and 1 https://github.com/seoyeon-p/mtl-kd-skic were among the first to create a dataset annotated for terminology extraction and classification from acl anthology papers to stir research in this area. augenstein et al. (2017) proposed a shared task on scienceie at semeval 2017, which includes keyphrase identification and keyphrase classification as subtask a and subtask b, respectively. this shared task attracted a large number of participating teams and achieved a highest f1-score of 56% and 44% for subtasks a and b, respectively, using an rnn with a crf layer on top @xcite . these low f1 scores proved the difficulty of the subtasks. other works participating in this shared task used hand-crafted features for the keyphrase identification and classification. for example, @xcite used combined features of pretrained word embeddings and a set of linguistic features with support vector machines. @xcite and @xcite used crf with linguistically motivated features such as part-of-speech tags and word lemmas. one observation that was made apparent was that the existing annotated datasets for this task are small in size. in order to overcome the small data size problem, @xcite proposed to transfer knowledge from data-rich tasks through deep multi-task learning where keyphrase boudary classification represents the main task and several related tasks such as framenet, semantic supersense tagging, chunking, and multiword expression identification serve as auxiliary tasks (one at a time) to help guide the main task. still, the results of this approach are low @xcite , e.g., yielding an f1 of 45.49% on the semeval 2017 dataset @xcite , and an f1 of 58.95% on the acl dataset (qasemizadeh and schumann, 2016) for keyphrase classification. @xcite proposed to use intermediate task transfer learning using pre-trained language models @xcite to overcome the data scarcity problem by using one intermediate task at a time. in another line of research, @xcite aimed at addressing the same small data size problem and proposed a graph-based semi-supervised algorithm with a data selection strategy to leverage unannotated articles. lai et al. (2020) introduced a semi-supervised approach for scholarly keyphrase identification employing self-knowledge distillation. @xcite also used semi-supervised learning for entity and relation extraction from scientific papers. other works used supervised or semi-supervised keyphrase extraction or generation @xcite @xcite @xcite @xcite @xcite @xcite . moreover, ammar et al. (2018) focused on building a literature (heterogeneous) graph to organize published scientific literature for easy manipulation and discovery. luan et al. (2018) created a new dataset annotated for three tasks, sentence-level scientific entity recognition, relation extraction, and coreference resolution, and used a multi-task learning framework (sciie) to jointly learn from all three tasks. jain et al. (2020) extended sentence-level information extraction to document-level, creating a comprehensive dataset for extracting information from scientific articles. pan et al. (2023) introduced a scholarly benchmark for dataset mention detection to address the limitations of existing corpora in terms of size, diversity of dataset mentions, and entity linking information. in contrast, we propose a cosine embedding constraint added to multi-task knowledge distillation that enforces teacher-student similarity in the embedding space. similar to our work, some works @xcite explored teacher-student similarity at the internal representations. however, their focus is on model compression (to approximate the teacher with a smaller student model) while our work is focused on teacher-student semantic space similarity that ensures the student's final representations do not diverge (too much) from those of the teachers. accordingly, we aim to preserve the beneficial information from the teachers that may not be retained in the multi-task knowledge distillation due to catastrophic forgetting @xcite . along similar lines, mahmoud and hajj (2022) presented multi-objective learning which leveraged knowledge distillation in a multi-task autoencoder network to minimize catastrophic forgetting of pretrained tasks. @xcite introduced a regularization term in multi-task learning, utilizing task uncertainty to enhance model generalization and mitigate the risk of catastrophic forgetting. ## approach in this section, we first describe our baseline multitask learning model ( §4.1). we then present multitask learning that distills knowledge from multiple related auxiliary single-tasks (teachers) into a multi-task (student) model ( §4.2). after that, we describe our multi-task knowledge distillation with embedding constraints ( §4.3). last, we describe the auxiliary tasks ( §4.4). ## experiments 6.1 baselines we use the following baselines for comparison. to compare our method with existing ner approaches, we consider the following two ner methods that utilize a backbone of bilstm and bert, respectively. ## conclusion in this paper, we addressed the task of keyphrase boundary classification using multi-task knowledge distillation. we further enhance this approach by proposing a novel extension in which the student model not only imitates teachers' output distribution over classes but also produces similar language representations. the results of experiments on three datasets showed that our model achieves stateof-the-art results. in the future, we plan to expand our method to various settings such as few-shot learning and semi-supervised settings.