General-to-Specific Transfer Labeling for Domain Adaptable Keyphrase Generation (2023)

Abstract: Training keyphrase generation (KPG) models require a large amount of annotated data, which can be prohibitively expensive and often limited to specific domains. In this study, we first demonstrate that large distribution shifts among different domains severely hinder the transferability of KPG models. We then propose a three-stage pipeline, which gradually guides KPG models’ learning focus from general syntactical features to domain-related semantics, in a data-efficient manner. With domain-general phrase pre-training, we pre-train Sequence-to-Sequence models with generic phrase annotations that are widely available on the web, which enables the models to generate phrases in a wide range of domains. The resulting model is then applied in the Transfer Labeling stage to produce domain-specific pseudo keyphrases, which help adapt models to a new domain. Finally, we fine-tune the model with limited data with true labels to fully adapt it to the target domain. Our experiment results show that the proposed process can produce good quality keyphrases in new domains and achieve consistent improvements after adaptation with limited in-domain annotated data. All code and datasets are available at https://github.com/memray/OpenNMT-kpg-release .

## introduction the last decade has seen major advances in deep neural networks and their applications in natural language processing. particularly, the subarea of neural keyphrase generation (kpg) has made great progress with the aid of large language models @xcite and large-scale datasets @xcite . due to the high cost of data annotation, most, if not all, of the large-scale kpg datasets are constructed by scraping domain-specific data from the internet. for example, @xcite collected more than 500k scientific papers of which keyphrases are provided by paper authors. gallina et al. crawled about 280k news articles from new york times with editor-assigned keyphrases. following gururangan et al. (2020), we use "domain" to denote a distribution over language characterizing a given topic or genre. specifically in kpg tasks, domains can be "computer science papers", "online forum articles", "news" etc. despite recent neural models can to some extent learn kpg skills from existing datasets @xcite @xcite , because most of these datasets are limited to a single domain, it remains unclear how the trained models can be transferred to new domains, especially in a real-world setting. some existing studies claim their models demonstrate a certain degree of transferability across domains. for instance, @xcite show that models trained with scientific paper datasets can generate decent quality keyphrases from news articles, in a zero-shot manner. @xcite that training with open-domain web documents can improve the model's generalizability. however, there is a lack of systematic studies on domain transferring kpg, and thus the observations reported in prior works do not support a comprehensive understanding of this topic. to investigate this question, we conduct an empirical study on how well kpg models can transfer across domains. we utilize commonly used kpg datasets covering four different domains (science, news, web, q&a). we first show experiment results ( §2.2) that suggest models trained with data in a specific domain do not generalize well to other domains, even in cases where they are initialized with pre-trained language models such as bart @xcite . we also visualize the domain gaps among datasets by inspecting their phrase overlaps. keyphrases often represent the specific knowledge of a domain and this may result in the failure of transferring models across domains. the empirical study motivates us to explore novel methods that can help models possess the ability of generating high quality keyphrases and more importantly, can quickly adapt to a new domain with limited amount of annotation. we propose a three-stage training pipeline, in which we gradually guide a kpg model's learning focus from general syntactical features to domain-specific information. first, we pre-train the model using community labeled phrases in wikipedia ( §3.1). then, we use a novel self-training-based domain adaptation method, namely transfer labeling, to adapt the model to the new domain. note this domain adaptation method does not require ground-truth labels, we leverage the model pre-trained in the previous stage to generate pseudo-labels for training itself. finally, we use a limited amount of in-domain data with true annotations to fully adapt the model to the new domain. we report extensive experiment results and thorough analyses to demonstrate the effectiveness of the proposed methods. ## experiments we reuse the model architecture described in §2.2 throughout this paper. and most models apply a single iteration of transfer labeling. we discuss the effect of multi-iteration transfer labeling in §4.2.5. see appendix a.1 for implementation details. ## conclusion in this study, we investigate domain gaps in the kpg task that hinder models from generalization. we attempt to alleviate this issue by proposing a three-stage pipeline to strategically enhance models' abilities on keyness and phraseness. essentially, we consider phraseness as a domain-general property and can be acquired from wikipedia data as distant supervision. then we use self-labeling to distill the phraseness into data in a new domain, and the resulting pseudo labels are used for domain adaptation, as the labels can reflect the keyness and phraseness of the new domain. finally, we finetune the model with limited amount of target domain data with true labels. by taking the advantage of open-domain knowledge on the web, we believe this general-to-specific paradigm is generic and can be applied to a wide variety of machine learning tasks. as a next step, we plan to employ the proposed method for text classification and information retrieval, to see whether the domain-general phrase model can produce reliable class labels and queries for domain adaptation.