Representation Learning for Resource-Constrained Keyphrase Generation (2022)

Abstract: State-of-the-art keyphrase generation methods generally depend on large annotated datasets, limiting their performance in domains with limited annotated data. To overcome this challenge, we design a data-oriented approach that first identifies salient information using retrieval-based corpus-level statistics, and then learns a task-specific intermediate representation based on a pre-trained language model using large-scale unlabeled documents. We introduce salient span recovery and salient span prediction as denoising training objectives that condense the intra-article and inter-article knowledge essential for keyphrase generation. Through experiments on multiple keyphrase generation benchmarks, we show the effectiveness of the proposed approach for facilitating low-resource keyphrase generation and zero-shot domain adaptation. Our method especially benefits the generation of absent keyphrases, approaching the performance of models trained with large training sets.

## introduction keyphrases of a document are the phrases that summarize the most important information. in the keyphrase generation task, given a document, a model is required to generate a set of keyphrases, each of which can be classified as a present keyphrase if it appears as a contiguous text span in the document or an absent keyphrase otherwise. the generated keyphrases can facilitate a wide range of applications, such as document clustering @xcite , recommendation systems @xcite , information retrieval tasks @xcite @xcite ), text summarization @xcite , and text classification @xcite @xcite . ## methods problem definition let d kp denote a keyphrase generation dataset, which is a set of tuples (x i , p i ), where x i is an input document and a typical way to train bart-like sequence-tosequence plms for keyphrase generation is to directly fine-tune using the formulation above. designed for small d kp , our method first extracts salient spans using d aux and further trains bart using these spans. the resulting model with taskspecific and domain-specific knowledge is then fine-tuned in the same way. section 2.1 introduces the salient span extraction method, and section 2.2 details the intermediate learning objectives. ## results and analysis we aim to address the following questions. how much do they overlap with salient spans? ## conclusion this paper considers the problem of low-resource keyphrase generation. we design an innovative retrieval-based method to extract salient information from unlabeled documents and perform continued bart pre-training. we verify that the method facilitates low-resource keyphrase generation and zero-shot cross-domain generalization. our method consistently outperforms the baselines in a range of resource schemes. future works may consider investigating dense embeddings for extracting salient spans, composing the proposed objectives, or designing specialized methods for finetuning on small datasets.