Learning Rich Representation of Keyphrases from Text (2022)

Abstract: In this work, we explore how to train task-specific language models aimed towards learning rich representation of keyphrases from text documents. We experiment with different masking strategies for pre-training transformer language models (LMs) in discriminative as well as generative settings. In the discriminative setting, we introduce a new pre-training objective - Keyphrase Boundary Infilling with Replacement (KBIR), showing large gains in performance (upto 8.16 points in F1) over SOTA, when the LM pre-trained using KBIR is fine-tuned for the task of keyphrase extraction. In the generative setting, we introduce a new pre-training setup for BART - KeyBART, that reproduces the keyphrases related to the input text in the CatSeq format, instead of the denoised original input. This also led to gains in performance (upto 4.33 points in F1@M) over SOTA for keyphrase generation. Additionally, we also fine-tune the pre-trained language models on named entity recognition (NER), question answering (QA), relation extraction (RE), abstractive summarization and achieve comparable performance with that of the SOTA, showing that learning rich representation of keyphrases is indeed beneficial for many other fundamental NLP tasks.

## introduction and background keyphrases capture the most salient topics of a document and facilitates extreme summarization. identifying them in an automated way from a text document can be useful for several downstream tasks -classification @xcite , clustering @xcite , summarization @xcite , reviewer and document recommendation @xcite , and many different information retrieval tasks such as enabling semantic and faceted search @xcite , query expansion @xcite , and interactive document retrieval @xcite . keyphrases could either be extractive (part of the document) or abstractive (not part of the document). prior works have referred to them as present and absent keyphrases, respectively. automatically identifying them entails the process of detecting the extractive @xcite and generating the abstractive keyphrases (c ¸ano and bojar, 2019a) from a given document. while extractive approaches have mostly dominated over the generative ones with higher accuracies (c ¸ano and bojar, 2019b), the task is far from solved and the performances of the present systems are worse in comparison to many other nlp tasks @xcite . some of the major challenges are the varied length of the documents to be processed, their structural inconsistency and developing strategies that can perform well in different domains. most of the prior work on identifying keyphrases using deep learning techniques have concentrated on developing new architectures and frameworks based on different training paradigms such as seq2seq @xcite @xcite @xcite @xcite , sequence tagging @xcite , reinforcement learning @xcite , adversarial training @xcite and game theory @xcite . although, there has been tremendous progress in learning better semantic and syntactic representation of language at different levels -characters, words, phrases, sentences and documents @xcite , there hasn't been any effort in learning rich pre-trained representations of keyphrases, which is the major focus of this work. transformer language models when pre-trained on large corpora with different pre-training objectives @xcite have shown great success in various downstream tasks on fine-tuning, including the tasks of keyphrase extraction @xcite @xcite and generation @xcite . however, pre-training objectives tailored towards learning better representation of keyphrases that can result in improving the performance of identifying and generating keyphrases from text have not yet been explored. this motivated us to look into this specific problem and make an attempt to answer the following questions: q1 -can we formulate a pre-training objective for language models that can learn better representation of keyphrases? previous work explored training language models for learning better representation of text spans @xcite , summary sentences @xcite , and tokens for named entity recognition @xcite . to effectively learn rich representation of keyphrases in a bert like discriminative setup, we propose a new pretraining objective -keyphrase boundary infilling with replacement (kbir) (section 2) which utilizes a multi-task learning setup for optimizing a combined loss of random token masked language modeling (mlm) @xcite , keyphrase boundary infilling (kbi) (section 2.1) and keyphrase replacement classification (krc) (section 2.2). we also propose a new setup for pre-training bart @xcite ) -keybart (section 3), focused towards learning better representation of keyphrases in a generative setting. instead of reproducing the denoised input text as proposed in the original setup, we produce the keyphrases associated with the input document in the catseq @xcite format from a corrupted input. q2 -does learning rich representation of keyphrases in a language model lead to performance gains for the tasks of keyphrase extraction and generation? one of the key contributions of this work is the introduction of kbir, which is the combination of the kbi and krc objectives with mlm that helps to learn good representation of keyphrases. this is validated by obtaining sota performance for the task of keyphrase extraction on three benchmark datasets (section 4.2.1), surpassing the existing sota @xcite by at most 8.16 f1 points on the semeval 2017 corpus @xcite . we also evaluated the keybart approach across five benchmark datasets for the task of keyphrase generation and obtained sota performances for both present and absent keyphrases (section 4.2.2). our best model surpassed the sota one2seq model @xcite by 4.33 f1@m points and 0.72 f1@m points on inspec @xcite for present and absent keyphrases respectively. q3 -do rich keyphrase representations aid other fundamental tasks in nlp such as ner, qa, re and summarization? it is to be noted, that although we trained our models on a large corpus of 23 million scientific articles, we find that it performs reasonably well when fine-tuned on datasets that do not belong to the scientific domain for different nlp tasks as shown in section 4.2.3, 4.2.4, 4.2.5 and 4.2.6. this also suggests that identifying keyphrases in the context of an input text is a fundamental nlp task and a language model trained to learn optimal representation of keyphrases can aid many other tasks. to summarize the main contributions that we make in this work are: we have made our models 1 2 publicly avail-able. we also make our pre-training code foot_2 available. next, we give a detailed description of the methods that we propose in this work. in the previous section, we mentioned various methods that aim at learning representations of text spans. unlike lms like spanbert @xcite and pegasus @xcite ) whose primary objective is to learn representations of random or heuristically chosen spans of text, the intuition behind learning good keyphrase representation is to provide the lm the ability to learn spans as well as to identify important phrases (in this case keyphrases) in the context of an input text. this motivated us to devise a framework that can optimize both of these objectives. towards this effort, we propose a new pre-training objective keyphrase boundary infilling with replacement (kbir) which is composed of two individual tasks -keyphrase boundary infilling (kbi) and keyphrase replacement classification (krc) jointly learnt in a multi-task learning setup as shown in figure 1 . we build our framework on top of roberta which implements random token masked language modeling (mlm), therefore making our lm essentially optimizing mlm along with kbi and krc objectives. in the following section, we describe the individual components of our framework. ## qualitative analysis we perform a qualitative analysis on the semeval-2010 dataset as it is the only common dataset between ke and kg tasks by leveraging predictions from the best performing models. we present examples in ## other experiments we also attempted to use a combination of the wikipedia (english) dump and s2orc @xcite corpora for pre-training our models. in order to obtain keyphrase tags for data at a large scale, we employed textrank @xcite on each document in the corpora. we set the maximum number of keyphrases to 10 for the textrank algorithm and considered all keyphrases tagged by textrank. we created random splits for our dataset to generate a train and development (dev) set. however, we found that keyphrases tagged in this manner added a lot of noise to the dataset and resulted in only marginal overall gains. to train a keyphrase specific language model, we also used a combined generative and discriminative approach as introduced in electra @xcite . the generative approach made the model predict the masked tokens that are part of a keyphrase. in the discriminative approach, the original sequence was perturbed by replacing the tokens of a keyphrase with another semantically unrelated keyphrase. we were unable to stabilize the training for such a setup and didn't get promising results. ## conclusion and future work we explored lms capable of learning rich representations of keyphrases that achieve sota performance across multiple datasets for keyphrase extraction and generation tasks. towards this effort, we proposed a new pre-training objective kbir and a new training setup keybart. the trained lms demonstrate their effectiveness by achieving sota or near sota performance for various downstream nlp tasks when fine-tuned on benchmark datasets spanning across multiple domains. as a next step, we would like to probe our lms to understand them more and also gauge their effectiveness for the tasks of cross-domain keyphrase extraction and generation. we would also like to explore scaling these approaches to more datasets by revisiting more sophisticated unsupervised keyphrase tagging methods.