An Empirical Study on Neural Keyphrase Generation (2021)

Abstract: Recent years have seen a flourishing of neural keyphrase generation (KPG) works, including the release of several large-scale datasets and a host of new models to tackle them. Model performance on KPG tasks has increased significantly with evolving deep learning research. However, there lacks a comprehensive comparison among different model designs, and a thorough investigation on related factors that may affect a KPG system’s generalization performance. In this empirical study, we aim to fill this gap by providing extensive experimental results and analyzing the most crucial factors impacting the generalizability of KPG models. We hope this study can help clarify some of the uncertainties surrounding the KPG task and facilitate future research on this topic.

## introduction keyphrases are phrases that summarize and highlight important information in a piece of text. keyphrase generation (kpg) is the task of automatically predicting such keyphrases given the source text. the task can be (and has often been) easily misunderstood and trivialized as yet another natural language generation task like summarization and translation, failing to recognize one key aspect that distinguishes kpg: the multiplicity of generation targets; for each input sequence, a kpg system is expected to output multiple keyphrases, each a mini-sequence of multiple word tokens. despite this unique nature, kpg has been essentially "brute-forced" into the sequence-to-sequence (seq2seq) @xcite framework in the existing literature @xcite @xcite @xcite @xcite .the community has approached the unique challenges with much ingenuity in problem formulation, model design, and evaluation. for example, multiple target phrases have been reformulated by either splitting into one phrase per data point or joining into a single sequence with delimiters (figure 1 ), both allowing straightforward applications of existing neural techniques such as seq2seq. in accordance with the tremendous success and demonstrated effectiveness of neural approaches, steady progress has been made in the past few years -at least empiricallyacross various domains, including sub-areas where it was previously shown to be rather difficult (e.g., in generating keyphrases that are not present in the source text). meanwhile, with the myriad of kpg's unique challenges comes an ever-growing collection of studies that, albeit novel and practical, may quickly proliferate and overwhelm. we are therefore motivated to present this study as -to the best of our knowledge -the first systematic investigation on such challenges as well as the effect of interplay among their solutions. we hope this study can serve as a practical guide to help researchers to gain a more holistic view on the task, and to profit from the empirical results of our investigations on a variety of topics in kpg including model design, evaluation, and hyper-parameter selection. the rest of the paper is organized as follows. we first enumerate specific challenges in kpg due to the multiplicity of its target, and describe general setups for the experiments. we subsequently present experimental results and discussions to answer three main questions: 1. how well do kpg models generalize to various testing distributions? 2. does the order of target keyphrases matter while training one2seq? 3. are larger training data helpful? how to better make use of them? problem definition formally, the task of keyphrase generation (kpg) is to generate a set of keyphrases {p 1 , . . . , p n } given a source text t (a sequence of words). semantically, these phrases summarize and highlight important information contained in t, while syntactically, each keyphrase may consist of multiple words. a keyphrase is defined as present if it is a sub-string of the source text, or as absent otherwise. training paradigms to tackle the unique challenge of generating multiple targets, existing neural kpg approaches can be categorized under one of two training paradigms: one2one @xcite or @xcite , both based on the seq2seq framework. their main difference lies in how target keyphrase multiplicity is handled in constructing data points (figure 1 ). specifically, with multiple target phrases {p 1 , . . . , p n }, one2one takes one phrase at a time and pairs it with the source text t to form n data points (t, p i ) @xmath0:n . during training, a model learns a one-to-many mapping from t to p i 's, i.e., the same source string usually has multiple corresponding target strings. in contrast, one2seq concatenates all ground-truth keyphrases p i into a single string: @xmath1.e., prefixed @xmath2, joint @xmath3, and suffixed @xmath4, thus forming a single data point (t, p ). a system is then trained to predict the concatenated sequence p given t. by default, we con-struct p follow the ordering strategy proposed in @xcite . specifically, we sort present phrases by their first occurrences in source text, and append absent keyphrases at the end. this ordering is denoted as pr e s-ab s in §4. architecture in this paper, we adopt the architecture used in both @xcite and @xcite , using rnn to denote it. rnn is a gru-based seq2seq model @xcite with a copy mechanism @xcite and a coverage mechanism @xcite . we also consider a more recent architecture, transformer @xcite , which is widely used in encoderdecoder language generation literature @xcite . we replace both the encoder gru and decoder gru in rnn by transformer blocks, and denote this architecture variant as trans. both the rnn and trans models can be trained with either the one2one or one2seq paradigm. in recent years, a host of auxiliary designs and mechanisms have been proposed and developed based on either one2one or one2seq (see §6). in this study, however, we focus only on the "vanilla" version of them and we show that given a set of carefully chosen architectures and training strategies, base models can achieve comparable, if not better performance than state-of-the-art methods. we assume that kpg systems derived from either one2one or one2seq model would be affected by these factors of model designing in similar ways. decoding strategies kpg is distinct from other nlg tasks since it expects a set of multi-word phrases (rather than a single sequence) as model predictions. depending on the preference of po- as an example, we illustrate the two decoding strategies in figure 1 . specifically, a one2one model typically collects output keyphrases from all beams and use the top k phrases as the model output (@xmath5. in one2seq, either beam search or greedy decoding can be applied. for beam search, we use both the order of phrases within a beam and the rankings of beams to rank the outputs. in the shown example, top 5 beam search outputs are obtained from the 2 beams with highest rankings. as for greedy decoding, the decoder uses a beam size of 1, and takes all phrases from the single beam as outputs. in this way, the one2seq model can determine the number of phrases to output by itself conditioned on t. evaluation due to the multiplicity of targets in kpg task, the evaluation protocols are distinct from typical nlg tasks. a spectrum of evaluation metrics have been used to evaluate kpg systems, including metrics that truncate model outputs at a fixed number such as f 1 @5 and f 1 @ @xcite ; metrics that evaluate a model's ability of generating variant number of phrases such as f 1 @o and f 1 @m (yuan et al., 2020); metrics that evaluate absent keyphrases such as recall@50 (r@50). detailed definitions of the metrics are provided in appendix a. due to space limit, we mainly discuss f 1 @o, f 1 @10 and r@50 in the main content, complete results with all common metrics are included in appendix e. we save model checkpoints for every 5,000 training steps and report test performance using checkpoints that produce the best f 1 @o or r@50 on the kp20k validation set. datasets a collection of datasets in the domain of scientific publication (kp20k, in s p e c, kr a p i v i n, nus, and se mev a l) and news articles (duc) have been widely used to evaluate kpg task. following previous work, we train models using the training set of kp20k since its size is sufficient to support the training of deep neural networks. evaluation is performed on kp20k's test set as well as all other datasets without fine-tuning. details of the datasets are shown in appendix b. ## generalizability in this section, we show and analyze the generalization performance of kpg systems from 2 dimensions: model architecture and training paradigm. specifically, we compare the two model architectures (i.e., rnn and trans) as described in §2. for each model architecture, we train the kpg model using either of the training paradigms (i.e., one2one or one2seq) also as described in §2. to better understand model variants' generalization properties, we categorize the 6 testing sets into 3 classes according to their distribution similarity with the training data (kp20k), as shown in ## training with more data in this section, we further explore the possibility of improving kpg performance by scaling up the training data. data size has been shown as one of the most effective factors for training language models @xcite but it has yet been discussed in the context of kpg. magkp dataset we construct a new dataset, namely magkp, on the basis of microsoft academic graph @xcite . we filter the original mag v1 dataset (166 million papers, multiple domains) and only keep papers in computer science and with at least one keyphrase. this results in 2.7 million data points (5× larger than kp20k). this dataset remains noisy despite the stringent filtering criteria, this is because 1) the data is crawled from the web and 2) some keywords are labeled by automatic systems rather than humans. this noisy nature brings many interesting observations. ## related work traditional keyphrase extraction keyphrase extraction has been studied extensively for decades. a common approach is to formulate it as a twostep process. specifically, a system first heuristically selects a set of candidate phrases from the text using some pre-defined features @xcite @xcite . subsequently, a ranker is used to select the top ranked candidates following various criteria. the ranker can be bagged decision trees @xcite , multi-layer perceptron, support vector machine (lopez and romary, 2010) or pagerank @xcite @xcite . compared to the newly developed data driven approaches with deep neural networks, the above approaches suffer from poor performance and the need of dataset-specific heuristic design. ## conclusion and takeaways we present an empirical study discussing neural kpg models from various aspects. through extensive experiments and analysis, we answer the three questions ( §1). results suggest that given a carefully chosen architecture and training strategy, a base model can perform comparable with fancy sota models. further augmented with (noisy) data in the correct way, a base model can outperform sota models (appendix e). we strive to provide a guideline on how to choose such architectures and training strategies, which hopefully can be proven valuable and helpful to the community. we conclude our discussion with the following takeaways: one2one performs better on absent kpg. see section 3. wang chen, hou pong chan, piji li, and irwin king. 2020. exclusive hierarchical decoding for deep keyphrase generation. in proceedings of the 58th annual meeting of the association for computational linguistics. association for computational linguistics. wang chen, yifan gao, jiani zhang, irwin king, and michael r lyu. 2019b. title-guided encoding for keyphrase generation. in proceedings of the aaai conference on artificial intelligence, volume 33, pages 6268-6275. kyunghyun cho, bart van merriënboer, caglar gulcehre, dzmitry bahdanau, fethi bougares, holger schwenk, and yoshua bengio. 2014. learning phrase representations using rnn encoder-decoder for statistical machine translation. in proceedings of the 2014 conference on empirical methods in natural language processing (emnlp), pages 1724-1734, doha, qatar. association for computational linguistics. sebastian gehrmann, yuntian deng, and alexander rush. 2018. bottom-up abstractive summarization. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 4098-4109, brussels, belgium. association for computational linguistics. sujatha das gollapalli, xiaoli li, and peng yang. 2017. incorporating expert knowledge into keyphrase extraction. in aaai, pages 3180-3187. aaai press. jiatao gu, zhengdong lu, hang li, and victor o.k. li. 2016. incorporating copying mechanism in sequence-to-sequence learning. in proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1631-1640, berlin, germany. association for computational linguistics. caglar gulcehre, sungjin ahn, ramesh nallapati, bowen zhou, and yoshua bengio. 2016. pointing the unknown words. arxiv preprint arxiv:1603.08148. tho thi ngoc le, minh le nguyen, and akira shimazu. 2016. unsupervised keyphrase extraction: introducing new kinds of words to keyphrases. 29th australasian joint conference, hobart, tas, australia, december 5-8, 2016. zhiyuan liu, xinxiong chen, yabin zheng, and maosong sun. 2011. automatic keyphrase extraction by bridging vocabulary gap. the fifteenth conference on computational natural language learning. patrice lopez and laurent romary. 2010. humb: automatic key term extraction from scientific articles in grobidp. the 5th international workshop on semantic evaluation. yi luan, mari ostendorf, and hannaneh hajishirzi. 2017. scientific information extraction with semisupervised neural tagging. in proceedings of the 2017 conference on empirical methods in natural language processing, pages 2641-2651, copenhagen, denmark. association for computational linguistics. olena medelyan, eibe frank, and ian h. witten. 2009. human-competitive tagging using automatic keyphrase extraction. in proceedings of the 2009 conference on empirical methods in natural language processing, pages 1318-1327, singapore. association for computational linguistics. rui meng, sanqiang zhao, shuguang han, daqing he, peter brusilovsky, and yu chi. 2017. deep keyphrase generation. in proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers), pages 582-592. association for computational linguistics. rada mihalcea and paul tarau. 2004. textrank: bringing order into text. in proceedings of the 2004 conference on empirical methods in natural language processing, pages 404-411, barcelona, spain. association for computational linguistics. myle ott, sergey edunov, david grangier, and michael auli. 2018. scaling neural machine translation. arxiv preprint arxiv:1806.00187. colin raffel, noam shazeer, adam roberts, katherine lee, sharan narang, michael matena, yanqi zhou, wei li, and peter j liu. 2019. exploring the limits of transfer learning with a unified text-to-text transformer. arxiv preprint arxiv:1910.10683. abigail see, peter j. liu, and christopher d. manning. 2017. get to the point: summarization with pointergenerator networks. in proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1073-1083, vancouver, canada. association for computational linguistics. arnab sinha, zhihong shen, yang song, hao ma, darrin eide, bo-june hsu, and kuansan wang. 2015. an overview of microsoft academic service (mas) and applications. in proceedings of the 24th international conference on world wide web, pages 243-246. emma strubell, ananya ganesh, and andrew mccallum. 2019. energy and policy considerations for deep learning in nlp. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 3645-3650, florence, italy. association for computational linguistics. sandeep subramanian, tong wang, xingdi yuan, saizheng zhang, adam trischler, and yoshua bengio. 2018. neural models for key phrase extraction and question generation. in proceedings of the workshop on machine reading for question answering, pages 78-88, melbourne, australia. association for computational linguistics. zhiqing sun, jian tang, pan du, zhi-hong deng, and jian-yun nie. 2019. divgraphpointer: a graph pointer network for extracting diverse keyphrases. in proceedings of the 42nd international acm si-gir conference on research and development in information retrieval, sigir'19, page 755-764, new york, ny, usa. association for computing machinery. ilya sutskever, oriol vinyals, and quoc v. le. 2014. sequence to sequence learning with neural networks. in proceedings of the 27th international conference on neural information processing systems -volume 2, nips'14, pages 3104-3112, cambridge, ma, usa. mit press. avinash swaminathan, haimin zhang, debanjan mahata, rakesh gosangi, rajiv shah, and amanda stent. 2020. a preliminary exploration of gans for keyphrase generation. in proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pages 8021-8030. ashish vaswani, noam shazeer, niki parmar, jakob uszkoreit, llion jones, aidan n gomez, ł ukasz kaiser, and illia polosukhin. 2017. attention is all you need. in advances in neural information processing systems 30. oriol vinyals, samy bengio, and manjunath kudlur. 2016. order matters: sequence to sequence for sets. in international conference on learning representations (iclr). xiaojun wan and jianguo xiao. 2008. single document keyphrase extraction using neighborhood knowledge. aaai. minmei wang, bo zhao, and yihua huang. 2016. ptr: phrase-based topical ranking for automatic keyphrase extraction in scientific publications. 23rd international conference, iconip 2016. yue wang, jing li, hou pong chan, irwin king, michael r. lyu, and shuming shi. 2019. topicaware neural keyphrase generation for social media language. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 2516-2526, florence, italy. association for computational linguistics. ian h. witten, gordon w. paynter, eibe frank, carl gutwin, and craig g. nevill-manning. 1999. kea: practical automatic keyphrase extraction. in proceedings of the fourth acm conference on digital libraries, dl '99, pages 254-255, new york, ny, usa. acm. zhilin yang, junjie hu, ruslan salakhutdinov, and william w. cohen. 2017. semi-supervised qa with generative domain-adaptive nets. in the 55th annual meeting of the association for computational linguistics. association for computational linguistics. hai ye and lu wang. 2018. semi-supervised learning for neural keyphrase generation. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 4142-4153, brussels, belgium. association for computational linguistics. xingdi yuan, tong wang, rui meng, khushboo thaker, peter brusilovsky daqing he, and adam trischler. 2020. one size does not fit all: generating and evaluating variable number of keyphrases. in proceedings of the 58th annual meeting of the association for computational linguistics. association for computational linguistics. qi zhang, yang wang, yeyun gong, and xuanjing huang. 2016. keyphrase extraction using deep recurrent neural networks on twitter. in proceedings of the 2016 conference on empirical methods in natural language processing, pages 836-845. association for computational linguistics. jing zhao and yuxiang zhang. 2019. incorporating linguistic constraints into keyphrase generation. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 5224-5233.