Keyphrase Generation for Scientific Document Retrieval (2020)

Abstract: Sequence-to-sequence models have lead to significant progress in keyphrase generation, but it remains unknown whether they are reliable enough to be beneficial for document retrieval. This study provides empirical evidence that such models can significantly improve retrieval performance, and introduces a new extrinsic evaluation framework that allows for a better understanding of the limitations of keyphrase generation models. Using this framework, we point out and discuss the difficulties encountered with supplementing documents with -not present in text- keyphrases, and generalizing models across domains. Our code is available at https://github.com/boudinfl/ir-using-kg

## introduction with the exponential growth of the scientific literature @xcite , retrieving relevant scientific papers becomes increasingly difficult. keywords, also referred to as keyphrases, provide an effective way to supplement paper indexing and improve retrieval effectiveness in scientific digital libraries @xcite @xcite . however, only few documents have assigned keyphrases, and those who do were, for the most part, selflabeled by their authors, thus exhibiting annotation inconsistencies @xcite . this has motivated an active line of research on automatic keyphrase extraction (see @xcite for an overview) and, more recently, keyphrase generation @xcite , where the task is to find a set of words and phrases that represents the main content of a document. although models for predicting keyphrases have been extensively evaluated on their ability to reproduce author's keywords, it still remains unclear whether they can be usefully applied in information retrieval. one reason for this lack of evidence may have been their relatively low performance discouraging attempts at using them for indexing @xcite ). yet, recently proposed models not only achieve much better performance, but also display a property that may have a significant impact on retrieval effectiveness: the capacity to generate keyphrases that do not appear in the source text. these absent keyphrases do not just highlight the topics that are most relevant, but provide some form of semantic expansion by adding new content (e.g. synonyms, semantically related terms) to the index @xcite . the goal of this paper is two-fold: to gather empirical evidence as to whether current keyphrase generation models are good enough to improve scientific document retrieval, and to gain further insights into the performance of these models from an extrinsic perspective. our contributions are listed as follows: ## methodology this section presents our methodology for assessing the usefulness ( §2.3) of keyphrase generation ( §2.2) in scientific document retrieval ( §2.1). ## results results for retrieval models using keyphrase generation are reported in overall, bm25+rm3 achieves the best retrieval effectiveness, confirming previous findings on adhoc retrieval in limited data scenarios @xcite . for clarity and conciseness, we focus on this model in the rest of this paper. encouraging diversity in keyphrases seems not to be appropriate for retrieval, as seq2seq+corr consistently gives lower results than seq2seq+copy. it is also interesting to see that the effectiveness gains of query expansion (rm3) and document expansion are additive, suggesting that they provide different but complementary relevance signals. moreover, our results show that query expansion is more effective, which is in line with past work @xcite . one hyper-parameter that we have deliberately left untouched so far is the number n of predicted keyphrases that directly controls the precisionrecall trade-off of keyphrase generation models. to understand how this parameter affects retrieval effectiveness, we repeated our experiments by varying n within the range [0, 9], and plotted the results in figure 1 . without author keywords, we observe that all models achieve gains, but only seq2seq+copy does yield significant improvements. with author keywords, seq2seq+copy is again the only model that achieves significance, while the others show mixed results, sometimes even degrading scores. one likely explanation for this is that these models produce keyphrases that cause documents to drift away from their original meaning. we note that results are close to optimal for @xmath0, supporting our initial setting for this parameter. from our experiments, it appears that unsupervised keyphrase extraction is not effective enough to significantly improve retrieval effectiveness. the fact that keyphrase generation does so, suggests that the ability to predict absent keyphrases may be what enables better performance. yet counterintuitively, we found that most of the gains in retrieval effectiveness are due to the high extractive accuracy of keyphrase generation models. results in neural models for keyphrase generation exhibit a limited generalization ability, which means that their performance degrades on documents that differ from the ones encountered during training (cf. ## conclusion we presented the first study of the usefulness of keyphrase generation for scientific document retrieval. our results show that keyphrases can significantly improve retrieval effectiveness, and also highlight the importance of evaluating keyphrase generation models from an extrinsic perspective. other retrieval tasks may also benefit from using keyphrase information and we expect our results to serve as a basis for further improvements.