Redefining Absent Keyphrases and their Effect on Retrieval Effectiveness (2021)

Abstract: Neural keyphrase generation models have recently attracted much interest due to their ability to output absent keyphrases, that is, keyphrases that do not appear in the source text. In this paper, we discuss the usefulness of absent keyphrases from an Information Retrieval (IR) perspective, and show that the commonly drawn distinction between present and absent keyphrases is not made explicit enough. We introduce a finer-grained categorization scheme that sheds more light on the impact of absent keyphrases on scientific document retrieval. Under this scheme, we find that only a fraction (around 20%) of the words that make up keyphrases actually serves as document expansion, but that this small fraction of words is behind much of the gains observed in retrieval effectiveness. We also discuss how the proposed scheme can offer a new angle to evaluate the output of neural keyphrase generation models.

## introduction searching the scholarly literature for documents of interest is becoming frustratingly difficult and timeconsuming as the volume of published research grows exponentially. one promising approach to address this problem and improve the retrievability of documents is to supplement paper indexing with automatically generated keyphrases @xcite @xcite . traditionally, keyphrases are defined as a short list of terms that represent the main concepts in a document @xcite . in recent years, this definition was further refined to differentiate between keyphrases that are present in the source document or not, and in turn, proposed models for producing keyphrases were divided into extractive @xcite @xcite @xcite and generative models @xcite @xcite el asri, 2020, inter alia) based on their ability to output absent keyphrases. obviously, keyphrases have different effects on retrieval models depending on whether or not they occur in the document: present keyphrases highlight important parts of the input and make weighting terms easier, while absent keyphrases add new terms to the input and provide some form of document expansion. intuitively, assigning absent keyphrases is more appealing since it may alleviate the vocabulary mismatch problem between query terms and relevant documents @xcite , hence enabling the retrieval of relevant documents that otherwise would have been missed. this is especially true for scholarly collections, in which documents are mostly short texts (i.e. scientific abstracts) due to licensing issues and/or resource limitations @xcite . yet, the extent to which present and absent keyphrases contribute to improved retrieval effectiveness has not been thoroughly explored. worse still, there is no unique and rigorous definition of what exactly makes a keyphrase absent. although not stated explicitly, many recent studies adopt the definition by @xcite , in which keyphrases that do not match any contiguous subsequence of source text are regarded as absent. from an information retrieval (ir) perspective where stemmed content words are used to index documents, this definition is not sufficiently explicit, as demonstrated by the example shown in figure 1 . we see that, under this definition, some absent keyphrases can have all of their words occurring in the source document, and therefore act no differently from present keyphrases on indexing. in fact, only a fraction of the words that compose these absent keyphrases are genuinely expanding the document, which in our example are the set of words retrieval, behavior, support . from a keyphrase generation point of view, this definition is not entirely satisfactory either, since training a ## (re)defining absent keyphrases telling absent and present keyphrases apart may seem quite easy at first, but actually there are several intricacies to the process that should be noted. starting from @xcite 's definition, "we denote phrases that do not match any contiguous subsequence of source text as absent keyphrases, and the ones that fully match a part of the text as present keyphrases", it is apparent that simple string matching between keyphrases and source document is not acceptable since it produces false positives (e.g. "supervised learning" matches "unsupervised learning"). instead, token-level sequence matching is to be used and combined with stemming to deal with different inflectional forms of the same word. using stemming is critical here since it is carried out as a standard procedure in indexing documents for ir, but also in evaluating the precision of keyphrase generation models against gold standard annotations @xcite . looking back at our example in figure 1 , we see that absent keyphrases can be further divided into three sub-categories depending on the proportion of present words they contain. indeed, some absent keyphrases have some, or even all, of their constituent words (in stemmed forms) present in the text, while others are composed entirely of unseen words. accordingly, we propose the following fine-grained categorization scheme (illustrated with the example from figure 1 and explained in more depth with pseudo-code in appendix a): present: keyphrases that match contiguous sequences of words in the source document (e.g. "search system"). reordered: keyphrases whose constituent words occur in the source document but not as contiguous sequences (e.g. "information sharing"). mixed: keyphrases from which some, but not all, of their constituent words occur in the source document (e.g. "information retrieval"). unseen: keyphrases whose constituent words do not occur in the source document (e.g. "retrieval support"). in contrast to the previously-used binary classification (i.e. present or absent), this finer-grained categorization scheme draws a distinction between keyphrases that expand the document (i.e. mixed and unseen) and those that don't (i.e. present and reordered). it thus allows us to better understand how keyphrases affect the retrieval process by making it possible to numerically quantify the contribution of each category to the overall retrieval effectiveness. at the same time, this scheme provides a new angle to evaluate the ability of keyphrase generation models to output absent keyphrases by contrasting their prmu distributions against those observed in the gold standard annotations. in other words, a model has to mimic the distribution of absent keyphrases in manual annotation in order to perform well. ## experiments here, we outline our experimental setup ( §3.1), examine the distribution of keyphrases in commonlyused datasets with respect to the proposed categorization scheme ( §3.2), show the influence of each category on the retrieval effectiveness ( §3.3), and explore how these categories fit into the outputs of neural keyphrase generation models ( §3.4). ## related work until recently, most previous models for predicting keyphrases were doing so by extracting the most salient noun phrases from documents @xcite . keyphrase extraction models are usually divided into supervised models that cast keyphrase extraction either as a binary classification problem @xcite @xcite @xcite or as a sequence labelling problem @xcite @xcite , and unsupervised models that rely predominantly on graphbased ranking approaches @xcite @xcite @xcite . note that none of these models can produce absent keyphrases. a related line of research focuses on keyphrase assignment, that is, the task of selecting entries from a predefined list of keyphrases (i.e. a controlled vocabulary) @xcite @xcite . here, predicting keyphrases is treated as a multi-class classification task, and models can produce both present and absent keyphrases. further in that direction is @xcite that jointly performs keyphrase extraction and assignment using an unsupervised graph-based ranking model. also closely related to our work is previous research on document expansion @xcite , and particularly recent work on supplementing document indexing with automatically generated queries @xcite . these latter models augment texts with potential queries that, just as keyphrases, mitigate vocabulary mismatch and reweight existing terms @xcite . on the term weighting side, recent work shows that deep neural language models, in this case bert @xcite , can be successfully applied to estimate document-specific term weights @xcite . ## conclusion in this paper, we investigated the usefulness of absent keyphrases for document retrieval. we showed that the commonly accepted definition of absent keyphrases is not sufficiently explicit in the context of ir, and proposed a finer-grained categorization scheme that allows for a better understanding of their impact on retrieval effectiveness. our code and data are publicly available at https://github.com/boudinfl/ redefining-absent-keyphrases .