Exploiting Position and Contextual Word Embeddings for Keyphrase Extraction from Scientific Papers (2021)

Abstract: Keyphrases associated with research papers provide an effective way to find useful information in the large and growing scholarly digital collections. In this paper, we present KPRank, an unsupervised graph-based algorithm for keyphrase extraction that exploits both positional information and contextual word embeddings into a biased PageRank. Our experimental results on five benchmark datasets show that KPRank that uses contextual word embeddings with additional position signal outperforms previous approaches and strong baselines for this task.

## introduction keyphrase extraction is the task of automatically extracting a small set of descriptive words or phrases that can accurately summarize the topics discussed in a document @xcite . keyphrases are useful in many applications such as document indexing and summarization @xcite @xcite , topic tracking @xcite , contextual advertising @xcite , and opinion mining @xcite . most of the previous approaches to keyphrase extraction are either supervised or unsupervised. while supervised approaches perform generally better @xcite , the unsupervised ones have the advantage that they do not require large human-annotated corpora for training reliable models. unsupervised keyphrase extraction methods usually use graph-based ranking algorithms such as pagerank that work on the word graph constructed from the target document @xcite . various pagerank extensions have been proposed that incorporate different types of information @xcite . for example, @xcite proposed to incorporate a local neighborhood of the target document into the graph construction, with the neighborhood being determined based on the textual similarity between documents. @xcite exploited topical information to select keyphrases from all major topics. more recently, @xcite proposed a theme-weighted biased pagerank, called key2vec, for keyphrase extraction. in key2vec, a theme-vector is computed by averaging the embeddings of words and phrases from the title of a scientific document to capture its theme and the pagerank is biased based on the similarity of candidate words or phrases to the computed theme vector. however, this model is oblivious to the position of words in a scientific document, in which more important words appear not only frequently, but also close to the beginning of the document @xcite . inspired by the transformer models @xcite that infuse positional information into the word embeddings to produce embeddings with time signal, we propose an extension of key2vec that incorporates words' positions into a biased pagerank. moreover, different from @xcite , who used non-contextual fasttext embeddings @xcite , we propose to integrate scibert contextual embeddings @xcite into our biased pagerank extension. our contributions are as follows: (1) we propose kprank, an unsupervised graph-based algorithm that exploits both the position of words in a document and the contextual word embeddings for computing a biased pagerank score for ranking candidate phrases; (2) we show empirically that infusing position information into our biased kprank model yields better performance compared with its counterpart that does not use the position information. in addition, kprank with contextual scib-ert embeddings performs better than fasttextbased kprank; (3) finally, we show that kprank outperforms many previous unsupervised models. ## proposed approach in this section, we describe our unsupervised graphbased algorithm called kprank, that exploits both position information of the words in a document along with contextual word embeddings for computing a biased pagerank score for each candidate word. our approach consists of three steps: (1) candidate word selection and word graph construction; (2) word scoring by biased pagerank; and (3) candidate phrase formation. ## data for evaluation, we use five datasets, which we describe below. we use the combination of controlled (author assigned) and uncontrolled (reader assigned) keyphrases as gold-standard phrases. we used uncontrolled keyphrases when available. inspec @xcite ) contains 2,000 research papers. it has a train-validation-test split of 1,000, 500 and 500 papers, respectively. krapivin @xcite ) contains 2,304 acm research papers with full text and authorassigned keyphrases. similar to @xcite , since the dataset does not have a train-test split, we sampled 400 papers as the test set. nus (nguyen and @xcite ) contains 211 research papers. this dataset does not have a train and test split and it is relatively small. hence, consistent with @xcite we used entire dataset as the test set. for each dataset we use its test set for evaluation. ## experimental setup and results evaluation metrics. to evaluate the performance of different methods, we use micro avg. f1-score. we report the performance for the top 5 and 10 candidate phrases returned by different methods as in @xcite . to create a word graph for a given document, we use its title and abstract. to match the predicted keyphrases with gold-standard keyphrases, we do exact match between the stemmed version of each. the effect of position, contextual embeddings, and the comparison with previous works. to see the effect of positional information, we compare the performance of kprank that uses contextual scibert (sb) embeddings along with positional information (denoted as kprank(sb)) with that of its counterpart that does not use positional information (denoted as kprank(sb-pos)). moreover, to see the effect of contextual embeddings, we compare the performance of scibert-based kprank (kprank(sb)) with that of kprank that uses fasttext non-contextual word embeddings @xcite (denoted as kprank(fasttext)). for fasttext, we used pretrained 300 dimensional embeddings trained on subword information on common crawl. note that kprank(scibert) and kprank(fasttext) use positional information along with the theme score. last, we compare the performance of kprank with tf-idf and six pagerank based unsupervised methods as baselines: positionrank @xcite , key2vec @xcite , textrank @xcite , singlerank @xcite , expandrank @xcite , topicrank @xcite . tables 2 shows these comparisons on semeval, inspec, krapivin, nus, and acm. it can be seen from the table that adding position information shows much higher improvement in the performance of kprank, i.e. kprank(sb) substantially outperforms kprank(sb-pos). moreover, kprank(sb) outperforms kprank(fasttext) on all the datasets except for krapivin. importantly, kprank(sb) outperforms most baseline methods, including key2vec (by a large margin) e.g., on se-meval, kprank(sb) achieves an f1@5 of 22.51% as compared with 17.54% achieved by key2vec. we can also notice from figure 2 shows the confusion matrices of kprank(sb) using @5 predictions on all five datasets. each matrix is represented as a heat map, i.e., the darker the blue color the higher the value at that position and the darker the blue on the main diagonal, the more accurate the model is. ## conclusion and future work in this paper, we proposed a novel unsupervised graph-based algorithm, named kprank, which incorporates both positional appearances of the words along with contextual word embeddings for computing a biased pagerank score for each candidate word. our experimental results on five datasets show that incorporating position information into our biased kprank model yields better performance compared with a kprank that does not use the position information, and scibertbased kprank usually outperforms fasttext-based kprank on this task. moreover, kprank outperforms strong baseline methods. in the future, it would be interesting to explore kprank on other domains, such as biology, and social science.