Word centrality constrained representation for keyphrase extraction (2021)

Abstract: To keep pace with the increased generation and digitization of documents, automated methods that can improve search, discovery and mining of the vast body of literature are essential. Keyphrases provide a concise representation by identifying salient concepts in a document. Various supervised approaches model keyphrase extraction using local context to predict the label for each token and perform much better than the unsupervised counterparts. Unfortunately, this method fails for short documents where the context is unclear. Moreover, keyphrases, which are usually the gist of a document, need to be the central theme. We propose a new extraction model that introduces a centrality constraint to enrich the word representation of a Bidirectional long short-term memory. Performance evaluation on 2 publicly available datasets demonstrate our model outperforms existing state-of-the art approaches.

## introduction keyphrase extraction is an important information extraction task that identifies single or multi-word linguistic units that concisely represent a document. they can also serve to provide a brief summary of the document content. keyphrases are widely used in variety of natural language processing tasks such as document summarization @xcite , query formulation @xcite ), text classification @xcite , clustering @xcite , and recommendation systems @xcite . keyphrases have become increasingly important for biomedical documents as there has been an exponential growth with over 32 million articles indexed by pubmed (nlm). figure 1 shows a pubmed document with the authorspecified keyphrases highlighted in blue. existing keyphrase extraction methods mainly fall either under a supervised or unsupervised approach. common unsupervised approaches use word co-occurrence statistics to build graph-based ranking algorithms. each word is mapped to a node and edges connect words that co-occur within a specified window size. even though unsupervised approaches are desirable for datasets which do not have manually-labeled ground truth values, most such methods perform worse compared to the supervised counterparts. the supervised approaches use classification to label every token as being part of a keyphrase or not by using features such as part-of speech tags, termfrequency inverse document frequency (tf-idf), and the position of the token in the document. recently, supervised methods based on deep learning have been employed for keyphrase extraction. in @xcite and @xcite , the authors posed the problem as a sequence labeling task and applied a long short-term memory network (lstm) and conditional random fields (crf) to tag each token in document as positive (i.e., part of a keyphrase) or negative. while these approaches achieve much better performance, they still suffer from a major limitation when applied on biomedical literature. the task of labelling each token does not consider how central the token is to the document contents. for figure 1 , the main theme of the keyphrases are genes associated with breast cancer. thus, the document theme can be used as additional information to improve the keyphrase extraction performance. to this end, we propose to address the problem of keyphrase extraction as a sequence labelling task with an additional component to capture the centrality of each token. we design a centrality layer built on top of a bidirectional lstm (bilstm) layer to constrain each token with regards to the central theme of the document. the output dependencies are then modeled using a crf layer. the contributions of our work are: the next section presents a brief description of the related work. the proposed keyphrase extraction method is introduced in section 3. sections 4, and 5 present experimental results and conclusion respectively. ## related work keyphrase extraction methods mainly take either supervised or unsupervised approach. unsupervised approaches generate candidates and rank using features such as tf-idf and topic proportions @xcite , graph-based centrality measures @xcite , topic modeling @xcite , and document's citation network @xcite . unsupervised, graph-based methods build a graph from the input document where all the candidate keyphrases are nodes and the connection between each candidate is represented by edges. a graph-based ranking method then determines the weights for each node based on the relatedness between the candidates. alternatively, topic-based approaches cluster candidate keyphrases into topics in the document so that all the topics in the input document are represented by the selected keyphrases. recently @xcite proposed a sentence embedding model named sifrank that uses autoregressive pre-trained language model to extract keyphrases from short documents. yet unsupervised methods often fail to achieve state-of-the-art performance. under the supervised approach, the keyphrase extraction problem is treated as a binary classification task @xcite @xcite , where learning algorithms such as support vector machines @xcite and maximum entropy @xcite are used. supervised keyphrase extraction can also be posed as a ranking problem between candidates @xcite . the candidates keys are extracted using statistical features (tf-idf, number of occurrences, first occurrence of the key) and structural features (part of speech tags). deep learning based models have also been used for keyphrase extraction. word embeddings are used to measure the relatedness between words in graph-based models @xcite . zhang et al. (2016) 2020 ) extended the lstm-crf to utilize bilstm and incorporated an attention mechanism to retrieve additional information from other sentences within the same document. @xcite evaluated the effect of various pre-trained word embeddings for the bilstm-crf architecture in extracting keyphrases from benchmark datasets and found contextual embeddings offered better performance. while these models offer better performance, they fail to capture the centrality of the keyphrases which represent a salient feature of the document. ## methodology the keyphrase extraction task is formulated as a sequence labelling task. given a document @xmath0, w 2 , • • • , w t where w i is the i th word and t is the number of words in the document, we predict the labels @xmath1, y 2 , • • • , y t where each label y i is whether word w i is a keyphrase or not. ## experiments datasets. we ran our experiment on 2 publicly available keyphrase datasets: pubmed @xcite and inspec @xcite . pubmed consists of 2532 articles from pubmed central open access subset with at least 5 author-provided keyphrases while inspec contains 200 abstracts of scientific journal papers from computer science collected between the years 1998 and 2002. each document in inspec has two sets of keywords assigned: the controlled keywords, which are manually controlled assigned keywords that appear in the inspec thesaurus but may not appear in the document, and the uncontrolled keywords which are freely assigned by the editors. the union of both sets is considered as the ground-truth in this work. summary statistics for the datasets are shown in since we use a sequence labeling formulation of the keyphrase extraction problem, the abstract/keyphrases data pairs are prepared such that each document is a sequence of word tokens, each with positive labels if it occurs in a keyphrase (k b , k i ), or with a negative label (k o ). experiment settings. as baseline models, we train bilstm and bilstm-crf with 100dimension glove pre-trained embedding vectors @xcite . we also train bilstm-crf with two 768-dimension contextual embeddings, bert @xcite and biobert @xcite . @xcite , a state-of-the art baseline, uses a sentence enriching process from all the documents using sentence embedding. to replicate their work, we used the bert model to extract sentence embeddings for each document and enrich the representation. finally, our model is trained using bert word embeddings for the inspec dataset and biobert embeddings for the pubmed dataset. the results reported are from three runs using 80/20/20 split for train/val/test sets respectively. the bilstm, and bilstm-crf are optimized during training using stochastic gradient descent with the learning rate 0.0001. gradient clipping and drop-out are used to prevent overflow and overfitting. we select the model with the best f1 score on the validation set over three runs. the final test scores reported are the averages running the best model on the test sets. the code was implemented in tensorflow 2.4.1 and the code is available at https://github.com/zhgero/ keyphrases_centrality.git . ## results the performance comparisons between the baselines and our model are shown in we also compared our models with several state-of-the-art unsupervised approaches including singlerank @xcite , position-rank @xcite , topicrank @xcite , and sifrank @xcite in figure 3 , we compare keyphrases tagged by the biobert model and our model on a sample abstract. the true positives are colored blue while false negatives are in red. we observe that the biobert model fails to identify 'chronic thromboembolic pulmonary hypertension' as an important keyphrase whereas our model correctly identifies it. this may be due to the single occurrence of 'pulmonary hypertension' in the input text. meanwhile our model leverages the document embedding to 'understand' that pulmonary hypertension is semantically relevant in the context of the entire abstract. we also observe a similar pattern with the keyphrase 'duration of anticoagulation'. even though both models fail to capture the entire phrase, our model identifies 'anticoagulation' as a strong candidate because of its semantic meaning in the context of the whole abstract. the figure also illustrates the limitation of the models as both struggle with common words such as 'post' and 'high' that are attached as prefixes to important keywords. 'high risk', 'duration of' and 'post-' are considered unimportant by both models. this can be explained by the fact that such words usually occur outside a keyphrase boundary and get overlooked even when they appear with important words. false positives by both models are important terms as the phrases are very relevant in the context of abstract but were not selected by the authors. ## conclusion in this paper, we proposed a keyphrase extraction method that focuses on identifying words which are central to the document semantics. the problem of keyphrase extraction is posed as a sequence labeling task where each token is tagged as either a keyphrase or not. in addition to our novel centrality constraint layer, we have used bi-lstm layers to capture the long term dependencies among the input sequences. finally, we have a crf layer which is well suited to capture the dependencies from the output labels. empirical results on two datasets show that our method gains significant improvement in the pubmed dataset while performing slightly better on the inspec dataset.