S a SAKE : Syntax and Semantics Aware Keyphrase Extraction from Research Papers (2020)

Abstract: Keyphrases in a research paper succinctly capture the primary content of the paper and also assist in indexing the paper at a concept level. Given the huge rate at which scientific papers are published today, it is important to have effective ways of automatically extracting keyphrases from a research paper. In this paper, we present a novel method, Syntax and Semantics Aware Keyphrase Extraction (SaSAKE), to extract keyphrases from research papers. It uses a transformer architecture, stacking up sentence encoders to incorporate sequential information, and graph encoders to incorporate syntactic and semantic dependency graph information. Incorporation of these dependency graphs helps to alleviate long-range dependency problems and identify the boundaries of multi-word keyphrases effectively. Experimental results on three benchmark datasets show that our proposed method SaSAKE achieves state-of-the-art performance in keyphrase extraction from scientific papers.

## introduction keyphrases are words or phrases that capture important concepts of a document. the task of keyphrase extraction, i.e., automatically extracting a collection of keyphrases from a document, has attracted considerable attention from the research community due to its pivotal importance in various applications like text document retrieval @xcite , document categorization @xcite , opinion mining @xcite and summarization @xcite . keyphrase extraction from research papers is especially important due to their ability to concisely capture the main tenets of the complex scholarly documents. although it is common for authors to specify keyphrases in research papers, they are not present in all publications. in these cases, it is beneficial to automatically infer the most relevant keyphrases. traditional methods for keyphrase extraction follow a two-step procedure where important phrases from the document are extracted as potential keyphrase candidates by heuristic rules @xcite @xcite , and then the extracted candidate phrases are ranked either by unsupervised approaches @xcite @xcite or supervised approaches @xcite . they typically label each candidate phrase independently without taking into account the dependencies that could potentially exist between neighbouring labels, and they also ignore the semantic meaning of the text. to overcome the above stated limitation, recently @xcite formulated keyphrase extraction as a sequence labeling task and used linear-chain conditional random fields for this task. however, this approach does not explicitly take into account the long-term dependencies and semantics of the text. more recently, to capture both the semantics of the text as well as the dependencies among the labels of neighboring words, @xcite used a deep learning-based approach called bilstm-crf that combines a bi-directional long short-term memory (bilstm) layer, which models the sequential input text, with a conditional random field (crf) layer, which captures the dependencies in the output. while the above models are promising, they still suffer from some limitations which we address in this work. for example, consider the sentence in fig. 1 . the previous models @xcite @xcite fail to recognize "recurrent elman neural network" as a keyphrase. this can be attributed to two reasons. firstly, due to the long-range dependency between the words "we", "extracted" and "network", the model fails to understand the importance of the phrase with respect to the sentence. secondly, due to their inability to detect the boundaries of phrase accurately, they fail to predict multi-word keyphrases correctly. we notice long-range dependencies such as those illustrated in the first limitation are shortened in semantic dependency graphs in which nodes represent words and edges represent semantic relations between the words, thereby representing predicate-argument relations between content words in a sentence. in fig. 1 , lower pink edges show the semantic dependency graph using dm (delph-in minimal recursion semantics) representation schema @xcite shortening the distance between those long-range dependencies. these properties of a semantic dependency graph allow neural network models to capture long-range semantic dependencies effortlessly @xcite . to address the second limitation, we posit that syntactic dependency graphs in which nodes represent words and edges represent syntactic relations between the words provide the ability to find the boundaries of keyphrases accurately. in fig. 1 , the upper red arrows indicate the syntactic dependency relations among words, providing the ability to recognize the boundaries of the keyphrase "recurrent elman neural network" with the help of the syntactic labels that act as important cues to identify such multi-word keyphrases in the sentence. though historically, feature based approaches (nguyen and @xcite did rely on syntactic and semantic dependency information to extract various features, the latest generation of keyphrase extraction models put syntax and semantics aside in favour of neural sequence models which outperformed syntactically-driven feature engineering methods. we believe that one of the reasons for this choice is the lack of simple and effective methods to incorporate syntactic and semantic dependency information into sequential neural networks. in this paper, we propose sasake, an acronym for syntax and semantic aware keyphrase extraction from research papers based on transformer architecture @xcite which stacks up sentence encoders to incorporate sequential information and graph encoders to incorporate syntactic and semantic dependency graph information. firstly, the sentence encoder reads the sentence producing context-level representations for each word through multi-head self-attention mechanism that is then fed as input to both the syntactic and the semantic graph encoders along with their corresponding dependency graphs. then the semantic (syntactic) graph encoder captures the semantic (syntactic) relations among words and enhances the context-level representations incorporating semantic (syntactic) dependency information to produce semantics (syntax)-aware representations. we adopt a multi-head graph attention mechanism in graph encoders to aggregate the information of the relation triples (head, type, tail) into the corresponding head and tail nodes to construct a semantics/syntax-aware representation. however, the propagation of semantic and syntactic relations from distant nodes may introduce noise into the representations. so, we employ an aggregation layer that balances the influence of the local contextual information and the additional information obtained from the syntactic and the semantic relations, thereby producing more balanced final representations for each word that are then passed to a crf layer which models the dependencies among the labels for prediction. experimental results on three datasets of research papers show that sasake outperforms previous state-of-the-art approaches for keyphrase extraction. ## related work keyphrase extraction has been an active area of research since two decades @xcite . unsupervised approaches score the extracted potential candidate phrases based on graph-based ranking algorithms @xcite wherein each word in the document is mapped to a node in the graph and the connecting edges in the graph represent the association patterns among the words in the document. then, the scores of the individual words are estimated using various graph centrality measures and a combination of other heuristic rules based on tf-idf scores, word co-occurrence measures, extraction of specific lexical patterns and clustering. such methods include textrank @xcite , lexrank @xcite , topicrank @xcite , sgrank @xcite and singlerank @xcite . on the other hand, supervised approaches use binary classification to label the extracted candidate phrases as keyphrases or non-keyphrases based on textual features such as term frequencies @xcite and syntactic properties (nguyen and @xcite . additionally, @xcite incorporate external knowledge like document citation context to mine features. ## problem definition given a sentence as the sequence of words @xmath0, x 2 , • • • , x n ] as input, we parse the input sentence x to obtain the syntactic dependency graph g 1 = (v 1 , e 1 ) and the semantic dependency graph g 2 = (v 2 , e 2 ) using existing tools, where v 1 , v 2 denote the sets of nodes and e 1 , e 2 denote the sets of edges in the two cases, respectively. each edge in g 1 represents a triple (head, type, tail) where @xmath1, @xmath2, nmod, dobj, . . .}. each edge in g 2 represents a triple (head, type, tail) where @xmath3, @xmath4, arg2, compound, . . .} as per dm annotation scheme. keyphrase extraction task is formulated as a sequence labeling task as follows: given an input sentence x, its syntactic dependency graph g 1 and its semantic dependency graph g 2 , our aim is to output a sequence @xmath5, y 2 , . . . , y n ] where each y i is a label from the set {kp,nkp}. here kp denotes that the word x i is a keyphrase word and nkp denotes otherwise. every longest sequence of kp words in a sentence constitutes a keyphrase. ## our approach: sasake our proposed approach, sasake consists of the following components: (1) sentence encoder (2) syntactic graph encoder (3) semantic graph encoder (4) aggregation layer and (5) label sequence prediction layer. the sentence encoder takes a sequence of words as input to produce a context level representation of the words. then the syntactic and semantic graph encoders work in parallel taking the context level representation of the words along with the syntactic and the semantic dependency graphs to produce syntax-aware and semantics-aware representation of the words, respectively. the aggregation layer combines the obtained syntax-aware and semantics-aware representations; it also helps to mitigate the influence of the distant supporting information as we believe that the prediction should be based primarily on the local context. finally, the above representations are fed into a crf layer which acts as a decoder to predict the label, kp or nkp, associated with each word.