Heterogeneous Graph Neural Networks for Keyphrase Generation (2021)

Abstract: The encoder–decoder framework achieves state-of-the-art results in keyphrase generation (KG) tasks by predicting both present keyphrases that appear in the source document and absent keyphrases that do not. However, relying solely on the source document can result in generating uncontrollable and inaccurate absent keyphrases. To address these problems, we propose a novel graph-based method that can capture explicit knowledge from related references. Our model first retrieves some document-keyphrases pairs similar to the source document from a pre-defined index as references. Then a heterogeneous graph is constructed to capture relations with different levels of granularity of the source document and its retrieved references. To guide the decoding process, a hierarchical attention and copy mechanism is introduced, which directly copies appropriate words from both source document and its references based on their relevance and significance. The experimental results on multiple KG benchmarks show that the proposed model achieves significant improvements against other baseline models, especially with regard to the absent keyphrase prediction.

## introduction keyphrase generation (kg), a fundamental task in the field of natural language processing (nlp), refers to the generation of a set of keyphrases that expresses the crucial semantic meaning of a document. these keyphrases can be further categorized into present keyphrases that appear in the document and absent keyphrases that do not. current kg approaches generally adopt an encoderdecoder framework @xcite with attention mechanism @xcite and copy mechanism @xcite ; although the previous methods for keyphrase generation have shown promising results on present keyphrase predictions, they are not yet satisfactory on the absent keyphrase predictions, which also occupy a large proportion. @xcite to simultaneously predict present and absent keyphrases @xcite @xcite @xcite . although the proposed methods for keyphrase generation have shown promising results on present keyphrase predictions, they often generate uncontrollable and inaccurate predictions on the absent ones. the main reason is that there are numerous candidates of absent keyphrases that have implicit relationships (e.g., technology hypernyms or task hypernyms) with the concepts in the document. for instance, for a document discussing "lstm", all the technology hypernyms like "neural network", "rnn" and "recurrent neural network" can be its absent keyphrases candidates. when dealing with scarce training data or limited model size, it is nontrivial for the model to summarize and memorize all the candidates accurately. thus, one can expect that the generated absent keyphrases are often suboptimal when the candidate set in model's mind is relatively small or inaccurate. this problem is crucial because absent keyphrases account for a large proportion of all the ground-truth keyphrases. as shown in figure 1 , in some datasets, up to 50% of the keyphrases are absent. d final distribution gate d d iterative updating source attention distribution reference attention distribution heterogeneous graph encoder reference-aware decoder tf-idf bigru bigru d reference node w d d vocabulary distribution figure 2 : graphical illustration of our proposed gater. we first retrieve references using the source document, where each reference is the concatenation of document and keyphrases pair from the training set. then we construct a heterogeneous graph and perform iterative updating. finally, the source document node is extracted to decode the keyphrase sequence with a hierarchical attention and copy mechanism. our heterogeneous graph is designed to incorporate knowledge from the related references. it contains source document, reference and keyword nodes, and has the following advantages: (a) different reference nodes can interact with the source document regarding the explicit shared keyword information, which can enrich the semantic representation of the source document; (b) a powerful structural prior is introduced as the keywords are highly overlapped with the groundtruth keyphrases. statistically, we collect the top five keywords from each document on the validation set, and we find that these keywords contain 68% of the tokens in the ground-truth keyphrases. on the decoder side, as a portion of absent keyphrases directly appear in the references, we propose a hierarchical attention and copy mechanism for copying appropriate words from both source document and its references based on their relevance and significance. the main contributions of this paper can be summarized as follows: (1) we design a heterogeneous graph network for keyphrase generation, which can enrich the source document node through keyword nodes and retrieved reference nodes; ## methodology in this work, we propose a heterogeneous graph attention network based on references (gater) for keyphrase generation, as shown in figure 2 . given a source document, we first retrieve related document from a predefined index 2 and concatenate each retrieved document with its keyphrases to serve as a reference. then we construct a heterogeneous graph that contains document nodes 3 and keyword nodes based on the source document and its references. the graph is updated iteratively to enhance the representations of the source document node. finally, the source document node is extracted to decode the keyphrase sequence. to facilitate the decoding process, we also introduce a hierarchical attention and copy mechanism, with which the model directly attends to and copies from both the source document and its references. the hierarchical ar-rangement ensures that more semantically relevant words and those in more relevant references will be given larger weights for the current decision. ## conclusions in this paper, we propose a graph-based method that can capture explicit knowledge from related references. our model consists of a heterogeneous graph encoder to model different granularity of relations among the source document and its references, and a hierarchical attention and copy mechanism to guide the decoding process. extensive experiments demonstrate the effectiveness and portability of our method on both the present and absent keyphrase predictions.