Exclusive Hierarchical Decoding for Deep Keyphrase Generation (2020)

Abstract: Keyphrase generation (KG) aims to summarize the main ideas of a document into a set of keyphrases. A new setting is recently introduced into this problem, in which, given a document, the model needs to predict a set of keyphrases and simultaneously determine the appropriate number of keyphrases to produce. Previous work in this setting employs a sequential decoding process to generate keyphrases. However, such a decoding method ignores the intrinsic hierarchical compositionality existing in the keyphrase set of a document. Moreover, previous work tends to generate duplicated keyphrases, which wastes time and computing resources. To overcome these limitations, we propose an exclusive hierarchical decoding framework that includes a hierarchical decoding process and either a soft or a hard exclusion mechanism. The hierarchical decoding process is to explicitly model the hierarchical compositionality of a keyphrase set. Both the soft and the hard exclusion mechanisms keep track of previously-predicted keyphrases within a window size to enhance the diversity of the generated keyphrases. Extensive experiments on multiple KG benchmark datasets demonstrate the effectiveness of our method to generate less duplicated and more accurate keyphrases.

## introduction keyphrases are short phrases that indicate the core information of a document. as shown in figure 1 , the keyphrase generation (kg) problem focuses on automatically producing a keyphrase set (a set of keyphrases) for the given document. because of the condensed expression, keyphrases can benefit various downstream applications including opinion mining @xcite , doc-figure 1 : an example of an input document and its expected keyphrase output for keyphrase generation problem. present keyphrases that appear in the document are underlined. keyphrases of a document can be categorized into two groups: present keyphrase that appears in the document and absent keyphrase that does not appear in the document. recent generative methods for kg apply the attentional encoderdecoder framework @xcite with copy mechanism @xcite to predict both present and absent keyphrases. to generate multiple keyphrases for an input document, these methods first use beam search to generate a huge number of keyphrases (e.g., 200) and then pick the top n ranked keyphrases as the final prediction. thus, in other words, these methods can only predict a fixed number of keyphrases for all documents. however, in a practical situation, the appropriate number of keyphrases varies according to the content of the input document. to simultaneously predict keyphrases and determine the suitable number of keyphrases, @xcite adopts a sequential decoding method with greedy search to generate one sequence consisting of the predicted keyphrases and separators. for example, the produced sequence may be "hemodynamics [sep] erectile dysfunction [sep] ...", where "[sep]" is the sep-arator. after producing an ending token, the decoding process terminates. the final keyphrase predictions are obtained after splitting the sequence by separators. however, there are two drawbacks to this method. first, the sequential decoding method ignores the hierarchical compositionality existing in a keyphrase set (a keyphrase set is composed of multiple keyphrases and each keyphrase consists of multiple words). in this work, we examine the hypothesis that a generative model can predict more accurate keyphrases by incorporating the knowledge of the hierarchical compositionality in the decoder architecture. second, the sequential decoding method tends to generate duplicated keyphrases. it is simple to design specific post-processing rules to remove the repeated keyphrases, but generating and then removing repeated keyphrases wastes time and computing resources. to address these two limitations, we propose a novel exclusive hierarchical decoding framework for kg, which includes a hierarchical decoding process and an exclusion mechanism. our hierarchical decoding process is designed to explicitly model the hierarchical compositionality of a keyphrase set. it is composed of phrase-level decoding (pd) and word-level decoding (wd). a pd step determines which aspect of the document to summarize based on both the document content and the aspects summarized by previouslygenerated keyphrases. the hidden representation of the captured aspect is employed to initialize the wd process. then, a new wd process is conducted under the pd step to generate a new keyphrase word by word. both pd and wd repeat until meeting the stop conditions. in our method, both pd and wd attend the document content to gather contextual information. moreover, the attention score of each wd step is rescaled by the corresponding pd attention score. the purpose of the attention rescaling is to indicate which aspect is focused on by the current pd step. we also propose two kinds of exclusion mechanisms (i.e., a soft one and a hard one) to avoid generating duplicated keyphrases. either the soft one or the hard one is used in our hierarchical decoding process. both of them are used in the wd process of our hierarchical decoding. besides, both of them collect the previously-generated k keyphrases, where k is a predefined window size. the soft exclusion mechanism is incorporated in the training stage, where an exclusive loss is em-ployed to encourage the model to generate a different first word of the current keyphrase with the first words of the collected k keyphrases. however, the hard exclusion mechanism is used in the inference stage, where an exclusive search is used to force wd to produce a different first word with the first words of the collected k keyphrases. our motivation is from the statistical observation that in 85% of the documents on the largest kg benchmark, the keyphrases of each individual document have different first words. moreover, since a keyphrase is usually composed of only two or three words, the predicted first word significantly affects the prediction of the following keyphrase words. thus, our exclusion mechanisms can boost the diversity of the generated keyphrases. in addition, generating fewer duplications will also improve the chance to produce correct keyphrases that have not been predicted yet. we conduct extensive experiments on four popular real-world benchmarks. empirical results demonstrate the effectiveness of our hierarchical decoding process. besides, both the soft and the hard exclusion mechanisms significantly reduce the number of duplicated keyphrases. furthermore, after employing the hard exclusion mechanism, our model consistently outperforms all the sota sequential decoding baselines on the four benchmarks. we summarize our main contributions as follows: (1) to our best knowledge, we are the first to design a hierarchical decoding process for the keyphrase generation problem; (2) we propose two novel exclusion mechanisms to avoid generating duplicated keyphrases as well as improve the generation accuracy; and (3) our method consistently outperforms all the sota sequential decoding methods on multiple benchmarks under the new setting. ## notations and problem definition we denote vectors and matrices with bold lowercase and uppercase letters respectively. sets are denoted with calligraphy letters. we use w to represent a parameter matrix. we define the keyphrase generation problem as follows. the input is a document x, the output is a keyphrase set @xmath0,...,|y| , where |y| is the keyphrase number of x. both the x and each y i are sequences of words, i.e., @xmath1, ..., x lx ] and y @xmath2, ..., y i l y i ], where l x and l y i are the word numbers of x and y i correspondingly. ## our methodology we first encode each word of the document into a hidden state and then employ our exclusive hierarchical decoding shown in figure 2 to produce keyphrases for the given document. our hierarchical decoding process consists of phrase-level decoding (pd) and word-level decoding (wd). each pd step decides an appropriate aspect to summarize based on both the context of the document and the aspects summarized by previous pd steps. then, the hidden representation of the captured aspect is employed to initialize the wd process to generate a new keyphrase word by word. the wd process terminates when producing a "[eowd]" token. if the wd process output a "[eopd]" token, the whole hierarchical decoding process stops. both pd and wd attend the document content. the pd attention score is used to re-weight the wd attention score to provide aspect guidance. to improve the diversity of the predicted keyphrases, we incorporate either an exclusive loss when training (i.e., the soft exclusion mechanism) or an exclusive search mechanism when inference (i.e., the hard exclusion mechanism). ## experiment setup our model implementations are based on the opennmt system @xcite using py-torch @xcite . experiments of all models are repeated with three different random seeds and the averaged results are reported. ## conclusion and future work in this paper, we propose an exclusive hierarchical decoding framework for keyphrase generation. unlike previous sequential decoding methods, our hierarchical decoding consists of a phrase-level decoding process to capture the current aspect to summarize and a word-level decoding process to generate keyphrases based on the captured aspect. besides, we also propose a soft and a hard exclusion mechanisms to enhance the diversity of the generated keyphrases. extensive experimental results demonstrate the effectiveness of our meth-soc hw/sw co-verification based debugging technique. purpose -increasingly complex and sophisticated vlsi design, coupled with shrinking design cycles, requires shorter verification time and efficient debug method. … soc hw/sw co-verification technique seems to draw a balance, but design under test (dut) still resides in fpga and remains hard for debugging. the purpose of this paper is to study a run-time rtl debugging methodology for a fpgabased co-verification system. … targets {computer hardware; computer software; co-verification; debugging} figure 3 : an example of generated keyphrases by baselines and our exhird-h. the correct predictions are bold and the present keyphrases are underlined. the digit in parentheses represents the frequency that the corresponding keyphrase is generated by the model (e.g., "debugging (3)" means the keyphrase "debugging" is generated three times by the model).