One Size Does Not Fit All: Generating and Evaluating Variable Number of Keyphrases (2020)

Abstract: Different texts shall by nature correspond to different number of keyphrases. This desideratum is largely missing from existing neural keyphrase generation models. In this study, we address this problem from both modeling and evaluation perspectives. We first propose a recurrent generative model that generates multiple keyphrases as delimiter-separated sequences. Generation diversity is further enhanced with two novel techniques by manipulating decoder hidden states. In contrast to previous approaches, our model is capable of generating diverse keyphrases and controlling number of outputs. We further propose two evaluation metrics tailored towards the variable-number generation. We also introduce a new dataset StackEx that expands beyond the only existing genre (i.e., academic writing) in keyphrase generation tasks. With both previous and new evaluation metrics, our model outperforms strong baselines on all datasets.

## introduction keyphrase generation is the task of automatically predicting keyphrases given a source text. desired keyphrases are often multi-word units that summarize the high-level meaning and highlight certain important topics or information of the source text. consequently, models that can successfully perform this task should be capable of not only distilling high-level information from a document, but also locating specific, important snippets therein. to make the problem even more challenging, a keyphrase may or may not be a substring of the source text (i.e., it may be present or absent). moreover, a given source text is usually associated with a set of multiple keyphrases. thus, keyphrase generation is an instance of the set generation problem, where both the size of the set and the size (i.e., the number of tokens in a phrase) of each element can vary depending on the source. similar to summarization, keyphrase generation is often formulated as a sequence-to-sequence (seq2seq) generation task in most prior studies @xcite @xcite . conditioned on a source text, seq2seq models generate phrases individually or as a longer sequence jointed by delimiting tokens. since standard seq2seq models generate only one sequence at a time, thus to generate multiple phrases, a common approach is to over-generate using beam search @xcite with a large beam width. models are then evaluated by taking a fixed number of top predicted phrases (typically 5 or 10) and comparing them against the ground truth keyphrases. though this approach has achieved good empirical results, we argue that it suffers from two major limitations. firstly, models that use beam search to generate multiple keyphrases generally lack the ability to determine the dynamic number of keyphrases needed for different source texts. meanwhile, the parallelism in beam search also fails to model the inter-relation among the generated phrases, which can often result in diminished diversity in the output. although certain existing models take output diversity into consideration during training @xcite , the effort is significantly undermined during decoding due to the reliance on over-generation and phrase ranking with beam search. secondly, the current evaluation setup is rather problematic, since existing studies attempt to match a fixed number of outputs against a variable number of ground truth keyphrases. empirically, the number of keyphrases can vary drastically for different source texts, depending on a plethora of factors including the length or genre of the text, the granularity of keyphrase annotation, etc. for the several commonly used keyphrase generation datasets, for example, the average number of keyphrases per data point can range from 5.3 to 15.7, with variances sometimes as large as 64.6 ( to overcome these problems, we propose novel decoding strategies and evaluation metrics for the keyphrase generation task. the main contributions of this work are as follows: ## model architecture given a piece of source text, our objective is to generate a variable number of multi-word phrases. to this end, we opt for the sequence-to-sequence (seq2seq) @xcite framework as the basis of our model, combined with attention and pointer softmax mechanisms in the decoder. since each data example contains one source text sequence and multiple target phrase sequences (dubbed on e2ma n y, and each sequence can be of multi-word), two paradigms can be adopted for training seq2seq models. the first one @xcite is to divide each on e2ma n y data example into multiple on e2on e examples, and the resulting models (e.g., copyrnn) can generate one phrase at once and must rely on beam search technique to produce more unique phrases. to enable models to generate multiple phrases and control the number to output, we propose the second training paradigm on e2se q, in which we concatenate multiple phrases into a single sequence with a delimiter sep , and this concatenated sequence is then used as the target for sequence generation during training. an overview of the model's structure is shown in figure 1 . 1 ## evaluating keyphrase generation formally, given a source text, suppose that a model predicts a list of unique keyphrases ŷ = (ŷ 1 , . . . , ŷm ) ordered by the quality of the predictions ŷi , and that the ground truth keyphrases for the given source text is the oracle set y. when only the top k predictions ŷ:@xmath2, . . . , ŷmin(k,m) ) are used for evaluation, precision, recall, and f 1 score are consequently conditioned on k and defined as: a simple remedy is to set k as a variable number which is specific to each data example. here we define two new metrics: ## datasets and experiments in this section, we report our experiment results on multiple datasets and compare with existing models. we use catseq to refer to the delimiter3 we acknowledge that f1@o scores of @xcite and @xcite might be not completely comparable with ours. this is due to additional post-processing and filtering methods might have been applied in different work. we elaborate the data pre-processing and evaluation protocols used in this work in appendix e. to construct target sequences for training catseq and catseqd, ground truth keyphrases are sorted by their order of first occurrence in the source text. keyphrases that do not appear in the source text are appended to the end. this order may guide the attention mechanism to attend to source positions in a smoother way. implementation details can be found in appendix d. as for the pre-processing and evaluation, we follow the same steps as in @xcite . more details are provide in appendix e for reproducing our results. we include a set of existing models @xcite @xcite @xcite as baselines, they all share same behavior of abstractive keyphrase generation with our proposed model. specially for computing existing model's scores with our proposed new metrics (f 1 @o and f 1 @m), we implemented our own version of copyrnn @xcite based on their open sourced code, denoted as copyrnn*. we also report the scores of models from @xcite . based on their publicly released outputs. we also include a set of models that use similar strategies but can not directly compare with. this includes four non-neural extractive models: tfidf @xcite , textrank @xcite , kea @xcite , and maui @xcite ; one neural extractive model @xcite ; and two neural models that use additional data (e.g., title) @xcite . in section 5.3, we apply the self-terminating decoding strategy. since no existing model supports such decoding strategy, we only report results from our proposed models. they can be used for comparison in future studies. ## conclusion and future work we propose a recurrent generative model that sequentially generates multiple keyphrases, with two extra modules that enhance generation diversity. we propose new metrics to evaluate keyphrase generation. our model shows competitive performance on a set of keyphrase generation datasets, including one introduced in this work. in future work, we plan to investigate how target phrase order affects the generation behavior, and further explore set generation in an order invariant fashion.