O ne2 S et: G enerating Diverse Keyphrases as a Set (2021)

Abstract: Recently, the sequence-to-sequence models have made remarkable progress on the task of keyphrase generation (KG) by concatenating multiple keyphrases in a predefined order as a target sequence during training. However, the keyphrases are inherently an unordered set rather than an ordered sequence. Imposing a predefined order will introduce wrong bias during training, which can highly penalize shifts in the order between keyphrases. In this work, we propose a new training paradigm One2Set without predefining an order to concatenate the keyphrases. To fit this paradigm, we propose a novel model that utilizes a fixed set of learned control codes as conditions to generate a set of keyphrases in parallel. To solve the problem that there is no correspondence between each prediction and target during training, we propose a K-step label assignment mechanism via bipartite matching, which greatly increases the diversity and reduces the repetition rate of generated keyphrases. The experimental results on multiple benchmarks demonstrate that our approach significantly outperforms the state-of-the-art methods.

## introduction keyphrase generation (kg) aims to generate of a set of keyphrases that expresses the high-level semantic meaning of a document. these keyphrases can be further categorized into present keyphrases that appear in the document and absent keyphrases that do not. @xcite proposed a sequence-to-sequence (seq2seq) model with a copy mechanism @xcite to predict both present and absent keyphrases. however, the model needs beam search during inference to overgenerate multiple keyphrases, which cannot determine the dynamic number of keyphrases. to @xcite and predictions (lower) under one2seq and one2set training paradigm. for the one2seq training paradigm, although the predictions are correct in each keyphrase, they will still be considered wrong due to the shift in keyphrase order, and the model will receive a large penalty. however, as the keyphrases are inherently an unordered set rather than an ordered sequence, imposing a predefined order usually leads to the following intractable problems. first, the predefined order will give wrong bias during training, which can highly penalize shifts in the order between keyphrases. as shown in figure 1 (a), the model makes correct predictions in each keyphrase but can still receive a large loss during training. second, this increases the difficulty of model training. for example, the absent keyphrases are appended to the end in an author-defined order in @xcite , however, different authors can have various sorting bases, which makes it difficult for the model to learn a unified pattern. third, the model is highly sensitive to the predefined order, as shown in @xcite , and can suffer from error propagation during inference when previously having generated keyphrases with an incorrect order. lately, @xcite proposed a reinforcement learningbased fine-tuning method, which fine-tunes the pre-trained models with metric-based rewards (i.e., recall and f 1 ) for generating more sufficient and accurate keyphrases. however, this method can alleviate the impact of the order problems when fine-tuning but needs to be pre-trained under the one2seq paradigm to initialize the model, which can still introduce wrong biases. to address this problem, we propose a new training paradigm one2set where the ground-truth target is a set rather than a keyphrase-concatenated sequence. however, the vanilla seq2seq model can generate a sequence but not a set. hence, we introduce a set prediction model that adopts transformer @xcite as the main architecture together with a fixed set of learned control codes as additional decoder inputs to perform controllable generation. for each code, the model generates a corresponding keyphrase for the source document or a special ∅ token that represents the meaning of "no corresponding keyphrase". during training, the cross-entropy loss cannot be directly used since we do not know the correspondence between each prediction and target. hence, we introduce a k-step target assignment mechanism, where we first auto-regressively generate k words for each code and then assign targets via bipartite matching based on the predicted words. after that, we can train each code using teacher forcing as before. compared with the previous models, the proposed method has the following advantages: (a) there is no need to predefine an order to concatenate the keyphrases, thus the model will not be affected by the wrong biases in the whole training stage; and (b) the bipartite matching forces unique predictions for each code, which greatly reduces the duplication ratio and increases the diversity of predictions. we summarize our main contributions as follows: (1) we propose a new training paradigm one2set without predefining an order to concatenate the keyphrases; (2) we propose a novel set prediction model that can generate a set of diverse keyphrases in parallel and a dynamic target assignment mechanism to solve the intractable training problem under the one2set paradigm; (3) our method consistently outperforms all the state-of-the-art methods and greatly reduces the duplication ratio. our codes are publicly available at github foot_0 . ## methodology this paper proposes a new training paradigm one2set for keyphrase generation. a set prediction model based on transformer (settrans) is proposed to fit this paradigm, as shown in figure 2 . given a fixed set of learned control codes as input conditions, the model generates a keyphrase or a special ∅ token for each code in parallel. during training, a k-step target assignment mechanism is proposed to dynamically determine the target corresponding to each code. the main idea is that the model first freely predicts k steps without any supervision to see what keyphrase each code can roughly generate, and then use bipartite matching to find the optimal allocation based on the model's conjecture and target. given the correspondence of each code and target, a separate set loss is then used to correct the model's conjecture, where half of the codes are trained to predict the present keyphrase set and the others are trained to predict the absent keyphrase set. ## conclusions in this paper, we propose a new training paradigm one2set without predefining an order to concatenate the keyphrases, and a novel model settrans that predicts a set of keyphrases in parallel. to successfully train under one2set paradigm, we propose a k-step target assignment mechanism and a separate set loss, which greatly increases the number and diversity of the generated keyphrases. experiments show that our method gains significantly huge performance improvements against existing state-of-the-art models. we also show that settrans has great advantages in the inference efficiency compared with the transformer under one2seq paradigm.