WR - O ne2 S et: Towards Well-Calibrated Keyphrase Generation (2022)

Abstract: Keyphrase generation aims to automatically generate short phrases summarizing an input document. The recently emerged ONE2SET paradigm (Ye et al., 2021) generates keyphrases as a set and has achieved competitive performance. Nevertheless, we observe serious calibration errors outputted by ONE2SET, especially in the over-estimation of ∅ token (means “no corresponding keyphrase”). In this paper, we deeply analyze this limitation and identify two main reasons behind: 1) the parallel generation has to introduce excessive ∅ as padding tokens into training instances; and 2) the training mechanism assigning target to each slot is unstable and further aggravates the ∅ token over-estimation. To make the model well-calibrated, we propose WR-ONE2SET which extends ONE2SET with an adaptive instance-level cost Weighting strategy and a target Re-assignment mechanism. The former dynamically penalizes the over-estimated slots for different instances thus smoothing the uneven training distribution. The latter refines the original inappropriate assignment and reduces the supervisory signals of over-estimated slots. Experimental results on commonly-used datasets demonstrate the effectiveness and generality of our proposed paradigm.

## introduction keyphrases are short phrases fully encoding the main information of a given document. they can not only facilitate readers to quickly understand the document, but also provide useful information to many downstream tasks, including document classification @xcite , summarization @xcite , etc. with the rapid development of deep learning, keyphrase generation @xcite has attracted increasing attention due to its ability to produce phrases that even do not match any contiguous * * corresponding author. represents the multiple feasible target permutations generated by k-step target assignment mechanism at different training iterations. in this case, both "slot2" and "slot3" are expected to generate keyphrases. however, they often use ∅ token as supervisory signals, and thus over-estimate and output ∅ token. despite of its success, we observe serious overestimation problem on ∅ token, which significantly affects the generation quality. for example, in figure 1 , both "slot2" and "slot3" are expected to generate keyphrases, but ∅ token is over-confidently given. two questions naturally arise: 1) what are reasons behind the over-estimation problem in one2set? and 2) how can we alleviate them? in order to answer the first question, we conduct extensive analyses, and conclude two reasons. firstly, the over-estimation is a by-product inherently carried by the parallel generation. more concretely, excessive ∅ tokens have been introduced as the padding tokens and served as supervisory signals in training data. the unbalanced data and the lack of dependency among slots leads each slot to learn to commonly generate ∅ token. secondly, the k-step target assignment mechanism provides multiple feasible target permutations that are assigned to slots. as shown in figure 1 , the targets of the given document can be assigned in different permutation at each training iteration, which further increases the probability of ∅ token to be assigned as supervisory signal for each slot, thus exacerbating the over-estimation problem. both problems make the learned probabilities of the assigned targets deviate from its ground truth likelihood, finally constructing a miscalibrated model. consequently, we approach the above problems from the calibration perspective and propose two strategies that extend one2set to wr-one2set. specifically, an adaptive instance-level cost weighting is first introduced to penalize the over-estimated slots of different instances. according to the seriousness of the issue, instances are rendered different weights, therefore dynamically balancing the model training. besides, we propose a target re-assignment mechanism to refine the original inappropriate assignment and reduce the supervisory signals of ∅ token. in particular, we re-assign targets for the slots potentially generating fresh keyphrases but being pre-assigned with ∅ token. in these ways, wr-one2set is encouraged to produce well-calibrated probabilities on keyphrase generation. overall, major contributions of our work are three-fold: ## related work early studies mainly focus on automatic keyphrase extraction @xcite @xcite , which aims to directly extract keyphrases from the input document. recently, with the rapid development of deep learning, neural network-based models have been widely used in keyphrase generation. typically, these models are based on an attentional encoder-decoder framework equipped with copy mechanism, which is able to generate both present and absent keyphrases @xcite . generally, these models are constructed under the following paradigms: 1) one2one @xcite . under this paradigm, the input document is paired with each target keyphrase to form an independent training instance for model training. during inference, the models are encouraged to produce multiple keyphrases via beam search. 2) one2seq @xcite @xcite . it considers keyphrase generation as a sequence generation task, where different keyphrases are concatenated into a sequence in a predefined order. in this way, the semantic dependence between keyphrases can be exploited to benefit keyphrase generation. 3) one2set @xcite . unlike one2seq, this paradigm considers the keyphrases as a set, which can be predicted from slots in a parallel manner and partial target matching algorithm. considering that one2one neglects the correlation among keyphrases, the most popular paradigm one2seq exploits the correlation by pre-defining the keyphrases order for model training and inference. nevertheless, one2seq is the opposite of the flexible and unordered properties of keyphrases, increasing the difficulty of the model training. due to the parallelism and the conditional independence, one2set attracts much attention in the keyphrase generation community, and achieves the sota performance. as this method has just been put forward, it is inevitable to exist imperfections. hence, we are committed to analyses and further optimizing this framework. to the best of our knowledge, this is the first attempt to improve one2set. ## background here, we briefly introduce settrans @xcite , which is based on the one2set paradigm. it is a transformer-based, semi-autoregressive model. typically, it introduces n slots, each of which introduces a learnable control code as the additional decoder input, to generate keyphrases or ∅ tokens in parallel. its training involves two stages: 1) a k-step target assignment mechanism is firstly used to determine the correspondence between each prediction and target, and then 2) a new training objective is introduced to optimize the whole model. it contains two set losses to separately deal with two kinds of keyphrases: present keyphrases appearing in the input document, and absent keyphrases that do not match any contiguous subsequence of the document. ## preliminary analyses although one2set has achieved competitive performance, it still faces one major problem, i.e. ∅ token over-estimation. this occurs in such slots that produce ∅ tokens via the vanilla prediction while are able to generate correct keyphrases through the non-∅ prediction foot_2 . for illustration, we force all slots to generate non-∅ predictions during inference, where 14.6% of slots can produce correct ones. however, if we remove this restriction, 34.5% of these slots directly output ∅ tokens, revealing the over-estimation of ∅ token. such kind of miscalibration @xcite is a common drawback in neural network based models, which not only seriously hurts the generation quality of the one2set paradigm, but also limits the users' trust towards it. to understand the reasons behind this, we use the commonly-used kp20k dataset @xcite to train a standard settrans model, where the assigned targets to the slots of each instance are recorded during the last 80,000 training steps with an interval of 8,000 steps. here, we can obtain two crucial observations. observation 1: excessive ∅ tokens have been introduced as the padding tokens and served as supervisory signals in training data. one2set models keyphrase generation in a parallel computation fashion, therefore extensive padding ∅ to observation 2: the k-step assignment mechanism is unstable and further increases the possibility of ∅ tokens being served as supervisory signals for some slots. in spite of the clever design of kstep assignment mechanism, it unstably provides different feasible target permutations to slots at the training time. we argue that this further widens the gap between the distribution of supervisory signals and that of the ground-truth. to illustrate this, we classify the slots of each instance into three categories according to its target assignments: 1) slot(∅), each slot of this category is always assigned with ∅ tokens. apparently, these slots hardly generate keyphrases after training; 2) slot(target kp), each slot of this category is always assigned with target keyphrases and thus it has high probability of generating a keyphrase; 3) slot(∅+target kp), each slot is assigned with ## wr-one2set as discussed above, the parallelism and the training mechanism of one2set bring the advantages of conditional independence, but inherently lead to the miscalibration of the model. our principle is to maintain the primary advantages, and meanwhile, calibrating the model with lightweight strategies. to this end, we propose wr-one2set that significantly extends the conventional one2set paradigm in two training aspects, including an adaptive instance-level cost weighting strategy, and a target re-assignment mechanism. to facilitate subsequent descriptions, we summarize all related formal definitions in appendix, table a .2 for better understanding this paradigm. ## conclusion in this paper, we in-depth analyze the serious calibration errors of the one2set paradigm and point out its underlying reasons. to deal with this issue, we then significantly extend the conventional one2set into the wr-one2set paradigm with an adaptive instance-level cost weighting strategy and a target re-assignment mechanism. extensive experiments verify the effectiveness and generality of our extended paradigm. in the future, we plan to further refine our wr-one2set paradigm by considering the semantic relation between keyphrases. besides, we will improve our model by introducing variational neural networks, which have been successfully applied in many nlp tasks @xcite @xcite . finally, we will leverage the abundant knowledge from pre-trained models to further enhance our model.