Improving Absent Keyphrase Generation with Diversity Heads (2024)

Abstract: Keyphrase Generation (KPG) is the task of automatically generating appropriate keyphrases for a given text, with a wide range of real-world applications such as document indexing and tagging, information retrieval, and text summarization. NLP research makes a distinction between present and absent keyphrases based on whether a keyphrase is directly present as a sequence of words in the document during evaluation. However, present and absent keyphrases are treated together in a text-to-text generation framework during training. We treat present keyphrase extraction as a sequence labeling problem and propose a new absent keyphrase generation model that uses a modified cross-attention layer with additional heads to capture diverse views for the same context encoding in this paper. Our experiments show improvements over the state-of-the-art for four datasets for present keyphrase extraction and five datasets for absent keyphrase generation among the six English datasets we explored, covering long and short documents.

## introduction predicting keyphrases in a given text is useful in many application scenarios related to indexing and information retrieval. while some of the research on this topic focused on extracting only the keyphrases that directly appear in the document i.e., present keyphrases (see @xcite for a survey of extractive methods), more recent work on this topic treated it as an end-to-end text generation task, covering both present and absent keyphrases i.e., those that are not directly seen in the text. existing approaches can be categorized into three text representation paradigms: one2one, where the model learns to generate one keyphrase at a time per text @xcite ; one2seq, where all the keyphrases are generated as a single delimiterseparated sequence @xcite ; and one2set, where the model generates the keyphrases as a set @xcite . architecturally, kpg research falls into three categories, depicted in figure 1 . a common approach is to treat both present and absent keyphrases together, generating them autoregressively (e.g., @xcite , and figure 1 (a) illustrates this. a small number of approaches treat extraction and abstraction separately, but in a multi-task learning setup (e.g., @xcite where extraction is treated as sequence labeling and abstraction as autoregressive generation, illustrated by figure 1 @xcite . treating them as completely distinct (figure 1 (c )) has the potential to achieve increased performance on both, which we explore in this paper. a recurring problem in kpg research is that of repetition and duplication among the generated keyphrases. various approaches have been proposed in the literature to address this issue, which often result in high recall, low precision scenarios. an absent keyphrase generation approach that offers a good trade-off between precision and recall while generating diverse keyphrases is needed, which is also addressed in this paper. to summarize, we treat present keyphrase extraction and absent keyphrase generation as separate tasks, and focus on improving the diversity of absent keyphrase generation by introducing what we call diversity heads in this paper. the main contributions of the paper are listed below: the rest of the paper provides some background (section 2), describes our algorithm (section 3) and our experimental setup (section 4), followed by a discussion of our results (section 5), main conclusions (section 6) and brief notes on the limitations, and an ethics statement (section 7 and 8). ## our approach we separate present keyphrase extraction and absent keyphrase generation in this paper, by treating the former as sequence labeling, and the latter as sequence-to-set generation. details of our approach are explained below. ## experimental setup this section describes the implementation details, and provides the details of the datasets used, evaluation measures, and the baselines we compare against. implementation details all experiments were conducted in a distributed training setup with four nvidia tesla v100 gpus, and run for 5 epochs, with a batch size of 128 (per-gpu batch size of 32), and a text length of 384 tokens. the decoder is configured to have 8 independent decoder units with a maximum keyphrase length of 5 tokens (total 40 output tokens). these numbers were set after an empirical analysis with the validation datasets. the model is based on the transformer encoder-decoder backbone @xcite with 12 layers. a learning rate of 3e-05 is used with the adam optimizer (kingma and ba, 2014) and a non-linear learning rate scheduler with warm-up proportion of 0.1. ## results and discussion we present the main findings about present keyphrase extraction and absent keyphrase generation, followed by an analysis of the results in this section. considering the multiple train/test sets, we separate the discussion by short (abstracts) vs long (full text) documents. the first three approaches we compare against (unikp, transset and kpdrop-a in tables 2 3 4 5 ) were replicated and re-run and the next three (zero-shot chatgpt, seg-net and @xcite are taken as-is from the cited papers. ## conclusions we proposed to improve the keyphrase generation performance by a) developing a new approach for absent keyphrase generation with diversity heads and b) separating present keyphrase extraction as a plain sequence labeling problem. our experiments with six standard datasets consisting of short as well as long documents showed that our present keyphrase generation model outperformed the stateof-the-art on four of the six datasets, and our absent keyphrase generation model outperformed the stateof-the-art on five of the six datasets. the insights from further analysis of results on the evaluation measures and the nature of the datasets point to interesting directions for future work. improving cross-domain transfer, and exploring the portability of our approach to other languages are other possible future directions. it is important to note that the performance for this task reports much lower overall scores compared to other nlp tasks even with the recent generative llms like chat-gpt. this emphasizes the need for more focused research on the nature of the task itself along with model development. 7 limitations our experiments in this paper have focused only on english datasets and supervised learning scenarios, which can be considered a limitation. our model, while performing over the state-of-the-art for many datasets, still does poorly with crossdomain datasets (e.g., kp20k trained model tested on kptimes or viceversa). finally, our evaluation (which is the standard procedure for kpg) is more of a surface comparison and not a semantic comparison of the ground truth and predicted output, thus ignoring the keyphrases that semantically closer but lexically different from ground truth. there are also potential coverage issues in the original labeled keyphrases in one of the datasets, as qualitative analysis revealed some instances where the model's labels seem appropriate but do not figure in the ground truth (examples in table a10), which our evaluation process does not address at this point. additionally, while we separate extraction and generation, the two tasks indeed share some commonalities, which the complete separation ignores. a better multi-task framing of the task, where extraction can learn from generation and vice-versa may result in a better performance. finally, our experimental choices (e.g., number of epochs, not doing multiple runs of experiments) are limited by the compute availability, which can be considered a limitation in terms of exploring the experimental space in full. all our results and findings are to be understood in the context of these limitations.