Structure-Augmented Keyphrase Generation (2021)

Abstract: This paper studies the keyphrase generation (KG) task for scenarios where structure plays an important role. For example, a scientific publication consists of a short title and a long body, where the title can be used for de-emphasizing unimportant details in the body. Similarly, for short social media posts (, tweets), scarce context can be augmented from titles, though often missing. Our contribution is generating/augmenting structure then injecting these information in the encoding, using existing keyphrases of other documents, complementing missing/incomplete titles. We propose novel structure-augmented document encoding approaches that consist of the following two phases: The first phase, generating structure, extends the given document with related but absent keyphrases, augmenting missing context. The second phase, encoding structure, builds a graph of keyphrases and the given document to obtain the structure-aware representation of the augmented text. Our empirical results validate that our proposed structure augmentation and augmentation-aware encoding/decoding can improve KG for both scenarios, outperforming the state-of-the-art.

## introduction keyphrases not only help human readers to gain immediate insights about a given document, but also make documents queryable, e.g., by hashtagging social posts with predicted keyphrases. among keyphrase tasks, we focus on tasks that allow words that are not present in the given document, which often be called absent words, to be keyphrases, as the input document can be often context-scarce: many potentially relevant keyphrases are missing in the given document, since the keyphrase can be expressed in terms different from those present in the document (i.e., vocabulary mismatch). our goal is to generate keyphrases that are likely to be words absent from the document, especially field text title methods to tell if a question can be answered from a paragraph question ... is there any existing research in telling whether or not a question is answerable ? ... ... i considered textual entailment, but it doesn't seem to be exactly what i'm looking for ... existing ..., "natural language processing", keyphrases "formal languages", ... gold "question answering", keyphrases "natural language processing" in the following scenarios: (a) scientific publications with diverse vocabularies, or (b) short social posts, where we found that, from public and reallife datasets we used in our evaluation, 37% of keyphrases in scientific publications, 67% of hashtags, on average, in social media posts are absent in the given document. thus, we formulate our problem as keyphrase generation (kg) @xcite of predicting keyphrases, including absent words as well as present words, adopting encoderdecoder architecture. a recent trend is leveraging document structure for kg @xcite , where metadata in documents, e.g., document title, clarifies the meaning of documents @xcite . for illustration, our contribution is to construct a structured document x + , even when the structure is not given, by leveraging observed keyphrases from other documents in the training dataset, to improve both encoding and decoding. for encoding, keyphrases can be used to emphasize relevant parts in x , similarly done with the title. for decoding, the keyphrases can augment vocabulary, e.g., "natural language processing", from which the decoder can copy words. we stress that our work is designed to work for both closed and open set scenarios, represented by social and scientific document scenario respectively, with the following distinctions: our empirical results validate that generating and encoding document structures significantly improve performance, outperforming the state-of-theart, for both social and scientific documents. ## related work we briefly explain our target task of kg and introduce our distinction of leveraging structures. observing words in the document is a crucial signal for keyphrase tasks, especially for keyphrase extraction (ke) requiring keyphrases to appear in the given document. in contrast, we focus on kg, as our target scenarios require not having such restrictions. kg, to generate absent words, is often modeled as neural encoder-decoder architecture @xcite , which generates the keyphrase sequence given the input document @xcite . kg approaches can be further categorized into two settings: one-to-one (o2o) and one-to-seq (o2s) @xcite . in o2o, a model is trained to generate a single keyphrase for each document, and then, for evaluation, the model generates multiple keyphrases using beam search decoding with a large beam size (e.g., 200). on the other hand, in o2s, a model is trained to generate multiple keyphrases where multiple keyphrases for each document are concatenated into a single sequence with a predefined delimiter. our model follows o2o, as o2o is known to be better for predicting absent keyphrases @xcite and as absent keyphrases are frequently observed in our target scenario with scarce context. the distinction of our proposal for kg is to leverage structures in documents, to improve both encoder and decoder. structures in documents are essential sources of prediction signals @xcite . for example, for a scientific publication consisting of a title and the main body, the structured document enjoys the complementary strength of the two fields: while the main body contains many keywords (i.e., high recall), title, though much shorter, concisely describes the main focus of the paper (i.e., high precision). to leverage such structure, tgnet @xcite uses the title to guide the encoder to accurately capture core contents. however, the given title is observed to be often insufficient due to length limitation @xcite , which we consistently observed in our evaluations. as a further extreme, short social media posts may not have titles. furthermore, because of the length limitation (e.g., 140 character tweets), most keywords may not appear in the given post (i.e., low recall). in such scenarios, one can construct structured posts, to augment posts with missing keywords. for example, takg @xcite utilizes topic modeling, where topics shared across other documents enable the encoder to leverage contexts observed in other related posts. however, a small, fixed number of topics (e.g., 15 or 30) are limited in differentiating diverse documents with similar topics. our distinction is overcoming incomplete or missing structure, by generating a virtual structure from existing keyphrases (keyphrases in the train set), from which we "retrieve" terms that can serve as titles or topics for encoding. kg-ke-kr @xcite similarly leverages keyphrase retrieval, but they do not use this for structure-aware encoding and use only for decoding. in contrast, we jointly contextualize the given document and the retrieved keyphrases, to allow both fields to exchange contexts from each other. we empirically validate that our proposed structure-augmented encoding significantly boosts performance. ## approach given a document @xmath0, kg models aim to output a set of target keyphrases, which can be implemented using encoder-decoder architecture. we train a model to predict a single keyphrase, then, for inference, we generate multiple keyphrases using beam search decoding. we denote a single keyphrase with length t by @xmath1, . . . , y t ]. in the following sections, we first describe a baseline encoding targeting plain text x ( §3.1), and then, we explain our distinction of generating structure ( §3.2.1) and encoding structure ( §3.2.2). in this section, we do not consider pre-existing title t , but our framework straightforwardly extends for such titles, as we later discuss in §4.4.1. ## conclusion we studied the problem of kg for scientific text and social posts, representing context-scarce scenarios with open and closed keyphrase set, respectively. our work is two-phased, for augmenting and encoding missing/incomplete structure. empirical evaluation results validate that our proposed model outperforms the state-of-the-art in both problems.