T eg T ok: Augmenting Text Generation via Task-specific and Open-world Knowledge (2022)

Abstract: Generating natural and informative texts has been a long-standing problem in NLP. Much effort has been dedicated into incorporating pre-trained language models (PLMs) with various open-world knowledge, such as knowledge graphs or wiki pages. However, their ability to access and manipulate the task-specific knowledge is still limited on downstream tasks, as this type of knowledge is usually not well covered in PLMs and is hard to acquire. To address the problem, we propose augmenting TExt Generation via Task-specific and Open-world Knowledge (TegTok) in a unified framework. Our model selects knowledge entries from two types of knowledge sources through dense retrieval and then injects them into the input encoding and output decoding stages respectively on the basis of PLMs. With the help of these two types of knowledge, our model can learn what and how to generate. Experiments on two text generation tasks of dialogue generation and question generation, and on two datasets show that our method achieves better performance than various baseline models.

## introduction enabling natural models to generate natural and informative sequences is a challenging yet intriguing problem of artificial intelligence and has attracted increasing attention due to its promising potentials and alluring commercial values @xcite @xcite @xcite . thanks to the achievements on neural sequence modeling and pre-training technologies, current generative models are able to generate nature and fluency target sequences using either encoder-decoder architectures @xcite @xcite or language models @xcite @xcite despite these methods being the state-of-the-art frameworks for nlg, they are often provided limited knowledge to generate the desired output. thus, the performance of text generation is still far from satisfaction in many real-world scenarios @xcite . recently, much effort has been dedicated into incorporating traditional generative models or pretrained language models (plms) with a variety of open-world knowledge, such as structural knowledge bases (e.g., conceptnet) @xcite or unstructured documents (e.g., documents from wikipedia) @xcite . by providing the supplementary knowledge of an entity mentioned within or the background knowledge of a source text, it can help to better understand the input text and its surrounding context, and to ameliorate the informativeness of the generated text. although the open-world knowledge brings improvement to the generation process in most cases, its effect is still limited to the cases involving fewer entities or abstract semantics. on the other hand, the process of generating text by humans is often grounded by more than one single type of knowledge perception. in addition to world knowledge, the task-specific knowledge also acts as an important information source, and is usually not well covered in plms and is hard to acquire through fine-tuning. for example, in dialogue systems, what people have said or responded before can be reused as an important knowledge source, where these utterances talked before can be retained as the task-related knowledge in the mind of an interlocutor; for question generation, what part of a document makes people curious most and then ask specific questions, can often get enlightened by the existing questions raised from their corresponding passages. intuitively, these related task-specific examples can bring additional information associated with the given source mes-sages and provide exemplary information for neural generative models, but this useful information source is neglected in previous studies. on account of the above issues, we propose augmenting text generation via task-specific and open-world knowledge (tegtok). specifically, the world knowledge is assumed to be unstructured wikipedia documents that provide supplementary information of an entity mentioned within or background knowledge of an input sequence. the task-specific knowledge is a prebuilt index that is domain-relevant and acts as an exemplary information source for guiding text generation. it can be flexibly adjusted according to different tasks or domains, e.g., context-response pairs in dialogue generation or passage-question pairs in question generation. inspired by the success of dense retrieval methods for the task of open-domain question answering @xcite @xcite , we use pre-trained encoders to convert input texts and knowledge entries into dense representation vectors and employ fast maximum inner-product search (mips) @xcite to complete the retrieval, so as to ensure effectiveness and efficiency of knowledge selection. finally, these two types of knowledge are injected into source text encoding and target text decoding stages respectively. by this means, our model can learn how and what to generate in a unified framework with the help of two types of knowledge. to measure the effectiveness of our proposed framework, we evaluate it on the tasks of dialogue generation and question generation, which are both important research issues of text generation. experimental results show that our proposed method outperforms the gpt-2 @xcite and bart @xcite baseline models, and can generate more informative texts including entities that do not appear in the input texts. in summary, our contributions in this paper are three-fold: (1) a proposal of a general and unified text generation framework named tegtok that incorporates both task-specific and world knowledge through dense retrieval. (2) the proposed framework is verified on two text generation tasks. dialogue generation. the generation-based dialogue models synthesize a response with a nlg model by maximizing its generation probability given the previous conversation context. the pioneer researchers formulated the dialogue generation task as a sequence-to-sequence translation problem @xcite @xcite @xcite where encoder is designed for dialogue context modeling, and decoder is constructed to conduct the target response prediction. expanded from the general dialogue generation problem, more interesting and challenging tasks relying on external knowledge have been explored to improve the anthropomorphic characteristic of dialogue systems. a line of work introduced personalized information into dialogue generation to help deliver better dialogue response such as emotion @xcite @xcite and persona @xcite . in addition, to further enhance and enrich the response generation, researchers have studied grounding dialogue generation on knowledge graphs @xcite or unstructured documents @xcite @xcite @xcite . question generation. this task aims at generating a question from a given passage @xcite ) in an answer-aware or answer-unaware manner. in this paper, we work on the answerunaware setting, encouraging diversity of generated questions. researchers have explored statistical keyword extraction techniques to select salient words from input documents, and then incorporated the extracted keywords into question generation @xcite recent work has applied reinforcement learning to natural question generation @xcite . different from previous text generation models that either incorporate unstructured wikipedia knowledge or enhance the generation with exemplar cases, to the best of our knowledge, this paper makes the first attempt to retrieve and exploit both the task-specific and world knowledge for text generation in a unified framework. our knowledge retrieval process is conducted through dense representations which can help to capture deep and latent semantics. ## method formulation the task of text generation is to output an appropriate target text given a source text as input. given a dataset d, an example is represented as (s, t). specifically, s represents a source text and t represents a target text. a source text is used as a query to retrieve task-specific and world knowledge. technically, the retrieved task-specific and world knowledge entries can be treated as two latent variables z 1 and z 2 respectively that are marginalized to get the seq2seq probability p(t|s) via a top-m approximation as ## tegtok model figure 1 shows the overview architecture of tegtok which consists of a retriever and a generator. the retriever uses the input source text as a query to retrieve the world knowledge and task-specific knowledge, the former of which is concatenated with the source text as additional background knowledge and the latter is fed into the decoder as exemplary information to guide the target text decoding. details about each component are provided in the following subsections. ## conclusion in this paper, we study retrieving relevant external knowledge for enhancing text generation. two types of knowledge, i.e., task-specific and world knowledge, are retrieved using dense representations to ensure effectiveness and efficiency of knowledge selection, and are further incorporated into the input encoding and output decoding stages respectively, providing the supplementary information to guide text generation. experimental results on two tasks of dialogue generation and question generation show that our method achieves better performance than baseline models and can generate more informative texts. in the future, we will explore applying this framework to more text generation tasks and other modalities such as image caption, to further verify its effectiveness and generalization.