Focused Attention Improves Document-Grounded Generation (2021)

Abstract: Document grounded generation is the task of using the information provided in a document to improve text generation. This work focuses on two different document grounded generation tasks: Wikipedia Update Generation task and Dialogue response generation. Our work introduces two novel adaptations of large scale pre-trained encoder-decoder models focusing on building context driven representation of the document and enabling specific attention to the information in the document. Additionally, we provide a stronger BART baseline for these tasks. Our proposed techniques outperform existing methods on both automated (at least 48% increase in BLEU-4 points) and human evaluation for closeness to reference and relevance to the document. Furthermore, we perform comprehensive manual inspection of the generated output and categorize errors to provide insights into future directions in modeling these tasks.

## introduction natural language generation (nlg) systems are increasingly expected to be naturalistic, contentful, and situation-aware due to their popularity and pervasiveness in human life @xcite . this is particularly relevant in dialogue systems @xcite , machine translation systems @xcite , story generation @xcite , and question answering systems @xcite . despite these mainstream applications, nlg systems face the challenges of being bland, devoid of content, generating generic outputs and hallucinating information @xcite @xcite . grounding the generation in different modalities * work done during internship at salesforce. like images @xcite @xcite , videos @xcite , and structured data @xcite alleviates some of these issues. generating natural language from schematized or structured data such as database records, slot-value pair, and wikipedia infobox has been explored in prior work @xcite @xcite . although useful, these tasks encounter difficulties such as general applicability (databases may not be available for all domains) and are constrained by the available resources (size of the database). document grounded generation mitigates these applicability issues by exploiting the vast availability of data in unstructured form (e.g. books, encyclopedias, news articles, and wikipedia articles). this enhances the applicability of document grounded generation to a wide range of domains with limited (or no) availability of structured data. hence, recent work has focused on defining new tasks and carving the scope of the problems @xcite @xcite @xcite . we focus on two different document grounded generation tasks: (1) wikipedia update generation task @xcite and (2) dialogue response generation @xcite . prior work has studied these two tasks independently and focused on task specific modeling techniques @xcite . our work unifies these tasks and formally shows the similarity in them: presence of a context and a document to ground the information in the generation process. our work introduces two novel improvements to the architectures of large scale pre-trained models @xcite : (1) we focus on building context driven representation of the document, where the context is taken into account while building the representation of the document, and (2) during generation we provide specific attention to the information in the document. we provide a stronger bart-based @xcite baseline for these tasks. this work shows that pre-trained models albeit good at text generation, can be further improved by providing grounding specific improvements. our main contributions are the two new proposed techniques for the document grounded generation tasks ( §3.2 and §3.3). we also provide a new baseline which is stronger than the previous state-of-the-art methods @xcite for the two tasks. we formally show how the two independent tasks studied in this paper are identical and similar modeling techniques can be used to solve them ( §3). automated and human evaluation results on three different datasets demonstrate substantial improvements ( §5.1 and §5.2). specifically, we achieve an improvement of 19.7 bleu-4 points compared to zhao et al. (2020b) on the dialogue generation task. additionally, significant gains are observed in bleu-4 compared to bart-based baseline. a comprehensive manual analysis of the generated output is presented in this work which paves way for future work ( §6). we will release our code on github. our task is to generate text given a context and a source of content (document). additionally, the generated text should coherently fit the context and contain information from the document. we focus on content present in unstructured form in documents to ground text generation. figure 1 illustrates such an example. dialogue response generation is traditionally conditioned on the dialogue context @xcite . as figure 1 demonstrates, the generative model is conditioned on both the document as well as the dialogue context. note that the context and document play different roles in impacting the generationthe context sets the background while the document provides the content necessary to generate the text. formally, each sample i of our task is defined as a tuple (d i , c i , x i ) containing context c i , document d i and text x i to be generated. note that each d i can be a single document or a set of documents. the task is to generate x i such that it coherently follows c i and contains information from d i . the task can be modeled as the following conditional text generation model: p θ (x i |c i , d i ), where θ is a set of model parameters. figure 1 illustrates that the generator has to account for two inputs the dialogue context c i (shown in blue) and the document d i (shown in red) to generate the response x i grounded in d i (text shown in green). if the generative model was only conditioned on dialogue context, then it could produce generic responses like "do you think they did the right thing?" or "yes, i agree." or hallucinate information like "yes, and the times published it on the front page.". these which would be appropriate to the given context but are devoid of content or contain wrong information. document grounded models are capable of responding with interesting facts like "yes, but it was dangerous for the white house to ban the post from the white house." ## methodology a natural way to model p θ (x i |c i , d i ) is to train an encoder-decoder model using cross-entropy loss -log p θ with respect to the ground-truth output text. we discuss two ways of building effective representations for encoder-decoder models to focus on d i : (1) combine encoder representations of c i and d i , (2) include an additional attention multihead at each layer of the transformer to specifically focus on the content in d i . ## document grounded generation tasks document grounded generation can leverage unstructured data as a source of grounding and can hence be applied to a variety of generation tasks such as dialogue responses, wikipedia articles, reports and legal argument. this work focuses on wikipedia update generation and dialogue response generation which have been studied independently in prior work. we discuss the similarities in these two tasks and design a common modeling technique for them. ## experiments and results we implement all our models with the transformers tool @xcite , and the details are in §a. ## analysis and discussion we manually inspect the outputs of the codr model on the development set of cmu_dog and wikipedia update generation dataset to understand the their quality. we inspect 60 samples in each dataset which have rouge-l @xmath0. these are chosen such that we have 10 samples in each of the 6 buckets of rouge-l score (buckets are range of 10 points: 0-9, 10-19, 20-29, 30-39, 40-49 and 50-59). we analyse the generated outputs along the two aspects of appropriateness of the generation to the context and its grounding in the document. cmu_dog: we find that 52/60 (86.7%) responses were appropriate to the given chat context. these 52 responses are further categorized in ## related work generation grounded in document has been studied through a large body of summarization work @xcite and similar tasks such as headline generation @xcite . multiple new works have extended this research in new directions; wikipedia update generation (prabhumoye et al., 2019) introduces the task of generating an update to the wikipedia context based on a news document; wikipedia article generation (liu et al., 2018) introduces the task of generating an entire wikipedia article based on multiple documents; text editing by command @xcite introduces the task of generating a particular type of wikipedia edit conditioned on a command provided in natural language and a grounding consisting of snippets of 200 web page results. parallely, new tasks have also emerged focusing on document grounding for dialogue response generation @xcite ting and use pre-training along with a disentangled decoder. the disentangled decoder consists of a context processor, knowledge processor and a language model. a dialogue manager is used to combine the vocabulary distributions provided by these three components. @xcite propose a knowledge selection module integrated with pre-trained language models for this task. cao et al. (2020) use pre-trained language model gpt-2 @xcite and explore various attention fusion techniques for persona-based dialogue generation @xcite . our doha technique also introduces an additional attention multi-head but does not use any additional weights to fuse attention heads. similarly, junczys-dowmunt and grundkiewicz (2018) use an additional attention multi-head in transformer architecture for automatic post-editing task. we demonstrate how attention can be enhanced in pre-trained models. the codr model fuses the representations of the document and the context in the decoder which is inspired by the fusion-in-decoder model in open-domain qa @xcite . although @xcite introduce the usage of bart for knowledge grounded dialogues, it is primarily from the perspective of improving knowledge retrieval. we provide benchmark bart numbers ( ## conclusion and future work this paper proposes two novel improvements for document grounded generation and provides a stronger baseline. this paper demonstrates how similar modeling techniques could be used for two previously separately modeled tasks. our proposed models outperform the previous techniques and the new stronger baseline on automated metrics and human evaluation for the three datasets discussed in the paper. we present a comprehensive manual inspection which reveal certain data artifacts and provides us with insight on how to model these tasks in future. particularly, future work can focus on designing better evaluation metrics which don't penalize linguistic variations in generation. better models can also be constructed to focus on cases of partial hallucination or incorrect responses.