Space Efficient Context Encoding for Non-Task-Oriented Dialogue Generation with Graph Attention Transformer (2021)

Abstract: To improve the coherence and knowledge retrieval capabilities of non-task-oriented dialogue systems, recent Transformer-based models aim to integrate fixed background context. This often comes in the form of knowledge graphs, and the integration is done by creating pseudo utterances through paraphrasing knowledge triples, added into the accumulated dialogue context. However, the context length is fixed in these architectures, which restricts how much background or dialogue context can be kept. In this work, we propose a more concise encoding for background context structured in the form of knowledge graphs, by expressing the graph connections through restrictions on the attention weights. The results of our human evaluation show that this encoding reduces space requirements without negative effects on the precision of reproduction of knowledge and perceived consistency. Further, models trained with our proposed context encoding generate dialogues that are judged to be more comprehensive and interesting.

## introduction building on the idea of attention-based seq2seq models @xcite , recent language models such as bert @xcite and gpt-2 @xcite enable neural conversational models to generate responses that appear human-like and engaging @xcite . a closer look, however, reveals that the lack of long-term memory to represent consistent (world) knowledge and personality over multiple speaker turns can lead to incoherent content being generated @xcite . initiated by the conversational intelligence challenge @xcite , the research focus therefore shifted towards knowledge-grounded dialogue generation, resulting in first promising approaches using transformer-based architectures @xcite @xcite . the basic idea of these approaches is to provide the required background knowledge together with the current dialogue context when decoding the next system utterance. as the underlying language model's input sequence length is limited -for instance, to 1024 tokens in the case of gtp-2 -the presentation of the background knowledge to the model highly impacts the amount of context information that can be fed into a transformer network. in these earlier attempts, the knowledge was paraphrased into pseudo-utterances, on a par with the utterances from the dialogue history. in this paper, we show that a structured knowledge representation offers advantages over unstructured text: facts and complex relationships between different entities can be encoded concisely without performance drop in key indicators, such as knowledge correctness, consistency, and interestingness. @xcite showed the general feasibility of integrating knowledge graphs into domain-specific dialogues. with this work, we integrate arbitrary knowledge graphs into open-domain knowledgegrounded dialogues, preserving the information encoded in their structure. space efficient context encoding for our proposed encoding, we generate dialogue-specific local knowledge graphs (subgraphs of a background knowledge graph) that capture the information relevant to the dialogue (similar to @xcite ). we transform these subgraphs into a concise representation that fits the input sequence encoding for the underlying language model (gpt-2): labels of the distinct nodes and edges (entities and corresponding relations) are concatenated with the dialogue history. to preserve the graph structure, we fit the attention mask to force the self-attention layers for each node to attend to only connected nodes in the original graph (if there is a connection, attention weight is set to 1, otherwise to 0). this resembles the message-passing approach of graph neural networks @xcite . naive concatenation of graph triples has a space complexity of o(n • k), with n being the number of triples and k the number of word tokens per verbalized triple. paraphrasing these triples into pseudo-utterances results in even larger space complexity. our proposed encoding has a space complexity of o(l), with l being the number of distinct node and edge labels (entities and relations). this reduces the required context space compared to triple concatenation or paraphrasing if entities are repeated in the triples (and hence @xmath0, which can be assumed to be the case in knowledge graphs (see discussion below). the space savings grow with the size and average degree (connectedness) of the graph. empirical results with two different knowledge-grounded dialogue datasets confirm our theoretical considerations and show that we can reduce the required space by a factor of up to 3.6. these results imply that we can feed more context information into the model, which should result in higher accuracy. we discuss these results in detail in section 4.3. ## knowledge-augmented neural conversational models neural conversational models can be categorized into retrieval-based approaches @xcite that choose a next utterance from a set of suitable candidates, and generative approaches @xcite @xcite which decode the next utterance token by token out of a fixed vocabulary. the architectures are based on recurrent neural networks such as lstm @xcite or gru @xcite cells or self-attention layers @xcite in sequence-to-sequence structures. to integrate knowledge in addition to the dialogue history these models can be augmented by additional recurrent cells to encode the knowledge into a fixedsized vector representation @xcite @xcite . this can be traced back to first endto-end approaches reading documents for questionanswering @xcite or more general sequential data @xcite . @xcite embedded knowledge graphs (stored as triples) with lstm cells and message-passing, and then used a decoder lstm to generate a suitable answer. @xcite used a cnn architecture to encode external knowledge instead. the recent success of unsupervised pre-trained language generation models such as gpt-2 yielded a variety of conversational models using selfattention based on the idea of fine-tuning the models with specific knowledge-grounded dialogue datasets (which we will discuss in section 3). these models concatenate the additional context information as plain text to the input sequence @xcite @xcite . to differentiate context from dialogue, additional tokens are learned during fine-tuning and added to the word tokens. for bigger knowledge graphs, the limitation of the input sequence length of these models makes an information retrieval system necessary to estimate a small subset of relevant information that can be fed into the model. ## knowledge-grounded dialogue datasets the increasing availability of conversational content on social media platforms such as twitter or reddit led to the construction of many dialogue datasets, with open-subtitles @xcite and twitter-corpus @xcite being some popular examples (see also @xcite ). some recently published datasets emphasize knowledgeable dialogues by integrating external information sources. the objective is to create models that generate consistent dialogues with a high knowledge retrieval accuracy (utilizing information from user profiles or knowledge graphs). @xcite released the wizard of wikipedia dataset with over 22k open-domain dialogues. in each dialogue, one participant is playing the "wizard", i.e. an expert who is presented with potentially interesting and relevant wikipedia article excerpts, while the chat partner is the curious apprentice. the textual knowledge passages that were shown to the wizard are part of the dataset. the persona-chat dataset @xcite contains over 10k dialogues that are conditioned on profile information (personas), which ranges from hobbies or favorite food to family background. the information is shown to the participants as a set of sentences and they are tasked to integrate them into the dialogues. in addition, the dataset contains revised personas, which are rephrased, generalized, or specialized versions of the original personas. ## automated evaluation, and its limits we trained 25 models with both datasets with series-encoding, parallel-encoding, base-triples and base-paraphrased (only komodis) and with graph depths d0, d1 and d2. as we were also interested to investigate the effect of different decoding strategies, we used beam-search and topk-sampling when generating the dialogues. these were created by four colleagues (who were not involved in the creation of the models and did not know what the innovation was) interacting with the models. in sum, we created 500 dialogues. at training time, we use perplexity on the validation subset as the stopping criterion. in information retrieval, precision/recall and f1 score are typically chosen as automated metrics to evaluate the retrieval capabilities of a system. in our case, we are interested in the ability of reproducing entities and relations from the knowledge graph. commonly used word-overlap metrics, such as rouge-n/rouge-l @xcite , or bleu @xcite , aren't capable of measuring these. evaluating precision and recall requires precise co-reference resolution, parse tree annotation and question labelling (e.g. entities, relations, intent). assume following background facts and generated utterance: without knowing the intent (asking for the genre of 'pulp fiction' vs. asking for the preferred genre) of the previous utterance, we cannot determine if the occurrences of crime and comedy are true or false positives concerning the underlying knowledge graph. moreover, in a conversation knowledge is not only used when answering a question dialogue context inference relevant utterance-context pairs are shown together with annotations from our human evaluators. line one is labeled with "neutral", as a "drama" might be a "musical", the utterance in line three entails with the context regarding the genre and is therefore labeled with "entailment". line seven is labeled with "contradiction" as the value of the budget is wrong, but also as "entailment", as the proactive use of the release year is correct. but also proactively ("did you know that bruce willis worked on the movie only for 18 days?"). however, the resources that we use offer no such annotations and to the best of our knowledge, no published dataset does. without it, automated metrics don't work well. to tackle these challenges, we put our effort into a comprehensive human evaluation and annotation, which is described in the next section. participants were paid for their effort. ## conclusion we proposed a new and concise encoding for knowledge triples from a knowledge graph, which can be integrated into a for consistent non-goal-driven dialogue generation. in our encoding, we reduce the context length by avoiding repetition by concatenating the whole triples with the dialogue history. by manipulating self-attention layers to reflect connections between nodes in the graphs, we preserve the graph structure. the evaluation results prove that our encoding reduces space requirements without negative effects on the precision of reproduction of knowledge and perceived consistency. for reproducibility, we publish the source code and data.