Section-Aware Commonsense Knowledge-Grounded Dialogue Generation with Pre-trained Language Model (2022)

Abstract: In knowledge-grounded dialogue generation, pre-trained language models (PLMs) can be expected to deepen the fusing of dialogue context and knowledge because of their superior ability of semantic understanding. Unlike adopting the plain text knowledge, it is thorny to leverage the structural commonsense knowledge when using PLMs because most PLMs can only operate plain texts. Thus, linearizing commonsense knowledge facts into plan text is a compulsory trick. However, a dialogue is always aligned to a lot of retrieved fact candidates; as a result, the linearized text is always lengthy and then significantly increases the burden of using PLMs. To address this issue, we propose a novel two-stage framework SAKDP. In the first pre-screening stage, we use a ranking network PriorRanking to estimate the relevance of a retrieved knowledge fact. Thus, facts can be clustered into three sections of different priorities. As priority decreases, the relevance decreases, and the number of included facts increases. In the next dialogue generation stage, we use section-aware strategies to encode the linearized knowledge. The powerful but expensive PLM is only used for a few facts in the higher priority sections, reaching the performance-efficiency balance. Both the automatic and human evaluation demonstrate the superior performance of this work.

## introduction dialogue systems strive to facilitate human-like dialogue responses @xcite . one essential precondition for generating high-quality dialogue is a sufficient cognition of the contextually-relevant knowledge besides the literal surface. the dialogue is grounded on both the given user query and the context-related knowledge @xcite . for example, given a query 'are you thirsty?', rather than 'yes/no', a meaningful response should have more information, such as 'yes, and i'm going to drink some water.' (context-related @xmath0. seeking information from external sources is a feasible way to enhance the machine's cognition of knowledge @xcite . as for the choice of knowledge source, the structural commonsense knowledge @xcite is a proven option, which consists of a lot of knowledge facts that are frequently used in daily life. as shown in figure 1 , a commonsense knowledge base consists of various triplets, where each triplet is a real-world fact. commonsense knowledge can contribute to many aspects of the opendomain dialogue generation, such as the semantic understanding @xcite , reasoning @xcite , topic transition @xcite . meanwhile, pre-trained language models (plms) @xcite can learn a lot of implicit knowledge from the massive pretraining data @xcite @xcite . hence, another feasible way is to transfer the knowledge hidden in plms to the dialogue generation @xcite . prior works have shown plms can significantly promote the generation of high-quality dialogue responses @xcite @xcite , stylized responses @xcite , etc. a model can simultaneously adopt the aforementioned two ways to take a step further. knowl-edgpt @xcite adopted a bert @xcite to select text knowledge and a gpt2 @xcite to generate responses. ke-blender @xcite can implicitly infer knowledge by fine-tuning blender @xcite ) with text knowledge. nonetheless, most plms can only process plain texts, which makes it hard to infuse the structural knowledge @xcite . as a compromise, linearizing structural knowledge into plain text is a compulsory trick @xcite . the linearized knowledge text is much more verbose than the original, bringing new challenges in commonsense knowledge-grounded dialogue generation. a dialogue session is often paired with a lot of commonsense fact triplets; for example, in reddit dataset @xcite , the average number of 1/2-hop facts is 98.6/782.2, respectively. as shown in figure 1 , linearizing a fact into text often requires 5+ tokens. thus, lengthy linearized text can significantly aggravate the burden of transformer-based plms foot_0 and often exceeds the limits of most general plms (e.g., 512/1024 tokens). according to our empirical study, fact candidates retrieved for a dialogue query are always redundant, where most responses (98.25%) use no more than 3 facts, but 77.65 facts are given on average. inspired by such an observation, this paper proposes a novel two-stage framework sakdp (section-aware knowledge-grounded dialogue generation with pre-trained language model). the powerful but expensive plm is only used to encode a few facts of higher relevance. first, in the prescreening stage, we train a priorranking network using the contrastive learning scheme @xcite to estimate the relevance and then cluster fact candidates into three sections of different priorities: high, moderate, and low. second, considering the investment benefit ratio, we use different encoding solutions in the following dialogue generation stage. we propose a bert-based context-knowledge joint encoder to jointly encode the dialogue query and the relevant facts included by the high/moderate priority sections, bringing deeper infusing and interaction between dialogue and relevant knowledge. then, we employ a lightweight non-pretrained side-way encoder to encode facts in the low priority section. third, we use a hybrid selection to select the encoded context/knowledge memories and a multi-source generator to generate diverse dialogues. experiments on a chinese dataset @xcite prove sakdp can outperform baselines by a large margin. meanwhile, we conduct extensive studies to analyze the ranking performance and the necessity of pre-screening. sakdp is still competitive even only using three facts. the novelty/contribution of this work is three-fold: 1) we propose a novel sakdp to investigate the potential of both commonsense knowledge and pre-trained language models; 2) we propose to rank and cluster knowledge into three sections of different priorities. it can maximize cost-effectiveness and flexibility by using section-aware schemes; 3) extensive experiments and studies demonstrate the performance of sakdp. knowledge retrieval: commonsense fact candidates are usually retrieved by matching the name @xcite : 1) all entity words appearing in the query x are denoted as a set {e i }. 2) {e i } are adopted as keys to retrieve fact candidates from the base k. if the head entity or the tail entity of a fact k @xmath1, then k i will be added to the candidate set k. ## methodology problem definition: inspired by the above observation, as illustrated in figure 2 , sakdp is a two-stage approach: ## conclusion in this paper, we present an efficient two-stage section-aware commonsense knowledge-grounded dialogue generation framework sakdp. we propose a ranking network to cluster knowledge candidates into different priority sections and adopt different use schemes. subsequently, sakdp can benefit from both bert and commonsense knowledge with a balance of efficiency and performance. extensive experiments demonstrate the performance leadership of our approach. in the future, we will continue to promote the integration of plms and knowledge: 1) we will continue to improve the efficiency of knowledgegrounded and plm-based dialogue response generation; 2) we will explore more solutions to select knowledge in the pre-screening stage, for example, using gnns; 3) current sakdp is not fully plmbased because it uses a gru decoder. we will also try to explore the option of fully plm-based solutions.