Knowledge-Consistent Dialogue Generation with Knowledge Graphs
Anonymous Author(s)
AffiliationAddress email
Abstract
Pre-trained generative language models have achieved impressive performances ondialogue generation tasks.
However, when generating responses for a conversationthat requires complicated factual knowledge, they are far from perfect, due to thelack of mechanisms to retrieve, encode, and reflect the knowledge in the generatedresponses.
Unlike the methods working with unstructured text that are inefficient in retrieving and encoding the knowledge, some of the knowledge-groundeddialogue generation methods tackle this problem by leveraging the structuredknowledge from the Knowledge Graphs (KGs).
However, existing methods donot guarantee that the language model utilizes a relevant piece of knowledgefor the given dialogue, and that the model generates dialogues which are consistent with the knowledge, from the KG.
To overcome this limitation, we propose
SU bgraph R etrieval-augmented GE neration ( SURGE ), a framework for generating knowledge-consistent, context-relevant dialogues with a KG.
Specifically, ourmethod first retrieves the relevant subgraph from the given KG, and then enforcesconsistency across the facts by perturbing their word embeddings conditionedon the retrieved subgraph.
Then, it learns the latent representation space usinggraph-text multi-modal contrastive learning which ensures that the generated textshave high similarity to the retrieved subgraphs.
We validate the performance ofour SURGE framework on the OpendialKG dataset and show that our method doesgenerate high-quality dialogues that faithfully reflect the knowledge from the KG.
Introduction
Dialogue systems aim at conversing with humans by generating human-like responses, consideringthe context and history of the dialogue.
Recently, thanks to the development of pre-trained languagemodels (PLMs) for text generation [32, 34], neural dialogue agents are able to generate fluent responses.
However, despite their satisfactory fluency, they often generate factually incorrect responsesdue to a lack of explicit knowledge.
The problem can become worse, when the conversation requiresaccurate knowledge about certain subjects.
Thus, to overcome such limitations, some of the recentmethods access the external knowledge sources, for example, Wikipedia [7] or Web [21], and thenretrieve the documents containing the relevant knowledge for ongoing conversations.
While retrieving the relevant documents from a large-scale text corpus with information retrievaltechniques significantly boosts the performance of dialogue agents [18, 24], the computational burdenof searching for the relevant documents and embedding them on the fly could be high, which maycompromise the responsiveness of the conversation agent.
Thus, we instead consider the approachthat utilizes the pre-compiled Knowledge Graph (KG) [2, 43] consisting of symbolic facts, whichrepresent the entities as nodes and their relations as edges, in the form of a triplet, e.g., (Pride &
Prejudice, written by, Jane Austen) .
Such KG-augmented dialogue generation models are highlyefficient compared to retrieving from and augmenting with unstructured texts.
This is because we
[Figure]
Jane Austen writer
[Figure]
can directly retrieve entities from the context without searching for all candidate documents from alarge text corpus (e.g. Wikipedia), and the retrieved facts succinctly encode the required knowledgein the most compact and effective form.
Figure 1 shows an example which illustrates how the KGrepresents facts as relations among entities, that help generate a knowledge-grounded response.
Few recent works [41, 9, 50] use the KG to provide facts associated with the entities in the dialoguecontext to the conversation agents.
However, they utilize all the triplets associated to the given entity,whose facts are mostly irrelevant to the dialogue context (e.g., Jane Austen was born in Steventon in
Figure 1), which could mislead the model into generating factually incorrect responses.
We foundthat about 87% of facts from 1-hop KG are irrelevant to the context in the OpendialKG dataset [29].
Moreover, encoding all the facts including the unnecessary ones is computationally inefficient [9].
Even after correctly retrieving the relevant facts from the KG, it is not straightforward to combinethe representations from two heterogeneous modalities: the dialogue context is represented as a text,meanwhile, the knowledge is represented as a graph.
Moreover, since the PLMs already have tons ofpre-trained parameters trained on the unstructured texts, properly conditioning the structured graph tothe PLM is highly important.
If not done so, PLMs may generate inconsistent responses with regardto the knowledge from the retrieved subgraph, whose phenomenon is known as hallucination [37, 8]where PLMs generate responses with their own memorized yet incorrect knowledge.
In this work, we tackle such challenging and fundamental issues of knowledge-consistent dialoguegeneration with KG 1 .
In particular, at the first step, we propose a context-relevant subgraph retrievalthat retrieves only the relevant triplets from a large KG to prevent the model from generating contextirrelevant responses.
Notably, our subgraph retrieval method is end-to-end trainable jointly withthe generation objective by marginalizing the likelihood of the generated sentences over the latentvariable of the retrieved subgraph [24].
Then, to encode the retrieved subgraph along with the inputtext sequence, we propose a graph encoding that is permutation and relation inversion invariant yetefficient.
Furthermore, to ensure that the model does make use of the encoded knowledge whengenerating responses, we propose a multi-modal contrastive learning objective between the twodifferent graph-text modalities to enforce the consistency across the retrieved facts and the generatedtexts.
We refer to our framework as SU bgraph R etrieval-augmented GE neration ( SURGE ).
We validate our SURGE framework on the OpendialKG [29] dataset against relevant baselines, withthe T5-small [34] model as the base PLM.
When evaluating the generated responses from the dialogueagents, the conventional metrics (e.g. BLEU [31], Rouge [27]) can not measure how faithfully thegenerated responses reflect the world knowledge.
Thus, we introduce an additional performancemetric, referred to as Knowledge-verifying Question Answering (KQA), which evaluates whethergenerated responses contain the correct knowledge with an extractive question answering model.
The experimental results show that SURGE generates responses that not only agree with the goldknowledge but are also consistent with the retrieved knowledge from the KG.
Our main contributions can be summarized as follows:
• We propose a context-relevant subgraph retrieval method for knowledge graph-augmented dialoguegeneration, to extract only the relevant piece of the knowledge for the given context from the entire
knowledge graph, for generating more appropriate responses to the ongoing conversation.
• We propose an invariant yet efficient graph encoder and a multi-modal graph-text contrastive
learning objective to ensure that the generated responses faithfully reflect the retrieved knowledge.
• We validate SURGE against relevant baselines, demonstrating its efficacy in generating responses
that are more informative by retrieving and reflecting the relevant knowledge from the KG.
Related Work
Pre-trained Language Model Large Pre-trained Language Models (PLMs) [32, 23, 34] that use theencoder-decoder architecture based on Transformers [42] have achieved great successes on languagegeneration tasks.
As they can accurately contextualize the given context and then generate human-likesentences, they are often used as the base architecture for the neural dialogue systems [49, 15].
Moreover, when the PLMs become larger, dialogue generation models have shown to generatehigh-quality responses [1], suggesting that pre-trained parameters do contain certain knowledge.
However, despite the fluency of such PLM-based dialogue agents, they often generate factuallyincorrect responses that are unfaithful to the dialogue context but look plausible – widely knownas a hallucination problem [28].
Thus, generating responses requiring specific and valid factualknowledge is still challenging.
To tackle this, recent works propose to retrieve knowledge fromexternal sources, and then use it to augment the neural dialogue agents [37, 40], discussed below.
Knowledge-Grounded Dialogue
The sources of external knowledge can be categorized into twotypes: documents from large unstructured corpora such as Wikipedia [7] or Web [30], and symbolicfacts from Knowledge Graphs (KGs) [2, 43].
Firstly, Dinan et al.
[7] propose a retrieval-baseddialogue generation model, which links the pre-compiled documents retrieved from Wikipedia articleswith the given dialogue context using the information retrieval [3].
Further, several works [19, 25, 40]propose to learn the document retrievers in an end-to-end fashion, to generate the knowledge-groundedresponses for the given dialogue.
However, KG-augmented dialogue generation models, which usestructured KGs, are more efficient than the previous methods utilizing unstructured texts thanks to theefficacy of KG for encoding knowledge, and consequently thus more preferable when responsivenessis important [26].
Regarding the dialogue generation with the KG, Moon et al.
[29] introduce aknowledge-grounded dialogue dataset where each dialogue comes with the large-scale KG.
Beforethe era of pre-trained language models, several works [41, 45, 48, 5, 50] have suggested recurrentneural network based sequence-to-sequence models that generates dialogue by conditioning theoutput word distribution with the entities from the KG.
Further, Galetzka et al.
[9] propose an efficientway to encode all of the facts in the 1-hop neighbors of the entities that appear in the dialogue historyin the given KG, in order to reduce the number of input tokens used in the pre-trained languagemodel [32].
However, all of these methods simply match and retrieve all the facts for entities includingirrelevant ones that appear in the dialogue history, which may mislead the agent into generatingout-of-context responses.
Our work differs from these existing works, since we aim at retrievingonly the context-relevant subgraph among the 1-hop facts with a novel subgraph retriever, which isend-to-end trainable along with the dialogue generation model.
Method
We first discuss the basic ingredients: graph neural networks and transformers.
We then formalize thedialogue generation problem and describe the key components for our SU bgraph R etrieval-augmented
GE neration ( SURGE ) framework: context-relevant subgraph retrieval, invariant graph encoding, andgraph-text contrastive learning.
Figure 2 illustrates the overview of our framework.
3.1 Preliminaries
As we use two different modalities, namely text and graph, we first define them, and then describethe neural networks to encode them.
In particular, a text is defined as a sequence of tokens x =[ x 1 , ..., x N ] , ∀ x i ∈ V , where x i is a token and V is a pre-defined vocabulary formed with specifictokenization algorithms [39].
On the other hand, a knowledge graph (KG) is a type of multi-relationalgraphs G = { ( e h , r , e t ) } ∈ E × R × E , where e h and e t are head and tail entities along with theirrelation r , and E and R are sets of entities and relations, respectively, i.e., e h , e t ∈ E and r ∈ R .
To easily access different modalities in the same framework, we define the mapping function that mapsentities and relations in the KG to the tokens in the text as follows: q e : E → V l and q r : R → V l .
Inother words, any entity e ∈ E and relation r ∈ R can be mapped to a sequence of l tokens x ∈ V l :q e ( e ) =
x e and q r ( r ) =
x r .
Such functions enable us to associate the KG symbol with the text.
Transformer A Transformer [42] is a neural architecture that embeds a sequence of tokens whiletaking their relationships into account.
It is a basic building block of recent PLMs [6, 32].
Formally,assume that we have a sequence of tokens x = [ x 1 , ..., x N ] , ∀ x i ∈ V , then a goal of generativetransformers is to generate a sequence of tokens y <t =
[ y 1 , ..., y t − 1 ] , ∀ y i ∈ V , with encoder Enc ,decoder Dec and tokens’ embedding function f .
Thus, a hidden state at time t for generating
Negative Subgraph (𝒵′)
Could you recommend any book by Jane Austen ?
User b
KnowledgeGraph ( 𝒢 )
Dialogue History ( 𝒙 )
Subgraph (𝒵)
Context-Relevant Subgraph Retriever 𝑝 !
(𝒵|𝒙)
Encoder
Decoder
Generator (PLM)
Generate 𝑝 " (𝒚|𝒙,𝒵)
Sure, she wrote Pride & Prejudice and Lady
Susan.
Graph-Text Contrastive Learning
Decoder Hidden States ℎ a d e f
Place_of_birth written_by
Jane Austen; a
Lady Susan; d written_by
Iron man
Stan Lee
Iron Man 2 sequel c
Pride & Prejudice; e written_by written_by is_a written_by has_
[Figure]
a is_a a c
[Figure]
d
[Figure]
by f
[Figure]
y t is h t =
Dec ( Enc ( X ) , Y <t ) , where X = f ( x ) = [ f ( x 1 ) , ..., f ( x N )] and Y <t = f ( y <t ) =[ f ( y 1 ) , ..., f ( y t − 1 )] .
We note that both Enc and Dec functions are permutation sensitive withpositional encodings as in generic Transformer architecture [42, 46].
Graph Neural Network A Graph Neural Network (GNN) represents a node with its neighboringnodes over the graph structure [12], which is formalized as follows:
[Equation]
where e t and e h are embeddings of entities (nodes) e t and e h , respectively, N ( e t ; G )
= { e h |( e h , r , e t )
∈ G} is a set of neighboring entities of e t , AGG is a function that aggregates embeddings ofe t ’s neighboring entities, and UPD is a function that updates a representation of e t with the aggregatedmessages from AGG , at each iteration k .
3.2 Problem Statement
Here we formalize the problem of context-relevant subgraph retrieval for knowledge-groundeddialogue generation.
Given a dialogue history x
= [ x 1 , . . .
, x N ] , a model with generative PLMs firstencodes the input tokens, and then models a probabilistic distribution p ( y | x ) to generate an outputresponse y =
[ y 1 , . . .
, y T ] .
This problem requires a piece of specific knowledge for a conversation.
To that end, given a dialogue history x , we aim at retrieving a subgraph Z ⊆ G consisting of a set oftriplets z ∈ Z where z = ( e h , r , e t ) , which encodes relevant knowledge for ongoing conversation.
Thus, the distribution of the context-relevant facts Z is p ( Z| x ) , and our final likelihood of generatingresponses then becomes p ( y | x , Z ) .
Then, to jointly optimize the objective of graph retrieval withresponse generation, we treat Z as a latent variable and then marginalize the likelihood of thegenerative model over all possible latent variables for retrieved subgraphs Z , formalized as follows:
[Equation]
where p ϕ ( Z| x ) is an output distribution of the context-relevant subgraph retriever, and p θ ( y | x , Z ) isthe target distribution of a knowledge-augmented generator, parameterized as ϕ and θ , respectively.
3.3 Context-Relevant Subgraph Retriever
We now provide a concrete description of our context-relevant subgraph retriever formalized in Eq. 2.
We assume that a retrieval probability of each triplet in Z = { z 1 , . . .
, z n } is independent.
Then, forsimplicity, we decompose the probability of retrieving a set of triplets p ( Z| x ) into the product ofindividual triplet retrieval probabilities, as follows: p ( Z| x )
=
p ( z 1 | x ) p ( z 2 | x ) . . .
p ( z n | x ) .
From the aforementioned Eq.
about p ( Z| x ) , we can now focus on retrieving the only one triplet.
Therefore, we define the retrieval of one triplet with an inner product of dense vectors between thedialogue history x
and the candidate triplet z , similarly to a dense retrieval model [11], as follows:
[Equation]
where d is a triplet embedding function and q is a dialogue context embedding function.
We can usea PLM for implementing q , but we need another effective method for d that can reflect the propertyof the graph.
Therefore, we propose the GNN-based triplet embedding method for realizing d .
Let consider a set of triplets associated to the entities that appear in the given dialogue context{ ( e , r , e t ) or ( e h , r , e ) | q e ( e ) ⊆
x } , as the retrieval candidates.
To effectively represent the tripletsconsisting of entities and their relations as items, we use GNNs described in Section 3.1 for the tripletembedding function d .
In our triplet retrieval, representing both nodes and edges, which are equallyessential components for the multi-relational graph, is worthwhile to represent an entire triplet.
Todo so, we adopt the existing edge message passing framework [17] that transforms edges of theoriginal graph to nodes of the dual hypergraph [38] (i.e., transforming G to G ∗ ), which allows us touse existing node-level GNNs for representing edges of the original graph (See Section D.1 of the
Supplementary File for more details).
Formally, our triplet embedding function is denoted as follows:
[Equation]
where z = ( e h , r, e t ) , and ∥ is the concatenation operator.
3.4 Invariant Graph Encoding
In this subsection, we then now specify the remaining operations for p θ ( y | x , Z ) , which generates yconditioned on the two different modalities, namely text x and graph Z .
Before doing so, we firstdefine the notion of graph encoding, whose goal is to leverage the retrieved subgraph informationalong with the dialogue history for response generation, which is formalized in Definition 3.1.
Definition 3.1.
(Graph Encoding) Let ψ ( x , Z ) be a graph encoding function.
Then, givena sequence of tokens x = [ x 1 , ..., x N ] and a subgraph Z , it first yields a new sequencex ′ =
[ x ′ 1 , ..., x ′ m , x 1 , ..., x N ] where [ x ′ 1 , ..., x ′ m ] comes from q e ( e ) = x
′ e and q r ( r ) = x ′ r∀ ( e , r , ∗ ) ∈ Z .
Then, it embeds a sequence X ′ =
[ f ( x ′ 1 ) , ..., f ( x ′ m ) , f ( x 1 ) , ..., f ( x N )]
=f
([ x ′ 1 , ..., x ′ m , x 1 , ..., x N ]) , where f is the token embedding function.
Consequently, X ′ =
ψ ( x , Z ) .
For instance, given a sequence x = [ x 1 , . . .
, x N ] and a subgraph Z
= { ( a , d , b ) , ( b , e , a ) , ( a , d , c ) }from the retriever, ψ ( x , Z ) =
f
([ a, d, b, b, e, a, a, d, c, x 1 , ..., x N ]) with a = q e ( a ) , b = q e ( b ) ,c = q e ( c ) , d = q r ( d ) , e = q r ( e ) , which we term as the naïve encoding.
Due to its simplicity, it iswidely used for a text-conditioned generation [24].
However, for graph encoding, it violates twoimportant invariance properties: permutation invariance [47] and relation-inversion invariance, whichare formalized in Definition 3.2, 3.3.
Definition 3.2.
(Permutation Invariance)
For any set permutation π , ψ ( x , Z ) = ψ ( x , π · Z ) , i.e.,an order of elements in a subgraph does not affect a representation.
Definition 3.3.
(Relation Inversion Invariance) Let a relation ¬ d be an inverse relation to d , if( a , d , b ) =
( b , ¬ d , a ) ∀ a , b ∈ E .
Then, ψ ( x , Z ∪ { ( a , d , b ) } ) =
ψ ( x , Z ∪ { ( b , ¬ d , a ) } ) for anysubgraph Z .
Invariant Graph Encoding To meet both properties, we consider two additional operations on aset of triplets up to the naïve encoding.
We first define a SORT operator that returns the same outputregardless of the order of input set elements, as follows:
[Equation]
where S n is a set of all possible permutations for n elements.
Moreover, we define a INV operatorthat adds the inverse triplet of each triplet in the subgraph Z , as follows:
[Equation]
With above operations, we now define a more solid graph encoding function: ψ ( x , SORT ( INV ( Z ))) ,which satisfies both permutation and relation inversion invariance.
Invariant and Efficient Graph Encoding However, above encoding is not efficient since it requiresthe O ( n ) space complexity for encoding a graph with n triplets.
To be more efficient, we newlydefine ˜ ψ that only encodes the unique nodes (entities) along the sequence, formalized as follows:
[Equation]
where ENT ( Z ) returns the set of unique nodes in Z and SORT is used to preserve the permutationinvariance.
This encoding is thus invariant but efficient since it only costs O ( k ) , for a k -entity sequence where k < n .
However, as it does not consider the relational information in Z , we furtherperturb the entities’ token embeddings with respect to their representations in Z .
Specifically, foreach entity a ∈ ENT ( Z ) , we apply affine transformations from learnable Multi-Layer Perceptrons( MLP ) on the token embedding of a as follows:
[Equation]
where β : R d → R d perturbs the embedding according to Z , AGGR is the relation-aware aggregationfunction for triplet ( b , r ,
a ) ∈
Z with a = q e ( a ) and b = q e ( b ) .
In sum, we denote a relation-awareinvariant and efficient encoder ψ ∗ , formally represented as follows:
[Equation]
where β can be applied to the sequence of representations, β : R n × d → R n × d .
We conclude that ourgraph encoding satisfies both properties.
For proofs, please see Section C of the Supplementary File.
3.5 Consistent Generation with Graph-Text Contrastive Learning
Although the previous schemes allow retrieving and encoding subgraphs that are relevant to theinput dialogue history, the consistent generation with the given subgraph is further required, whengenerating responses with the factual knowledge.
In other words, the model should be able to generatedifferent sequences given different graphs with the same dialogue history.
However, we only access the single ground-truth response regardless of the retrieved knowledge,while the generative model is trained with a teacher forcing.
Thus, this setting can give rise tothe problem of exposure bias [35]: the model is never exposed to other generated tokens duringtraining.
To overcome such limitations, we introduce a novel graph-text contrastive learning methodmotivated by multi-modal contrastive learning [33].
Formally, for a single pair of a graph and text,the contrastive learning objective is defined as follows:
[Equation]
where z = 1 m (cid:80) mi =1 z
i is the mean of graph representations from [ z ′ 1 , . . .
, z ′ m , z 1 , . .
z N ] =
Enc ( ψ ∗ ( x , Z )) , h = 1 T (cid:80) Tt
=
h t is the mean of decoder representations, sim is the cosine similarity,ζ and ξ are learnable linear projection layers, and τ is a learnable temperature parameter.
Furthermore, 234 (cid:80) h ′ and (cid:80) z
′ indicate the summation over negative samples, which are other texts or graphs withina same mini-batch as in previous contrastive literature [4, 10, 13, 20, 22, 33].
With Eq. 8, the modelcan embed the correlated pairs closer together in order to generate a consistent response to a givengraph, i.e., given a different graph, the model would generate different tokens for the same text.
3.6 Training
We train the entire model by maximizing the log-likelihood log p ( y | x ) defined in Eq. 2 with respectto parameters of the retriever ϕ and the generator θ .
However, computing the marginal probabilityover all possible subgraphs (cid:80)
Z⊆G p ϕ ( Z| x ) p θ ( y | x , Z ) is impractical.
Therefore, as in existingworks [11, 24], we approximate this by instead summing over k sampled subgraphs.
Moreover,for each subgraph, we samples n triplets without replacement from the categorical distribution,parameterized by ϕ : z 1 , . . .
, z n ∼ Cat ( |G| , p ϕ ( z | x )) , which results in Z = { z 1 , . . .
, z n } .
Our whole end-to-end training objective for retrieval-augmented generation is then defined as follows:
[Equation]
where we simplify the sampling over n triplets as the sampling over the subgraph distributionp ϕ
( Z| x ) .
We assume that we can access the gold subgraph for some data in training.
Thus, wefurther add the supervised retrieval loss to introduce a semi-supervised retriever learning as follows:
[Equation]
where Z ∗ is the available ground-truth subgraph.
Combining all objectives in Eq.
8, 9, and 10, ourfinal training objective is then defined as follows:
L =
L ret + L sup + L cont .
Could you recommend a book written by Jane Austen ?
User (a) Dialogue History
Sure, Lady Susan.
Jane Austen
Lady Susan
Jane Austen
Lady Susan write (b) Corresponding Fact (c) Candidates in KG
Jane Austen write
Emma
Sense & Sensibility
QA Context: Could you recommend a book written by Jane Austen?
<
Generated Response>
Question: (Jane Austen , write , ? )
Jane Austen
Answer Candidates: (d) KQA example
Lady Susan
Sense & Sensibility Emma
[Figure]
A Novel Metric: Knowledge-verifying QA
Existing automatic evaluation metrics, namely BLEU and ROUGE [31, 27], are limited in that theyonly consider the lexical overlaps of words without measuring the factual correctness of the generatedresponses.
As shown in Figure 3 (a), there could be multiple correct responses, but existing metricsscore them lower due to the lexical mismatch.
To solve this issue, we propose K nowledge-verifying
Q uestion A nswering ( KQA ) which measures whether generated responses contain factually correctknowledge given the dialogue history.
Compared to the existing metrics using question generationmethods [14, 44], we automatically derive QA pairs for evaluation from the dialogue and the largescale KG [2].
In particular, at first, the ground truth is an entity mentioned in the gold response, and aquestion is composed of an entity in a dialogue and the existing relation between two entities.
Oncethe question is determined, more answer candidates can be found through the searching in KG, whichallows more acceptable responses that contain the correct knowledge as in Figure 3 (d).
Experiment
[Figure]
5.1 Experimental Setup
We conduct experiments on the OpendialKG dataset [29], which is a dialogue corpus associatedwith a large-scale Knowledge Graph (KG), namely Freebase [2] with 100k entities and 1M facts.
This dataset has 15K dialogues with 91K utterances.
We note that 49% of the responses come withthe gold knowledge, whereas others are not.
Since the dataset does not provide a predefined data split,we randomly split it into train (70%), validation (15%), and test sets (15%).
We use T5-small [34] forall experiments.
We also conduct experiments on another dataset and language model and present theresults in Section E of the Supplementary File.
For details, see Section D of the Supplementary File.
5.2 Baselines and Our Models
We compare different variants of our SURGE framework against various KG-augmented dialoguegeneration models.
No Knowledge.
This model is only provided with the dialog history, thus noexternal knowledge is used.
All Knowledge.
This model is provided with entire facts within a 1-hopsubgraph of entities associated with the dialog history.
Gold Knowledge.
This model is providedwith the exact gold knowledge, even in the test time if the gold knowledge exists.
Space Efficient Encoding.
This model takes all facts from the 1-hop subgraph of the entities as input.
We use twodifferent encoding methods introduced in [9], namely Space Efficient (series) and Space Efficient(parallel).
EARL.
This is an RNN-based model, where the entities are conditioned in responsegeneration [50].
Random/Sparse Retrieval.
These models are provided with selected facts from a1-hop subgraph, via the random sampling or the sparse retrieval – BM25 [36].
Text-based Retrieval.
This model is a variant of our framework where T5 encoder [34] is used for d in Eq.
4 instead of
GNNs similar to [16].
SURGE (unsupervised).
Ours with retrieved context-relevant facts from1-hop subgraph, where the retrieval is trained without any supervision.
SURGE (semi-supervised).
Ours but the retriever is trained with supervision if the data has a gold fact.
SURGE (contrastive).
Our full model jointly trains the retriever in a semi-supervised manner with the contrastive learningterm.
By default, all our models are trained with an invariant and efficient graph encoding.
5.3 Evaluation Metrics
We evaluate the generated responses using BLEU [31], ROUGE [27] and unigram overlap (F1)
withthe gold response.
Along with these conventional text evaluation metrics, we also evaluate the results [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Figure] [Figure]
[Table]
using our new metric, KQA (Section 4) which measures whether the generated responses containproper knowledge.
Lastly we compute the Knowledge F1 (KF1) similarly as in Shuster et al.
which is a measure of unigram overlap between the retrieved knowledge and generated response.
5.4 Experimental Results and Analysis
In Table 1, we report the knowledge-grounded response generation performances of baselines and our
SURGE.
As shown in Table 1, our models significantly outperform all the baseline models, excludingthe upper bound measure, Gold Knowledge , in all evaluation metrics.
The high BLEU, ROUGE, and
F1 refer that ours sufficiently learns the syntactic and semantic structure of the responses.
Our modelsalso achieve high F1 and EM scores in KQA.
The high KQA scores indicate that the generatedresponses are formed with the correct facts, which are relevant to the dialog context.
Even thebaseline models such as All Knowledge , Space Efficient Encoding [9], and EARL [50], which areprovided with all of 1-hop facts, underperform than ours.
The result demonstrates that selectingrelevant knowledge is critical in knowledge-augmented response generation.
Figure 4 examines the generation performance further by categorizing the dialogues into two groups:those with gold knowledge and those without.
When there is no gold knowledge, the model using
Gold Knowledge suffers a significant drop in all metrics and performs similarly to No Knowledge.
On the contrary, ours significantly improves the unigram F1 and KQA scores even with the retrievedknowledge without using the exact gold knowledge.
Knowledge Retrieval Figure 5 shows the performances of knowledge retrieval methods, where weonly measure the retrieval performance on dialogues that contain the gold knowledge.
We use Mean
Reciprocal Rank (MRR) and Hits@k as metrics.
Note that our SURGE is a differentiable retriever,which jointly learns to retrieve the context-relevant knowledge and then generate the correspondingresponses, whereas Random and BM25 [36] retrieve the knowledge without learning.
Therefore, ourmodels outperform other retrieval approaches by a large margin (See Section G with Figure 4 of the
Supplementary File for examples of baselines and ours).
In addition, when our retriever is trained ina semi-supervised manner, we observe the substantial performance gains from unsupervised learning,as the model can learn to retrieve the ground truth knowledge during training.
Knowledge-Consistent Generation
We conduct an ablation study on our models to validate theknowledge consistency performance of the response generation by computing the Knowledge F1(KF1) score [40]. To focus solely on the case where a given knowledge is consistently reflected in thegenerated responses, we use the gold knowledge rather than the retrieved one.
We randomly perturb [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Figure]
) contrastive learning.
the gold knowledge to ensure that responses are generated from the given knowledge rather than thetrained knowledge.
Table 2 shows that our model with a contrastive learning term outperforms allothers in the KF1, implying that the generated responses accurately reflect the encoded knowledge.
Sensitive Analysis on Graph Encoding We further conduct an analysis on graph encoding variantsintroduced in Section 3.4.
The knowledge length in Table 3 indicates the average token length usedfor graph encoding.
Our Invariant & Efficient ψ ∗ performs the best against other variants, whileusing the lesser space at the graph encoding phase.
Notably, simple Invariant achieves a comparableperformance against Invariant & Efficient , but yields a longer sequence.
Retrieval and Generation Examples Figure 6 shows the examples of generated responses alongwith the retrieved knowledge.
In particular, we compare our SURGE against Space Efficient (parallel)baseline.
In example (a), the baseline response contains an incorrect fact distracted by the contextuallyirrelevant entity ‘sailor’.
Contrarily, SURGE successfully retrieves relevant facts from the KG thengenerates the factually correct response.
This tendency is similar in example (b), where the baselineincorrectly generates the response with a wrong fact containing ‘King Kong’, meanwhile our SURGEretrieves context-relevant facts and generates a informative response.
Human Evaluation We sample 30 responses of SURGE, All Knowledge , and Space Efficient onthe OpendialKG test dataset [29], then conduct a human study of them.
We recruit 46 annotators, andask them to evaluate the quality of the generated responses by the 3 models given in a random order,with 3 criteria – consistency, informativeness, and fluency – using a 3 point Likert-like scale.
Asshown in Table 4, ours obtains significantly ( p-value < 0 .
higher scores than others in all criteria,which is another evidence that our framework generates consistent, informative, and fluent responses.346 Embedding Space Visualization We further visualize the multi-modal graph-text latent space in
Figure 7.
The visualization shows that, for the same dialogue with different subgraphs, our SURGEwith graph-text contrastive learning (right) generates distinct response embeddings pertraining todifferent subgraphs, unlike the one without graph-text contrastive learning which shows less varietyover responses for the same dialogue (left).
We include zoomed Figure 7 in the Supplementary File .
Conclusion
We proposed a novel end-to-end framework for knowledge graph-augmented dialogue generationwhich retrieves context-relevant subgraph, encodes a subgraph with the text, and generates knowledgeconsistent responses, called as SU bgraph R etrieval-augmented GE neration ( SURGE ).
Our resultsdemonstrate the effectiveness of our framework in both quantitative and qualitative experiments inknowledge retrieval and response generation tasks.
The analysis shows the contribution of eachproposed component: retrieval, encoding, and graph-text representation learning.
Our work suggestsa new direction to generate informative responses for knowledge graph-based dialogue task byempirically showing the importance of retrieving the more relevant subgraph knowledge rather thanusing all the relevant knowledge graphs when generating knowledge-grounded responses.