Variational Learning for Unsupervised Knowledge Grounded Dialogs
Abstract
Recent methods for knowledge grounded dialogs generate responses by incorporating information from an external textual document [Lewis et al. , 2020; Guu et al. , 2020].
These methods do not require the exact document to be known during training and rely on the use of a retrieval system to fetch relevant documents from a large index.
The documents used to generate the responses are modeled as latent variables whose prior probabilities need to be estimated.
Models such as RAG [ Lewis et al. , 2020 ] and REALM [ Guu et al. , 2020 ] , marginalize the document probabilities over the documents retrieved from the index to define the log likelihood loss function which is optimized end-to-end.
In this paper, we develop a variational approach to the above technique wherein, we instead maximize the Evidence Lower bound (ELBO).
Using a collection of three publicly available openconversation datasets, we demonstrate how the posterior distribution, that has information from the ground-truth response, allows for a better approximation of the objective function during training.
To overcome the challenges associated with sampling over a large knowledge collection, we develop an efficient approach to approximate the ELBO.
To the best of our knowledge we are the first to apply variational training for open-scale unsupervised knowledge grounded dialog systems.
Introduction
In this paper we focus our attention on the task of generating responses, grounded on information present in a large collection of external textual documents [Lewis et al. , 2020].
In real-world scenarios, the exact document that one must access for generating the response is often unknown and one only has access to conversation logs and a document collection.
Hence during training, given a dialog context, the primary challenge is first figuring out the correct document needed to generate the response, and then using that document for generating the actual response.
Figure 1 shows an example dialog from the OR-QuAC dataset [Qu et al. , 2020].
Here, the user is asking the system about what happened when a lawsuit filed by Billy Graham 1 failed.
Responding to this question in the dialog context, without relying on external knowledge isn’t possible.
The figure shows the ground truth response along with the correct document needed to generate the response.
A straightforward baseline approach would be to use an out-of-the-box retriever (for instance, a tf-idf based retriever such as BM25 [ Robertson et al. , 1994 ] or a neural retriever such as DPR [ Karpukhin et al. , 2020 ] ) for first retrieving the document and then using a retrieved document for generating the response.
While this is fairly easy to implement, it cannot be trained in an end-to-end manner and thus, the retriever never improves as the model learns to generate responses.
To overcome this limitation, methods such as RAG [ Lewis et al. , 2020 ] , model documents as latent variables and learn a distribution over these variables (Figure 2a).
This distribution is referred to as the document prior .
Specifically, the document-prior distribution is defined by querying a knowledge index [ Johnson et al. , 2017; Karpukhin et al. , 2020 ] using the dialog context (history), and then converting the retrieval scores of the top-k 2 documents into a probability distribution.
The response-likelihood can be defined using any neural language generator such as GPT2 [ Radford et al. , 2019 ] .
It then performs a marginalisation of the latent variable over the retrieved documents to compute the approximate probability of the response, given the context.
The negative log likelihood under this approximation forms the loss function to train.
However, one of the weaknesses of this approach is that, using the document-prior to query the index during training, ignores crucial information present in the ground-truth response which could have aided document retrieval.
As a result, the response-likelihood network parameters may receive a weaker signal during training, which, in-turn, can cause models to try reducing their dependence on external knowledge by ‘memorizing’, especially if the correct document is rarely fetched by the retriever.
Variational Retrieval-Augmented Generation (VRAG):
In this paper, we propose an approach that overcomes this limitation.
We incorporate the ground truth response with the dialog context for retrieving the documents during training in a secondary retriever.
This increases the chances of retrieving the correct document during training.
The distribution over the documents defined by this retriever is referred to as the document posterior .
The document posterior guides the training of the document prior while the documents sampled by the posterior are fed to the decoder for generating the response.
Such a formalism emerges naturally in the variational setting, wherein the evidence lower bound (ELBO) is optimized instead of the maximum likelihood objective.
Hence, we refer to the model as Variational Retrieval-Augmented
Generation (VRAG).
One of the advantages of variational training is that it provides a low variance estimate of the objective (as compared to sampling from the document-prior distribution), for the same number of samples.
Although this has been used in supervised settings ( [ Chen et al. , 2020 ] and [ Kim et al. , 2020a ] ), we note that directly training under the variational objective may be prohibitively expensive in case of a very large 3 document collection (sampling an element from the posterior distribution, would require retrieval scores for each document in the index collection).
In fact, related approaches for variational training therefore only use a small set of pre-retrieved documents [ Lian et al. , 2019; Li et al. , 2020 ] to overcome this bottleneck.
In particular, such approaches first use an out-ofthe box retriever to fetch a small set of documents (typically 5-10 documents) from the entire document collection.
The methods then learn a prior as well as posterior distribution over the small set of pre-retrieved documents only by optimizing the variational objective (Figure 2b).
A major weakness of this approach is that the out-of-thebox retriever does not benefit from training.
As a result, if the recall of the out-of-the box retriever is low, that is, the correct document is not present in the pre-retrieved subset for most of the training data, the mapping from the document to the response learnt will be highly noisy and not very useful (we also demonstrate this experimentally in this paper).
Contributions:
In this paper we describe our approach called VRAG or Variational Retrieval-Augmented Generation 4 which allows us to extend variational optimization to cases where documents are retrieved from large document collections.
Instead of pre-retrieving a small set of documents to facilitate variational training, we retrieve documents from the entire collection but perform a summation over the top-k retrieved documents from the posterior distribution as well as the prior distribution to approximate the variational objective (Figure 2c).
Top-k retrieval can be performed efficiently using an index for nearest neighbor search such as
Faiss [ Johnson et al. , 2017 ] , and we find that this simple trick performs significantly better than other approaches.
We present experiments on three, publicly available, conversational QA datasets and we show that variational training helps build better knowledge grounded dialog systems.
Our experiments show that not only does VRAG perform better on the end-task, it also learns a better retriever.
To the best of our knowledge, we are the first to apply variational training for open-scale unsupervised knowledge grounded dialog systems.
Background
As is commonly done in dialog modeling tasks, we represent the collection of dialogs as a set of context (dialog history) and response pairs;
T = { ( x ( i ) , y ( i ) ) } mi =1 where each context x ( i ) as well as its response y ( i ) is a sequence of tokens.
Further let D = { d j } Nj =1 be a set of documents in the form of a large indexed document collection.
We assume that each context-response pair requires exactly 1 document d j ∈ D (where 1 ≤ j ≤ N ) to generate the corresponding response.
Let z ( i ) denote a discrete variable which in- dicates the document (from the indexed collection) needed for training instance i i.e, d z i ∈ D .
We can now model the joint likelihood of a response and document pair ( y ( i ) , z ( i ) ) as p ( y ( i ) , z ( i ) | x ( i ) )
=
p ( z ( i ) | x ( i ) ) p ( y ( i )
| x ( i ) , z ( i ) ) .
In the absence of document-level supervision, the z ( i ) variables are unknown or ‘latent’.
Here we will be maximizing log p ( y ( i ) | x ( i ) )
= (cid:80) z p ( z ( i )
| x ( i ) )
p ( y ( i ) |
z ( i ) , x ( i ) ) .
However, since an explicit summation over the entire document collection can be computationally intractable, one needs to resort to a few approximation techniques.
For ease of notation, we will drop the superscript ( i ) from now on.
Retrieval based approaches:
Approaches such as
RAG [Lewis et al. , 2020] and REALM [Guu et al. , 2020], maintain an index which allows one to retrieve the top-k documents with high prior probability.
The objective is then approximated as a sum over these retrieved documents.
Specifically, the document-prior distribution p ( z | x ) is defined based on scores returned by the Dense Passage Retriever (DPR)
[Karpukhin et al. , 2020].
The top-k most relevant documents ( S pk ) for a query (dialog context) are retrieved from an index that allows efficient retrieval [ Johnson et al. , 2017 ] using MIPS search.
We denote the approximate document-prior distribution, normalized over the set S pk , as ˆ p ( z | x ) .
The overall objective for generating the response can then be written as log (cid:104)(cid:80) z ∈ S pk ˆ p ( z | x ) p ( y | z , x ) (cid:105) .
RAG suffers from a drawback that it does not use the information from responses in order to retrieve documents for a given training instance.
Variational techniques: An alternative approach is to maximize a variational lower bound on the objective.
Here we need to define an Evidence Lower Bound (ELBO) on the likelihood as log p ( y | x ) ≥ E z ∼ q ( z | x , y ) (cid:104) log (cid:16) p ( y , z
| x ) q ( z | x , y ) (cid:17)(cid:105) .
This lower bound holds for any distribution q .
Variational autoencoders [ Kingma and Welling, 2013 ] define another network to model the distribution q .
To train such networks, the ELBO is split as:
[Equation]
The first term is an expectation that can be estimated by sampling documents from the document-posterior q ( z | x , y ) distribution.
The response-likelihood network is then run only using these sampled documents.
One can either use re-parameterization trick (with Gumbel softmax distribution)
[Jang et al. , 2016; Maddison et al. , 2016] or policy gradient method to back propagate through the sampling step.
However, in order to sample a document, one would need to compute the entire distribution over the documents.
This can be prohibitively expensive when using a large document collection.
The second term (KL-divergence) is also computed as an explicit sum, given access to prior and posterior probability distributions, and is also intractable for large document collections.
Insummary, variational training which uses the posterior distribution to retrieve documents while training, can help retrieve more relevant documents for training the responsegenerator.
One of the trivial ways to extend these approaches (for large document corpus) is to identify the candidate knowledge documents for each training instance via an
[Figure] [Figure]
existing out-of-the-box retriever ( [ Li et al. , 2020 ] , [ Lian et al. , 2019 ] ).
One can then create prior and posterior distributions on the restricted set of documents.
However this does not allow us to train the retriever and we also show in our experiments, that the trained distribution in such a setting does a poor job of generalizing as a retriever (Section 4).
In our approach we generalize the variational technique to open domain setting without fixing or pre-retrieving those candidate documents.
In order to do so, we would first need to be able to compute the ELBO objective more efficiently (over the entire document corpus) as described in the next section.
Variational RAG (VRAG)
Variational training involves using both, the document-prior ( p ( z | x ) ) and document-posterior distributions ( q ( z | x , y ) ).
We model each of these based on scores from a Dense Passage retriever (DPR)
[ Karpukhin et al. , 2020 ] i.e. p ( z | x ) =
[Equation]
(cid:0)
[Equation]
(2) and q ( z | x , y )
= softmax (cid:0) f
( z ) T h ( x , y ) (cid:1) (3) where f and g are parameterized representations of documents ( z ) and dialog contexts ( x ).
h ( x , y ) denotes the joint embedding of the context-response pair.
These are created using neural models such as BERT [Devlin et al. , 2018].
We use a single network to compute the document embeddings f ( z ) for both prior and posterior.
In order to efficiently compute expectation and KL divergence terms in the ELBO objective (Equation 1), we need to approximate the above distributions.
To do so, we maintain an index on document embeddings.
This allows us to retrieve the set of top-k documents under prior and posterior distributions (equations 2 and 3) using MIPS search [Johnson et al. , 2017].
Note that since the query embeddings g and h are trainable, the retriever is trained over the epochs as well.
We denote the sets of top documents under prior and posterior distributions by S pk and S qk respectively.
The overall cost is then computed using Evidence Lower
Bound (Equation 1).
Here for the first term, we normalize ˆ q over the set of top-k documents (denoted by S qk ) returned by the index when queried using the posterior (i.e, the response and the dialog context).
The first term is then approximated as (cid:80) z ∈ S qk ˆ q ( z | x , y ) log p ( y | z , x ) .
To approximate the KLdivergence we use the top-k knowledge instances retrieved by querying the index using the dialog-context (for prior) and the context-response pair (for posterior).
We then take union of the two sets S pk and S qk to form the set S KL .
We use this set ( S KL ) to approximate the KL-divergence.
Thus, the KLdivergence in Equation 1 is given by:
[Equation]
where the approximate posterior ( ˆ q ) and prior ( ˆ p ) in this case are obtained by normalizing the retrieval scores on S KL .
The intuition behind this approximation is that the documents with low posterior probability do not contribute much to the
KL objective and hence, can safely be ignored.
Similar to other variational models, VRAG is trained end-to-end.
3. Architecture
We now describe the neural networks used to model the prior and posterior distributions, and the response generator.
Prior Distribution Encoders: We need to create document and context representations defined by functions f and g , respectively for modelling the prior.
We pass the input context through BERT model [Devlin et al. , 2018] to create context representation.
We use special markers to separate the turns.
The embedding at final layer of [ CLS ] token is passed through a linear layer to create context representation.
Similarly the document is passed through (separate) BERT Model to create a document representation at [ CLS ] token.
Posterior Distribution Encoders: Similar to the modeling of the prior distribution, we use another (separate) BERT to model the posterior.
Here we create the input representation of the context-response pair ( x , y ), where a special marker is used to separate the context and response.
Response Likelihood:
We use the standard sequence to sequence formulation for response likelihood, where log p ( y | z , x ) = (cid:80) j log p ( y j | y <
j , z , x ) .
Here, we use the
GPT2 [Radford et al. , 2019] model as our decoder with input sequence consisting of context and response.
3. Training details
We train our network to maximize the ELBO objective (Equation 1).
We initialize our document-prior (for both RAG and VRAG) and document-posterior (for VRAG) networks with the pretrained DPR-Multiset {+model 
+}
pre-trained using data from the Natural Questions [Kwiatkowski et al. , 2019], TriviaQA
[Joshi et al. , 2017], WebQuestions [Berant et al. , 2013] and CuratedTREC [Baudiˇs and ˇSediv`y, 2015] datasets.
During the training of the model, it can be difficult to rebuild the document index after every change to the document representation parameters in f , therefore similar to Lewis et al. , the parameters in f are kept constant.
We used early stopping with patience = 5 on recall of the validation sets to prevent overfitting of models.
The loss was optimized using AdamW Optimizer [Loshchilov and
Hutter, 2017].
We also found it useful to continue training the response-likelihood for both RAG and VRAG after the joint training is complete.
This is because while training, the decoder-likelihood function often lags behind prior (for
RAG) and posterior (for VRAG).
3. Response Generation
At test time we need to generate the response for a given context by first retrieving a document – thus, in both RAG and VRAG models we use the trained document-prior model to retrieve the top-k documents using the dialog context as query.
We experiment with two different decoding strategies to generate the response:
Top Document Decoding:
In this case the document with the highest prior probability is used to condition the generator.
The response is then generated using beam search (beam width=3) on the trained GPT-2 response generation model; the most likely beam is taken to be the prediction of the model.
We refer to this method as the ‘top-1 decoding’ in our experiments.
Top-k Documents Decoding:
Here top (k=5) documents are retrieved from the prior distribution, say z 1 , ..., z 5 .
A beam search is then run to generate the top response from each of these say r 1 , ..., r 5 .
We use the estimate of p ( r i | x ) ≈ p
( z | x ) p ( r i | z , x ) .
The most likely response under the estimated distribution is taken to be the response generated by the model 7 .
We refer to this as ‘top-5 decoding’ in our experiments.
Experiments
Our experiments aim to answer the following questions: (1)
Does Variational RAG (VRAG), which uses samples from the approximated document-posterior distribution, perform better than vanilla RAG?
(2) Does the quality of generated responses improve by decoding using the top-k documents?
(3) How do the trained document retriever modules for RAG and VRAG compare with each other?
(4) Are the quality of samples returned by the document-posterior of VRAG better than document-prior of RAG as hypothesized?
(5) How does VRAG compare with other approximations for variational training?
4. Datasets
OR-QuAC
[Qu et al. , 2020]: This dataset is a modified version of the QuAC [Choi et al. , 2018] dataset.
The dataset consists of dialog conversations, where each conversation is associated with the top-5 most relevant documents (retrieved using TF-IDF [Robertson et al. , 1994] based BM25 ranking) from the QuAC dataset.
To create an open-scale collection for our task, we index the set of all the documents available in the train, validation and test splits.
In some cases of the test and validation set, the ground-truth document may be missing in the top-5 list associated with each conversation.
In such cases, we obtain the ground-truth document from the original
QuAC dataset and add it to the indexed collection.
DSTC9 [ Kim et al. , 2020b ] : This dialog dataset was released as part of the DSTC9 challenge.
The dataset comprises of dialog conversation turns in which the system: (i) needs to identify the turns in which to consult a collection for textual FAQs, (ii) retrieve the FAQ if required, (iii) and then generate the response based on the retrieved FAQ.
The universe of knowledge documents in this dataset is the set of
FAQs and each FAQ also includes the entity name because the same question can occur multiple times for different entities (eg: “Is parking available?”).
The training dataset consists of conversations based on 4 different domains i.e hotels, restaurants, trains and taxis.
The test dataset contains an additional domain, attractions, which is not found in the train and validation splits.
DoQA [ Campos et al. , 2020 ] : This dataset comprises of open-ended dialog conversations on different domains like cooking, travel and movies.
Unlike, the OR-QuAC and
DSTC9 datasets, most questions in this dataset are not factoid/specific questions, and are open-ended.
We only use the cooking split for both training and testing.
We preprocess all the datasets by removing all the examples if the ground truth response is “CANNOTANSWER” (unanswerable).
Each ‘instance’ refers to a context-response pair.
Question-Answering Task :
The OR-QuAC dataset also contains non-contextual variations of questions at each dialog turn
and we use them in a QA setting (referred to as ORQuAC-QA).
4. Baselines
Apart from our approach (VRAG), we also study the performance of RAG, as well as, a pipeline model which uses the pre-trained DPR-Multiset Retriever and a GPT2 based decoder which is fine-tuned to generate responses.
We refer to this as the DPR + GPT2 baseline in our experiments.
In addition, we also show the importance of using a trainable retriever for variational setting by comparing gainst a variational model where a fixed set of candidate documents are retrieved using a pre-trained DPR as a retriever (conditioned
[Table]
on context and response) during training [Lian et al. , 2019;
Li et al. , 2020].
This model retrieves from the entire document collection using the prior distribution during inference.
We refer to this baseline as Variational Model with preretrieval (“pre-VM”).
Thus, the distributions being trained do not change the set of candidate documents used as training progresses.
All models use their respective document-prior distributions during testing.
4. Evaluation metrics
For each of our experimental runs, we report the Mean Reciprocal Rank@5 (MRR@5), Recall@1 (R@1), Recall@ (R@5) to evaluate prior’s performance.
We also report BLEU scores (both with top-1 and top-5 decoding) to assess the performance of generator.
The BLEU-1 and BLEU-4 scores in our tables are denoted by B-1 and B-4 respectively.
We also consider “BLEU-penalized” scores (indicated by BP-1 anad
BP-4) which consider the BLEU score at a given test instance as 0 if the document retrieved for the given instance is incorrect.
These help ensure that a model is not able to produce a high score by memorizing on a particular given domain.
4. Results
As can be seen in Table 1, VRAG outperforms RAG on each dataset for document retrieval.
We also note that both RAG and VRAG significantly improve the performance of the initial DPR based retriever.
Table 2 shows the performance of the models on the response generation task.
Note that all results in Table 2 are obtained after further fine tuning the generator networks (after joint training is complete).
The results show that the VRAG model outperforms the RAG model on language generation tasks (BLEU metrics) on all datasets except DoQA.
We believe this difference is due to the nature of documents used in this dataset - other datasets have a lot more fact based questions to be answered using knowledge in documents, while more than 66% of the questions in the
DoQA dataset are non-fact based and open-ended.
In addition, it is also possible that the RAG model is memorizing and overfitting on this dataset.
For instance, see gains in penalized BLEU scores (BP-1 and BP-4) of RAG and
[Table] [Table]
[Table]
VRAG over DPR in OR-QuAC and DSTC9 datasets in Table 2 – the relative gain of VRAG (over RAG) is significantly higher.
This suggests that VRAG model is more likely to generate the response using the correct document and not by merely memorize on a given domain.
Alternative approximations for Variational Training:
In
Table 1 we see that the recall scores of pre-VM model are much worse than our VRAG model.
This observation validates our hypothesis that, because the retrieved samples are not improved during training, the prior and posterior distributions in pre-VM end up focusing only on the few (potentially incorrect) initially-retrieved documents.
This is especially problematic if the initial recall is low.
As training progresses, this may worsen the distributions over the initial model (eg: see DSTC9 recall scores for DPR and pre-VM in Table 1) because the correct document isn’t present in the retrieved sample, thus giving it an incorrect signal.
Effect of Top-5 Decoding: From Table 2 we find that in almost all cases, using top-5 decoding to generate responses performs better than using the single (best scored) document to generate responses.
This indicates that models are to able to incorporate information from the correct document even if it is not returned as the top-ranked document.
Benefit of Document Posterior:
In Section 3, we motivated the VRAG model by suggesting that using the posterior to sample documents while training the decoder could help train a better model.
We find that recall of the document posterior in VRAG is nearly 14-70% higher than the document prior of RAG (depending on the dataset).
Further, we find that the use of the responses by VRAG (posterior), to query the index during training, results in significantly better retrieval accuracy than RAG (prior) at every epoch during training.
While this is perhaps intuitive and expected, our results on recall in
Table 2 demonstrate the VRAG (prior) which is trained indirectly via the KL-divergence with VRAG (posterior) also outperforms the RAG (prior).
We attribute this gain to the fact that the generator in VRAG learns to focus well on the generated documents which further reinforces the posterior network (and indirectly the prior network through KL term) to improve.
Study of Memorization: One of the issues in such an unsupervised learning is that generator may fail to use the retrieved knowledge.
Instead the parameters might have been trained to internally model the external knowledge sources themselves, refered to as ‘memorization’.
In order to study memorization in the models, we compared the results on response generation of all models in the absence of the correct document – if the correct document is missing, the models should perform very poorly.
A good performance even without obtaining correct document, would indicate lesser reliance on external knowledge and hence a higher tendency for ‘memorization’.
We, thus rebuild the document index without including any of the documents from the test set and then reevaluate the performance of our models.
Table 3, shows the drop in BLEU scores for each of the models after removing the correct document on OR-QuAC dataset.
As can be seen, the percentage drop is highest in the VRAG model indicating a higher usage of knowledge instance and thus, possibly lesser memorization.
Conclusion
In this paper we described an approach to run variational training on knowledge grounded dialog with a large corpus.
Our experiments on three conversational QA datasets indicate that variational training is helpful as it produces better document samples while training.
We find that our model, VRAG (having access to superior samples from posterior while training), not only generates better responses, it also learns a better retriever (prior distribution).
We believe that such sampling approximations could also be helpful in other tasks; for instance, it could also be interesting to apply them to other approaches such as Reinforcement
Learning to the setting of a large corpus.
This would require overcoming similar challenges in sampling as we did in this paper for variational training.
A Hyperparameters and Training Details A. Architecture
We now describe the neural networks used to model the prior and posterior distributions, and the response generator.
Prior Distribution
We need to create document and context representations defined by functions f and g , respectively for modelling the prior.
To define the function g we use the BERT Model [ Devlin et al. , 2018 ] .
We consider the context x as a sequence of turns x 1 , . . .
x i , . . .
x n , where each turn x i , is an utterance from either one of the two speakers ‘S1’ or ‘S2’ in the dialog.
We then create an input representation of the context =
[ CLS ] , 〈 S 1 〉 , x 1 , 〈 S 2 〉 , x 2 , 〈 S 1 〉 , ..., x n , [ SEP ] , where [ CLS ] , [ SEP ] , 〈 S 1 〉 , 〈 S 2 〉 are marker tokens.
This input representation is then fed to BERT; the embedding at the final layer of [ CLS ] token is then passed through a linear layer to obtain the embedding representation g ( x ) of the context x .
Similarly enc( z ) =
[ CLS ] , z , [ SEP ] forms the input encoding for document z .
This is also passed through a (separate) BERT model, followed by the application of a linear layer to the embedding at [ CLS ] token.
This gives the embedding representation f ( z ) for document z .
Posterior Distribution
Similar to the modeling of the prior distribution, we use another (separate) BERT to model the posterior.
Here we create the input representation of the context-response pair ( x , y ) as: [ CLS ] , 〈 S 1 〉 , x 1 , 〈 S 2 〉 , x 2 , 〈 S 1 〉 , ..., x n , 〈 RSEP 〉 , y , [ SEP ] .
Here, 〈 RSEP 〉 is a special marker token used to separate the response from the dialog-context.
This input representation is passed through BERT model and we obtain the representation g ( x , y ) by applying a linear layer on the embedding of the [ CLS ] token at last layer.
Response Likelihood
We use the standard sequence to sequence formulation for response likelihood, where
[Equation]
[Equation]
Here, we use the GPT2 [Radford et al. , 2019] model as our decoder.
We encode the input sequence as 〈 bos 〉 , enc( x ), 〈
KSEP 〉 , enc( z ), 〈 eos 〉 , where 〈 bos 〉 and 〈 eos 〉 are the begin- and end-of-sequence markers, and 〈 KSEP 〉 is a special marker token to separate the dialog context from the document (knowledge).
For each response token index j , log p ( y j | y <
j , z , x ) is then defined by passing the output at the j th index of the decoder through a linear layer.
A. Network Hyperparameters
We train both RAG and VRAG with AdamW optimizer.
We train for 10 epochs with early stopping (patience = 5).
The learning rates and time per epoch are given in Table 7.
We use BERT base (110 million parameters) for the context encoders in both prior and posterior networks and GPT2 small (117 million parameters) for the decoder network.
The document encoder is also a BERT base (110 million parameters) which is fixed at training time.
All of our experiments are conducted on a single machine with 100GB of system memory and Nvidia V100 GPU.
B Further discussion of experiments and results B. Dataset Statistics
The statistics of datasets we used are shown in Table 4.
B. Effect of Decoder fine-tuning
Training VRAG involves optimizing two objectives - reducing the KL-divergence between the document-prior and document-posterior, and, maximizing the log likelihood of the responses.
VAE models often end up prioritizing the
KL-divergence over the likelihood objective and sometimes end up with zero KL-divergence by forcing the documentposterior to match the prior (called posterior-collapse)
[Lucas et al. , 2019; Bowman et al. , 2015; Chen et al. , 2016;
Oord et al. , 2017].
However, we hypothesize even in cases where there is no posterior collapse, the joint training could result in the response-generator (likelihood term) being inadequately trained.
We observed that fine-tuning the response decoder after freezing the weights of priors, results in an improvement in both RAG and VRAG.
The results shown in the main paper were obtained after fine tuning the generator after the joint training is complete.
We show the results both with and without fine tuning in Table 2 here.
It is interesting to note on the DoQA dataset, RAG initially reported a B-4 of only 0 .(top-5 decoding) which goes up to 17 .
04 after fine-tuning .
5 k u s e d fo r t r a in in g 667777r @
R A G V R A G
B. Effect of top-k OR-QuAC
[Table]
We also consider the case when we reduce the number of samples ( k ) while training.
Figure 3 shows the variation in R @ versus k on the DoQA dataset.
Here we observe that recall decreases with decrease in k .
This is because the computation of objective becomes less accurate for decrease in k .
We also note that when only one document is sampled, RAG outperforms VRAG.
We hypothesize, that in this case the VRAG prior trains poorly due to an extremely weak approximation of the KL divergence 8 .
B. Recall scores versus training epoch
As shown in Figure 4, the recall scores improve over epoch.
Moreover the posterior recall scores are always higher then prior recall scores.
This shows both the benefit of using a posterior during training, as well as the benefit of training the retriever as opposed to keeping it fixed.
B. RAG prior vs VRAG posterior
Table 5 compares the recall of RAG prior with VRAG posterior.
The VRAG (posterior scores) being higher than RAG (prior) help in generating better document samples for the response generator during training.
As discussed in the main section this explains the improved scores of VRAG over
RAG.
C Dialog examples
Here, we show the retrieved documents for a given dialog context and the generated responses using top-1 decoding (using best document).
All documents in the tables are in decreasing order of probabilities.
C. OR-QuAC
Refer to table 8 for examples.
Dialog context
User : What disputes did Billy Graham have?
Agent : Graham personally sued Zahorian and the WWF
User :
Why
Agent :
He claimed that they had forced him to take steroids to maintain his position in the company.
User :
Did he win the lawsuit?
Agent
His lawsuit was unsuccessful.
User : What happened after the suit failed?
Retrieved documents
RAG
VRAG
Late in 2003, Giambi was
In the early 1990s named by FBI officers
US federal agents were investigating the Bay Area ... investigating Dr. Geo ...
Landis testified at the
Late in 2003, Giambi hearing that Geoghegan was named by FBI came to know of LeMond’s ...
officers investigating ...
In the early 1990s US
Since 2003, Bonds has federal agents were been a key figure in the investigating Dr. George ...
Bay Area Laboratory ...
ResponseGroundtruth response : Graham went on a public awareness campaign regarding the dangers of steroids.
Response (RAG) :
The prosecution in the Barry Bonds perjury case indicated they intended to call both Jason and
Jeremy Giambi to testify against Bonds in his Marchtrial.
Response (VRAG) :
Graham went on a public awareness campaign regarding the dangers of steroids during this time, including an appearance with McMahon on The Phil
Donahue Show in 1992.
C. DSTC
Since a lot of entities contain similar documents in the
DSTC9 [Kim et al. , 2020b] dataset, we create a document collection in which the documents are represented as
ENTITY NAME : question : answer (6) to distinguish between the documents of different entities.
Refer to table 9 for examples.
DoQA
[Table] [Table]
RAG 32.
pre
[Table] [Table]
[Kim et al. , 2020b]
RAG
[Table]
pre
[Table]
VRAG
[Table]
DPR
[Table]
RAG 2.
pre
[Table] [Table]
DoQA
VRAG 17.
[ Campos et al. , 2020 ]
RAG
[Table]
tuned) 23.
pre
[Table]
VRAG
[Table]
[Table]
OR-QuAC-QA
[Table]
[ Qu et al. , 2020 ]
[Table]
5 E p o c h s 0 .
r @
R A G p rio r V R A G p rio r V R
A G p o s te rio r
[Table]
5 E p o c h s 0 .50 .50 .
.70 .8r @
R A G p rio r V R A G p rio r V R
A G p o s te rio r
[Table]
5 E p o c h s 0 .
m rr @
R A G p rio r V R A G p rio r V R
A G p o s te rio r
(c
[Figure]
C. DoQA
Both models, RAG and VRAG generate the correct response as shown in the example, however, in this example, RAG does so by using the incorrect document which shows response memorization.
Refer to table 10 for examples.
[Table] [Table]
User :
Hi.
I’m trying to find a cheap place to eat that serves Izakaya food in Laurel Heights.
Agent : There isn’t a cheap Izakaya place in Laurel Heights.
Would you like for me to check a different
price
range?
User : Moderate price range should be fine.
Agent : I found the Kaiju Eats Ramen and Izakaya in Laurel Heights.
It is moderately priced.
User : Do they take advanced reservations?
Retrieved documents
RAG
VRAG
KAIJU EATS RAMEN KAIJU EATS RAMEN
AND IZAKYA :
Can I order
AND IZAKYA :
Can I food for delivery at Kaiju make an advance
Eats Ramen and Izakaya?
: reservation? :
Yes,
Delivery service is not reservations taken.
available at Kaiju Eats
Ramen and Izakaya.
MENSHO TOKYO : Are
KAIJU EATS RAMEN reservations necessary there?
AND IZAKYA :
Can I : No, Mensho Tokyo doesn’t order food for delivery at accept reservations.
Kaiju Eats Ramen and
Izakaya?
Delivery service is not available at Kaiju Eats
Ramen and Izakaya.
TABOUN :
Do you accept
USHIO RAMEN : reservations?
: No, they do
Do you accept not take reservations.
reservations?
: Yes, the restaurant does take reservations.
ResponseGroundtruth response : Yes, they do take reservations.
Would you like me to make one for you?
Response (RAG) :
No, they do not take advanced reservations.
Anything else I can do for you?
Response (VRAG) :
Yes, you may make a reservation.
Would you like to make one?
[Table]
User : Should chicken be cooked beforehand while making pizza?
Agent : Cook the chicken ahead of time.
User : Why should it be cooked ahead of time?
Retrieved
[Table]
RAG
VRAG
I’d suggest skinless
Cook the chicken ahead bone-in chicken thighs, of time.
I doubt your as they have plenty of fat pizza cooking time and collagen to keep them and your chicken moist and tasty.
... cooking time will ...
Cook the chicken ahead
I’d say it depends.
If of time.
I doubt your you are dead-set on pizza cooking time observing all food and your chicken safety rules, the pizzas cooking time will ... are probably unsafe to eat. ...
Don’t cook the chicken
If your chicken burns, the pieces for so long.
Add issue is likely not that you them ten or twenty minutes cooked it too long, but before serving.
For that that you cooked it too ...
matter, I wouldn’t ...
ResponseGroundtruth response : I doubt your pizza cooking time and your chicken cooking time will be a perfect match, and it is more likely than anything that your chicken would be undercooked.
That would not be good for anyone.
response (RAG) : I doubt your pizza cooking time and your chicken cooking time will be a perfect match, and it is more likely than anything that your chicken would be undercooked. response (VRAG) : I doubt your pizza cooking time and your chicken cooking time will be a perfect match, and it is more likely than anything that your chicken would be undercooked.