Variational Learning for Unsupervised Knowledge Grounded Dialogs
Abstract
Recent methods for knowledge grounded dialogs generate re- sponses by incorporating information from an external textual document (Lewis et al. 2020; Guu et al. 2020).
These meth- ods do not require the exact document to be known during training and rely on the use of a retrieval system to fetch rel- evant documents from a large index.
The documents used to generate the responses are modeled as latent variables whose prior probabilities need to be estimated.
Models such as RAG (Lewis et al. 2020), marginalize the document probabilities over the documents retrieved from the index to define the log likelihood loss function which is optimized end-to-end.
In this paper, we develop a variational approach to the above technique wherein, we instead maximize the Evidence Lower bound (ELBO).
Using a collection of three publicly available open-conversation datasets, we demonstrate how the posterior distribution, that has information from the ground-truth response, allows for a better approximation of the objective function during training.
To overcome the challenges asso- ciated with sampling over a large knowledge collection, we develop an efficient approach to approximate the ELBO.
To the best of our knowledge we are the first to apply variational training for open-scale unsupervised knowledge grounded di- alog systems.
Introduction
Dialog systems are typically modeled as sequence-tosequence tasks where the dialog context (history) is encoded and then used to generate responses.
These models are trained end-to-end by maximizing the likelihood of responses on a given training set of conversational logs (Sordoni et al. 2015).
However real-world conversational systems also need to be able to incorporate external (structured or unstructured) knowledge while generating responses (Gangi Reddy et al. 2019;
Budzianowski et al.
Qu et al. 2020;
Kim et al. 2020).
Existing work for knowledge grounded tasks typically relies on the availability of conversation logs and the associated knowledge source instance (Qu et al.
Gangi Reddy et al. 2019), though some recent methods relax the dependence on annotated logs (Raghu, Gupta et al. 2020; Lewis et al. 2020; Guu et al.
Unstructured Knowledge Grounded Dialog: In this paper we focus our attention on the task of generating responses grounded on information present in external textual documents.
Similar to recent models such as RAG (Lewis et al.
and REALM (Guu et al. 2020)
we do not require the exact document to be known during training and we rely on the use of a retrieval system to fetch documents from a large (indexed) collection.
In methods such as RAG (Lewis et al. 2020), the documents used to generate responses are modeled as latent variables and, the document-prior and response-likelihood probabilities are defined using outputs of their respective subnetworks.
Specifically, the document-prior distribution is defined by querying a knowledge index (Johnson, Douze, and
J ´ egou 2017; Karpukhin et al. 2020) using the dialog context (history), and then converting the retrieval scores of the topk 1 documents into a probability distribution.
The responselikelihood can be defined using any neural language generator such as GPT2 (Radford et al. 2019).
It then performs a marginalisation of the latent variable over the retrieved documents to compute the approximate probability of the response, given the context.
The negative log likelihood under this approximation forms the loss function to train.
One of the weaknesses of such approaches is that, using the document-prior to query the index during training, ignores crucial information present in the ground-truth response.
As a result, the response-likelihood network parameters may receive a weaker signal while training, which, inturn, can cause models to try reducing their dependence on external knowledge by ‘memorizing’.
Variational Training:
Querying the index with additional information from the ground-truth response while training can help retrieve more relevant documents, and thus, provide a stronger signal to the response-likelihood parameters.
Therefore, in this paper, instead of using the documentprior distribution to retrieve documents while training, we retrieve the document under a document-posterior distribution and optimize the Evidence Lower Bound or ELBO, which forms the basis of variational auto encoders (Kingma and Welling 2013).
The loss function in this case consists of an expectation of negative log-likelihood (responselikelihood) under the document-posterior distribution and a Kullback–Leibler (KL) divergence between documentposterior and document-prior distributions.
One of the advantages of variational training is that it provides a low variance estimate of the objective (as compared to sampling from the document-prior distribution), for the same number of samples.
However, we note that directly training under the variational objective may be prohibitively expensive in case of a very large 2 document collection (sampling an element from the posterior distribution, would require retrieval scores for each document in the index collection).
Inthis work we overcome this challenge by performing a sum over the top-k retrieved documents (instead of all documents) to approximate the variational objective.
Topk retrieval can be performed efficiently using an index for nearest neighbor search such as Faiss (Johnson, Douze, and
J´egou 2017).
Contributions:
In this paper we describe our approach called VRAG or Variational Retrieval-Augmented Generation 3 which allows us to extend variational optimization to cases where documents are retrieved from large document collections.
We present experiments on three, publicly available, conversational QA datasets and we show that variational training helps build better knowledge grounded dialog systems.
Our experiments show that, not only does
VRAG perform better on the end-task, it also learns better document-priors.
To the best of our knowledge, we are the first to apply variational training for open-scale unsupervised knowledge grounded dialog systems.
Background
Open conversational QA (OQA) tasks require systems to retrieve a document from a large collection and then ground the generated response based on information from the document.
Training such models end-to-end without the groundtruth document is challenging – the only source of supervision is in the generated responses and they present a weak signal for improving the retriever.
As is commonly done in dialog modeling tasks, we represent the collection of dialogs as a set of context (dialog history) and response pairs;
T = { ( x ( i ) , y ( i ) ) } mi =1 where each context x ( i ) as well as its response y ( i ) is a sequence of tokens.
Further let D = { d j } Nj =1 be a set of documents in the form of a large indexed document collection.
We assume that each context-response pair requires exactly 1 document d j ∈ D (where 1 ≤ j ≤ N ) to generate the corresponding response.
Let z ( i ) denote a discrete variable which indicates the document (from the indexed collection) needed for training instance i i.e, d z i ∈ D .
We can now model the joint likelihood of a context response pair ( x ( i ) , y ( i ) ) as p ( y ( i ) , z ( i ) | x ( i ) )
=
p ( z ( i ) | x ( i ) ) p ( y ( i )
| x ( i ) , z ( i ) ) .
With document-level supervision: If we are given the ground-truth labels of z ( i ) for each training instance, we can maximize log (cid:81) i p ( y ( i ) ,
z ( i ) | x ( i ) ) =
(cid:80) i log p ( z ( i ) | x ( i ) )
+ log p ( y ( i ) |
z ( i ) , x ( i ) ) .
Since the sum splits, we can model the two distributions (document-prior and response-likelihood) separately, by training neural networks for each of them.
However for unsupervised settings the z ( i ) variables are unknown or ‘latent’.
Without document-level supervision: In the absence of document-level supervision we will be maximizing log p ( y ( i ) | x ( i ) ) .
To compute this likelihood we marginalize over the latent variable z ( i ) i.e., p ( y ( i ) | x ( i ) )
= (cid:80) z p ( z ( i ) | x ( i ) ) p ( y ( i ) |
z ( i ) , x ( i ) ) .
However, since an explicit summation over the entire document collection can be computationally intractable, one needs to resort to a few approximation techniques.
For ease of notation, we will drop the superscript ( i ) from now on.
Retrieval based approaches: Approaches such as
RAG (Lewis et al. 2020) and REALM (Guu et al. 2020), maintain an index which allows one to retrieve the top-k documents with high prior probability.
The objective is then approximated as a sum over these retrieved documents.
Specifically, the document-prior distribution p ( z | x ) is defined based on scores returned by the Dense Passage
Retriever (DPR)
[Equation]
Karpukhin et al.
[Equation]
where f and g are parameterized representations of documents ( z ) and dialog contexts/query ( x ), created using neural models such as BERT (Devlin et al. 2018).
The top-k most relevant documents ( S pk ) for a query (dialog context) are retrieved from an index that allows efficient retrieval (Johnson, Douze, and J ´ egou 2017) using MIPS search.
We denote the approximate documentprior distribution, normalized over the set S pk , as:
[Equation]
The overall objective for generating the response can then be written as:
[Equation]
Variational techniques: Another approach is to maximize a variational lower bound on the objective.
Here we need to define an Evidence Lower Bound (ELBO) on the likelihood as log p ( y | x ) ≥
E z ∼ q ( z | x , y ) (cid:104) log (cid:16) p ( y , z
| x ) q ( z | x , y ) (cid:17)(cid:105) .
This lower bound holds for any distribution q .
Variational autoencoders (Kingma and Welling 2013) define another network to model the distribution q .
To train such networks, the ELBO is split as E z ∼ q [log p
( y | z , x )] − KL [ q ∥ p ( z | x )] .
The first term is an expectation that can be estimated by sampling documents from the document-posterior q ( z | x , y ) distribution.
The response-likelihood network is then run only using these sampled documents.
One can either use re-parameterization trick (with Gumbel softmax distribution)
(Jang, Gu, and Poole 2016; Maddison,
Mnih, and Teh 2016) or policy gradient method to back propagate through the sampling step.
However, in order to sample a document, one would need to compute the entire distribution over the documents.
This can be prohibitively expensive when using a large document collection.
The second term (KL-divergence) is also computed as an explicit sum, given access to prior and posterior probability distributions, and is also intractable for large document collections.
In summary, variational training which uses the posterior distribution to retrieve documents while training, can help retrieve more relevant documents for training the responsegenerator.
However, we would first need to be able to compute the ELBO objective more efficiently.
In the next section we describe our approach to overcome this challenge.
Variational RAG (VRAG)
Variational training involves using both, the document-prior and document-posterior distributions.
In our VRAG model, we continue to use the document-prior distribution as defined by RAG (Equation 2) and similary define the approximate document-posterior distribution ( ˆ q ):
[Equation]
Here, h ( x , y ) denotes the joint embedding of the contextresponse pair while f ( z ) indicates the document embeddings.
In practise, h and f are often implemented using neural networks.
We use a single network to compute the document embeddings f ( z ) for both prior and posterior.
This allows us to use a common document index as well.
The overall cost is then computed using Evidence Lower Bound as
[Equation]
Here for the first term, we normalize ˆ q over the set of topk documents (denoted by S qk ) returned by the index when queried using the posterior (i.e, the response and the dialog context).
The first term is then approximated as:-
[Equation]
To approximate the KL-divergence we use the top-k knowledge instances retrieved by querying the index using the dialog-context (for prior) and the context-response pair (for posterior).
We then take union of the two sets S pk and S qk to form the set S KL .
We use this set ( S KL ) to approximate the KL-divergence.
Thus, the KL-divergence in Equationis given by:
[Equation]
where the approximate posterior ( ˆ q ) and prior ( ˆ p ) in this case are obtained by normalizing the retrieval scores on S KL .
Similar to other variational models, VRAG is trained endto-end.
3. Architecture
We now describe the neural networks used to model the prior and posterior distributions, and the response generator.
Prior Distribution We need to create document and context representations defined by functions f and g , respectively for modelling the prior.
To define the function g we use the BERT Model (Devlin et al. 2018).
We consider the context x as a sequence of turns x 1 , . . .
x i , . . .
x n , where each turn x i , is an utterance from either one of the two speakers ‘S1’ or ‘S2’ in the dialog.
We then create an input representation of the context =
[ CLS ] , 〈 S 1 〉 , x 1 , 〈 S 2 〉 , x 2 , 〈 S 1 〉 , ..., x n , [ SEP ] , where [ CLS ] , [ SEP ] , 〈 S 1 〉 , 〈 S 2 〉 are marker tokens.
This input representation is then fed to BERT; the embedding at the final layer of [ CLS ] token is then passed through a linear layer to obtain the embedding representation g ( x ) of the context x .
Similarly enc( z ) =
[ CLS ] , z , [ SEP ] forms the input encoding for document z .
This is also passed through a (separate) BERT model, followed by the application of a linear layer to the embedding at [ CLS ] token.
This gives the embedding representation f ( z ) for document z .
Posterior Distribution Similar to the modeling of the prior distribution, we use another (separate) BERT to model the posterior.
Here we create the input representation of the context-response pair ( x , y ) as: [ CLS ] , 〈 S 1 〉 , x 1 , 〈 S 2 〉 , x 2 , 〈
S 1 〉 , ..., x n , 〈 RSEP 〉 , y , [ SEP ] .
Here, 〈 RSEP 〉 is a special marker token used to separate the response from the dialog-context.
This input representation is passed through
BERT model
and we obtain the representation g ( x , y ) by applying a linear layer on the embedding of the [ CLS ] token at last layer.
Response Likelihood We use the standard sequence to sequence formulation for response likelihood, where
[Equation]
Here, we use the GPT2 (Radford et al.
2019) model as our decoder.
We encode the input sequence as 〈 bos 〉 , enc( x ), 〈
KSEP 〉 , enc( z ), 〈 eos 〉 , where 〈 bos 〉 and 〈 eos 〉 are the begin- and end-of-sequence markers, and 〈 KSEP 〉 is a special marker token to separate the dialog context from the document (knowledge).
For each response token index j , log p ( y j | y <
j , z , x ) is then defined by passing the output at the j th index of the decoder through a linear layer.
3. Training details
We train our network to maximize the ELBO objective (Equation 4).
We initialize our document-prior (for both
RAG and VRAG) and document-posterior (for VRAG) networks with the pretrained DPR-Multiset model.
This model is pre-trained using data from the Natural Questions (Kwiatkowski et al. 2019), TriviaQA (Joshi et al. 2017),
WebQuestions (Berant et al. 2013), and CuratedTREC (Baudiˇs and ˇSediv`y 2015) datasets.
During the training of the model, it can be difficult to rebuild the document index after every change to the document representation parameters in f , therefore the parameters in f are kept constant (Lewis et al. 2020).
We used early stopping with patience = 5 on recall of the validation sets to prevent overfitting of models.
The loss was optimized using AdamW Optimizer (Loshchilov and Hutter 2017).
More details about the hyper-parameter settings are available in Appendix B .
We also found it useful to continue training the response-likelihood for both RAG and VRAG after the joint training is complete.
This is because while training, the decoder-likelihood function often lags behind prior (for RAG) and posterior (for VRAG) – we discuss this in more detail in Section 5.4.
3. Response Generation
At test time we need to generate the response for a given context by first retrieving a document – thus, in both RAG and VRAG models we use the trained document-prior model to retrieve the the top-k documents using the dialog context as query.
We experiment with two different decoding strategies to generate the response:
Top Document Decoding:
In this case the document with highest prior probability is used to condition the generator.
The response is then generated using beam search (beam width=3) on the trained GPT-2 response generation model; the most likely beam is taken to be the prediction of the model We refer
to this method as the ‘top-1 decoding’ in our experiments
Top-k Documents Decoding:
Here top (k = 5) doc uments are retrieved from the prior distribution, say z 1 , ..., z 5 .
A beam search is then run to generate the top response from each of these say r 1 , ..., r 5 .
We use the estimate of p ( r i | x ) ≈ p
( z | x ) p ( r i | z , x ) .
The most likely response under the estimated distribution is taken to be the response generated by the model 5 .
We refer to this as ‘top-5 decoding’ in our experiments.
Related Work
In this paper we have focused on the task of open domain dialog question answering with large document corpus.
Previously, (Chen et al. 2020) and (Kim, Ahn, and Kim 2020) had used VAE loss to improve the performance of knowledge grounded dialogs in supervised settings.
Here they had access to the ground truth knowledge to query the index while training which they then used to compute the ‘knowledge’ loss.
There has also been work done to use variational objective for unsupervised training of knowledge grounded dialogs (Lian et al. 2019), but such methods rely on a small list of 5 − 10 candidate knowledge documents that was provided for each training instance.
As a result, such models were not required to query the entire knowledge corpus.
In contrast, in our work the challenge was to apply variational training to the setting where we don’t have any specific document annotations for training instances and we had to also query a document index.
The REALM model (Guu et al. 2020) has previously used latent variables to train a knowledge retriever with language model in a non-variational manner.
Retrieval-Augmented
Generator (RAG) (Lewis et al. 2020) had also used a similar objective but had used a pre-trained retriever and extended it to generative tasks.
To the best of our knowledge, we are the first to use the variational objective for open domain document-grounded response generation, using a large document collection.
Experiments
Our experiments aim to answer the following questions: (1)
Does Variational RAG (VRAG), which uses samples from the approximated document-posterior distribution, perform better than vanilla RAG?
(2) Does the quality of generated responses improve by decoding using the top-k documents?
(3) How do the trained document retriever modules for RAG and VRAG compare with each other?
(4) Are the quality of samples returned by the document-posterior of VRAG better than document-prior of RAG as hypothesized?
(5) Is further fine-tuning of the response-likelihood possible using the trained prior-distributions of the joint models?
5. Datasets
We present our experiments on the following datasets:
OR-QuAC (Qu et al.
This dataset is a modified version of the QuAC (Choi et al. 2018) dataset.
The dataset consists of dialog conversations, where each conversation is associated with the top-5 most relevant documents (retrieved using TF-IDF based BM25 ranking (Robertson and
Zaragoza 2009)) from the QuAC dataset.
To create an openscale collection for our task, we index the set of all the documents available in the train, validation and test splits.
In some cases of the test and validation set, the ground-truth document may be missing in the top-5 list associated with each conversation.
In such cases, we obtain the ground-truth document from the original QuAC dataset and add it to the indexed collection.
DSTC9 (Kim et al. 2020):
This dialog dataset was released as part of the DSTC9 challenge.
The dataset comprises of dialog conversation turns in which the system: (i) needs to identify the turns in which to consult a collection for textual
FAQs, (ii) retrieve the FAQ if required, (iii) and then and generate the response based on the retrieved FAQ.
The universe of knowledge documents in this dataset is the set of
FAQs.
The training dataset consists of conversations based ondifferent domains i.e hotels, restaurants, trains and taxis.
The test dataset contains an additional domain, attractions, which is not found in the train and validation splits.
DoQA (Campos et al. 2020):
This dataset comprises of open-ended dialog conversations on different domains like cooking, travel and movies.
Unlike, the OR-QuAC and
DSTC9 datasets, most questions in this dataset are not factoid/specific questions, and are open-ended.
We only use the cooking split for both training and testing.
We preprocess all the datasets by removing all the examples if the ground truth response is “CANNOT-ANSWER” (unanswerable).
The statistics for the different datasets used are summarized in Table 1.
Each ‘instance’ refers to a context-response pair.
[Table]
Question-Answering Task:
The OR-QuAC dataset also contains non-contextual variations of questions at each dialog turn
and we use them in a QA setting (referred to as
OR-QuAC-QA).
5. Baselines
Apart from our approach (VRAG), we also train and evaluate on the original RAG model for each dataset.
We also experiment with a pipeline model which uses the pre-trained
DPR-Multiset Retriever and a GPT2 based decoder which is fine-tuned to generate responses.
We refer to this as the DPR + GPT2 baseline in our experiments.
5. Evaluation metrics
For each of our experimental runs, we report the Mean Reciprocal Rank@5 (MRR@5), Recall@1 (R@1), Recall@ (R@5) to evaluate prior’s performance.
We also report
BLEU scores (both with top-1 and top-5 decoding) to assess the performance of generator.
The BLEU-1 and BLEU4 scores in our tables are denoted by B-1 and B-4 respectively.
We also consider BLEU-penalized scores (indicated by BP-1 anad BP-4) which consider the BLEU score at a given test instance as 0 if the document retrieved for the given instance is incorrect.
These help ensure that a model is not able to produce a high score by memorizing on a particular given domain.
5. Results
As can be seen in Table 3, VRAG outperforms RAG on all retrieval tasks (Recall and MRR metrics) on every dataset.
We also note that both RAG and VRAG significantly improve the performance of the baseline DPR based retriever model.
Further Table 7 shows the results of language generation tasks on all datasets.
Note that all results in Tableare obtained after further fine tuning the generator networks for both RAG and VRAG after joint training is complete.
The results shows that the VRAG model outperforms the
RAG model on language generation tasks (BLEU metrics) on all datasets except DoQA.
We believe this difference is due to the nature of documents used in this dataset - other datasets have a lot more fact based questions to be answered using knowledge in documents, while more than 66% of the questions in the DoQA dataset are non-fact based and openended.
Perhaps the use of a stronger reasoning model which is capable of answering such questions better, may be required on this dataset.
Another, possibility is that the RAG model is memorizing and overfitting better on the domain.
Note that on other datasets, the relative gain of VRAG (over
[Table] [Table] RAG) is also significantly higher in penalized BLEU scores (BP-1 and BP-4).
This indicates that VRAG model is more likely to generate the response using the correct document and not by merely memorizing on a given domain.
Effect of Top-5 Decoding From Table 7 we find that in almost all cases, using top-5 decoding to generate responses performs better than using the single (best scored) document to generate responses.
This indicates that both RAG and VRAG are to able to incorporate information from the correct document even if it is not returned as the top-ranked document.
Lastly, as mentioned previously, VRAG performs better than RAG when using, both, top-1 and top-5 decoding.
Benefit of Document Posterior
In Section 3, we motivated the VRAG model by suggesting that using the posterior to sample documents while training the decoder could help train a better model.
Table 4 compares the performance of the VRAG (posterior) against the RAG (prior) on a retrieval task using the test data.
Figure 2 shows the retrieval performance versus epoch for VRAG prior and posterior as well as RAG prior during training.
As can be seen, the use of the responses by VRAG (posterior), to query the index during training, results in significantly better retrieval accuracy than RAG (prior) at every epoch.
The better samples from posterior help in training the generator network as well.
While this is perhaps intuitive and expected, our results on recall in Table 7 demonstrate the VRAG (prior) which is trained indirectly via the KL-divergence with VRAG (posterior) also outperforms the RAG (prior).
We attribute this gain to the fact that the generator in VRAG learns to focus well on the generated documents which further reinforces the posterior network (and indirectly the prior network through KL term) to improve.
Effect of Decoder fine-tuning Training VRAG involves optimizing two objectives - reducing the KL-divergence between the document-prior and document-posterior, and, maximizing the log likelihood of the responses.
VAE models often end up prioritizing the KL-divergence over the likelihood objective and sometimes end up with zero KLdivergence by forcing the document-posterior to match the prior (called posterior-collapse)
(Lucas et al. 2019; Bowman et al. 2015; Chen et al. 2016;
Oord, Vinyals, and
Kavukcuoglu 2017).
However, we hypothesize even in cases where there is no posterior collapse, the joint training could result in the response-generator (likelihood term) being inadequately trained.
We observed that fine-tuning the response decoder after freezing the weights of priors, results in an improvement in both RAG and VRAG.
The results shown in Table 7 were obtained after fine tuning the generator after the joint training is complete.
In appendix C, we show the results both with and without fine tuning.
It is interesting to note on the
DoQA dataset, RAG initially reported a B-4 of only 0 .(top-5 decoding) which goes up to 17 .
04 after fine-tuning (refer to Table in Appendix C).
Study of Memorization One of the issues in such an unsupervised learning is that generator may fail to use the retrieved knowledge.
Instead the parameters might have been trained to internally model the external knowledge sources themselves, a phenomenon we call ’memorization’.
In order to study memorization in the models, we compared the
OR-QuAC
DSTC
DoQA
OR-QuAC-QA
R @
R @ R @
R @ R @
R @ R @
R @
RAG (Prior) 24.
VRAG (Posterior) 39.
5 Epochs 0.
r @
RAG prior VRAG prior VRAG posterior (
[Figure]
Recall@      5 Epochs 0.
r @
RAG prior VRAG prior VRAG posterior (
[Figure]
) Recall@      5 Epochs 0.
m rr @
RAG prior VRAG prior VRAG posterior (c) MRR@
[Figure]
[Table]
OR-QuAC
[Table]
results on response generation of the pipeline, RAG and VRAG models in the absence of the correct document – if the correct document is missing, the models should perform very poorly. A good performance even without obtaining correct document, would indicate lesser reliance on external knowledge and hence a higher tendency for ’memorization’.
We, thus rebuild the document index without including any of the documents from the test set and then re-evaluate the performance of our models.
Table 5, shows the drop in
BLEU scores for each of the models after removing the correct document on OR-QuAC dataset.
However the percentage drop is highest in the VRAG model.
This indicates a higher usage of knowledge instance and hence lesser memorization.
Effect of top-k
We also consider the case when we reduce the number of samples ( k ) while training.
Figure 3 shows the variation in R @3 versus k on the DoQA dataset.
Here we observe that recall decreases with decrease in k .
This is because the computation of objective becomes less accurate for decrease in k .
We also note that when only one document is sampled, RAG outperforms VRAG.
We hypothesize, that in this case the VRAG prior trains poorly due to an extremely weak approximation of the KL divergence 6 . 5 k used for training     r @ RAG VRAG
Conclusion
In this paper we described an approach to run variational training on knowledge grounded dialog with a large corpus.
Our experiments on three conversational QA datasets indicate that variational training is helpful as it produces better document samples while training.
We find that our model,
VRAG (having access to superior samples from posterior while training), not only generates better responses, it also learns a better retriever (prior distribution).
We believe that such sampling approximations could also be helpful in other tasks; for instance, it could also be interesting to apply them to other approaches such as Reinforcement Learning to the setting of a large corpus.
This would require overcoming similar challenges in sampling as we did in this paper for variational training.
A Error guarantee for KL approximation
In this appendix, we justify the our approximation to KL term.
Theorem 1.
Let q and p be distributions over sample space Ω
= { 1 , 2 , ..., n } .
Let δ > 0 .
Define S p,low ( δ ) =
{ z : p ( z ) ≤ δ } and S q,high ( δ ) =
{ z : q ( z ) ≥ δ } .
Let
S ( δ ) =
S p,low ( δ ) ∪ S q,high ( δ ) .
Let
KL [ q || p ] = (cid:88)
z ∈ Ω q ( z ) log (cid:18) q ( z ) p ( z ) (cid:19) and ˆ KL δ
[ q || p ] = (cid:88)
z ∈ S ( δ ) q ( z ) log (cid:18) q
( z ) p ( z ) (cid:19)
Then for δ = min (cid:110) ϵ/ 2 N log(2 N/ϵ ) , 1 e (cid:111)
we have | KL − ˆ KL δ | < ϵ .
Proof.
| KL − ˆ KL δ | = (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (cid:88) z ∈ Ω \ S ( δ ) q ( z ) log (cid:18) q ( z ) p
( z ) (cid:19)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) [Equation] [Equation]
p
[Equation]
9) (By Triangular inequality)
[Equation] [Equation]
[Equation] [Equation] (Using triangular inequality) (14)
Now for z ∈ Ω \ S ( δ ) , we have that q ( z ) <
δ and p ( z ) > δ .
Now for f ( x ) = x | log( x ) | = − x log( x ) , we have f ′ ( x ) = − 1 − log( x ) .
Thus f ′ ( x ) ≥ 0 for x ≤ 1 e .
Thus f ( x ) = x | log( x ) | is an increasing function of x for x < 1 e .
Thus for q ( z ) <
δ < 1 e , we have that q ( z ) | log q ( z ) | < δ | log( δ ) | = − δ log( δ ) .
Also the second term q ( z ) | log p ( z ) | = − q ( z ) log p ( z ) .
Thus for z ∈ S ( δ ) , we have q ( z ) <
δ and p ( z ) > δ .
Thus log p ( z ) > log( δ ) or − log p ( z ) <
− log( δ ) .
Overall, q ( z ) | log p ( z ) | < − δ log( δ ) .
Hence | KL − ˆ KL δ | ≤ (cid:88) z ∈ Ω \ S ( δ ) − 2 δ log( δ ) (15) ≤ N ( − 2 δ log( δ )) .
Now, lim x → 0 + x log( x ) = 0 .
Thus, given ϵ > 0 , ∃ δ 0 >s.t. | δ 0 log( δ 0 )
| < ϵ 2 N .
In fact, we can choose δ 0 =
− ϵ/ 2 N log( ϵ/ 2 N ) .
Now, choose δ = min (cid:8) δ 0 , 1 e (cid:9) .
Then
[Equation] [Equation]
[Equation]
B Hyperparameters and Training Details
Dataset
Learning rate
Time per epoch
OR-QuAC 10 −3 hrs DSTC 10 −2 hrs DoQA 10 −20 mins OR-QuAC-QA 10 −2.5 hrs
We train both RAG and VRAG with AdamW optimizer.
We train for 10 epochs with early stopping (patience = 5).
The learning rates and time per epoch are given in Table 6.
We use BERT base (110 million parameters) for the context encoders in both prior and posterior networks and GPT small (117 million parameters) for the decoder network.
The document encoder is also a BERT base (110 million parameters) which is fixed at training time.
All of our experiments are conducted on a single machine with 100GB of system memory and Nvidia V100 GPU.
C Comparison of language generation results both with and without fine tuning on all datasets
Table 7 shows the results on RAG and VRAG both with and without fine tuning.
[Table]
Campos et al. 2020)
[Table]
OR-QuAC-QA
[Table]
al. 2020)
[Table]
Table