H INDSIGHT : P OSTERIOR - GUIDED TRAINING OF RETRIEVERS FOR IMPROVED OPEN - ENDED GENERATION
A BSTRACT
Many text generation systems beneﬁt from retrieving passages from a textual knowledge corpus (e.g., Wikipedia) and using them to generate the output.
For open-ended generation tasks, like generating informative utterances in conversations, many varied passages z are relevant to the context x but few are relevant to the observed next utterance y (label). For such tasks, existing methods (that jointly train the retriever and generator) underperform: during training the top-k context-relevant retrieved passages might not contain the label-relevant passage and the generator may hence not learn a preference to ground its generated output in them.
We propose using an additional guide-retriever that also conditions on the observed label y and “in hindsight” retrieves label-relevant passages during training.
We maximize the evidence lower bound (ELBo) to jointly train the guide-retriever Q ( z | x, y ) with the standard retriever P η ( z | x ) and the generator P θ ( y | x, z ) and ﬁnd that ELBo has better inductive biases than prior work. For informative conversations from the Wizard of Wikipedia dataset, with our posteriorguided training, the retriever ﬁnds passages with higher relevance in the top- (23% relative improvement), the generator’s responses are more grounded in the retrieved passage (19% relative improvement) and the end-to-end system produces better overall output (6.4% relative improvement).
I NTRODUCTION
In knowledge-intensive NLP tasks, models must use open-domain knowledge to answer questions (Kwiatkowski et al., 2019; Joshi et al., 2017), fact-check claims (Thorne et al., 2018) or engage in informative conversations (Dinan et al., 2019; Zhou et al., 2018).
State-of-the-art models for open-domain question answering are retrieval-augmented : they extract relevant passages from a human-readable corpus (e.g., Wikipedia) using a learned retriever and process it with a task-speciﬁc reader .
If the relevant passage is known (e.g., human-annotated gold passage), the retriever can be supervised with it.
In this work we consider open-ended generation tasks where the gold-passages are unknown.
Figure 1 illustrates this one-to-many setting: for a conversational context x , many relevant passages z (dubbed context-relevant passages) could have generated many coherent responses.
But only z gold (dubbed label-relevant passage) generates the observed target output y .
Had we known z gold corresponding to the target output, we could have supervised the retriever with z gold and trained the generator conditioned on z gold – but we don’t!
Current methods for retrieval-augmented generation (Lewis et al., 2020) work well for short-answer
QA-like tasks: Natural Questions (Kwiatkowski et al., 2019) or fact-checking (Thorne et al., 2018).
Lewis et al. (2020) use the generator’s probability distribution P θ ( y | x, z ) as a proxy for label relevance and train the retriever P η ( z | x ) by marginalizing p ( y | x ) over retrieved documents z : P ( y | x ) = (cid:80)
z ∈ top-k
( P η ( .
| x ))
P η ( z | x ) P θ ( y | x, z ) .
However, for one-to-many tasks, this objective leads to suboptimal solutions: the generator is less grounded in the retrieved passages (Figure 3), the retriever performance saturates at low recall (Figure 3), and the top-k retrieved passages exclude many label-relevant passages weakening the supervision during training (Table 1).
In our work, as a proxy for z gold , we train a separate guide-retriever model to ﬁnd label-relevant passages. The guide-retriever uses both the input x and the output y and is represented by the label-posterior distribution Q ( z | x, y ) that captures label-relevance in “hindsight”. Modeling the label-posterior distribution Q ( z | x, y ) with a full-ﬂedged retriever generalizes weak supervision approaches and retrieves label-relevant passages from the entire collection.
We jointly optimize the retriever, posterior-guide, and generator using the evidence lower bound (ELBo): E z i ∼ Q ( .
| x,y )
[log P θ ( y | x, z )] − D KL
( Q | P η ) .
While the objective function is a lower bound, it encodes biases that improve joint-training on open-ended tasks: (1) conditioning the generator on the passages weighted by their label-relevance (from the label-posterior distribution) increases grounding and (2) training the retriever with a mode-seeking reverse-KL divergence encourages it to match some modes with the guide (label-relevant passages), with a lesser penalty for matching other modes (other context-relevant passages).
Our main contribution is a complete H INDSIGHT training system that: (1) uses a guide-retriever to provide a stronger learning signal for both the generator and the retriever and (2) is amenable to index-updates with iterative closed-set training (Section 3).
To evaluate one-to-many open-ended generation tasks, it is insufﬁcient to just evaluate the end-to-end performance of the joint system. Thus, we also evaluate the individual models (retriever and generator) and at varying passage depths.
Using H INDSIGHT on the Wizard of Wikipedia dataset of informative conversations: the retriever ﬁnds more relevant passages with a 23% relative improvement (r.i.) in success@10 (i.e., is the label-relevant passage among the top-10 retrieved passages?), the generator is more grounded with 19% r.i. in Novel-F1 overlap with the top-1 retrieved passage (i.e., its overlap with the retrieved passage excluding words that are common or in the input) and the combined system is overall better with a 6.4% r.i. in Novel-F1@1 overlap with the gold utterance (the best matching generation when considering top-1 retrieved passage).
H INDSIGHT also improves performance on the MS-MARCO NLGen dataset, a one-to-one free-form QA task.
B ACKGROUND
Open-domain Question Answering In the reading comprehension task, a passage is given and the models extract the answer span from it. In Open-domain QA (a.k.a. open-QA) no such passage is given; the models are expected to extract the answer from a large document corpus.
Dr. QA (Chen et al., 2017), the ﬁrst neural system for factoid open-QA, used an off-the-shelf retriever (e.g., TFIDF, BM25) to ﬁnd relevant passages and trained a reader to extract the answer span. Now, trainable neural retrievers have replaced the classical term-matching retrievers.
Here, pre-trained models (like
BERT) embed the document corpus and the query into a single vector space and efﬁcient nearestneighbour search algorithms (Jegou et al., 2010; Johnson et al., 2017) ﬁnd the relevant passages corresponding to the query.
The neural retriever can be trained variously: pretraining with the inverse cloze task then weakly supervising using span matches (Lee et al., 2019), using gold passages with in-batch negatives (Karpukhin et al., 2020), and retrieval-guided supervision with span-based positives (Khattab et al., 2021).
Natural language generation tasks provide some input (sequence of tokens, image) and expect the system to produce another sequence of tokens (or word-pieces) as output.
An open-ended task accepts a higher diversity of generations.
Factoid question-answering with a single correct short answer is less open-ended than free-form long answers.
Machine translation accepts a few correct translations (Bojar et al., 2014) but they are less diverse than informative dialogue, where the speakers can lead the conversation in many different directions (Dinan et al., 2019), making it more open-ended.
Many more generation tasks such as summarization (Narayan et al., 2018) and story generation (Mostafazadeh et al., 2016) lie on this spectrum.
Retrieval for Language Modeling
Khandelwal et al.
(2020) retrieve similar contexts from the training set at each time-step and increase the likelihood of tokens that were predicted in similar contexts.
Guu et al.
(2020) instead pre-train a retrieval-augmented masked language model using salient-span masking and ﬁne-tune it on downstream QA tasks.
Zheng et al.
(2020) use term-overlap with the label as a heuristic to identify the gold-passage from a small passage set ( ∼ 50 ) and train a reranker.
Prior work has also modeled the posterior of various probabilistic models (Lian et al., 2019;
Kim et al., 2020; Zhan et al., 2021) or used reinforcement learning (Zhao et al., 2020) to improve knowledge selection from the small passage set.
In Zheng et al.
(2021), the authors increase grounding by using the label to reweigh passage tokens and in Cai et al.
(2019) they increase grounding by feeding a corrupted version of the label to the generator as a stand-in for the label-relevant passage during training.
Lewis et al. (2020) introduce retrieval-augmented generation, where, for input x and output y , a retriever ﬁnds top-k passages ( z ) from a corpus and jointly train it with a generator ( P θ ) by maximizing the likelihood of the output marginalized over the top-k documents. In this work, we refer to this loss function as the M ARGINALIZED L OSS :
[Equation] Here P θ ( y | x, z ) is conceptually utilized in two roles: ﬁrst, supervising the retriever (i.e., teaching the retriever to score label-relevant passages higher than other passages) and keeping the generator grounded (i.e., maximizing the probability of the target output given the context-relevant passages). In the next section we introduce a guide-retriever to capture the label-relevance and we train it using ELB O L OSS , a lower bound to M ARGINALIZED L OSS , that has better inductive biases.
T RAINING WITH H INDSIGHT
To identify label-relevant passages, we explicitly model the posterior distribution Q ( z | x, y ) with a learned neural model.
Unlike the retriever P η ( z | x ) , the label-posterior model has access to the target output and in hindsight can differentiate the label-relevant from other context-relevant passages.
We learn the label-posterior jointly with the retriever and the generator by maximizing the evidence lower bound, ELB O L OSS , as given by the formula:
[Equation]
The ELB O L OSS has two terms with useful inductive biases.
The ﬁrst term maximizes the expectation of the generator’s log-likelihood P θ over the passages sampled from the label-posterior distribution Q .
The generator need attend only to the label-relevant passages, biasing it toward relying more on the retrieved passages rather than its internal language model.
The second term is the KL divergence from the retriever to the label-posterior, also referred to as the reverse KL divergence:
[Equation]
This term is again weighted by Q ( z | x, y ) , making it like a probabilistic implication: high Q ( z | x, y ) implies high P ( z | x ) , i.e., label-relevance implies context-relevance but not vice-versa.
In one-tomany tasks, which have many context-relevant passages but few label-relevant passages, this term captures the intuition that the retriever be penalized heavily if it doesn’t retrieve the label-relevant passage but lightly if it retrieves other context-relevant passages that happen to not be label-relevant.
Rather than modeling the label-posterior Q ( z | x, y ) as a re-reranker (that merely reranks documents as retrieved by the retriever P η ), we model it as a guide retriever that ﬁnds label-relevant passages from the entire corpus.
We sample passages from the label-posterior distribution, and estimate the ELB O L OSS more accurately than using passages from P η ( z | x ) .
The guide retriever generalizes weak supervision approaches (Lee et al., 2019; Guu et al., 2020) and relevance-guided supervision (Khattab et al., 2021), to posterior-guided supervision with a learned posterior retriever rather than brittle heuristics based on word-overlap.
Prior works (Guu et al., 2020; Khattab et al., 2021) intermittently update the passage index during training.
To allow for such a workﬂow, we organize our training into rounds (see Figure 2).
At the beginning of each round, in the outer loop, we encode the passages and the queries with various retrievers and ﬁnd the highest scoring r passages that we dub the closedset.
In the inner loop that runs for many epochs, we sample k ( = 8 ) passages from the closed-set ( r = 100 ).
This is fast because we are no longer retrieving from the entire corpus in the inner loop and also sufﬁcient because the closed-set has a high recall.
During the inner loop, we update the retrievers (both document and query encoders) and use the latest model parameters for computing the loss functions.
A round results in trained models that are then used for the next round.
We ﬁnd that 2 rounds are often sufﬁcient, with decreasing marginal utility from the third round onward.
Distributional repositioning before inference
We approximate the expectation terms in ELB O L OSS by sampling k passages from the closed-set Q top-r ( .
| x, y ) , which provides better supervision than P η ( z | x ) and leads to faster training.
However, the models only ever get exposed to passages from the Q ( .
| x, y ) distribution, which limits their ability to generalize over passages from P η ( .
| x ) during inference.
To remedy this, we instead sample passages from an α -mixture of the two distributions: with probability α , z ∼ P η ( .
| x ) and with probability 1 − α , z ∼ Q ( . | x, y ) .
In the initial rounds we set low values of α and increase it toward the end to reposition the passage distribution and better match with P η ( .
| x ) at test time.
The retriever and the generator can be trained by sampling passages from different α -mixtures
and we utilize this to avoid retriever overﬁtting (with α = 1 ) while maintaining generator groundedness (with α = 0 .
In practice, the retriever and the generator when jointly trained converge at different times.
The single loss term in M ARGINALIZED L OSS (Eq. 1) hides the convergence of individual models; one model starts to overﬁt while the other model still hasn’t converged.
With ELB O L OSS , there are two terms in Eq.
2: the ﬁrst connecting Q ( z | x, y ) and
P θ ( y | x, z ) , the second connecting Q ( z | x, y ) and P η ( z | x ) .
After training for a few epochs, we freeze the guide and train the models independently until convergence based on their individual losses.
E XPERIMENTAL E VALUATION
We evaluate on two open-ended knowledge-intensive tasks: informative conversations and free-form question answering.
We ask the following three research questions:
RQ 1 Relevance: Are the retrieved passages more relevant?
(Section 4.4) RQ 2 Groundedness: Does the generator make better use of the retrieved passages?
(Section 4.5) RQ 3 Generation Quality: Does this lead to better end-to-end performance?
(Section 4.6) 4.
M ODELS Retriever Models
We model the retriever P η ( z | x ) and the guide-retriever Q ( z | x, y ) using ColBERT (Khattab & Zaharia, 2020).
ColBERT encodes the query tokens q i and the document tokens
d j independently using BERT, normalizes to produce unit-vectors E q i and E d j , and deﬁnes similarity as S q,d = (cid:80) i max j E Tq i E d j .
Unlike DPR’s
[CLS] token embedding (Karpukhin et al., 2020), with ColBERT’s late-interaction paradigm the query and document tokens retain their identities and contribute to a ﬁner-grained term-wise similarity leading to state-of-the-art retrieval results on open-domain QA benchmarks (Khattab et al., 2021).
To convert similarity scores into a probability distribution, calculate the softmax of the scores over the k sampled passages.
For the posteriorretriever, we concatenate the input and the output into the query q =
[ x y ] .
ColBERT pre-trained on the MS-MARCO passage ranking dataset is widely used for other tasks and we use it for the Wizard of Wikipedia task.
However, the MS-MARCO NLGen task contains queries from the passage ranking pre-training dataset.
Therefore, following Lewis et al.
(2020), we use Natural Questions to pre-train ColBERT Khattab et al.
(2021) for the MS-MARCO NLGen task.
Following Lewis et al.
we use a pre-trained BART model and ﬁnetune it for the respective tasks during training.
It is conditioned on both the context and the document and trained to produce the target.
At test time, we decode using beam-search with a beam size of 4.
Informative conversations are open-ended because people have the agency to drive them in different directions at every turn (one-to-many) and are knowledge-intensive because the utterances contain speciﬁc bits of world knowledge.
We evaluate with Wizard of Wikipedia (WoW) dataset (Dinan et al., 2019), where an “apprentice” chats (via text) with a “wizard”, being curious about different topics, and the “wizard” grounds their response in a sentence from Wikipedia.
The input for this task is the conversational history x , the output is the wizard’s utterance y and the models can retrieve individual passages ( z ) from all of Wikipedia ( ≈ 26 million passages).
We use the version of this dataset provided in the KILT benchmark (Petroni et al., 2021) and report leaderboard performance on the held out test set. We use the dev set to answer the granular research questions. [Table]
Free-form Question Answering
We use the MS-MARCO
NLGen dataset (Nguyen et al., 2016) where the task is to generate natural-sounding answers to questions. This free-form open-QA task is one-to-one but more challenging than other extractive open-domain QA datasets.
The dataset is a subset of MS-MARCO questions whose answers were reviewed by a separate editor and rewritten if they had a high overlap with one of the provided passages (indicating that the original editor may have copied the passage directly).
These “well-formed answers” are meant to be complete sentences (such as can be read out by a conversational assistant) and are long (median length 11 words).
The input for this task is a query x , the output is a well-formed answer y [Table] and the models can retrieve from the MS-MARCO passage collection (8.8 million web passages).
The public benchmark and the test set is no longer available for evaluation.
Instead we split the public validation set into a validation and test set and show results on the test set.
While both datasets annotate the passages referred to by the person who wrote the target output (gold passages), we only use them for evaluation and validation and not for training.
4. B ASELINES
Apart from the two main methods – M ARGINALIZED L OSS and ELB O L OSS – we train two additional baselines: gold-supervised and generator-only . For the Gold-supervised baseline, we assume that the gold-passage z gold is available during training and train a retriever by maximizing its log-likelihood. We take random passages from ELBo Retriever’s closed-set as negatives (excluding top-10 to avoid false negatives).
We train the Gold-supervised generator by simply maximizing
P θ ( y | x, z gold ) .
For the generator-only baseline, we ignore the existence of passages and directly maximize P ( y | x ) with a sequence to sequence model.
4. R ELEVANCE E VALUATION
We evaluate the relevance of the retrieved passages ( RQ 1 ) using the gold passage labels supplied by each dataset.
For one-to-many tasks, we expect the label-relevant passage to be one of the top- k ( k = { 1 , 5 , 10 } ) retrieved passages.
Thus, we report Success@ k (S@ k for short) 1 , the percentage of inputs for which any gold provenance passage is retrieved within the top- k ( k = { 1 , 5 , 10 } ) passages.
We also report Mean Reciprocal Rank (MRR), a common IR evaluation metric.
Our results are shown in Table 1.
With Wizard of Wikipedia, ELB O L OSS retriever markedly outperforms M ARGINALIZED L OSS .
Both systems have a relatively high Success@1 and easily handle 38.9–41.1% of the examples, but the ELB O L OSS retriever continues to ﬁnd many more relevant passages at larger retrieval depths k widening the gap to [-M ARGINALIZED L OSS ’s success@
-]11 points for Success@10.
With MS
MARCO NLGen, ELBoLoss outperforms MarginalizedLoss by 1–2 points across our metrics, reﬂecting smaller—but nonetheless consistent—gains when compared with Wizard of Wikipedia, a one-to-many generation task.
Based on manual inspection, we ﬁnd many false negatives (corroborated by Arabzadeh et al.
(2021)), i.e. passages that contain the answer but aren’t marked as gold, leading to a lower Success@1 compared to Wizard of Wikipedia. [Table]
M ARGINALIZED L OSS depends on the retriever P η ( z | x ) to ﬁnd label-relevant passages during training and is therefore recall limited. M ARGINALIZED L OSS ’s success@100 on Wizard of Wikipedia saturates at 55.8% (not reported in the table) without much hope of further improvement because z gold that are never retrieved cannot provide positive examples for supervision. With ELB O L OSS , the guide-retriever retrieves label-relevant passages with >85% success@10 (
Table 8) for both the datasets providing better supervision than M ARGINALIZED L OSS . Consequently, ELB O L OSS retriever’s success@5 is higher than M ARGINALIZED L OSS despite containing 20 × less passages and reaches 69.3% for success@ [Table] Comparison with Gold-supervised retriever
We ﬁnd that the Gold-sup.
retriever quickly overﬁts during training resulting in a lower performance than using ELBo loss.
We also see lower performance when training with ELB O L OSS while sampling purely from Q ( .
| x, y ) (i.e., α = 0 , Table 7) because it is low-entropy and the same label-relevant passages get repeatedly sampled.
By sampling from P η ( .
| x ) , the KL divergence is minimized over a wider and realistic support set of passages.
Perhaps, there are many passages that have some label-relevant phrases making them partially relevant and Q ( z | x, y ) “teaches” P η ( z | x ) to capture these phrase-level relative differences.
Sampling from P η ( .
| x ) leads to better generalization, has similarities to distillation and is an interesting direction for future work.
Overall, we ﬁnd that ELB O L OSS improves relevance of retrieved passages over M ARGINALIZEDL OSS for two qualitatively different tasks, with larger gains for the one-to-many generation task.
4. G ROUNDEDNESS E VALUATION
We now examine RQ 2 , studying the degree to which the generator relies on the retrieved passages for producing its output.
To quantify this groundedness , we compute F1-overlap between a retrieved passage (not necessarily the gold passage) and the produced text when generation is conditioned on that passage.
We get the retrieved passages for each method (except generator-only) using the corresponding retriever.
As an analogue of Success@ k , we propose Max.
F1@k , the largest F1-overlap exhibited by any generated output with the corresponding retrieved passage fed to the generator.
We also propose
Novel-F1, a new metric that discounts words that occur frequently and words that already appear in the context x , since otherwise these tokens dominate raw F1 in practice (up to 80%, see Appendix A.5) but are not indicative of grounding.
Our results are shown in Table 2 and Figure 3.
For the Wizard of Wikipedia dataset, we observe that our ELB O L OSS generator outperforms M ARGINALIZED L OSS by 2.7 F1 (14.5% relative improvement) and 3.3 Novel-F1 (19% r.i.) when given the top retrieved passage.
For the MS MARCO NLGen dataset, we observe smaller but consistent gains in groundedness (1–2 F1, Novel-F1) with ELB O L OSS compared to M ARGINALIZED L OSS .
In Figure 3 (right), M ARGINALIZED L OSS generator’s overlap decays rapidly beyond the top passage, whereas the ELB O L OSS ( α = 0 .
25 ) generator’s overlap declines gradually.
This shows that the ELB O L OSS generator stays grounded beyond just the top passage, a desirable property in one-to-many generation systems.
We also see (in Figure 3, right) that groundedness increases as α decreases.
However, a generator trained with α = 0 , despite being maximally grounded (Appendix table 5), has lower end-to-end performance (Appendix table 6) because it is unduly “trusting” of the provided passage (nearly ﬂat line in Figure 3) and does not abstain from using irrelevant passages. We see the same effect with the Gold-supervised generator on MS-MARCO NLGen: it is more grounded but has lower downstream performance. Overall, we ﬁnd that ELB O L OSS improves grounding of the generator over M ARGINALIZED L OSS for two qualitatively different tasks, with larger gains for the one-to-many generation task.
4. E ND - TO - END E VALUATION
To evaluate the end-to-end quality of our systems, we calculate F1 and Novel-F1 (deﬁned in Section 4.5) of the decoded output with the human-written gold output .
To allow for the possibility of the generator using any part of the gold passage (and not just the human-written gold output) for the Wizard Of Wikipedia task, we use Knowledge-F1 (deﬁned by Shuster et al.
F1 between the sampled generation and the gold passage .
Since it is reasonable to expect the gold passage to be in the top- k for k >1 for one-to-many tasks (as in Section 4.5), we also compute the max.
over top- k retrieved passages.
The results are summarized in Table 3.
For the Wizard of Wikipedia dataset, using only the top retrieved passage ELB O L OSS leads to 6 .
7% relative improvement in Novel-F1@ 1 .
But in the oneto-many setting, the label-relevant passage is an arbitrary choice from amongst the context-relevant passages.
We account for that using max.
overlap over the top- 5 passages and see larger improvements for ELB O L OSS , namely 1 F1, 2 Novel-F1 ( ∼ 15% r.i.), and 1.5 K-F1 ( ∼ 10% r.i.).
For MS
Marco NLGen, we see a small but consistent increase due to ELB O L OSS over M ARGINALIZEDL OSS : 1.5 F1 and 2 Novel-F1 across passage depths.
We also submit the above models (ELB O L OSS and M ARGINALIZED L OSS ) to the Wizard of
Wikipedia task on the KILT leaderboard.
ELB O L OSS consistently outperforms the baseline M ARGINALIZED L OSS across all metrics (see Table 4).
The table also reports Recall@5, which evaluates retrieval at a coarser granularity, namely at the full Wikipedia page level, though so far we have investigated it directly at the passage level.
Consistent with the results in Table 1, our method also outperforms M ARGINALIZED L OSS in retrieval metrics.
In fact, our ELB O L OSS model achieves state-of-the-art performance across all the generation metrics (F1, ROUGE-L, KILT-F1,
KILT-ROUGE-L) on the leaderboard, though it is not the strongest on R-Prec and Recall@5.
To conclude, we have evaluated the ELB O L OSS and M ARGINALIZED L OSS using a one-to-one freeform QA dataset and a one-to-many dataset of informative conversations.
Our results show that our method ELB O L OSS trains a better retriever, a more grounded generator and improves end-to-end performance, especially in the one-to-many setting.
D ISCUSSION
Hallucination grounding and correctness Shuster et al.
(2021) show that providing retrieved passages to a generator reduces hallucination.
Our work increases grounding in the retrieved passage, promising to further reduce hallucination.
Even though the generator is now more likely to use content from the provided passage (rather than hallucinating from parametric memory), that does not guarantee correctness .
Our token-level overlap metrics that evaluate grounding do not capture this aspect either.
There is scope for future work to address this gap with better training methods (and evaluation metrics) that produce (and reward) grounded and correct outputs.
Practical matters: Trust and control
For tasks like QA, we want a “conservative” generator: it should abstain from using a passage that doesn’t contain the answer.
For more open-ended tasks like informative conversations, we want the generator make use of diverse passages.
In our work, we show that by reducing α , the distribution of the passages shifts from P η ( .
| x ) to Q ( .
| x, y ) and the generator increasingly trusts the retrieved passages (Figure 3).
System designers can use the [Table] [Table] ELBo (curr.
best)
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
92 † α -mixture as a tool to modulate the degree of trust placed by the generator in the retrieved passages.
Further, when a “trusting” generator is deployed in real-life settings (e.g., in open-domain socialbots;
Paranjape et al. 2020), external business logic can select an appropriate passage from the top- k retrieved passages and effectively control the generated content.
Izacard & Grave (2021a) provide multiple passages to the generator simultaneously and in Izacard & Grave (2021b)
they use the decoder’s attention weights over each passage for relevance supervision.
However, Sachan et al.
(2021) use Fusion-in-Decoder only at inference, forgoing the decoder’s attention weights and using an equivalent version of the M ARGINALIZED L OSS for training the retriever. In our experiments we found marginal improvements while using Fusion-in-decoder at inference and we believe our approach of conditioning on individual passages is more precise for relevance supervision in one-to-many generation tasks. Inductive biases of H INDSIGHT
A learned P η ( z | x ) that maximizes ELB O L OSS (a lowerbound) will also maximize M ARGINALIZED L OSS . It can therefore seem counter-intuitive at ﬁrst that ELB O L OSS empirically outperforms M ARGINALIZED L OSS .
We hypothesize that it better captures the necessary inductive biases (refer Section 3). The α -mixture further loosens the bound theoretically, inducing a bias toward the test distribution and improving generalization. Trading off theoretical tightness of bounds for desirable inductive biases proved fruitful for us and we hope that it inspires other researchers. In this paper, we propose H INDSIGHT , a system that introduces a guide-retriever to improve supervision for both the retriever and the generator for retrieval-augmented, open-ended generation.
During training, the guide retriever uses the target output of each example in order to ﬁnd relevant passages, leading to better retrieval and more grounded generation.
The resulting system achieves considerable empirical improvements over existing work, improving retrieval quality by up to 23%, grounding by up to 19%, and end-to-end output quality by up to 6.4%. A CKNOWLEDGEMENTS
We would like to thank the anonymous reviewers and Stanford colleagues: Eric Mitchell, Nandita Bhaskhar, Shikhar Murthy, Amelia Hardy and Antoine Bosselut for their valuable feedback. This research was supported in part by Stanford HAI, Samsung Electronics Co., Ltd, and afﬁliate members and other supporters of the Stanford DAWN project—Ant Financial, Facebook, Google, Virtusa and VMware—as well as Cisco, SAP, and the NSF under CAREER grant CNS-1651570.
Any opinions, ﬁndings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reﬂect the views of the National Science Foundation.
R EPRODUCIBILITY S TATEMENT
Section 3 contains key details of our method.
The code for recreating these experiments along with hyperparameters will be released at https://github.com/AshwinParanjape/ hindsight .
E THICS S TATEMENT
Any generative system that produces free-form text and interacts with humans needs to have safeguards in place against harmful speech.
In our work, we try to reduce hallucination and increase grounding which is a step in the right direction.
It also helps restrict the content and exert control over the generation.
But as we point out in the discussion (Section 5) increased grounding does not imply correctness.
These systems by themselves are not ready to be deployed in the real world and future work needs to address outstanding issues.
A A PPENDIX A. I S HIGHER GROUNDING PURELY DUE TO A BETTER RETRIEVER ?
To test this hypothesis, we ran an ablation test, where we used the best retriever we had (from
Hindsight, α = 0 ) and trained a generator using Marginalized loss while keeping the retriever itself ﬁxed.
This simulates the situation where we improved retrieval independently and we want to test if that is sufﬁcient to explain increased grounding.
We see in table 5 that that the better retriever in the ablation (Marg.
Gen. with ﬁxed ELBo Ret.) leads to increased grounding compared to the generator trained using M ARGINALIZED L OSS .
However, the ablation is worse than ELBo Gen. ( α = 0 . 25 ), demonstrating that the increased grounding is also due to ELB O L OSS .
A. W ITHOUT DISTRIBUTIONAL REPOSITIONING
ELB O L OSS doesn’t perform as well without distributional repositioning.
[Table]
[Table]
[Table] [Table] [Table] Bucatini > Abstract | Bucatini , also known as perciatelli , is a thick spaghetti-like pasta with a hole running through the center.
The name comes from , meaning "hole", while "bucato" or its Nea...
[Table]
Pasta con le sarde > Ingredients.
| The principal ingredients are olive oil, onions, pasta and a ﬁnely chopped mixture of sardines and anchovy.
Various types of pasta are used for the dish, but b...
[Table]
Bucatini > Preparation.
| Standard pasta machines will roll out sheets of ﬂat pasta which are then cut into ribbons to make ﬂat, ribbon-style pasta like fettuccine , tagliatelle, or pappardelle. ...
Bocconcini > Abstract |
This cheese is described by its Italian name, which means "small mouthfuls".
It is made in the "pasta ﬁlata" manner by dipping curds into hot whey, and kneading, pulling, ... 5.
Carbonara > Abstract | Carbonara () is an Italian pasta dish from Rome made with egg, hard cheese, guanciale (or pancetta) and black pepper.
The dish arrived at its modern form, with its current ...
A. E XAMPLES OF RETRIEVED PASSAGES A.3. C ONVERSATION 1: I TALIAN CUISINE
Other Ooh I like that!
Stick some nice spicy arrabbiata sauce with it, ahhhh!
Have you ever had bucatini before?
Self Oh yeah!
I love that spicy garlic and tomato sauce.
No I have not had bucatini.
Is that a type of cheese?
Other Now you’re speakin’ my language.
No no, it’s a style of noodle, like a really long straw.
Bucatini amatraciana is insanely good.
[Table]
text 1.
[Table]
Bucatini > Abstract | Bucatini , also known as perciatelli , is a thick spaghetti-like pasta with a hole running through the center.
The name comes from , meaning "hole", while "bucato" or its Nea... 2. Bucatini > Preparation. | Standard pasta machines will roll out sheets of ﬂat pasta which are then cut into ribbons to make ﬂat, ribbon-style pasta like fettuccine, tagliatelle, or pappardelle. Rotini > Abstract | Rotini is a type of helix- or corkscrew-shaped pasta.
The name comes from a 17th-century Italian word meaning "small wheels".
Rotini is related to fusilli, but has a tighter he... 4.
Vermicelli > History.:The Americas.
| The "ﬁdeo" is a type of noodle, produced in Europe ever since the Roman times, best known as "ﬁdeus" or "ﬁdelis", and then spread to Mexican and Latin Amer... 5.
Rollatini > Abstract | Rollatini (sometimes also spelled rolatini or rolletini) is an Italianstyle dish (called "rollatini di melanzane" in faux Italian) that is usually made with thin slices of ...
A.3. C ONVERSATION 2: R OCK AND R OLL
Self Do you mean Elvis Aaron Presley, the American singer and actor?
Other That’s the one.
I think his nickname was the king of rock ’n roll.
Self I had just heard of him being "The King".
There probably would not have been a Sun Records if not for Elvis and Sam Phillips.
Other He was revolutionary for his time.
Many older people thought he was straight from the devil.
[Table]
) was an American record producer who played an important role in the development of rock and roll during the
[Table]
Cultural impact of Elvis Presley > Abstract | Since the beginning of his career, Elvis
Presley has had an extensive cultural impact.
According to "Rolling Stone", "it was Elvis who made rock ’n’ r...
[Table]
Abstract | Freddie King (September 3, 1934 – December 28, 1976) was an American blues guitarist and singer.
He recorded several hits for Federal Records in the early 1960s [Table]
His
[Table]
Elvis
[Table]
Abstract | With a series of successful network television appearances and chart-topping records, he became the leading ﬁgure of the newly popular sound of rock and roll.
His
energ
[Table]
Elvis Presley > Abstract | Elvis Aaron Presley (January 8, 1935 – August 16, 1977), also known mononymously as Elvis, was an American singer, musician, and actor.
Regarded as one of the most signi...
[Table]
Elvis Presley > Abstract | Elvis Aaron Presley (January 8, 1935 – August 16, 1977), also known mononymously as Elvis, was an American singer, musician, and actor.
Regarded as one of the most signi... 2.
Sam Phillips > Abstract | Samuel Cornelius Phillips
(January 5, 1923 – July 30, 2003) was an American record producer who played an important role in the development of rock and roll during the 19... 3.
Johnny Otis > Abstract | Johnny Otis (born Ioannis Alexandres Veliotes; December 28, 1921 – January 17, 2012) was an American singer, musician, composer, arranger, bandleader, talent scout, disc j... 4.
Carl Perkins > Abstract | Called "the King of Rockabilly", he was inducted into the Rock and Roll Hall of Fame, the Rockabilly Hall of Fame, the Memphis Music Hall of Fame, and the Nashville Songw... 5.
Chubby Checker > Abstract | Chubby Checker (born Ernest Evans; October 3, 1941) is an American rock ’n roll singer and dancer.
He is widely known for popularising many dance styles including the t...
A. E XAMPLES OF G ENERATED OUTPUTS
We look at a conversation that illustrates the differences in grounding between M ARGINALIZEDL OSS generator and ELB O L OSS generator.
We see that the M ARGINALIZED L OSS generator hallucinates in 4 out of 5 utterances and only uses retrieved passage content in 2 out of 5 utterances.
On the other hand ELB O L OSS generator doesn’t hallucinate and uses retrieved passage content for all utterances.
Prior turns:
Self I’m still working on getting better proper nutrition (Cheetos get me) but moderate vigorous exercise feels great
Other Yes, we have a swimming pool
so I’ve been able to get a bit more exercise, while enjoying the cool water during this hot summer.
[Table]
Physical [Table]
Abstract
That’s great!
I love swimming too.
It’s a great way to stay hydrated and prevent hypokinetic diseases .
Outline of Exercise >
Abstract
That’s great!
I love swimming too.
It’s a great way to keep your immune system healthy and prevent diseases like heart disease and diabetes .
Underwater hockey >
Abstract
That’s great!
I love swimming too.
It’s a great way to keep your body hydrated .
Exercise Trends >
Abstract
That’s great!
I love swimming too.
It’s a great way to keep your body hydrated .
I’ve never been to a pool before.
Callanetics >
Abstract
That’s great!
I love swimming pools too.
They’re great for exercise, especially in the summer.
[Table]
Abstract
That’s great!
I’m also dedicated to physical ﬁtness , which is a state of health and well-being.
It’s important to be healthy, to resist hypokinetic diseases .
Exercise >
Abstract
That’s awesome!
I’ve been doing it for a long time.
It’s a great way to improve strength, prevent aging, develop muscles and the cardiovascular system .
Bodybuilding >
Abstract
That’s awesome!
I’m also into bodybuilding , which is the use of progressive resistance exercise to control and de
- velop one’s musculature for aesthetic purposes .
General ﬁtness training >
Abstract That’s great!
I’m also into general ﬁtness training , which is focused on broad goals of overall health and wellbeing .
Strength training >
Abstract That’s awesome!
I’m also into strength training , which is a type of physical exercise specializing in the use of resistance to induce muscular contraction .
A. N OVEL -F Rationale
We conducted a small experiment with the generated output on Wizard of Wikipedia dataset using top-8 retrieved passages.
We removed the gold passage and computed overlap of the generated output with the target output.
We consistently found (across models and passage ranks) the F1 overlap to be close to 15.
This meant that by conditioning on arbitrary passages the generator (likely by ignoring them altogether) is able to achieve around 80% of the F1-overlap of the best performing models ( ∼ 19 F1).
This can be a confounding factor for selecting models based on high
F1 overlap.
A model that simply copies content from the input x can achieve high F1-overlap but fail to using the retrieved passage to generate the output.
Removing commonly occurring words reduces it to 8 F1, but removing words from input context reduces it further down to 4 F1.
Thus we ﬁnd Novel-F1 to be the cleanest measure of overlap as it discounts two confounding factors and only looks at “Novel” words, words that are rare and were not in the input text x .
We construct the list of common words based on their frequency in the training corpus.
We sort words by frequency and take the most frequent words that contribute: 50% of the probability mass toward Wizard of Wikipedia utterances (amounting to 121 words) following Shuster et al.
However, we found that using the same heuristic for MS-MARCO NLGen answers included numbers and rarer tokens that could potentially be in the answer span.
So we instead use only 33% of the probability mass (amounting to 55 words).
We also ran evaluation using 50% of the probability mass but found the trends to be consistent.
MS Marco NLGen list of common words
[Table]
is, of, in, to, and, for, or, are, that, on, from, as, by, you, with, it, county, can, at, per, was, your, average, cost, be, between, which, used, one, united, states, there, years, located, name, not, new, have, takes, number, has, means, days, when, blood, system, year, should, no, most, ﬁrst, hours, up, minutes,
Wizard of Wikipedia list of common words (
[Table] frequency)
is, of, in, to, and, for, or, are, that, on, from, as, by, you, with, it, county, can, at, per, was, your, average, cost, be, between, which, used, one, united, states, there, years, located, name, not, new, have, takes, number, has, means, days, when, blood, system, year, should, no, most, ﬁrst, hours, up, minutes, 1 i, and, of, in, is, to, it, that, are, you, they, have, was, but, for, as, its, like, with, on, so, be, or, not, yes, do, can, from, there, by, well, also, one, my, know, has, some, he, their, love, most, people, think, really, all, about, just, too, them, im, which, sure, more, been, at, would, many, were, good, very, dont, when, thats, no,
yeah, what, other, great, if, because, used, actually, ﬁrst, since, lot, me, even, your, how, we, time, different, world, use, get, called, only, out, much, over, had, though, music, around, popular, his, am, made, than, such, back, up, us, make, usually, who, favorite, new, food, oh, long, she, now, did, pretty, any, where, years, this, way, go
B I NTUITION BEHIND IMPROVEMENTS DUE TO ELB O L OSS
To understand the intuition behind suboptimality of M ARGINALIZED L OSS for open-ended generation tasks consider the following: We would want a good retriever to assign similar but high probabilities to all context-relevant passages because they are similarly relevant but a good generator to only assign high probabilities when using label-relevant passages because only label-relevant passages are pertinent to the target output.
But the training signal to a model (partial derivative w.r.t the model and a passage) is modulated by the probability of the other model:
[Equation]
Since context-relevant passages have similar P ( z i | x ) the gradient encourages the generator to assign equal probabilities to the target output using all context-relevant passages.
We see this issue play out empirically when using M ARGINALIZED L OSS for two different tasks: Open-Domain QA (Natural
Questions by Kwiatkowski et al.
and informative dialogue (Wizard of Wikipedia by Dinan et al.
(Figure 4).
We see that on the Natural Questions dataset, where there is typically one correct answer, the generator produces distribution with a sharp peak that can potentially serve as an accurate proxy for label-relevance and in turn train a good retriever.
But on the Wizard of Wikipedia dataset, the generator produces a ﬂatter distribution which is a bad proxy for label-relevance.
This provides weaker supervision for the retriever which learns a ﬂatter probability distribution as well and is less able to differentiate context-relevant from irrelevant passages.
We see in Figure 5 that for the Wizard of Wikipedia dataset with ELB O L OSS we obtain a sharp distribution for Q ( z | x, y ) (nearly as good as P θ ( y | x, z ) on NQ from Figure 4) and that the P η ( z | x ) and P θ ( y | x, z ) are now sharper than M ARGINALIZED L OSS .
While a sharper distribution does not imply a better retriever and generator (they may still assign high probability to the wrong passage), a ﬂatter distribution limits their potential.
As we will see in Section 4, ELB O L OSS indeed utilizes the potential and trains a better retriever and more a grounded generator.