Recent advances in large-scale pre-training such as GPT-3 allow seemingly high quality text to be generated from a given prompt. However, such generation systems often suffer from problems of hallucinated facts, and are not inherently designed to incorporate useful external information. Grounded generation models appear to offer remedies, but their training typically relies on rarely-available parallel data where corresponding  documents are provided for context. We propose a framework that alleviates this data constraint by jointly training a grounded generator and document retriever on the language model signal. The model learns to retrieve the documents with the highest utility in generation  and attentively combines them in the output . We demonstrate that by taking advantage of external references our approach can produce more informative and interesting text in both prose and dialogue generation.
Recent advances in large-scale pre-training such as GPT-3 allow seemingly high quality text to be generated from a given prompt. However, such generation systems often suffer from problems of hallucinated facts, and are not inherently designed to incorporate useful external information. Grounded generation models appear to offer remedies, but their training typically relies on rarely-available parallel data where corresponding information-relevant documents are provided for context. We propose a framework that alleviates this data constraint by jointly training a grounded generator and document retriever on the language model signal. The model learns to reward retrieval of the documents with the highest utility in generation , and attentively combines them using a Mixture-of-Experts (MoE) ensemble to generate follow-on text . We demonstrate that both generator and retriever can take advantage of this joint training and work synergistically to produce more informative and relevant text in both prose and dialogue generation.
A
information-relevant
R
retrieve
reward retrieval of
A
,
R
in the output
using a Mixture-of-Experts (MoE) ensemble to generate follow-on text
R
by taking advantage of external references our approach can
both generator and retriever can take advantage of this joint training and work synergistically to
R
interesting
relevant