G ENERATE RATHER THAN R ETRIEVE : L ARGE L ANGUAGE M ODELS ARE S TRONG C ONTEXT G ENERATORS
A BSTRACT
Knowledge-intensive tasks, such as open-domain question answering (QA), require access to a large amount of world or domain knowledge.
A common approach for knowledge-intensive tasks is to employ a retrieve-then-read pipeline that ﬁrst retrieves a handful of relevant contextual documents from an external corpus such as Wikipedia and then predicts an answer conditioned on the retrieved documents.
In this paper, we present a novel perspective for solving knowledge-intensive tasks by replacing document retrievers with large language model generators.
We call our method generate-then-read (G EN R EAD ), which ﬁrst prompts a large language model to generate contextual documents based on a given question, and then reads the generated documents to produce the ﬁnal answer.
Furthermore, we propose a novel clustering-based prompting method that selects distinct prompts, in order to generate diverse documents that cover different perspectives, leading to better recall over acceptable answers.
We conduct extensive experiments on three different knowledge-intensive tasks, including open-domain QA, fact checking, and dialogue system.
Notably, G EN R EAD achieves 71.6 and 54.4 exact match scores on TriviaQA and WebQ, signiﬁcantly outperforming the state-of-the-art retrieve-thenread pipeline DPR-FiD by +4.0 and +3.9, without retrieving any documents from any external knowledge source.
Lastly, we demonstrate the model performance can be further improved by combining retrieval and generation.
Our code and generated documents can be found at https://github.com/wyu97/GenRead .
I NTRODUCTION
Knowledge-intensive tasks, such as open-domain question answering (QA) and fact checking, require access to a large amount of world or domain knowledge (Petroni et al., 2021).
These tasks are even challenging for humans without access to an external knowledge source such as Wikipedia.
A common thread of existing methods for knowledge-intensive tasks employ a retrieve-then-read pipeline that ﬁrst retrieves a handful of relevant contextual documents from Wikipedia and then conditions the prediction of the answer on these documents along with the question (Karpukhin et al., 2020; Lewis et al., 2020; Izacard & Grave, 2021).
Nevertheless, these methods mainly suffer from three drawbacks.
First, candidate documents for retrieval are chunked (e.g., 100 words) and ﬁxed, so the retrieved documents might contain noisy information that is irrelevant to the question.
Second, the representations of questions and documents are typically obtained independently in modern two-tower dense retrieval models (Karpukhin et al., 2020), leading to only shallow interactions captured between them (Khattab et al., 2021).
Third, document retrieval over a large corpus requires the retriever model to ﬁrst encode all candidate documents and store representations for each document.
These two operations limit the parameters of dense retrievers and the size of embedding vectors, and thus cannot enjoy the world knowledge or deduction capabilities of large language models (Levine et al., 2022).
In this paper, we propose to leverage large language models, such as InstructGPT (Ouyang et al., 2022), to directly generate contextual documents for a given question, instead of retrieving relevant documents from an external corpus, such as Wikipedia.
Our approach has two main advantages.
First, we show that generated contextual documents contain the correct answer more often than the top retrieved documents.
We believe this is because large language models generate contextual documents by performing deep token-level cross-attention between all the question and document contents, resulting in generated documents that are more speciﬁc to the question than retrieved documents.
Second, we show that our approach signiﬁcantly outperforms directly generating answers from large language models despite not incorporating any new external information.
This is mainly because the task of generating document-level contexts is close to the objective of causal language modeling pre-training, so the world knowledge stored in the model parameters can be better utilized.
We show, on multiple datasets, that generated documents are more likely to contain correct answers than the top retrieved documents.
Notably, in dense retrieval methods, as more documents are retrieved, the recall of documents containing the correct answer increases (Karpukhin et al., 2020).
However, the recall performance does not scale as well with generated documents because even with sampling methods, generated documents tend to contain duplicate information.
In order to improve the recall performance of generated documents, we propose a novel clustering-based prompt method.
We synthesize a prompt with in-context demonstrations of question-document pairs sampled from diverse clusters.
These prompts result in generated documents that cover different perspectives of the question and improve the scaling of performance as more documents are generated per question.
In contrast to the retrieve-then-read pipeline, our method is essentially a generate-then-read pipeline.
Speciﬁcally, it ﬁrst prompts a large language model to generate contextual documents based on a given question, and then reads the generated document to produce the ﬁnal answer.
The reader can still be a large model (e.g., InstructGPT (Ouyang et al., 2022)) used under a zero-shot setting, or a small one (e.g., FiD (Izacard & Grave, 2021)) ﬁne-tuned with generated documents on the training split of the target dataset.
We evaluate our proposed method on three different knowledge-intensive tasks and demonstrate its effectiveness on both zero-shot and supervised settings.
Overall, our main contributions can be summarized as follows:
We propose a novel generate-then-read pipeline for solving knowledge-intensive tasks,
i
e., replacing the process of retrieving documents from Wikipedia or searching for related documents on Google, by prompting a large language model to generate relevant contextual documents.
We propose a novel clustering-based prompting approach to generate multiple diverse contextual documents that increases the likelihood of covering the correct answer.
We demonstrate this approach can signiﬁcantly improve performance on end QA and other downstream tasks.
We conduct extensive experiments with three knowledge-intensive NLP tasks under both zeroshot and supervised settings.
Notably, our method can match or even outperform retrieve-then-read pipeline methods, without retrieving any documents from any external knowledge source.
R ELATED W ORK
K NOWLEDGE - INTENSIVE NLP VIA R ETRIEVE - THEN -R EAD P IPELINE .
Mainstream methods for solving knowledge-intensive NLP tasks employ a retrieve-then-read model pipeline.
Given a question, this model ﬁrst leverages a retriever over a large evidence corpus (e.g. Wikipedia) to fetch a set of relevant documents that may contain the answer.
A reader is then used to peruse the retrieved documents and predict an answer.
Recent follow-up work has mainly focused on improving the retriever (Karpukhin et al., 2020;
Qu et al., 2021; Sachan et al., 2022) or the reader (Izacard & Grave, 2021;
Cheng et al., 2021; Yu et al., 2022), or training the system end-to-end (Lewis et al., 2020; Singh et al., 2021).
Early retrieval methods mainly employed sparse retrievers, such as
BM25 (Chen et al., 2017).
Recently, ORQA (Lee et al., 2019) and DPR (Karpukhin et al., 2020) have revolutionized the ﬁeld by utilizing dense contextualized vectors for document indexing, leading to superior performance to traditional approaches.
We propose an alternative approach which forgoes retrieval, instead extracting the knowledge from the model parameters of a large language model.
We show that our approach is can be combine with dense retrievers to outperform both methods independently.
Our method can also be combined with any reader mechanism, allowing generated context documents to be plugged into any current knowledge-intensive NLP pipelines.
G ENERATOR AS R ETRIEVER FOR O BTAINING C ONTEXTUAL D OCUMENTS .
Recent works have investigated using auto-regressive language models to generate identiﬁer strings for documents, as an intermediate target for retrievals, such as entity names (De Cao et al., 2020) or distinctive n-grams that can be mapped to full passages (Bevilacqua et al., 2022).
However, one needs to create the identiﬁers, hence the structure was not thoroughly evaluated on a large-scale benchmark (Bevilacqua et al., 2022).
Other works have demonstrated that the knowledge stored in the parameters of pre-trained language models could be “retrieved” to some extent by directly generating text (Petroni et al., 2019; Roberts et al., 2020).
However, the previous work only used generation for query expansion (Mao et al., 2021), which did not exploit the potential of directly generating contextual documents for open-domain questions.
Different from the above approaches that aimed to train a generator model to produce contextual document identiﬁers (which is still using the original Wikipedia text) or provide data augmentation to retrievers, our work directly generates contextual documents for given questions.
NLP M ODELS E NHANCED BY
L ARGE L ANGUAGE M ODEL
O UTPUTS .
A line of recent work has shown that relevant knowledge can be elicited from large language models, especially for those domains that lack appropriate knowledge bases with sufﬁcient coverage (Liu et al., 2022b; Fang et al., 2022).
For example, Liu et al.
(2022b)
proposed leveraging GPT-3 to generate relevant contexts, then providing the contexts as additional input when answering a commonsense question.
Another line of work focused on prompting a large language model to generate a series of intermediate reasoning steps, often referred to as chain-of-thought (Wei et al., 2022b; Kojima et al., 2022; Li et al., 2022).
The prompt consists of an instruction (e.g., Let’s think step by step!), a few demonstrations that are ﬁxed for each task, and a new-question placeholder.
The demonstrations are human-written, and each consists of a question in the style of the task and a series of intermediate reasoning steps that is helpful for answering the question.
Our work does not require any human annotation, but adds to this line of work of leveraging model generated text to guide further generations.
In our case, we apply this approach to knowledge-intensive tasks, which have not been explored by previous work.
P ROPOSED M ETHOD
In this section, we present details of our proposed novel generate-then-read (G EN R EAD ) pipeline for solving various knowledge-intensive tasks.
Speciﬁcally, it ﬁrst prompts a large language model to generate contextual documents with respect to a given query, then reads the generated documents to predict the ﬁnal answer.
The reader can either be a large model (e.g., InstructGPT) used for the zero-shot setting, or a small one (e.g., FiD) ﬁne-tuned with generated documents on the training split of the target dataset.
We introduce the zero-shot setting in § 3.1 and supervised setting in § 3.2.
3. Z ERO - SHOT S ETTING
Under the zero-shot setting, there is no training data – neither questions nor contextual documents.
When tested on the open-domain QA task, most existing large language models directly encode the given question and predict the answer (Brown et al., 2020; Du et al., 2022; Chowdhery et al., 2022).
Speciﬁcally, the question q , associated with some text prompt, is input to the model, which then generates the answer, denoted as p ( a | q, θ ) , where θ represents the pre-trained model parameters.
In practice, the maximum a posteriori estimation (MAP) is the ﬁnal answer, i.e., ˆ a = arg max a p ( a | q, θ ) .
However, this way of directly asking large language models to output answers often leads to poor performance, as it leaves a considerable amount of additional world knowledge unexploited (Levine et al., 2022).
On the contrary, the zero-shot retrieve-then-read pipeline ﬁrst uses an off-the-shelf retriever to fetch relevant documents from an external knowledge source such as Wikipedia, then asks the large language model to read the documents and predict the answer.
In this work, we improve the performance by introducing an additional auxiliary generated document variable d , and then extend the model to have the form p ( a | q )
= (cid:80) i p ( a | d i , q ) p ( d i | q ) .
In practice, we cannot sum over all possible documents d .
Therefore, the most common approach is to compute the MAP estimate ˆ d = arg max ˆ p ( d ) using beam search, and then to approximate the sum over d with this single value.
This two step approach, we label it as a generate-then-read pipeline.
S TEP 1: G ENERATE .
In this step, we ﬁrst prompt a large language model (e.g., InstructGPT (Ouyang et al., 2022)) to generate documents based on the given question.
For example, the input to the language model could be “Generate a background document to answer the given question.
{ question placeholder } ”.
We can use any decoding strategy (e.g., greedy decoding, beam search), but we used greedy decoding throughout the zero-shot experiments for simplicity and reproducibility. [Figure]
S TEP 2: R EAD .
In the second step, we use generated sentence ˆ d along with the input question to produce the ﬁnal answer from the large language model.
This is actually the same setting as “zeroshot” reading comprehension, as widely studied in existing works (Brown et al., 2020; Lazaridou et al., 2022).
We choose appropriate prompts from P3 (Bach et al., 2022), such as “Refer to the passage below and answer the following question.
Passage: { background placeholder } Question: { question placeholder } ”.
Finally, the language model is fed the prompted text to generate an answer.
3. S UPERVISED S ETTING
Although large language models demonstrate impressive performance on zero-shot learning abilities, their performance still lag behind the supervised setting.
Therefore, we also explore how the generated documents from large language models can beneﬁt the supervised setting.
As directly ﬁne-tuning large language models on downstream datasets could be prohibitively expensive, we leverage a small reader model such as FiD to peruse the generated documents under the supervised setting.
Under the supervised setting, scaling the size of retrieved documents can lead to better performance (Karpukhin et al., 2020; Izacard & Grave, 2021).
This is mainly because retrieving more documents can cover more relevant information and knowledge, i.e., a higher recall score.
Nevertheless, asking a large language model to generate multiple high-quality contextual documents is a challenging task.
Dense retrieval methods can fetch multiple documents covering different perspectives of the answer.
Compared to dense retrievers, simply prompting a large language model to generate multiple contextual documents often leads to low knowledge coverage, since the contents generated by multiple decoding passes from the same input tend to be similar.
Sampling decoding methods, such as nucleus sampling 1 (Holtzman et al., 2020) can diversify the generation process to some extent, but the knowledge content of generated texts still tends to be highly repetitive when used to generate documents for a given question.
We further propose two novel solutions, including diverse human prompts and clustering-based prompts, which will be elaborated on in this section.
3.2. D IVERSE H UMAN P ROMPTS
In order to avoid similar token distributions under a single prompt, we ask human annotators to provide different prompts, in order to make the generated document diverse.
This method is simple, but can effectively vary the token distribution during generation.
In the experiments, we empirically found this method can bring improvement to the retrieval performance (Figure 2).
However, this method suffers from two drawbacks.
On one hand, it requires human annotators to write different prompts, which cannot be easily generalized to different knowledge-intensive tasks.
On the other hand, different large language models might be sensitive to different prompt words, which might cause a set of good prompt words not work on a different large language model.
3.2. C LUSTERING - BASED P ROMPTS
To increase knowledge coverage in generated documents, we propose a novel clustering-based prompt method.
It ﬁrst clusters the representations of a set of documents into K classes ( K = 2 in Figure [Table] + InstructGPT [Table]
Previous [Table] InstructGPT (no docs [Table]
can signiﬁcantly outperform the original InstructGPT, achieving new state-of-the-art performance on three open-domain QA benchmarks (previous
GLaM (Du et al., 2022)
2 FLAN (Wei et al., 2021)) under this setting without using any external document.
Our G EN R EAD can achieve comparable or even better performance than zero-shot retrieve-then-read models that use a retriever or search engine to ﬁrst obtain contextual documents.
To ensure reproducibility, we use greedy search in decoding.
All prompts used are shown in the § B.1.
1), where the number of classes is equal to the number of documents that need to be generated in the end.
Next, it randomly selects n question-document pairs ( n = 5 in Figure 1) from each cluster.
Lastly, a large language model presents the different n question-document pairs as in-context demonstrations for generating documents to a given question.
In this way, large language models are based on different distributions of examples, hence resulting in generated documents covering different perspectives.
We show this in Figure 1 and illustrate the details of each step as follows.
S TEP 1: G ET O NE I NITIAL D OCUMENT P ER Q UESTION .
Similar to the zero-shot setting, we ﬁrst ask a large language model to generate one contextual document d for each question q ∈ Q , where Q is the set of questions in the training split.
Alternatively, we can use an unsupervised retriever (e.g.,
BM25) to obtain a document from Wikipedia.
We now have a question-document pair set { q i , d i } |Q| i =1 .
S TEP 2: E NCODE EACH D OCUMENT , D O K-
MEANS C LUSTERING .
We then use a large language model (i.e., GPT-3) to encode each question-document pair, i.e., e i = GPT-
([ q i , d i ]) , resulting in a 12,288-dimensional vector per document.
Then, we use K-means to cluster all embedding vectors { e i } | Q | i =1 into K sets, so each question-document pair is assigned a unique cluster id c ∈ { 1 , ... , K } .
We vary the number of K in the experiments, which will be illustrated in Figure 2.
S TEP 3: S AMPLE AND G ENERATE K D OCUMENTS .
Lastly, we sample n question-document pairs from each cluster c , denoted as { q c 1 , d c 1 ; q c 2 , d c 2 ; ... ; q cn , d cn } , in which n is a hyperparameter 2 .
Then, the n sampled question-document pairs from the same cluster serve as in-context demonstrations for the large language model to generate a contextual document.
For example, the input to the large language model could be “ { q c 1 placeholder } { d c 1 placeholder } ... { q cn placeholder } { d cn placeholder } { input question placeholder } ”.
By enumerating the sampled documents in these K clusters, we can ﬁnally get K -generated documents.
By conditioning on different sampled in-context demonstrations collected from different clusters, the large language model has been biased for different perspectives.
Although these different perspectives exist in a latent manner, we empirically show it works well in practice, by comparing it with sampling methods, diverse human prompts (Figure 2 and Table 2) and randomly sampling n pairs from the entire dataset (Table 11).
E XPERIMENTS
In this section, we conduct comprehensive experiments on three knowledge-intensive NLP tasks, including open-domain QA (NQ (Kwiatkowski et al., 2019), TriviaQA
(Joshi et al., 2017)
and
WebQ (Berant et al., 2013)), fact checking (FEVER (Thorne et al., 2018) and FM2 (Eisenschlos et al., 2021)) and open-domain dialogue system (WoW (Dinan et al., 2019)).
More detailed dataset
R e c a ll @
K
TriviaQA
TopK documents
Google search
DPR-Multi
Ours (sampling)
Ours (human)
Ours (clustering)
R e c a ll @
K
WebQ
TopK documents
Google search
DPR-Multi
Ours (sampling)
Ours (human)
Ours (clustering) information can be found in Appendix A.1.
To evaluate the model performance, we use exact match (EM) score for evaluating open-domain QA (Zhu et al., 2021).
An answer is considered correct if and only if its normalized form has a match in the acceptable answer list.
We also employ Recall@K (R@K) as an intermediate evaluation metric, measured as the percentage of top-K retrieved or generated documents that contain the answer. This metric is commonly used in evaluations of previous works (Karpukhin et al., 2020; Izacard & Grave, 2020; Sachan et al., 2022). For other knowledge-intensive tasks, we follow the KILT benchmark (Petroni et al., 2021) to use accuracy (ACC) for fact checking and F1 / Rouge-L (R-L) score for open-domain dialogue system.
4. Z ERO - SHOT S ETTING E XPERIMENTS
We ﬁrst compare our proposed G EN R EAD approach with various large language models proposed in recent years, including GPT-3 (Brown et al., 2020), Gopher (Rae et al., 2021), FLAN (Wei et al., 2021), GLaM (Du et al., 2022), Chinchilla (Hoffmann et al., 2022), PaLM (Chowdhery et al., 2022) and InstructGPT (Ouyang et al., 2022).
Due to the space limitation, we only put the best performance on each dataset in Table 1, in which the line is called previous SoTA methods .
In addition, their corresponding model parameters and performance are listed in Table 9 in Appendix.
All of these baseline methods use the same input formats, i.e., [prompt words; question].
G EN R EAD is based on InstructGPT with 175B parameters.
In order to fully evaluate the effectiveness of our proposed method, we also compare with InstructGPT augmented with retrieved documents from Wikipedia or Google search.
The baseline methods (1) BM25 / Contriever + InstructGPT; (2) Google + InstructGPT; (3) DPR + InstructGPT have the same input format as our
G EN R EAD , i.e., [prompt words; contextual document; question].
BM25 is a traditional sparse retrieval method.
Contriever (Izacard et al., 2022a) is a state-of-the-art unsupervised dense retrieval model.
DPR (Karpukhin et al., 2020) is a supervised dense retrieval model directly trained on NQ,
TriviaQA and WebQ datasets.
We note that comparing with above three methods is challenging because our method only relies on the large language model itself, without using any external corpus.
4.1. E XPERIMENTAL R ESULTS
In the experiments, we use InstructGPT as our backbone model.
As shown in Table 1, compared with state-of-the-art large language models, our proposed G EN R EAD with the InstructGPT reader improves its performance by generating contextual documents and conditioning on the generated documents, even though no new data is introduced, and the generator and reader have the exact same parameters.
Speciﬁcally, G EN R EAD can improve the EM score by +6.9 on three open-domain
QA benchmarks, compared to the original InstructGPT.
We also make a similar observation on fact checking and open-domain dialogue system.
Our proposed G EN R EAD can consistently outperform the baseline InstructGPT model without retrieving any contextual documents.
To further validate the effectiveness of G EN R EAD , we compare against zero-shot retrieve-then-read pipeline models, which ﬁrst use a retrieval model or the Google search engine to get a relevant contextual document, then use InstructGPT to read the texts and produce the ﬁnal answer.
As shown in Table 1, G EN R EAD can achieve on-par performance with zero-shot retrieve-then-read pipeline models on the NQ and FM2 datasets, and outperform them on all other benchmarks. The knowledge [Table] [Table]
) can achieve better performance than baseline methods on TriviaQA and WebQ. Through our detailed analysis of NQ, we found the performance gap mainly due to the temporality issue, which will be elaborated in § A.8.
learned by the large language models can be retrieved via autoregressive text generation.
Without seeing any examples from these datasets, G EN R EAD can outperform using the supervised retrieval model (i.e., DPR) to recover relevant contextual documents.
4. S UPERVISED S ETTING E XPERIMENTS
We compare our proposed G EN R EAD with retrieve-then-read models, including DPR (Karpukhin et al., 2020), RAG (Lewis et al., 2020), and FiD (Izacard & Grave, 2021).
In addition, we compared with obtaining relevant documents from the internet using the Google search engine.
4.2. E XPERIMENTAL S ETUP
For our proposed method, we replace the retriever with a large language model to directly generate contextual documents.
In the experiments, we use InstructGPT (Ouyang et al., 2022).
After contextual documents are retrieved or generated, we employ a FiD reader with 770M parameter models (i.e.,
FiD-l) and 3B parameter models (i.e., FiD-xl) that are ﬁne-tuned on the training split of target datasets.
We note that we only use 10 documents during reading for the following reasons.
Why do we choose to use only 10 documents instead of 100 when reading?
As noted in Section 6.2 in DPR (Karpukhin et al., 2020) and Figure 3 in FiD (Izacard & Grave, 2021), increasing the number of documents can lead to better model performance and achieve state-of-the-art when using 100 documents.
However, there are two major drawbacks to using 100 documents during the reading step.
First, the operation is very expensive, leading to a signiﬁcant increase in memory consumption and training time.
As reported by Izacard & Grave (2021), the training process requires 64 Tesla V100 32GB running for around one day.
Second, generating documents by using a large language model is slow and expensive, so only using 10 documents can be a signiﬁcant cost saving in our method.
Therefore, in our experiments, we choose to use 10 documents during the reading process.
When using FiD-770M (i.e., FiD-large), the training process can be easily performed even on a single Tesla V100 32GB GPU.
Meanwhile, when only using 10 documents, we can also increase the size of FiD model from 770M to 3B, which takes about the same amount of GPU memory as using 100 documents on a 770M model, but at the same time signiﬁcantly shortens the training time.
We note that training T5-3B model needs a bigger cluster such as 8 Tesla V100 or A100 GPUs.
4.2. E XPERIMENTAL R ESULTS ON O PEN - DOMAIN QA
We ﬁrst use Recall@K to compare the retrieval accuracy of different models.
As shown in Figure 2,
G EN R EAD can signiﬁcantly outperform DPR and Google search for under 10 retrieved or generated
E M s c o r e
NQ
TopK documents
DPR only (K documents) DPR (K-10) + LLM (10)
E M s c o r e
TriviaQA
TopK documents
DPR only (K documents) DPR (K-10) + LLM (10)
E M s c o r e
WebQ
TopK documents
DPR only (K documents) DPR (K-10) + LLM (10)
In this experiment, we use FiD-l (i.e , FiD large) as the reader model because when the documents scale to more than -xl ( 3B) causes out of issues on A documents.
Compared to different G EN EAD variants, including nucleus sampling, human written prompts, and clustering-based prompts, clustering-based prompts achieve the best performance. At the same time, we notice that the language model inevitably has the problem that the slope of the curve decreases as the number of generated documents increases. On one hand, this is due to the similarity of token distributions when large language models generate multiple documents.
On the other hand, due to the shallow interaction characteristics of the dense retrieval model itself, the retrieved documents might not be completely relevant to the given question, so that the increase in recall might come from false positive documents, as also mentioned by Sachan et al.
As shown in Table 2, we can ﬁrst observe the FiD model performs the best among all baseline models.
Using FiD-xl with only 10 documents achieves comparable performance with using FiD-l withdocuments.
The average gap is less than 1% on three benchmarks.
Compared with both close-book models and Wikipedia-based retrieve-then-read pipelines, our proposed G EN R EAD can achieve state-of-the-art performance.
Furthermore, compared with using sampling methods to generate documents, the clustering-based prompt method can improve the EM score by +2.2 on average.
This indicates that the clustering-based prompt method is effectively increasing the knowledge coverage of generated documents, and also leading to better downstream QA performance.
We also show that G EN R EAD can outperform Google search on all benchmarks.
We observe both our method and
Google search perform worse than DPR, mainly due to the signiﬁcant portion of time-dependent questions in the dataset, which is described in the following analysis.
4.2. E XPERIMENTAL R ESULTS ON O THER T ASKS Models
[Table]
et al
[Table]
We demonstrate the experimental results in Table 3.
Under the supervised setting, G EN R EAD can achieve on par performance on the fact checking task and superior performance on the dialogue system task, indicating that large language model can be seen as a strong knowledge generator.
The main reason that G EN R EAD performs worse than the dense retriever for fact checking is that the task provides sufﬁcient semantic information to reach strong performance on this binary decision task.
So, there is a smaller semantic gap between the given factual statement and contextual documents than that of question and document pairs in open-domain QA, which is an easier retrieval setting for modern dense retrieval methods that are mainly based on vector similarity.
4. O BSERVATIONS AND E XPERIMENTAL A NALYSIS 4.3. C OMPLEMENTARITY OF G ENERATED AND R ETRIEVED D OCUMENTS
Generated documents can be combined with retrieved documents to outperform both.
Even with a very large number of retrieved documents, including few samples of generated knowledge leads to large improvements.
As shown in Table 2, merging retrieved documents with generated documents can achieve state-of-the-art performance compared to all baseline methods listed in the table.
Speciﬁcally, it can improve +5.7 averagely on three open-domain QA benchmarks compared to DPR alone, and improve +4.4 averagely compared to the large language model alone.
4.3. C OVERAGE A NALYSIS OVER A LL P OSSIBLE A NSWERS
The improvement in open-domain QA performance is due to the fact that correct answers are included more frequently in the generated text Recall@K is the most commonly used metric in existing works to measure the retrieval performance, which computes the percentage of top-K retrieved or generated documents that contain any possible answer at least once.
than in the retrieved documents.
However, as many questions contain multiple correct answers, recall@K cannot fully reﬂect the diversity of generated or retrieved documents.
Each question in the WebQ has 2.39 correct answers, 1.79 correct answers in NQ and 14.02 (including all entity alias) in the TriviaQA.
NQ and WebQ do not include alias names in the labels.
[Table]
et al., 2020)
[Table]
In this section, we also demonstrate the answer coverage performance of different models in Table 5.
Answer coverage measures
the percentage of the number of answers that are contained in the documents over all possible answers.
Coverage analysis showed that generated text tends to have lower coverage than retrieved documents because generated documents tends to have little diversity compared to retrieved documents.
To improve coverage, we propose G EN R EAD with clustering, where we include examples in the prompt from different clusters of the training data to elicit more diverse generations.
E PILOGUE
C ONCLUSION .
In this paper, we present a novel perspective for solving knowledge-intensive tasks by replacing the dense retrieval models with large language model generators.
We call it generate-thenread , which ﬁrst prompts a large language model to generate contextual documents, then read the generated document to infer the ﬁnal answer.
Notably, without retrieving any documents, it reaches 71.6 and 54.4 exact match scores on TriviaQA and WebQ, signiﬁcantly outperforming the current retrieval-reader model DPR-FiD , as well as on other two knowledge-intensive tasks.
L IMITATION AND F UTURE W ORK .
Despite the strong performance on the presented datasets, our approach is limited in its ability to update knowledge state and adapt to new domains.
A major feature of retrieve-then-read is the ability to swap in new documents when new information is learned, such as temporally more recent documents, or adding in documents from a new domain to quickly adapt to a new downstream task.
Our approach relies on a large language model to contain all this knowledge and adding new knowledge would likely require some retraining.
Future work will explore how to efﬁciently incorporate new knowledge into our generate-then-read method.
Besides, generated documents might suffer from hallucination error, resulting in incorrect predictions.
We demonstrated case study in Table 15.
Consideration in combination with recent approaches (Creswell & Shanahan, 2022) to boost generative faithfulness is a also direction worthy of future research.
A CKNOWLEDGEMENTS
This work was supported in part by NSF IIS-2119531, IIS-2137396, IIS-2142827, CCF-1901059, and ONR N00014-22-1-2507. Wenhao is supported in part by Bloomberg Data Science Ph.D Fellowship.
We are grateful to the reviewers for their insightful suggestions that have improved the quality of our paper.
Their dedication to ensuring the completeness of our research is greatly appreciated.
E THICS S TATEMENT
Large language models have a wide range of beneﬁcial applications for society, but they also have potentially harmful applications.
Previous work has shown various forms of bias, such as racial and gender bias, in large language models like GPT-3, even after explicit efforts to reduce toxic language (Chan, 2022).
The importance of addressing these societal harms is acknowledged by
OpenAI themselves in their 2020 paper introducing GPT-3 (Brown et al., 2020), which stated “we focus on two primary issues: the potential for deliberate misuse of language models like GPT-3 ... and issues of bias, fairness, and representation within models like GPT-3.”
on page 34.
The goal of this paper is to utilize knowledge stored in the parameters of large language models to answer open-domain questions and solve knowledge-intensive tasks.
Unlike retrieve-then-read where an external corpus can be curated to be trustworthy, the use of a model to generate contextual documents may further permeate existing biases in common models.
First, our work shows that generated documents suffer from challenges of stale information from outdated documents used for training.
Second, we show that generated documents tend to be less diverse, potentially biasing answers towards more common entities and terms from the training data.
Finally, we conducted experiments on only three large language models.
It is possible that some of our conclusions or observations may not necessarily hold for other models trained with different data or objectives.
Regarding ethical solutions, future work includes (i) further exploring potential bias and intentional or unintentional harm that may result from using generated contextual documents; (ii) better aligning language models with user intent to generate less biased contents and fewer fabricated facts.
A A PPENDIX
[Table]
A. D ATASETS AND S PLITS
– T RIVIA QA (TQA)
(Joshi et al., 2017) contains a set of trivia questions with answers that were originally scraped from trivia and quiz-league websites.
– W EB Q UESTIONS (WebQ) (Berant et al., 2013) consists of questions selected using Google Suggest API, where the answers are entities in Freebase.
– N ATURAL Q UESTIONS (NQ) (Kwiatkowski et al., 2019) were mined from real Google search queries and the answers are spans in Wikipedia articles identiﬁed by human annotators.
We explore the same train / dev / test splits for the open-domain QA setting as used by Izacard & Grave (2021); Karpukhin et al.
For TriviaQA, GPT-3 / GLaM / PaLM (Brown et al., 2020; Du et al., 2022; Chowdhery et al., 2022) evaluate on the Wikipedia dev set of 7,993 examples, so we ran an additional evaluation on that dev set in order to compare with their performance.
– F EVER (Thorne et al., 2018) is one of the largest datasets for fact checking that requires retrieving evidence from external corpus to support if a statement is supported or refuted.
– F OOL M E T WICE (FM2) (Eisenschlos et al., 2021) is a challenging fact checking dataset collected by gamiﬁcation.
Players write challenging claims either entailed or refuted by evidence from Wikipedia.
They are then tasked to spot the refuted claim among a group.
– W IZARD OF W IKIPEDIA (WoW) (Dinan et al., 2019) is an open-domain dialogue task for training agents that can converse knowledgeably about open-domain topics.
One speaker in the conversation must ground their utterances in a speciﬁc knowledge sentence from a Wikipedia page.
We use the same train / dev / test splits in KILT challenge (Petroni et al., 2021) for the FEVER and WoW datasets.
Their test labels are hidden, so the performance can only be evaluated through https://ai.facebook.com/tools/kilt .
For FM2, we use its ofﬁcial dataset splits.
A. I MPLEMENTATION D ETAILS
We use T5-770M (Raffel et al., 2020) and T5-3B as our backbone models to implement FiD (Izacard & Grave, 2021).
We use AdamW as the optimizer, with 2,000 warm-up steps.
We set the dropout probability to 0 .
1 and weight decay to 0 .
We use one A100 for running T5-770M and set the batch size of 16.
We use 8 A100 for running T5-3B and set the per GPU batch as 2, leading to the total batch size as 16.
We searched different learning rates, ranging from 5 e - 6 to 4 e - 5 , and we found 3 eto 6 e - 5 performed the best under the T5-3B setting and 5 e - 5 to 1 e - 4 performed the best under the
T5-770M setting.
We refer to more individual implementation details in Table 6.
We implement other baseline methods by using repositories:
– BM25: https://github.com/castorini/pyserini – DPR: https://github.com/facebookresearch/DPR – Contriever: https://github.com/facebookresearch/contriever
[Table]
NQ
TriviaQA
WebQ
FEVER
FM
WoW
[Table]
Validation
[Table]
[Table]
Validation performance
[Table]
A. R EPRODUCIBILITY VIA O PEN S OURCE L ARGE L ANGUAGE M ODELS
Documents obtained by ↓
TriviaQA
[Table]
DPR (Karpukhin et al., 2020)
[Table]
OPT (Zhang et al., 2022)
[Table]
[Table]
InstructGPT (Ouyang et al.,
[Figure]
[Table]
[Figure]
We note that reproducing experiments on the OpenAI API, though publicly available, costs money.
For this reason, we further add an evaluation on two open-source large language models OPT (Zhang et al., 2022) and
Codex (OpenAI, 2022).
As shown in Table 7, OPT performed worse than InstructGPT, but still achieved comparable performance with DPR; OpenAI Codex achieved the best performance on both TriviaQA and WebQ.
A. S CALING WITH N UMBER OF L ARGE L ANGUAGE M ODEL P ARAMETERS
[Figure]
Figure 4 shows the scaling of performance with InstructGPT generator parameters, including Ada-150M,
Babbage-1.3B, Curie-6.7B and Davinci-175B. We note that for both FiD and our G EN R EAD , we use the
FiD-xl with 10 input documents either retrieved from
Wikipedia or generated by InstructGPT.
The performance of both TriviaQA and WebQ continues to improve as the generator model parameters increase, as does the slope.
Only with the largest size InstructGPT,
G EN R EAD can outperform the DPR-FiD .
This indicates using large language model to generate contextual documents is an “emergent ability” of scaling, which is not present in smaller models but is only present in larger language models (Wei et al., 2022a).
A. R EADABILITY A NALYSIS OF R ETRIEVED AND G ENERATED D OCUMENTS
[Table]
After we manually compare some retrieved documents and
- erated documents
InstructGPT, we observe that the readability of different documents, when they contain the correct answer string, is different.
In other words, documents containing answers might also contain noisy information that is irrelevant to the question, which could affect both the model and human reading.
In order to further validate the readability of retrieved documents and generated documents, we extracted a subset of data examples from NQ, TriviaQA and WebQ datasets, in which both retrieved and generated documents contain the correct answer.
As shown in Table 8, when both retrieved and generated documents contain the correct answer, the FiD reader can produce more correct answers when reading the generated documents from large language models (e.g., InstructGPT).
We also provide some case studies in Tables 16-19.
For example, in Table 18, the question is “What city was Zeus the patron god of?”.
The ﬁrst document retrieved by DPR is “Like the other
Panhellenic Games, the ancient Olympic Games were a religious festival, held at the sanctuary of
Zeus at Olympia.”.
Although it contains the correct answer, it is hard to infer the answer “Olympia” from it.
On the contrary, InstructGPT generates the document “Zeus was the patron god of the city of
Olympia, which was located in the northwestern Peloponnese region of Greece.
Olympia was the site of the Olympic Games, held every four years in honor of Zeus.”, which is much easier to read.
A. A DDITIONAL N UMBERS FOR T ABLES IN THE M AIN P APER
– Table 9 contains additional evaluation results for Table 1.
It demonstrates zero-shot open-domain QA performance, compared to recent large language model.
– Figure 5 contains additional retrieval performance evaluation for Figure 3 of experiments on combining DPR retrieved documents and large language model generated document.
– Table 10 contains additional retrieval performance evaluated by Recall@K of baselines and different G EN R EAD variants.
Some numbers in the table overlaps with those in Figure 2.
Models
[Table]
Wei et
[Table]
[Table]
Hoffmann
[Table]
InstructGPT
[Table]
[Figure] [Figure]
Models
TriviaQA
WebQ
NQ
R@1 R@
R@20 R@1 R@
 R@20 R@1 R@10 R@
BM25 (Robertson et al., 2009) 46.
Contriever (Izacard et al., 2022a) 34.
DPR (Karpukhin et al., 2020) 53.
Google Search engine API 50.
G EN R EAD (nucleus, p=.95) 65.
G EN R EAD (10 human prompts) 65.
G EN R EAD (clustering prompts) 69.
Some numbers in the table overlaps with those in Figure 2.
The table aims to show the performance of more methods, and to provide accurate recall numbers for future research comparisons.
[Table]
A. D ISCUSSION ON I NFERENCE C OST OF DPR AND I NSTRUCT GPT
We now compare the costs of using DPR and InstructGPT to retrieve or generate contextual documents.
We consider DPR using the BERT-base (Devlin et al., 2019) version with 110M parameters and
InstructGPT using its largest version with 175B parameters.
For simplicity, we use the FLOPs-pertoken estimates for Transformer-based language models, which is introduced by Kaplan et al.
It should be noted that FLOPs are not a direct measure of real-world computing costs, as latency, power consumption, and other costs can vary widely based on other factors (Liu et al., 2022a).
For the DPR model, all Wikipedia documents (around 21M) only need to be encoded once.
Therefore, as the number of input questions increases, the marginal computational cost gradually decreases.
For fair comparison, we ﬁrst use DPR to encode all 21M Wikipedia documents once.
Encoding all
Wikipedia documents requires 110 e 6 (BERT-base parameters) × 21 e 6 (total number of documents) × 100 (tokens per document)
= 2 .
3 e 17 FLOPs.
When the embedding of all candidate documents are produced, retrieving documents for a given question requires 110 e 6 (BERT-base parameters) ×(tokens per question)
+21 e 6 (total number of documents)
×
( 768 + 768 − 1 ) = 3 .
2 e 10 FLOPs.
For InstructGPT, it requires 175 e 9 (InstructGPT parameters) × 10 (number of documents) ×(generated tokens per document)
= 9 .
6 e 13 FLOPs to generate 10 documents for a given question.
Therefore, the equation for the total cost Y DPR-cost to retrieve 10 documents using DPR versus the number of input questions X is: Y DPR-cost = 3 .
e 10 X
+ 2 . 3 e 17 .
Besides, the equation for the total cost Y GPT3-cost to generate 10 documents using InstructGPT versus the number of input questions X is: Y GPT3-cost = 9 .
e 13 X .
When Y DPR-cost = Y GPT3-cost , X ≈ 2473 .
In conclusion, if the number of input questions is less than 2473 , the total cost of InstructGPT is lower than the DPR; if the number of input questions is greater than 2473 , the total cost of InstructGPT exceeds the DPR.
A. E RROR A NALYSIS AND C ASE S TUDIES ON THE NQ DATASET
As stated in Zhang & Choi (2021), NQ contains a signiﬁcant proportion, roughly 16.5%, of questions that have time-dependent answers.
Similarly, Izacard et al.
(2022b) observed using the latest version of Wikipedia (12 / 2021) could lead to 4.4 drops of the EM score, compared to the Wikipedia version (12 / 2018) that the NQ questions are created from.
We provide case studies in Table 13 in Appendix.
We did case studies of 100 examples from the NQ dataset.
The results are shown in Table 12.
Among these 100 examples, we found that 29 examples have data collection and annotation mistakes, mainly including the temporal question issue (13 / 29) and the incomplete answer issue (16 / 29).
A typical temporal-dependent question is that no speciﬁc time condition is provided.
For example, “Who won the MVP for the National League?”
could have different answers in different years.
In 2017, the
MVP is Giancarlo Stanton, and in 2018, the MVP is Christian Yelich.
Besides, some answer labels provided in the NQ dataset are not complete.
For example, person names in the NQ dataset usually consist of ﬁrst, middle, and last names, but most names in the generated documents are ﬁrst and last names.
For the question “who played lionel in as time goes by?”, the labeled answer is “Geoffrey
Dyson Palmer”.
DPR-FiD produces “Geoffrey Dyson Palmer” but G EN R EAD produces “Geoffrey
Palmer”, both of which should be considered correct.
More examples are provided in Table 14.
Besides, G EN R EAD produced correct answers for 49 questions.
Among the 22 incorrect predictions, 12 of them could be classiﬁed as retrieval errors (i.e., step-I error) and 12 as reading errors (i.e., step-II error).
In all cases of retrieval errors, none of the generated documents contain the correct answer.
In all cases of reading errors, at least one generated document contains the correct answer but the reader model failed to infer the correct answer from the documents..
[Table]
- Query: Who got the ﬁrst Nobel Prize in Physics?
- Document: The ﬁrst Nobel Prize in Physics was awarded in 1901 to Wilhelm Conrad R¨ontgen for his discovery of the remarkable rays subsequently named after him.
- Predicted answer: Wilhelm Conrad R¨ontgen - Correct answer: Wilhelm Conrad R¨ontgen
Wrongretrieval
[Table]
[Table]
) - Query: Who died in the ﬁrst episode of Stranger Things? -
Document: In the ﬁrst episode of Stranger Things, the character Will Byers dies.
He is killed by Demogorgon, a monster from the Upside Down.
- Predicted answer: Will Byers - Correct answer: Benny Hammond
No hit answers (4%) - Query: When was coffee ﬁrst made into a drink? - Document: The history of coffee goes back to the 10th century, with coffee trees native to Ethiopia.
The earliest substantiated evidence of either coffee drinking or knowledge of coffee tree is from sixth century AD in Ethiopia.
- Predicted answer: the 10th century - Correct answer: the 15th century
[Table]
- Query: When is the fourth movie of the Divergent series coming out?
- Document: The fourth movie in the Divergent series was originally scheduled to be released in June 2017, but was delayed indeﬁnitely.
- Predicted answer:
June- Correct answer: never made
[Table]
- Query: Who won the MVP for the National League?
- Document:
In 2017, the National League MVP was won by Giancarlo Stanton of the Miami Marlins.
In 2018, the
National League MVP was won by Christian Yelich of the
Milwaukee Brewers.
- Predicted answer: Christian Yelich - Correct answer: Giancarlo Stanton
[Table]
- Query: Where do the greasers live in the Outsiders?
- Document: The Outsiders is a novel by S.E. Hinton.
It is about a gang of greasers in Oklahoma in the 1960s.
The
National League
MVP was won by Christian Yelich of the Milwaukee Brewers.
- Predicted answer: Oklahoma - Correct answer: Tulsa, Oklahoma
[Table]
Zhang & Choi (2021), which contains a subset of NQ data examples with time-dependent questions. [Table] as
[Table]
goes by
[Table]
Explanation: The labeled answer is “Geoffrey Dyson Palmer”, however, “Geoffrey Palmer” is also correct.
DPR retrieved documents: Geoffrey Dyson Palmer, (born 4 June 1927) is an English actor known for his roles in British television sitcoms playing Jimmy Anderson in “The Fall and Rise of Reginald Perrin”,
Ben Parkinson in “Butterﬂies” and Lionel Hardcastle in “As Time Goes By”.
His ﬁlm appearances include “A Fish Called Wanda”, “The Madness of King George”, “Mrs. Brown”, and “Tomorrow Never Dies”.
GPT generated documents: As Time Goes By is a British sitcom that aired on BBC One from 1992 to 2005.
The show starred Geoffrey Palmer and Judi Dench as Lionel and Jean Pargetter, a middle-aged couple who reunite after many years apart.
Lionel was played by Palmer, who was also a writer on the show.
Q
: How many cracker barrels in the united
states?
[Table]
Explanation: The labled answer is “639” or “over 600”, so “over 630” is also a reasonable answer.
Q
Where do the greasers live in the outsiders?
[Table]
Explanation: The labled answer is “Tulsa, Oklahoma”, but “Oklahoma” is also a correct answer.
DPR retrieved documents: The movie received mostly positive reviews from critics, and performed well at the box ofﬁce, grossing 33 million on a 10 million budget.
In Tulsa, Oklahoma, greasers are a gang of tough, low-income working-class teens.
They include Ponyboy Curtis and his two older brothers, Sodapop and Darrel, as well as Johnny.
GPT generated documents: The Outsiders is a novel by S.E. Hinton.
It is about a gang of greasers in
Oklahoma in the 1960s.
The greasers are from the poor side of town and constantly in trouble with the law.
Q
Where are unipolar neurons found in spinal cord?
the granule region
[Table]
Explanation: The labled answer is “the
Question: Who wrote the ﬁrst declaration of human rights?
Answer: Cyrus Cylinder
Generated document: The ﬁrst declaration of human rights was the Virginia Declaration of Rights, which was written by George Mason in 1776.
Retrieved document: John Peters Humphrey, OC (April 30, 1905 - 2013 March 14, 1995) was a
Canadian legal scholar, jurist, and human rights advocate.
He is most famous as the author of the ﬁrst draft of the Universal Declaration of Human Rights. ... ...
(Wikipedia page: John Peters Humphrey)
Golden Wikipedia evidence: - George Mason’s writings, including substantial portions of the Fairfax Resolves of 1774, the Virginia
Declaration of Rights of 1776, and his Objections to this Constitution of Government (1787) opposing ratiﬁcation, have exercised a signiﬁcant inﬂuence on American political thought and events.
Mason prepared the ﬁrst draft of the Virginia Declaration of Rights in 1776, and his words formed much of the text adopted by the ﬁnal Revolutionary Virginia Convention.
(Wikipedia page: George Mason) -
The Cyrus Cylinder was dubbed the “ﬁrst declaration of human rights” by the pre-Revolution Iranian government, a reading prominently advanced by Shah Mohammed Reza Pahlavi, in a 1967 book, The
White Revolution of Iran.
(Wikipedia page: Cyrus Cylinder)
Hallucination: In GPT generated documents, ﬁrst declaration of human rights was considered as
Virginia Declaration of Rights , which is factually incorrect.
This error might occur because Wikipedia page of George Mason contains similar descriptions, causing the model to confuse their meanings.
Model predictions: DPR-FiD John Peters Humphrey; G EN R EAD George Mason
Question: Who died in the ﬁrst episode of stranger things Answer: Benny or Barb
Generated document: In the ﬁrst episode of Stranger Things, the character Will Byers dies.
He is killed by the Demogorgon, a monster from the Upside Down.
Retrieved document: ... ... was the character of Barb, the nerdy friend and classmate of Nancy who is taken and killed by the monster early in the season.
According to actress Shannon Purser, Barb ¨wasn’t supposed to be a big deal¨, and the Duffer Brothers had not gone into great detail about the character since the focus was on ﬁnding Will. ... ...
(Wikipedia page: Stranger Things)
Golden Wikipedia evidence:
One area of focus from these fans was the character of Barb, the nerdy friend and classmate of Nancy who is taken and killed by the monster early in the season.
According to actress Shannon Purser,
Barb ¨wasn’t supposed to be a big deal¨, and the Duffer Brothers had not gone into great detail about the character since the focus was on ﬁnding Will. ... ...
(Wikipedia page: Stranger Things)
The owner, Benny, learns from a tattoo of ”011” on her arm that her name is Eleven.
Brenner, monitoring the phone lines, sends agents to the diner after Benny calls social services.
The agents kill
Benny, but Eleven manages to escape using telekinetic abilities.
Joyce’s phone short circuits after receiving a mysterious phone call that she believes is from Will.
While searching for Will in the woods, Mike, Dustin and Lucas come across Eleven.
(Wikipedia page: Stranger Things)
Hallucination: In GPT generated documents, Will Byers is said to be dead, which is actually incorrect.
Will Byers is actually a fake deaths in season 1 and he is still alive at the end of the season.
Model predictions: DPR-FiD Barb; G EN R EAD Will
Question: Who played the judge in good will hunting?
Answer: James P. Flynn
Generated document: The character of the Judge in Good Will Hunting was played by actor
Robert
Sean Leonard.
Leonard is best known for his role as Dr. James Wilson in the television series House.
Retrieved document: Janitor could show such facility with matrices applied this way.
Good
Will
Hunting Good Will Hunting is a 1997 American drama ﬁlm, directed by Gus Van Sant, and starring
Robin Williams, Matt Damon, Ben Afﬂeck, ... ...
(Wikipedia page: Good Will Hunting)
Golden Wikipedia evidence: - James P. Flynn appeared in many ﬁlms shot in the New England area.
Flynn was cast as a judge in the Boston-based ﬁlm
Good Will Hunting in 1997.
Later, he acted in the 1999 ﬁlm
The Cider House
Rules and What’s the Worst That Could Happen?
in 2001.
(Wikipedia page: James P. Flynn) - Cast: James P. Flynn as Judge George H. Malone (Wikipedia [Table]
Hallucination: In GPT generated documents, Robert Sean Leonard is considered as a judge in Good
Will Hunting , which is factually incorrect.
This error leads to wrong answer prediction.
[Table]
Query: What type of music did Mozart composed?
Answers: opera, classical music, art song, chamber music, ballet
DPR #1: The following is a list of surviving works that Mozart composed for performance at gatherings of Masons.
The story and music of his opera “The Magic Flute” is also considered to have strong
Masonic inﬂuences.
The following is a partial list of family members, patrons, and colleagues who were
Masons.
Mozart’s grandfather Johann Georg, a bookbinder, was raised among the extended Mozart family in Augsburg, in the house of Johann’s own grandfather David Mozart.
David and his children were distinguished architects and master ...
(Wikipedia id: 10988168 – “Mozart and Freemasonry”)
DPR #2: Progressively, and in large part at the hands of Mozart himself, the contrapuntal complexities of the late Baroque emerged once more, moderated and disciplined by new forms, and adapted to a new aesthetic and social milieu.
Mozart was a versatile composer, and wrote in every major genre, including symphony, opera, the solo concerto, chamber music including string quartet and string quintet, and the piano sonata.
These forms were not new, but Mozart advanced their technical sophistication and emotional reach.
He almost single-handedly developed and popularized the Classical piano concerto.
He wrote a great deal ...
(Wikipedia id: 451968 – “Wolfgang Amadeus Mozart”)
Among top-10 retrieved documents: opera: 2 times chamber music: 1 time
GPT nucleus
#1: Mozart was a proliﬁc composer and wrote in many different genres.
He composed symphonies, operas, concertos, chamber music, and more.
Many of his works are considered some of the greatest ever written in those genres.
GPT nucleus
#2: Mozart composed a variety of music, including operas, symphonies, chamber music, concertos, and sonatas.
His early works were inﬂuenced by the music of Haydn and Bach, and his later works show the inﬂuence of the French and Italian composers of his time.
Mozart’s music is characterized by its beauty, elegance, and precision, and he is considered one of the greatest composers of all time.
GPT nucleus
#3: Mozart composed a wide variety of music, including operas, symphonies, concertos, solo pieces, and chamber music.
He also wrote a number of religious works, including masses and motets.
Among top-10 generated documents: chamber music: 9 times
GPT cluster #1: Mozart found work in Vienna as a piano teacher and composer for opera and other works.
There he met a fellow Mason, Joseph Haydn, who was to become an important mentor.
Mozart married
Constanze Weber in 1782; their marriage is thought to have been happy, in contrast to his earlier unhappy arranged marriage to his cousin, Maria Anna ¨Nannerl ¨Mozart.
Mozart composed more than 600 works, including works widely.
GPT cluster #2: Mozart was a proliﬁc composer, and wrote in a wide variety of genres.
He composed over 600 works, including operas, symphonies, concertos, solo works, and chamber music.
His operas include The Marriage of Figaro, Don Giovanni, and Cosi fan tutte.
His symphonies include the well-known
Symphony No. 40 in G minor, and his concertos include the Piano Concerto No. 21 in C major, known as the ¨Elvira Madigan¨concerto.
GPT cluster #3: Mozart was a proliﬁc composer who wrote music in a wide variety of genres.
He is best known for his work in the classical genre, which includes such works as his Symphony No. 40 in G minor and his piano concertos.
He also wrote operas, including The Marriage of Figaro and The Magic Flute.
In addition to his work in classical music, Mozart also composed works in the popular genre, such as his Eine kleine Nachtmusik.
Among top-10 generated documents: opera: 2 times chamber music: 7 times classical music: 1 time
Query: What does Monsanto own?
Answers: agricultural chemicals, seed, agriculture, chemical industry
DPR #1:
In September 2016, Monsanto agreed to be acquired by Bayer for US$66 billion.
In an effort to receive regulatory clearance for the deal, Bayer announced the sale of signiﬁcant portions of its current agriculture businesses, including its seed and herbicide businesses, to BASF.
The deal was approved by the European Union on March 21, 2018, and approved in the United States on May 29, 2018.
The sale closed on June 7, 2018; Bayer announced its intent to discontinue the Monsanto name, with the combined company operating solely under the Bayer brand. ...
(Wikipedia id: 935126 – “Monsanto”)
DPR #2: Behind it all is of course an evil corporation: Monsanto.
GMO conspiracy theories GMO conspiracy theories are conspiracy theories related to the production and sale of genetically modiﬁed crops and genetically modiﬁed food (also referred to as genetically modiﬁed organisms or “GMOs”).
These conspiracy theories include claims that agribusinesses, especially Monsanto, have suppressed data showing that GMOs cause harm, deliberately cause food shortages to promote the use of GM food, or have co-opted government agencies ...
(Wikipedia id: 19304897 – “GMO conspiracy theories”)
Among top-10 retrieved documents: agriculture: 1 time seed: 2 times
GPT nucleus #1: Monsanto is an American agricultural biotechnology corporation founded in 1901.
It is headquartered in Creve Coeur, Missouri.
Monsanto is a leading producer of genetically engineered seed and Roundup, a glyphosate-based herbicide.
GPT nucleus #2: Monsanto is a multinational agrochemical and agricultural biotechnology corporation headquartered in Creve Coeur, Missouri.
It is one of the world’s leading producers of Roundup, a glyphosatebased herbicide.
The company also manufactures genetically engineered seed and Roundup Ready crops, including corn, soybeans, and cotton.
As of 2018, Monsanto is the world’s largest producer of glyphosate.
In 2018, Bayer acquired Monsanto for $63 billion.
Among top-10 generated documents: seed: 6 times
GPT cluster #1: Monsanto Company is an American multinational agrochemical and agricultural biotechnology corporation headquartered in Creve Coeur, St. Louis, Missouri.
It is a leading producer of genetically engineered (GE) seed and of the herbicide glyphosate, which it markets under the Roundup brand.
Founded in 1901 by John Francis Queeny, Monsanto spent most of the 20th century as a chemical company, becoming one of the world’s leading manufacturers of plastics, resins, and other chemicals.
GPT cluster #2: Monsanto is a biotechnology and agriculture company that specializes in genetically modiﬁed organisms (GMOs).
The company owns a number of patents on GMOs, and it produces a range of products that contain GMOs, including herbicides, pesticides, and seeds.
GPT cluster #3: Monsanto is a multinational agricultural biotechnology corporation.
It is one of the world’s leading producers of genetically engineered seed and Roundup, a glyphosate-based herbicide.
The company also manufactures other agricultural chemicals, such as insecticides and fungicides.
Among top-10 generated documents: agriculture: 2 times seed: 5 times agricultural chemicals: 1 time
Query: What city was Zeus the patron god of?
Answers: Olympia, Elis
DPR #1: Many were speciﬁc only to a particular deity or city-state.
For example, the festival of Lykaia was celebrated in Arcadia in Greece, which was dedicated to the pastoral god Pan.
Like the other Panhellenic
Games, the ancient Olympic Games were a religious festival, held at the sanctuary of Zeus at Olympia.
Other festivals centred on Greek theatre, of which the Dionysia in Athens was the most important.
More typical festivals featured a procession, large sacriﬁces and a feast to eat the offerings, and many included entertainments and customs such as visiting friends, wearing fancy dress and unusual behaviour in the ...
(Wikipedia id: 1945872 – “Ancient Greek religion”)
DPR #2: Theatre of Dionysus The Theatre of Dionysus Eleuthereus is a major theatre in Athens, considered to be the world’s ﬁrst theatre, built at the foot of the Athenian Acropolis.
Dedicated to Dionysus, the god of plays and wine (among other things), the theatre could seat as many as 17,000 people with excellent acoustics, making it an ideal location for ancient Athens’ biggest theatrical celebration, the Dionysia.
It was the ﬁrst theatre ever built, cut into the southern cliff face of the Acropolis, and supposedly the birthplace of Greek tragedy.
The remains of a restored and redesigned Roman version can still ...
(Wikipedia id: 3732515 – “Theatre of Dionysus”)
Among top-10 retrieved documents: Olympia: 1 time
GPT nucleus #1: Zeus was the patron god of the city of Olympia in Greece.
GPT nucleus
#2: The city of Olympia in Greece was the site of the ancient Olympics, and it was also the home of Zeus, the patron god of the Olympics.
Among top-10 generated documents: Olympia: 6 times
GPT cluster #1: Zeus was the patron god of the city of Olympia, which was located in the northwestern
Peloponnese region of Greece.
Olympia was the site of the Olympic Games, which were held every four years in honor of Zeus.
GPT cluster #2: Zeus was the patron god of the city of Olympia.
Olympia was a sanctuary of ancient
Greece in Elis on the Peloponnesos peninsula, in south-western Greece.
The sanctuary was known as the
Panhellenic Sanctuary of Zeus, and the altar of Zeus was located in the Temple of Zeus.
The Olympic
Games were held in Olympia every four years from 776 BC to 393 AD.
GPT cluster #3: Zeus was the patron god of the city of Olympia.
[Table]
Query: What is the collection of the districts to the east of the Jordan River?
Answers: Jordan, Golan Heights
DPR #1: Water politics in the Jordan River basin Water politics in the Jordan River basin refers to political issues of water within the Jordan River drainage basin, including competing claims and water usage, and issues of riparian rights of surface water along transnational rivers, as well as the availability and usage of ground water.
Water resources in the region are scarce, and these issues directly affect the ﬁve political subdivisions (Israel, the West Bank, Lebanon, Syria and Jordan) located within and bordering the basin, which were created since the collapse, , during World War I, of the former single controlling entity, the ...
(Wikipedia id: 12822413 – “Water politics in the Jordan River basin”)
DPR #2: Jordan’s western border follows the bottom of the rift.
Although an earthquake-prone region, no severe shocks had been recorded for several centuries.
By far the greatest part of the East Bank is desert, displaying the land forms and other features associated with great aridity.
Most of this land is part of the
Syrian Desert and northern Arabian Desert.
There are broad expanses of sand and dunes, particularly in the south and southeast, together with salt ﬂats.
Occasional jumbles of sandstone hills or low mountains support only meager and stunted vegetation that ...
(Wikipedia id: 210052 – “Geography of Jordan”)
Among top-10 retrieved documents: Jordan: 8 times
GPT nucleus
#1: The collection of the districts to the east of the Jordan River is known as the West Bank.
The West Bank is a landlocked territory situated in Western Asia.
It is bordered by Jordan to the east, Egypt to the southwest, and Israel to the west and north.
GPT nucleus #2: The collection of the districts to the east of the Jordan River is the West Bank.
The West
Bank is a landlocked territory near the Mediterranean coast of Western Asia, bordered by Lebanon to the north, Syria to the northeast, Jordan to the east, and Egypt to the southwest.
It is the smaller of two political entities that comprise the Palestinian territories, the other being the Gaza Strip.
Among top-10 generated documents: Jordan: 10 times
GPT cluster #1:
The collection of the districts to the east of the Jordan River is known as the West Bank.
The West Bank is a landlocked territory situated in Western Asia.
It is bordered by Jordan to the east, Egypt to the southwest, and Israel to the west and north.
GPT cluster #2: The Jordan River is a major river in the Middle East, the source of which is in the northern part of Israel.
The river ﬂows southward through Israel and then Jordan, emptying into the Dead Sea.
East of the river is the collection of districts known as the East Bank.
GPT cluster #3: There is no single answer to this question as the east bank of the Jordan River is home to a number of different districts and regions, each with its own unique history, culture, and customs.
However, some of the more well-known districts on the east bank include the West Bank, the Gaza Strip and the Golan Heights.
Among top-10 generated documents: Jordan: 10 times Golan Heights: 2 times
B P ROMPTS C HOICES B. Z ERO - SHOT L EARNING P ROMPTS ( FOR T ABLE 1) B.1. P ROMPTS FOR ”I NSTRUCT GPT ( NO DOCS .)”
We observed the prompts (i.e., “Q: { query }\ n \ nA:”) used in GPT-3 paper (Brown et al., 2020) perform poorly on its text - davinci - 002 version.
Therefore, we experimented with multiple prompts and found the following two prompts work best on open-domain QA datasets.
– (1) “ { query }\ n \
nThe answer is” (no space between { query } and \ n) – (2) “ { query } \ n \ n
The answer is” (performance reported in Table 1)
For fact checking and dialogue system, we used the following prompts.
– Fact Checking “ { claim } \ n \ n Is the claim true or false?”
– Open-domain Dialogue System “ { query } \ n \ n”
B.1. P ROMPTS FOR B ACKGROUND G ENERATION (S TEP -1)
– Open-domain Question Answering “Generate a background document from Wikipedia to answer the given question.
\ n \ n { query } \ n \ n” – Fact checking “Generate a background document from Wikipedia to support or refute the statement.
\ n \ n Statement: { claim } \ n \ n” – Open-domain Dialogue System “Generate a background document from Wikipedia to answer the given question.
\ n \ n { utterance } \ n \ n”
B.1. P ROMPTS FOR R EADING C OMPREHENSION (S TEP -2)
We collected the prompt from P3 (Bach et al., 2022), which includes over 2,000 open-source prompts for roughly 170 datasets.
For zero-shot QA, we experimented with three different reading comprehension prompts.
We reported the performance for each prompt in Table 20.
– (1) “Refer to the passage below and answer the following question with just a few
words.
Passage
{ background } \ n \ n Question: { query } \ n \ n The answer is” – (2) “Passage: { background } \ n \ n Question: { query } \ n \ n Referring to the passage above, the correct answer (just one entity) to the given question is” – (3) “Refer to the passage below and answer the following question with just one entity.
\ n \ n Passage: background \ n \ n Question: query \ n \ n The answer is”
For fact checking and dialogue system, we chose the simplest prompt from P
– Fact Checking “ { background } \ n \ n claim: { claim } \ n \ n Is the claim true or false?”
– Open-domain Dialogue System “ { background } \ n \ n utterance \ n \ n”
[Table]
. H UMAN P ROMPT A NNOTATIONS ( FOR S ECTION 3.2.1)
In order to get a better prompt for large language models to generate better contextual documents, we asked 30 students in the computer science department to write different prompts.
We ﬁrst constructed a small validation set with 200 examples by combining 50 random question-answer pairs from NQ, 100 random pairs from TriviaQA and 50 random pairs from WebQ. When an annotator wrote down a prompt, our system can immediately evaluate the prompt by using the validation set and return the performance to the annotator.
Then, the annotator can modify the previous prompt until the recall performance reaches a threshold, which is set as 50 in our experiments.
Finally, we got 29 prompts from human annotators due to two of them are the same.
We used the top-10 prompts (shown in
Table 21 and Table 22) in the human prompt setting, as described in § 3.2.1.
[Table] [Table]
Generate a background document from Wikipedia to answer the given question
[Table]
Generate a background document from web to answer the given question.
[Table]
Generate a Wikipedia document to support the given question
[Table]
Provide [Table] background document for the given question
[Table]
Prepare a background document to support the given question.
[Table]
To support the given question, prepare a background document.
[Table]
Create a background document that supports the
[Table] [Table]