CCQA: A New Web-Scale Question Answering Dataset for Model Pre-Training
Abstract
We propose a novel open-domain questionanswering dataset based on the Common
Crawl project.
With a previously unseen number of around 130 million multilingual question-answer pairs (including about 60 million English data-points), we use our largescale, natural, diverse and high-quality corpus to in-domain pre-train popular language models for the task of question-answering.
In our experiments, we ﬁnd that our Common Crawl
Question Answering dataset (CCQA) achieves promising results in zero-shot, low resource and ﬁne-tuned settings across multiple tasks, models and benchmarks 1 .
Introduction
Open-domain question-answering (ODQA) has evolved into a core problem in Natural Language Processing (NLP), receiving growing interest from the research community (Raffel et al., 2020; Roberts et al., 2020).
Despite the notoriously difﬁcult challenge to correctly answer opendomain questions on arbitrary topics, recent advances of pre-trained language models (such as
BERT (Devlin et al., 2019), BART (Lewis et al., 2020a) and T5 (Raffel et al., 2020)) have stimulated new research into additional, task-dependent pre-training steps.
Speciﬁcally, recent publications show that in-domain pre-training regimes can improve models for several downstream tasks (Gururangan et al., 2020).
For open-domain questionanswering, newly proposed pre-training tasks such as the Inverse Cloze Task (ICT) (Lee et al., 2019),
Body First Selection (BFS), Wiki Link Prediction (WLP)
(Chang et al., 2020) and Question Answering Infused Pre-training (QUIP) (Jia et al., 2021) show consistent improvements over baselines.
However, most of these approaches still rely on either unlabeled text, or synthetically generated question-answer (QA) pairs.
In this paper, we explore a second, somewhat orthogonal dimension to these lines of work, examining if a web-scale collection of natural QA pairs can support ODQA through in-domain pre-training.
Per deﬁnition, an ODQA system should be able to answer any question from an arbitrary domain.
We believe that to approach this ability with indomain pre-training, a suitable dataset should address the following 5 challenges: (1) Size; ODQA requires knowledge of a wide variety of topics.
The underlying dataset used for in-domain pretraining hence needs to cover this abundance of domains, requiring a web-scale dataset.
(2) Naturalness; While synthetic corpora can potentially capture a wide variety of language phenomena, to understand and generate truly natural language in all facets, synthetic datasets are not sufﬁcient.
(3) Quality; Given the requirement for a diverse, large-scale dataset, high data quality in terms of cleanliness and sensibility becomes a major challenge.
Given that web-scale data sources require highly automated approaches operating on noisy data, assuring data quality is non-trivial.
(4) Diversity; Besides size, another challenge for any
ODQA in-domain pre-training dataset is the generality of the corpus.
The dataset needs to support answering many diverse questions to allow models to learn general concepts.
(5) Evaluation Fairness;
A web-scale question-answering dataset potentially overlaps with existing benchmark corpora, leading to inﬂated performance measures and impeding the evaluation fairness (Lewis et al., 2021a).
To overcome these challenges, we propose a new large-scale dataset for open-domain questionanswering called the C ommon C rawl Q uestion
A nswering (CCQA) dataset.
Similar to popular datasets, such as C4 (Raffel et al., 2020), CCNet (Wenzek et al., 2020), CC-100 (Conneau et al., 2020), HTLM (Aghajanyan et al., 2022b), and CM3 (Aghajanyan et al., 2022a) we generate a large-scale, diverse and high-quality questionanswering dataset from Common Crawl.
More speciﬁcally, Common Crawl allows us to obtain a large number of truly natural questionanswer pairs, asked and answered by real humans on the web, rather than inferred through computational methods.
Using the abundantly available schema.org question annotation 2 , we generate question-answer pairs from explicit annotations, instead of heuristic rules, leading to high-quality data points.
In a large set of evaluations, we show that indomain pre-training on our CCQA dataset achieves promising results across different settings, models and benchmarks.
Using the rich information available on the web, we augment our dataset with additional data attributes beyond just questionanswer pairs, such as votes, multiple (competing) answers, question summaries and intra-textual
HTML markup, which can be used for a variety of tasks beyond question-answering in future work.
Furthermore, we evaluate the diversity and evaluation fairness of our dataset by computing topic distributions and train-test overlaps with benchmark datasets, providing additional rationale regarding the quality of our data and experiments.
To summarize, our main contributions in this paper are as follows:
• We generate the ﬁrst truly large-scale, natural question-answering dataset, containing around 130 million unﬁltered question-answer pairs ( 55 M unique pairs), including about 60 million English data points ( 24 M unique pairs).
• We present key dataset statistics, conﬁrming the high quality of our question-answer pairs, the wide range of diverse topics and a low overlap with existing benchmarks.
• We show the effectiveness of the dataset for in-domain pre-training by evaluating the performance of the unﬁltered English subset on two question-answering tasks, three different settings, four models and ﬁve diverse benchmarks.
Related Work
This work is inspired by a range of previous approaches using Common Crawl web-data, such as the Colossal Clean Crawled Corpus (C4) for language model pre-training (Raffel et al., 2020),
the word/sentence representation generation corpus CCNet (Wenzek et al., 2020), the CCdataset for translation (Conneau et al., 2020) and the markup-style language modelling HTLM corpus for zero-shot summarization (Aghajanyan et al., 2022b).
Despite all previously mentioned applications directly relying on large-scale web data from Common Crawl, their scope and application vary signiﬁcantly.
Compared to previously proposed datasets based on Common Crawl, we are the ﬁrst to extract well-structured question-answer pairs with additional meta-data, making our corpus a valuable resource for ODQA research, and a multitude of related tasks, such as question summarization, answer rating, and answer ranking.
Further web-based datasets outside the Common
Crawl domain are the TriviaQA (Joshi et al., 2017) and ELI5 corpora (Fan et al., 2019), extracting small-scale question-answer datasets from Trivia websites and Reddit threads respectively.
The large-scale GooAQ dataset (Khashabi et al., 2021) is similarly based on web data, however exploits the Google auto-complete feature and related answer boxes to generate semi-synthetic questionanswer pairs.
As a large-scale, completely syn- thetic dataset, the PAQ corpus (Lewis et al., 2021b) automatically generates a large set of Probably
Asked Questions from Wikipedia articles.
In contrast to these previously proposed datasets, our
CCQA corpus presents a large-scale, natural and diverse question-answering resource in the same order of magnitude as the largest synthetic datasets.
Besides the generation of the CCQA dataset, we evaluate its potential as an in-domain pre-training corpus for open-domain question-answering.
Our work is aligned with previous in-domain pretraining approaches, which have shown to improve a variety of downstream tasks (Gururangan et al., 2020).
Similar to in-domain pre-training, multiple domain-dependent pre-training tasks have been proposed for open-domain question-answering.
For example, Lee et al.
(2019) propose the Inverse Cloze Task (ICT), Chang et al. (2020) introduce Body
First Selection (BFS) and Wiki Link Prediction (WLP) and Jia et al.
(2021) describe a novel Question Answering Infused Pre-training (QUIP) task.
Along similar lines, Aghajanyan et al.
(2021) propose pre-ﬁnetuning, an alternative to in-domain pre-training, using around 50 domain-dependent datasets, showing that their MUPPET approach generalizes well to many tasks.
Khashabi et al.
(2020) introduce a similar concept for questionanswering in their UniﬁedQA framework.
While we propose a somewhat orthogonal dimension to most of these works, they nevertheless present us with strong intuition regarding the effectiveness of domain-dependent pre-training.
The Common Crawl Question Answering (CCQA) Dataset 3. Dataset Collection
Our Common Crawl Question Answering (CCQA) dataset contains around 130 million questionanswer pairs ( 55 M unique), extracted from 13 Common Crawl snapshots between May 2020 and May 2021 3 .
A high-level overview of the dataset generation process is depicted in Figure 1.
Starting from a set of raw HTML webpages, we make use of the schema.org deﬁnition to ﬁnd relevant tags, such as the question, answer, author and votes (for the full set of tags see Figure 2).
Using the explicit schema.org annotation (commonly used for searchengine optimization), instead of simple heuristics (e.g. question marks), we optimize the resulting corpus for high-quality data points.
Speciﬁcally, due to the added efforts for website creators to deﬁne schema.org conforming meta-data, we believe that annotated question-answer pairs are likely to be relevant to the general public, mostly exclude rhetorical and contextual questions, and as a result constitute high quality QA data, despite the noisy nature of webpages.
During the dataset processing steps, we remove all HTML elements that do not contain valid schema.org markers (red in Figure 1) and subsequently clean every question on the webpage to only conserve markup related to the textual content of schema.org tags 4 .
We further remove any unrelated markup attributes (e.g., CSS and JavaScript classes), before converting the content into a welldeﬁned JSON object, shown in Figure 2 and further described in section 3.2.
Using the 13 consecutive Common Crawl snapshots, we generate an initial dataset of 130 million question-answer pairs, naturally containing two types of potential duplicates: (1) Same-URL duplicates; where a webpage is updated between any two Common Crawl snapshots and (2) Content duplicates; where webpages from any Common Crawl snapshot contain same questions with potentially similar answers.
Here, we use the original, uncleaned version of the dataset, presenting a practical performance lower-bound, while leaving the exploration of additional ﬁltering steps for future work 5 .
Our dataset generation procedure is further outlined in Algorithm 1, found in Appendix A. For qualitative examples of our generated dataset format, we refer readers to Appendix G.
3. Dataset Format
The structured output of the dataset collection (shown in Figure 2), contains a three-level nested structure: (1) Every top-level data point represents a webpage in Common Crawl, encapsulating questions and answers found on the page, together with relevant metadata. On the second level, every question is represented as a tuple containing the question name (a short summary of the question) and question text (the main question). Questions also contain additional metadata as shown in Figure 2.
Every question can contain an arbitrary number of associated answers and answer attempts, located on the third and ﬁnal level of the nested structure. An answer thereby contains a mandatory accepted/suggested label, the answer text as well as optional metadata. With this nested structure of our CCQA dataset, we allow users to verify question-answer pairs and their metadata on the original webpage, utilize additional parts of the web-document and allow future research to tackle question-answering related tasks, such as answer selection, answer rating or answer ranking.
3. Dataset Dimensions
To gain better insights into the massive amount of data, we present a mix of automatically obtained dataset dimensions, a small-scale human pilot study, and a set of key dataset distributions.
Regarding the small-scale human pilot study, we analyze a random subset of 400 individual questionanswer pairs and evaluate their sensibility and answerability.
We deﬁne question sensibility as to whether the annotator understands the questions itself, while question answerability refers to whether the question provides enough context for a perfect question-answering system to correctly answer the question.
Furthermore, QA-sensibility denotes if the question-answer pair makes sense 6 . We refer [Table]
[Table] [Table] [Table] interested readers to Table 9 in Appendix E for further explanations on sensibility/answerability.
As shown in Table 1, our CCQA corpus contains nearly exclusively sensible questions, with the vast majority of them also answerable and sensible as a pair.
To complement our small-scale human annotation, we further explore key dataset dimension, including the fraction of samples with advanced markup, questions containing both, question name and question text (as deﬁned by the schema.org annotation), the number of questions without goldanswers, average question and answer length and the number of webpages with a valid language label, all indicating that the schema.org annotation highly correlates with carefully curated webpages.
Besides the key corpus-level statistics, we take a closer look at important dataset distributions in
Table 2.
Speciﬁcally, we present the top 5 domains at the top of Table 2, showing the largest number of webpages originating from the stackexchange domain, accounting for about 8% of data points.
Regarding the topical distribution of our dataset, we use the DMOZ/Curlie taxonomy, automatically extracting hierarchical topic information 7 .
We randomly sample 1 , 000 question webpages and show the top 5 topics in the second row of Table 2.
A more detailed topic distribution, also considering second-level assignments, can be found in Tablein Appendix B. Regarding the question-word distribution in our CCQA dataset, we observe that the majority of 36% of question words are what questions, followed by how , when , which and where .
A full list of all 8 questions words and their relative appearance in our corpus can be found in Tablein Appendix C. Lastly, expanding on the number of non-trivial markup tags presented in Table 1, we explore the frequency of HTML markup tags in our dataset in the last row in Table 2.
For a list of the top-25 tags found in our corpus, we point interested readers to Table 8 in Appendix D.
Evaluation
In this section, we showcase the value of our CCQA dataset with experiments on the closedbook question-answering (section 4.1) and passage retrieval for open-book QA (section 4.2) tasks.
4. Closed-Book Question-Answering 4.1. Task
The closed-book question-answering task challenges systems to answer questions without to use of additional information sources, such as knowledge bases or evidence documents.
As a result, models are solely relying on the question text and the information stored inside the model weights during training.
Here, we evaluate our new CCQA dataset as an in-domain pre-training corpus for this highly challenging task by converting the JSON representation into plain question-answer pairs, removing markup tags and additional metadata.
4.1. Models & Training
Using the question-answer pairs from the CCQA dataset, we in-domain pre-train large language models for question-answering.
We start with vanilla BART and T5 transformer models, shown on the left side (green) in Figure 3.
We then further in-domain pre-train the models using a denoising or sequence-to-sequence (seq2seq) setup (yellow box in Figure 3).
For the denoising task, we follow the vanilla BART approach (Lewis et al., 2020a), using a concatenation of Q: (cid:107) <question> (cid:107) A: (cid:107) <answer> as the model input.
For the seq2seq task, we train the model to predict the gold answer given a question as input.
With the additional in-domain pretraining step, a variety of training-ﬂows emerge, shown as numbered circles in Figure 3:
(1) Using a vanilla pre-trained language model to ﬁne-tune on the benchmark dataset.
(2) Using the CCQA dataset for in-domain pretraining and subsequently ﬁne-tune on the benchmark dataset.
(3) Using a pre-trained language model to directly infer answers on the benchmark dataset (zero-shot).
(4) Using the CCQA in-domain pre-trained model to directly infer answers on the benchmark dataset in zero-shot fashion
4.1. Datasets
We evaluate the performance of our CCQA corpus as an in-domain pre-training dataset on 5 common benchmarks, based on 4 publicly available datasets in the closed-book setting:
TriviaQA (TQA) is a short-form, factoid-style question-answering dataset (Joshi et al., 2017).
For the closed-book task, we ignore the available contexts and focus exclusively on question-answer pairs.
Since the ofﬁcial test-split of the dataset is not publicly available, we use the ofﬁcial validation set as our test split and randomly sample a validation set from the training data, as commonly done in previous work (Roberts et al., 2020).
Natural Questions (NQ) (Kwiatkowski et al., 2019) represents a popular corpus for questionanswering research.
Despite most recent work focusing on the short-form answers (NQ-Short), the
NQ corpus also provides additional long-form answers (NQ-Long) for a large subset of questions.
In this work, we use both, short, factoid answers and long-form responses.
ELI5 , introduced by Fan et al.
(2019), constitutes the ﬁrst large-scale long-form dataset for open-ended question-answering.
We again do not take available evidence documents into account, but focus on the question-answer pairs only.
GooAQ (Khashabi et al., 2021) contains semiautomatically extracted question-answer pairs from the Google question auto-complete feature. [Table] Top 5 Appearances in CCQA
[Table] Metrics
For datasets with short-form answers, we use the
Exact Match (EM) metric for ﬁne-tuned systems, in line with previous work by Roberts et al.
(2020) and Lewis et al.
(2021b).
While the EM metric works well for systems that are aware of the taskspeciﬁc format, it punishes potentially correct answers with additional context, which we believe is overly harsh in zero-shot settings, where the speciﬁc output format is not known (e.g., trainingﬂows (3) and (4)).
Therefore, we propose using the Answer-level Recall (AR) metric for our zeroshot experiments, while limiting the answer length with the max-length and length-penalty inference parameters.
As such, the AR metric requires the correct answer to be a continuous sub-sequence of the predicted tokens, while allowing for additional context.
Since AR operates on token-level, the prediction of super/sub-words, e.g., fun damental instead of fun , is considered incorrect.
For long-form question-answer datasets, we choose the Rouge-L (RL) score as our evaluation metric, which has shown strong correlation with
Rouge-1 and Rouge-2 scores, and is commonly used in previous work (Khashabi et al., 2021).
4.1. Hyper-Parameters
We use the default parameters of the BART (Lewis et al., 2020a) and T5 (Raffel et al., 2020) models for in-domain pre-training and ﬁne-tuning whenever possible.
Regarding the in-domain pre-training on our CCQA dataset, we limit training to 800 k steps using a batch-size of 1 , 024 .
During our ﬁne-tuning runs, we limit the number of updates to 20 k steps with a batch-size of 256 samples, with exception of the GooAQ dataset, which we ﬁne-tune for 100 k steps due to it’s large size.
We select the best model during our in-domain pre-training runs based on the perplexity measure, and pick the top ﬁne-tuned model according to the ﬁnal evaluation metric.
We do not perform any hyper-parameter search during in-domain pre-training and ﬁne-tuning.
For the inference step, our hyper-parameter setting is closely related to commonly used summarization parameters.
We use a beam-size of 4, maxlength of 140, and length-penalty of 2.0.
For the ﬁne-tuned short-form task, we choose a max-length of 30, following Xiong et al.
(2021) and a lengthpenalty of 1.0.
All model evaluations are based on
Huggingface Transformers 8 (Wolf et al., 2019).
4.1. Results
Our main results for the closed-book questionanswering task are presented in Table 3, showing the zero-shot and ﬁne-tuned performance of the BART Large (top), T5 Small (center) and T
Base (bottom) models for each of the 5 evaluation datasets.
Even though we present a wide variety of benchmark results, from short-form factoid questions to long-form answers, the CCQA seq2seq pretrained model consistently outperforms all other models on the zero-shot question-answering task.
Even more importantly, the additional in-domain pre-training step achieves better zero-shot performance than fully ﬁne-tuned, randomly initialized transformer models (as extensively used prior to 2018) in almost all settings.
Speciﬁcally, our model outperforms the randomly initialized transformers on all benchmarks for T5 Small and T5 Base, as well as on 4 out of 5 datasets using BART Large.
Comparing the fully ﬁne-tuned setting across models and datasets it becomes clear
that, although oftentimes performing comparably, our
CCQA seq2seq pre-trained model underperforms the vanilla models in most cases.
Seq2seq indomain pre-training on CCQA only reaches superior performance on the ELI5 dataset for all models, as well as on the GooAQ dataset for T5 Small. Showing that seq2seq pre-training on CCQA is effective in zero-shot scenarios, however only partially improves over baselines in the ﬁne-tuned set-
[Table]
Results from a Lewis et al.
(2021a)
[Table]
[Table] ting, we investigate: (1) Additional experiments using the CCQA dataset for denoising-style pretraining ( -d in Table 3) and (2) Evaluate additional low-resource scenarios, shown in Figure 4.
For our denoising-style in-domain pre-training experiment, we keep the available markup information, in line with HTLM (Aghajanyan et al., 2022b).
As shown in Table 3, the in-domain CCQA denoising objective outperforms the vanilla BART
Large model on 4 out of 5 benchmarks in the ﬁnetuned setting.
We believe that this result, alongside the zero-shot performance of the seq2seq CCQA model, clearly shows the usefulness and generality of our CCQA corpus for closed-book open-domain question-answering.
Taking a closer look at low-resource scenarios, we evaluate the vanilla T5 Small model against our in-domain pre-trained approach using 5 proper subsets of the NQ-Long, GooAQ and ELI5 benchmark datasets, drawn at random.
As presented in Figure 4, our CCQA model mostly outperforms the vanilla T5 Small model in low-resource scenarios with up to 10 , 000 data points.
While the performance of our CCQA model is consistently better on the ELI5 test-set, the vanilla baselines outperform our models fastest on the NQ-Long corpus.
Additional low-resource experiments on T5 Base are shown in Table 6, in Appendix F.
4. Passage Retrieval 4.2. Task
For the passage retrieval task, an important component of most open-book QA systems (e.g., Lewis et al.
(2020b) ; Izacard and Grave (2021)), models aim to extract a set of evidence passages from a large collection of documents through conditional ranking.
To align our corpus with the passage retrieval task, we aggregate every question into a single data point, consisting of the question itself, alongside all available answers as either positive or negative contexts.
If available, answer votes are used as a proxy to determine positive and negative (sometimes called “hard-negative") contexts.
Following the practice in Fan et al.
(2019), we assign every answer with at least 2 more upvotes than downvotes as a positive context and all other answer as negative.
If answer votes are not available, we use the accepted/suggested label (shown 1010full    9. NQ-Long Vanilla
CCQA 1010full  8.
GooAQ Vanilla CCQA 1010full  7. ELI
Vanilla
CCQA in Figure 2) as an indicator for positive and negative contexts.
In the absence of either criterion, we use all available answers as positive contexts.
4.2. Models & Training
For passage retrieval, we choose the Dense Passage Retriever (DPR) (Karpukhin et al., 2020), used in a variety of popular end-to-end open-book
QA models, such as RAG (Lewis et al., 2020b) and FiD (Izacard and Grave, 2021).
As shown in Figure 5, we start with the vanilla DPR model based on BERT (Devlin et al., 2019) and in-domain pre-train using questions and positive/negative passages from the CCQA dataset (yellow box in Figure 5), similar to O˘guz et al.
In line with the training-ﬂows of the closed-book models, we train
DPR using either the vanilla setup (pre-training [Table] [Table] → ﬁne-tuning) or the in-domain pre-training approach (pre-training → in-domain pre-training → ﬁne-tuning), shown as circles (1) and (2) in Figure 5, respectively.
4.2. Datasets & Metrics
Following the original DPR paper (Karpukhin et al., 2020), we evaluate the passage retrieval task on the
NQ-Short and TQA datasets presented in section 4.1.3, using the top-20 and top-100 retrieval accuracy (Acc@20/Acc@100) measures.
4.2. Hyper-Parameters
We use the default DPR hyper-parameters whenever possible (Karpukhin et al., 2020).
For indomain pre-training, we limit training to 800 k steps using a batch-size of 1 , 536 samples.
During ﬁnetuning, we restrict the number of updates to 20 k steps with a batch-size of 128 .
The best checkpoint is selected based on the Mean Reciprocal Rank (MRR) measure, following O˘guz et al.
We do not perform any hyper-parameter search.
4.2. Results
For the passage retrieval experiments, we compare our CCQA in-domain pre-trained DPR model against the vanilla DPR model published in
Karpukhin et al.
(2020), as well as the recently enhanced version (O˘guz et al., 2022).
Table 4 contains our empirical results, showing consistent improvements of our CCQA DPR model over the vanilla baselines.
More speciﬁcally, the in-domain
CCQA pre-training step increases the top-20 and top-100 accuracy score on the TQA benchmark dataset by over half a point, while the performance gap on NQ-Short shows consistent improvement of over 0 .
4. Evaluation Fairness: Dataset Overlap
With modern pre-training approaches using increasingly large datasets, accidental overlaps between pre-training corpora and benchmark datasets become more and more common (Lewis et al., 2021a).
To analyze this threat to the integrity of our dataset and empirical analysis, we follow Radford et al.
and evaluate the 8-gram question overlap of our CCQA training portion with the test-split of benchmark datasets using bloom ﬁlters.
Tableshows a consistently smaller question overlap between CCQA and the benchmark test set, compared to the benchmark training split itself.
Conclusion and Future Work
In this work, we presented our new web-scale
CCQA dataset for in-domain model pre-training.
Orthogonal to recent efforts on improving taskspeciﬁc pre-training objectives, we show our dataset generation process, followed by detailed insights into key corpus dimensions of this new, largescale, natural, and diverse question-answering dataset.
In a set of empirical evaluations, we conﬁrm the initial intuition that the corpus presents a valuable resource for open-domain questionanswering research.
In our zero-shot, low-resource and ﬁne-tuned experiments for open- and closedbook QA tasks, we show promising results across multiple model architectures.
With around 130 million question-answer pairs ( 55 M unique) as well as additional meta-data, our CCQA dataset presents a versatile source of information, which has a large variety of applications in future work (e.g., question summarization, answer rating, answer ranking and many more).  Ethical Considerations We now discuss the three major ethical considerations impacting this paper:
Hate-speech, Harmful Gender and Racial Biases:
With general web-data potentially containing hate-speech and harmful gender and racial biases, we believe that our extracted dataset based on the schema.org annotations is less impacted by these issues, with the schema.org annotation representing a good proxy for high-quality, professionally curated websites. As a result, we believe that the severity of this issue is signiﬁcantly reduced. Furthermore, in our human evaluation, we ﬁnd no signs of the above mentioned biases. We leave computational approaches to determine dataset biases for future work (e.g., the Word Embedding Association Test (Caliskan et al., 2017) and Sentence
Encoder Association Test (May et al., 2019)).
Intellectual Property
Since the Common Crawl project collects monthly additions to the world-wide-web independently of the source of the website and the original author, the intellectual property of websites contained in the Common Crawl corpus is not secured from the respective authors and publication rights are not obtained. As a result, we do not directly provide the CCQA dataset, but enable third parties to generate the corpus through our published dataset generation scripts available at https://github.com/ facebookresearch/CCQA .
Hallucinations and Factual Errors
As shown in the evaluation section, our model is able to generate reasonable answers for factoid and long-form questions.
The inferred answers are ﬂuent and human-like, but may also contain hallucinations and factual errors, especially for the challenging closed-book question-answering task.
Without a guarantee of the predicted answers being factually correct, they can potentially spread misinformation if not properly corrected.
Acknowledgments
We want to thank the anonymous reviewers for their valuable feedback and acknowledge our colleagues and peers for their help throughout this project, speciﬁcally Anchit Gupta, Gagan Aneja and Patrick Lewis for their valuable input on important decisions.
. Association for Computational Linguistics
Gautier Izacard and Édouard Grave. 2021.
Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume , pages 874–880. Robin Jia, Mike Lewis, and Luke Zettlemoyer. 2021. Question answering infused pre-training of generalpurpose contextualized representations.
arXiv preprint arXiv:2106.08190 .
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension.
In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Van couver
Armand Joulin, Edouard Grave, Piotr Bojanowski,
Matthijs Douze, Hérve Jégou, and Tomas Mikolov.
Armand Joulin, Edouard Grave
Piotr Bojanowski,
Tomas Mikolov
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick
Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih Daniel Khashabi, Sewon Min, Tushar Khot, Ashish
Sabharwal, Oyvind Tafjord, Peter Clark, and Han naneh Hajishirzi
Daniel Khashabi, Amos Ng, Tushar Khot, Ashish Sabharwal, Hannaneh Hajishirzi, and Chris CallisonBurch with diverse types.
In Findings of the for
Linguistics: EMNLP
, pages 421–433.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redﬁeld, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones,
Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai,
Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research Transactions of the Association for Computational Linguistics , 7:452–466. Kenton Lee, Ming-Wei Chang, and Kristina Toutanova.
weakly supervised
Proceedings of the 57th Annual Meeting of the Association for Computational
, pages
6096, Florence,
Italy
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer
Levy, Veselin Stoyanov, and Luke Zettlemoyer.
2020a
BART sequence sequence pretraining for natural language generation, translation, and comprehension.
In Proceedings of the 58th Annual Meeting of the Association for Computational
Linguistics , pages 7871–7880, Online.
Association for Computational Linguistics.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020b
Curran Associates, Inc.
Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel.
2021a
the
European
Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale
Minervini, Heinrich Küttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. 2021b.
PAQ:million probably-asked questions and what you can do with them.
Transactions of the Association for
Computational Linguistics , 9:1098–1115.
Chandler May, Alex Wang, Shikha Bordia, Samuel R.
Bowman, and Rachel Rudinger. 2019.
On measuring social biases in sentence encoders.
In Proceedings of the 2019 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume(Long and Short Papers) , pages 622–628, Minneapolis, Minnesota.
Association for Computational Linguistics.
Barlas O˘guz, Kushal Lakhotia, Anchit Gupta, Patrick
Lewis, Vladimir Karpukhin, Aleksandra Piktus,
Xilun Chen, Sebastian Riedel, Wen-tau
Yih, Sonal
Gupta, et al
Luan, much param
Resources
Sanh
Chaumond
Delangue
Moi
Pier ric Cistac
Rault
Rémi Louf
Fun towicz et al
Huggingface’s transformers
State of the-art natural language processing arXiv preprint arXiv:1910.03771 .
Wenhan Xiong, Xiang Li, Srini Iyer, Jingfei Du, Patrick
Lewis, William Yang Wang, Yashar Mehdad, Scott
Yih, Sebastian Riedel, Douwe Kiela, and Barlas Oguz. Answering Algorithm for document ∈ CommonCrawl do if "schema.org/Question" in document then (cid:46) Webpage contains schema.org annotation tree ← parse_html(document) questions ← ﬁnd_question_root(tree) for question_sub_tree in questions do question_sub_tree ← clean_question_subtree(question_sub_tree) end for questions ← convert_to_json(questions) save(questions) else skip document end if end for procedure FIND _ QUESTION _ ROOT (node) (cid:46)
Pre-order traversal, return when question found if node.itemtype =
= "https://schema.org/Question" then return node end if for child in node.children() do node ← ﬁnd_question_root(child) nodes.append(node) end for return nodes end procedure procedure CLEAN _ QUESTION _ SUBTREE (node) (cid:46) Post-order traversal, clean elements bottom-up for child in node do child ← clean_question_subtree(child) end for if "itemtype" | "itemprop" in node.attributes() then for attribute in node.attributes() do if not attribute.starts_with("item" | "content" | "date") then attribute.remove() end if end for else replace_node_with_children(node) end if end procedure
B Detailed Topic Distribution
[Table]
C Detailed Question Word Distribution
[Table]
D
[Table]
E Sensibility and Answerability Examples
[Table] [Table]
F
Set [Figure] { "Language":"-", "Fasttext_language":"en", "URI":"https://www.geograph.ie/faq3.php?q=multiple+account", "UUID":"a5e97da2-f688-42af-8626-73a38fa8d06f", "WARC_ID":"CC-MAIN-20201026031408-20201026061408-00221", "Questions":[ { "name_markup":"Can I change my name to a <b>pseudonym</b> on a submission ?", "Answers":[ { "text_markup":"You can submit all your photos under a pseudonym by changing the name on your
Profile<span><a>http://www.geograph.org.uk/profile.php</a></span>(link top write on most pages).
Note that by doing this, the name will be changed on all photos you have previously submitted from the account.
These may already have been used elsewhere, crediting the name originally shown.
<br> You can change the credit on an individual image, for instance if you asked someone else to take it for you, but the name on your profile will still be shown on the photo page and the photographer name will still link back to your profile.
<br>
You can open another account under a pseudonym but this will need to be done from a different email address and you will have to take care which account you are signed in with before submitting, making changes or posting in the forums.", "status":"acceptedAnswer" } ] } ] }
{ "Language":"en-US", "Fasttext_language":"en", "URI":"https://www.catholicfaithstore.com/Store/Products/SKU/b0d/
St-Olgas-Cross-Medal.html", "UUID":"94def557-e521-493a-babd-b63c5e030e62", "WARC_ID":"CC-MAIN-20210308174330-20210308204330-00337", "Questions":[ { "name_markup":"How do I care for my sterling silver?", "Answers":[ { "text_markup":"<p>
Sterling Silver Cleaning
Instructions</p><ul><li>NEVER use a sterling silver cleaning solution on your jewelry.
It will take off the protective coating.</li><li>
Take a half cup of warm water and a few drops of mild dishwashing liquid soap and mix together.</li><li>With a soft clean cotton cloth&#160;dip the cloth into the soapy water getting it moist.</li><li>
Use the moist cloth to wipe the surface of your sterling silver jewelry.</li><li>
Take the just cleaned jewelry and run under clear water for a few seconds to&#160;wash away any soap.</li><li>Allow jewelry to dry before storing</li></ul><p>Other things to remember: When not wearing your sterling silver jewelry, keep it in an air-tight container or zip lock bag.
Avoid household clean products getting in contact with the jewelry.
And take off your jewelry when you swim, shower or are washing dishes.</p><p>For a more detailed explanation see<a>5 Easy-To-Follow Steps for Cleaning Your
Sterling Silver Jewelry</a></p>", "status":"acceptedAnswer" } ] } ] }
{ "Language":"-", "Fasttext_language":"en", "URI":"https://quant.stackexchange.com/questions/39510/ software-for-american-basket-option-pricing-using-longstaff -schwartz-least-squar", "UUID":"e059deaf-3d73-4517-88a0-8abb8ad74972", "WARC_ID":"CC-MAIN-20210305183324-20210305213324-00585", "Questions":[ { "author":"Bananach", "name_markup":"<a>Software for American basket option pricing using Longstaff-Schwartz/Least Squares Monte Carlo method</a>", "text_markup":"<p>Is there free software (preferably in Python) that computes American basket (high-dimensional!)
option prices in the Black Scholes model using the
Longstaff-Schwartz algorithm (also known as Least Squares Monte
Carlo)?</p>~<p>
Optimally, I want to be able to control the number of basis functions, the number of Monte Carlo samples and the number of time steps used.</p>", "date_created":"2018-04-30T09:16:33", "upvote_count":"1", "answer_count":"1", "Answers":[ { "author":"byouness", "text_markup":"<p>
QuantLib is what you are looking for.
It is free/open source library written in C++, it is available in Python as well (via
SWIG):<a>https://www.quantlib.org/install/windows-python.shtml <
/a></p>~<p>
Examples are shipped with QuantLib and among them some show how to price options.</p><p>
To get a feel for what it’s like, you can check this blog post, explaining how to price an American option on a single asset using a binomial tree in Python :~<a>http://gouthamanbalaraman.com/blog/ american-option-pricing-quantlib-python.html</a></p>", "status":"acceptedAnswer", "upvote_count":"1", "comment_count":"1" } ] } ] }
{ "Language":"en", "Fasttext_language":"en", "URI":"https://wwwmybizpro.invoicera.com/expense-management.html", "UUID":"8cfe986c-4f33-4a2a-98f1-32aab3811533", "WARC_ID":"CC-MAIN-20210512100748-20210512130748-00544", "Questions":[ { "name_markup":"Do I need any new IT infrastructure to get the best use out of this software?", "Answers":[ { "text_markup":"NO!
Invoicera simply integrates with your current ERP and CRM.
It comes with the simplest self-explanatory user-interface for you to use.
You do not need any extra guidance with your Invoicera.", "status":"acceptedAnswer" } ] } ] }