CCQA : A New Web-Scale Question Answering Dataset for Model Pre-Training (2022)

Abstract: We propose a novel open-domain question-answering dataset based on the Common Crawl project. With a previously unseen number of around 130 million multilingual question-answer pairs (including about 60 million English data-points), we use our large-scale, natural, diverse and high-quality corpus to in-domain pre-train popular language models for the task of question-answering. In our experiments, we find that our Common Crawl Question Answering dataset (CCQA) achieves promising results in zero-shot, low resource and fine-tuned settings across multiple tasks, models and benchmarks.

## introduction open-domain question-answering (odqa) has evolved into a core problem in natural language processing (nlp), receiving growing interest from the research community @xcite . despite the notoriously difficult challenge to correctly answer opendomain questions on arbitrary topics, recent advances of pre-trained language models (such as bert @xcite , bart @xcite and t5 @xcite ) have stimulated new research into additional, task-dependent pre-training steps. specifically, recent publications show that in-domain pre-training regimes can improve models for several downstream tasks @xcite . for open-domain questionanswering, newly proposed pre-training tasks such as the inverse cloze task (ict) @xcite , body first selection (bfs), wiki link prediction (wlp) @xcite and question answering infused pre-training (quip) @xcite show consistent improvements over baselines. however, most of these approaches still rely on either unlabeled text, or synthetically generated question-answer (qa) pairs. in this paper, we explore a second, somewhat orthogonal dimension to these lines of work, examining if a web-scale collection of natural qa pairs can support odqa through in-domain pre-training. per definition, an odqa system should be able to answer any question from an arbitrary domain. we believe that to approach this ability with indomain pre-training, a suitable dataset should address the following 5 challenges: (1) size; odqa requires knowledge of a wide variety of topics. the underlying dataset used for in-domain pretraining hence needs to cover this abundance of domains, requiring a web-scale dataset. (2) naturalness; while synthetic corpora can potentially capture a wide variety of language phenomena, to understand and generate truly natural language in all facets, synthetic datasets are not sufficient. to overcome these challenges, we propose a new large-scale dataset for open-domain questionanswering called the common crawl question answering (ccqa) dataset. similar to popular datasets, such as c4 @xcite , cc-net @xcite , cc-100 @xcite , htlm @xcite , and cm3 @xcite we generate a large-scale, diverse and high-quality questionanswering dataset from common crawl. more specifically, common crawl allows us to obtain a large number of truly natural questionanswer pairs, asked and answered by real humans on the web, rather than inferred through computational methods. using the abundantly available schema.org question annotation foot_0 , we generate question-answer pairs from explicit annotations, instead of heuristic rules, leading to high-quality data points. in a large set of evaluations, we show that indomain pre-training on our ccqa dataset achieves promising results across different settings, models and benchmarks. using the rich information available on the web, we augment our dataset with additional data attributes beyond just questionanswer pairs, such as votes, multiple (competing) answers, question summaries and intra-textual html markup, which can be used for a variety of tasks beyond question-answering in future work. furthermore, we evaluate the diversity and evaluation fairness of our dataset by computing topic distributions and train-test overlaps with benchmark datasets, providing additional rationale regarding the quality of our data and experiments. to summarize, our main contributions in this paper are as follows: this work is inspired by a range of previous approaches using common crawl web-data, such as the colossal clean crawled corpus (c4) for language model pre-training @xcite , the word/sentence representation generation corpus ccnet @xcite , the cc-100 dataset for translation @xcite and the markup-style language modelling htlm corpus for zero-shot summarization @xcite . despite all previously mentioned applications directly relying on large-scale web data from common crawl, their scope and application vary significantly. compared to previously proposed datasets based on common crawl, we are the first to extract well-structured question-answer pairs with additional meta-data, making our corpus a valuable resource for odqa research, and a multitude of related tasks, such as question summarization, answer rating, and answer ranking. further web-based datasets outside the common crawl domain are the triviaqa @xcite and eli5 corpora @xcite , extracting small-scale question-answer datasets from trivia websites and reddit threads respectively. the large-scale gooaq dataset @xcite is similarly based on web data, however exploits the google auto-complete feature and related answer boxes to generate semi-synthetic questionanswer pairs. as a large-scale, completely syn-thetic dataset, the paq corpus @xcite automatically generates a large set of probably asked questions from wikipedia articles. in contrast to these previously proposed datasets, our ccqa corpus presents a large-scale, natural and diverse question-answering resource in the same order of magnitude as the largest synthetic datasets. besides the generation of the ccqa dataset, we evaluate its potential as an in-domain pre-training corpus for open-domain question-answering. our work is aligned with previous in-domain pretraining approaches, which have shown to improve a variety of downstream tasks @xcite . similar to in-domain pre-training, multiple domain-dependent pre-training tasks have been proposed for open-domain question-answering. for example, @xcite al. (2019 ) propose the inverse cloze task (ict), chang et al. (2020) introduce body first selection (bfs) and wiki link prediction (wlp) and jia et al. (2021) describe a novel question answering infused pre-training (quip) task. along similar lines, aghajanyan et al. (2021) propose pre-finetuning, an alternative to in-domain pre-training, using around 50 domain-dependent datasets, showing that their muppet approach generalizes well to many tasks. khashabi et al. (2020) introduce a similar concept for questionanswering in their unifiedqa framework. while we propose a somewhat orthogonal dimension to most of these works, they nevertheless present us with strong intuition regarding the effectiveness of domain-dependent pre-training. ## evaluation in this section, we showcase the value of our ccqa dataset with experiments on the closedbook question-answering (section 4.1) and passage retrieval for open-book qa (section 4.2) tasks. ## conclusion and future work in this work, we presented our new web-scale ccqa dataset for in-domain model pre-training. orthogonal to recent efforts on improving taskspecific pre-training objectives, we show our dataset generation process, followed by detailed insights into key corpus dimensions of this new, largescale, natural, and diverse question-answering dataset. in a set of empirical evaluations, we confirm the initial intuition that the corpus presents a valuable resource for open-domain questionanswering research. in our zero-shot, low-resource and fine-tuned experiments for open-and closedbook qa tasks, we show promising results across multiple model architectures. with around 130 million question-answer pairs (55m unique) as well as additional meta-data, our ccqa dataset presents a versatile source of information, which has a large variety of applications in future work (e.g., question summarization, answer rating, answer ranking and many more).