S 2 ORC : The Semantic Scholar Open Research Corpus (2020)

Abstract: We introduce S2ORC, a large corpus of 81.1M English-language academic papers spanning many academic disciplines. The corpus consists of rich metadata, paper abstracts, resolved bibliographic references, as well as structured full text for 8.1M open access papers. Full text is annotated with automatically-detected inline mentions of citations, figures, and tables, each linked to their corresponding paper objects. In S2ORC, we aggregate papers from hundreds of academic publishers and digital archives into a unified source, and create the largest publicly-available collection of machine-readable academic text to date. We hope this resource will facilitate research and development of tools and tasks for text mining over academic text.

## introduction academic papers are an increasingly important textual domain for natural language processing (nlp) research. aside from capturing valuable knowledge from humankind's collective research efforts, academic papers exhibit many interesting characteristics -thousands of words organized into sections, objects such as tables, figures and equations, frequent inline references to these objects, footnotes, other papers, and more. different types of resources have been used to support research over academic papers. citation graphs like aminer's open academic graph @xcite , the microsoft academic graph (mag) @xcite , and the semantic scholar literature graph @xcite , have had widespread application in bibliometrics, science-of-science, information retrieval, and network analysis. digital archives like arxiv, 2 pubmed central, 3 citeseerx @xcite , 4 and the acl anthology @xcite , 5 are popular resources for deriving large text corpora for summarization and language modeling or, with further annotation, development of datasets for tasks like entity extraction, text classification, parsing, and discourse analysis. we focus on bibliometrically-enhanced derivations of these corpora, such as the acl anthology network (aan) @xcite 6 derived from the acl anthology, refseer @xcite derived from citeseerx, and @xcite derived from arxiv, which combine useful aspects of citation graphs and raw text corpora. these resources provide citation mentions linked to paper identifiers in their corresponding digital archives, such as the acl anthology and citeseerx, or to nodes in citation graphs such as mag, enabling new forms of cross-paper discourse analysis (e.g., studying how or why papers are related). corpus papers w/ body text citation contexts references to tables / figures / equations linked to graph academic disciplines s2orc (pdf-parse) 8.1m full text yes s2orc (full) multi s2orc (latex-parse) 1.5m full text yes s2orc (full) physics, math, cs pubmed central (oa) 2.6m full text yes pubmed bio, med aan @xcite 25k full text no acl anthology comp ling saier and färber (2019) † 1.0m snippets no mag physics, math, cs refseer @xcite 1.0m snippets no citeseerx multi yet, existing corpora are not without their limitations. some cover a small number of papers (e.g. aan), are domain-specific (e.g. aan, pubmed central, @xcite ), or may not provide usable full text (e.g. @xcite and refseer). to address these issues, we introduce s2orc, foot_4 the semantic scholar foot_5 open research corpus, a large publicly-available collection of 81.1m academic papers covering dozens of academic disciplines. each paper is associated with metadata and abstracts aggregated from hundreds of trusted sources such as academic publishers and literature archives like pubmed and arxiv. notably, we release structured, machinereadable full text extracted from pdfs for 8.1m papers which we've identified as having open access status. s2orc full text preserves meaningful structure, e.g., paragraph breaks, section headers, inline citation mentions, references to tables and figures, and resolved citation links to other papers. additionally, we provide 1.5m full text latex parses from which we have extracted, in addition to citations and references, the source text of tables and mathematical formulas. as shown in in this paper, we describe the construction of s2orc ( §2). we provide summary statistics of the corpus ( §3) and evaluate the data quality ( §4). we then evaluate a bert model pretrained on s2orc ( §5), and discuss potential applications to a variety of nlp and analysis tasks over academic text ( §6). finally, we compare s2orc with other publicly-available academic text corpora ( §7). to construct s2orc, we must overcome challenges in (i) paper metadata aggregation, (ii) identifying open access publications, and (iii) clustering papers, in addition to identifying, extracting, and cleaning the full text and bibliometric annotations associated with each paper. the pipeline for creating s2orc is: ## the s2orc dataset the resulting corpus consists of 81.1m papers. our publisher-provided abstract coverage is 90.4%, or 73.4m papers. our pdf coverage is 35.6%, or 28.9m papers. these pdfs are processed using the pipeline discussed in §2.1. the total papers 81.1m papers w/ pdf 28.9m (35.6%) papers w/ bibliographies 27.6m (34.1%) papers w/ grobid full text 8.1m (10.0%) papers w/ latex full text 1.5m (1.8%) papers w/ publisher abstract 73.4m (90.4%) papers w/ dois 52.2m (64.3%) papers w/ pubmed ids 21.5m (26.5%) papers w/ pmc ids 4.7m (5.8%) papers w/ arxiv ids 1.7m (2.0%) papers w/ acl ids 42k (0.1%) the distribution of academic disciplines in s2orc is given in figure 2 using microsoft academic fields of study. not all papers in s2orc can be found in microsoft academic -those not found are denoted as unclassified. approximately 677k papers have more than one primary microsoft academic field of study; figure 2 represents only the top field of study for each paper. ## evaluation to evaluate the quality of our metadata selection, we randomly sample 500 paper clusters, restricting to those with pdfs. within each sampled cluster, we determine whether the canonical title and authors match the title and authors in the selected canonical pdf. inline citation detection and bibliography parsing are dependent on grobid @xcite . ahmad and afzal (2018) evaluate grobid for detecting inline citations using a corpus of 5k cite-seer papers, and found grobid to have an f1score of 0.89 on this task. @xcite report grobid as the best among 10 out-of-the-box tools for parsing bibliographies, also achieving an f1 of 0.89 in an evaluation corpus of 9.5k papers. we perform an evaluation over 200 randomly sampled papers from s2orc and found comparable f1-scores for grobid performance on both tasks. for bibliography linking, we randomly sample s2orc papers (500 grobid pdf parses and 100 latex parses) and select one linked bibliography entry from each sampled paper (while avoiding selecting multiple entries linked to the same paper). we determine whether the title and authors in the bibliography entry agree with the title and authors of the linked paper. ## pretraining bert on s2orc to demonstrate the suitability of s2orc for language model pretraining, we train bert-base @xcite on the parsed full text of s2orc and show that the resulting model (s2orc-scibert) performs similarly to sci-bert @xcite on a diverse suite of scientific nlp tasks and datasets. while scibert is a bert-base model also trained on multiple domains of scientific text, key differences in its pretraining corpus and vocabulary and those used for s2orc-scibert are: we follow a similar setup to @xcite for both pretraining and fine-tuning s2orc-scibert. like scibert, s2orc-scibert is pretrained from scratch using the original bert code 15 and default bert-base configurations on a single tpu v3-8 for one week. also like scibert, s2orc-scibert is finetuned on all tasks by optimizing a cross entropy loss using adam @xcite , a linear learning rate decay with 10% warm-up, batch size of 32, and dropout of 0.1. we search over an equal-sized grid of hyperparameters as @xcite . we fine-tune for 1 to 4 epochs with a maximum learning rate of 1e-5, 2e-5, 3e-5, or 5e-5. for each task, we select the optimal combination of these two hyperparameters using the development set and report the corresponding test set results. for details, we refer the reader to scibert code, 16 which we use for all experiments. the results in ## applications of s2orc s2orc can be used for many nlp and analysis tasks over academic text. we give a summary of potential applications below. the combination of structured full text annotated with linked inline citations makes s2orc well-suited for a variety of citation-related textbased tasks. without any additional supervision, s2orc can be used directly for both inline (he other tasks that leverage citation contexts in-clude classifying citation intent @xcite @xcite , identifying citation sentiment @xcite , identifying meaningful citations @xcite , extracting key phrases @xcite , and citation context-based paper summarization @xcite @xcite . the models in these papers require labeled citation contexts for training. s2orc could potentially benefit task performance without additional annotation, for example, by pretraining language models on s2orc citation contexts before fine-tuning to these tasks. @xcite find that long citation contexts (beyond sentence boundary) are important for tasks like summarization; the wider citation contexts available in s2orc could be used to augment existing datasets for document-level tasks. citation contexts can also be used for the more general tasks of identifying similar papers @xcite @xcite or bibliometric analysis @xcite @xcite . towards these tasks, the citation contexts in s2orc can provide insight into how and why papers are cited. we illustrate this by following @xcite in training a word2vec skip-gram model @xcite using full text citation contexts in s2orc, where each inline citation span is replaced with its linked paper identifier. when training over this modified text, the word2vec model learns embeddings corresponding to each unique paper identifier, which can be leveraged as paper embeddings. the resulting embeddings shown in figure 3 and the latex subset of s2orc also provides unique opportunities for research. in addition to citations and references, we also extract and parse tables from latex source into a structured format. there is an opportunity to use these ta-bles for corpus-level results extraction and aggregation. the latex subset also has fine-grained extraction and labeling of mathematical formulas, which can be used to understand proof construction, or to assist in symbol co-reference resolution. ## related work the acl anthology network (aan) @xcite is a bibliometric-enhanced corpus covering papers in the field of computational linguistics. it is built from the acl anthology @xcite and consists of 24.6k papers manually augmented with citation information. the pubmed central open access corpus is a large corpus of 2.6m papers in the biomedical domain with citations linked to pubmed identifiers. 17 citeseerx @xcite , consists of papers collected primarily via web crawl, without integrating metadata provided by sources outside of the pdf. although citation contexts are no longer available through citeseerx, the ref-seer dataset @xcite 18 is a dataset of short citation context snippets derived from 1.0m papers from citeseerx. more recently, @xcite introduce a corpus built using 1.0m arxiv publications. they use latex source to extract text, citation spans and bibliography entries, which are linked to papers in the microsoft academic graph. the citation context they provide are extracted snippets and no bibliography parses are provided. an updated version of this dataset @xcite released concurrently with this work now includes full text. compared with these resources, s2orc represents a significantly larger dataset of linked papers covering broad domains of science by leveraging pdf parsing in addition to latex source. s2orc also provides clean full text for text mining and nlp needs with additional enhancements such as annotations of table and figure references and captions. s2orc's wealth of metadata and structured text allows it to be flexibly adapted to a variety of downstream tasks. s2orc consists of 81.1m papers, 380.5m resolved citation links, and structured full text from 8.1m open-access pdfs and 1.5m latex source files. we aggregate metadata and abstracts from hundreds of trusted sources. full text is augmented with sections, citation mentions, and references to tables and figures. we demonstrate that s2orc can be used effectively for downstream nlp tasks in academic paper analysis. the pipeline for creating s2orc was used to construct the cord-19 corpus @xcite , which saw fervent adoption as the canonical resource for covid-19 text mining. cord-19 is aimed at assisting biomedical experts and policy makers process large amounts of covid-19 literature in the search for effective treatments and management policies. with over 75k dataset downloads, dozens of search and question-answering systems, and hundreds of participating teams across two shared tasks 19 in the first month of its release, there is little doubt of the resource's impact. our hope with the release of s2orc is to ensure such text mining resources are available to researchers even beyond periods of global crisis.