Original: Foundation Models of Scientific Knowledge for Chemistry: Opportunities, Challenges and Lessons Learned
Revised: Foundation Models of Scientific Knowledge for Chemistry: Opportunities, Challenges and Lessons Learned

Original: Anonymous ACL submission

Original: Abstract
Revised: Abstract

Original: Foundation models pre-trained on large corporademonstrate significant gains across many natural language processing tasks and domains e.g.,law, healthcare, education, etc.
Revised: Foundation models pre-trained on large corpora demonstrate significant gains across many natural language processing tasks and domains e.g., law, healthcare, education, etc.

Original: However, onlylimited efforts have investigated the opportunities and limitations of applying these powerfulmodels to science and security applications.
Revised: However, only limited efforts have investigated the opportunities and limitations of applying these powerful models to science and security applications.

Original: Inthis work we develop foundation models of scientific knowledge for chemistry to augment scientists with the advanced ability to perceive andreason at scale previously unimagined.
Revised: In this work, we develop foundation models of scientific knowledge for chemistry to augment scientists with the advanced ability to perceive and reason at scale previously unimagined.

Original: Specifically, we build large-scale (1.47B parameter)general-purpose models for chemistry that canbe effectively used to perform a wide range ofin-domain and out-of-domain tasks.
Revised: Specifically, we build large-scale (1.47B parameter) general-purpose models for chemistry that can be effectively used to perform a wide range of in-domain and out-of-domain tasks.

Original: Evaluatingthese models in a zero-shot setting, we analyzethe effect of model and data scaling, knowledgedepth, and temporality on model performancein context of model training efficiency.
Revised: Evaluating these models in a zero-shot setting, we analyze the effect of model and data scaling, knowledge depth, and temporality on model performance in context of model training efficiency.

Original: Our novel findings demonstrate that (1)unlikeprevious work (Kaplan et al., 2020) we showthat modelperformance is driven by the modelsize rather than the data size  when evaluatedin a zero-shot setting; (2) data quality (aka diversity) affects model performance more thandata quantity; (3) similarly, unlike previouswork (Luu et al., 2021) temporal order of thedocuments in the corpus boosts model performance only for specific tasks, e.g., SciQ; and(4) models pre-trained from scratch performbetter on in-domain tasks than those tuned fromgeneral-purpose models like Open AI’s GPT-2.
Revised: Our novel findings demonstrate that (1) model size significantly contributes to the task performance when evaluated in a zero-shot setting; (2) data quality (aka diversity) affects model performance more than data quantity; (3) similarly, unlike previous work (Luu et al., 2021) temporal order of the documents in the corpus boosts model performance only for specific tasks, e.g., SciQ; and (4) models pre-trained from scratch perform better on in-domain tasks than those tuned from general-purpose models like Open AI’s GPT-2.

Original:  Introduction
Revised:  Introduction

Original: The emergence of foundation models (Bommasani et al., 2021) such as large-scale autoencoding models ( e.g., BERT (Devlin et al., 2018),
Revised: The emergence of foundation models (Bommasani et al., 2021) such as large-scale autoencoding models ( e.g., BERT (Devlin et al., 2018),

Original: RoBERTa (Liu et al., 2019)) and autoregressivelanguage models ( e.g., GPT-2 (Radford et al. ,2019), GPT-3 (Brown et al., 2020), MegatronTuring (Smith et al., 2022) and Gopher (Raeet al., 2021)) as well as multimodal vision andlanguage models, such as FLAVA (Singh et al.,2021) and Perceiver (Jaegle et al., 2021), established a paradigm shift in Artificial Intelligence (AI).
Revised: RoBERTa (Liu et al., 2019)) and autoregressive language models ( e.g., GPT-2 (Radford et al., 2019), GPT-3 (Brown et al., 2020), MegatronTuring (Smith et al., 2022) and Gopher (Rae et al., 2021)) as well as multimodal vision and language models, such as FLAVA (Singh et al., 2021) and Perceiver (Jaegle et al., 2021), established a paradigm shift in Artificial Intelligence (AI).

Original: These foundation models, also called neural platforms , are built using self-supervised pretraining at scale.
Revised: These foundation models, also called neural platforms , are built using self-supervised pretraining at scale.

Original: They are then able to be easily adapted to a wide range of downstream tasks via transfer learning (Bommasani et al., 2021) and finetuning (Lee et al., 2019).
Revised: They are then able to be easily adapted to a wide range of downstream tasks via transfer learning (Bommasani et al., 2021) and finetuning (Lee et al., 2019).

Original: The wide community adoption of foundation models can be explained by their key properties, two of which are emergent behavior and homogenization – which also make foundation models appealing for adaption across science and security domains.
Revised: The wide community adoption of foundation models can be explained by their key properties, two of which are emergent behavior and homogenization – which also make foundation models appealing for adaption across science and security domains.

Original: Emergence, or emergent behavior, reflect new behaviors that a model introduces or is capable of that it was not explicitly trained to perform.
Revised: Emergence, or emergent behavior, reflect new behaviors that a model introduces or is capable of that it was not explicitly trained to perform.

Original: Homogenization is the consolidation of methods for building machine learning systems across a wide range of tasks.
Revised: Homogenization is the consolidation of methods for building machine learning systems across a wide range of tasks.

Original: Another key advantage of scaling language models is that they perform competitively on language tasks using in-context learning without fine-tuning or gradient updates.
Revised: Another key advantage of scaling language models is that they perform competitively on language tasks using in-context learning without fine-tuning or gradient updates.

Original: Thus, in-context learning allows foundation models to be effectively used across new downstream tasks with only simple instructions and a few optional examples.
Revised: Thus, in-context learning allows foundation models to be effectively used across new downstream tasks with only simple instructions and a few optional examples.

Original: In this work we focus on a science domain (chemistry) and demonstrate the value and limitations of large-scale language models evaluated across a wide range of in-domain (science-focused) and out-of-domain tasks.
Revised: In this work we focus on a science domain (chemistry) and demonstrate the value and limitations of large-scale language models evaluated across a wide range of in-domain (science-focused) and out-of-domain tasks.

Original: Unlike the majority of work on foundation models that focuses on pretraining these models on book corpora, web pages,
Revised: Unlike the majority of work on foundation models that focuses on pretraining these models on book corpora, web pages,

Original: Wikipedia and mixed sources, e.g., the Pile (Gao et al., 2020), we pretrain our models on scientific literature.
Revised: Wikipedia and mixed sources, e.g., the Pile (Gao et al., 2020), we pretrain our models on scientific literature.

Original: Using scientific literature presents unique opportunities and challenges.
Revised: Using scientific literature presents unique opportunities and challenges.

Original: Opportunities include the scale and diversity of scientific literature, the explicit structure, and explicit alignment across different modalities in the papers, e.g., table and figure references.
Revised: Opportunities include the scale and diversity of scientific literature, the explicit structure, and explicit alignment across different modalities in the papers, e.g., table and figure references.

Original: Challenges include limited benchmarks that can be used to perform model evaluation, model prompting and interactions.
Revised: Challenges include limited benchmarks that can be used to perform model evaluation, model prompting and interactions.

Original: There are three major contributions of this work:(1) we collect and release a 0.67TB dataset covering research publication data across 10+ sourcesfor chemistry; (2) we release 27 auto-regressivefoundation models for chemistry that have beenpretrained from scratch; and (3) we present a rigorous evaluation of model performance on 15+ indomain and out-of-domain tasks that investigatesthe effects of model and data scaling, knowledgedepth (aka diversity), and temporal order on performance as described in research questions below.(RQ1) Science-Focused Benchmarks
Revised: There are three major contributions of this work: (1) we collect and release a 0.67TB dataset covering research publication data across 10+ sources for chemistry; (2) we release 28 auto-regressive foundation models for chemistry that have been pretrained from scratch; and (3) we present a rigorous evaluation of model performance on 15+ indomain and out-of-domain tasks that investigates the effects of model and data scaling, knowledge depth (aka diversity), and temporal order on performance as described in research questions below. (RQ1) Science-Focused Benchmarks

Original: What arethe strengths and weaknesses of foundation modelspretrained on scientific literature when evaluatedon out-of-domain vs. in-domain tasks?(RQ2) Scaling Effect
Revised: What are the strengths and weaknesses of foundation models pretrained on scientific literature when evaluated on out-of-domain vs. in-domain tasks? (RQ2) Scaling Effect

Original: How does model scale affect the downstream performance?
Revised: How does model scale affect the downstream performance?

Original: Do neural scaling laws presented in (Kaplan et al., 2020; Tay et al.,2021) hold for the foundation models for science?(RQ3)
Revised: Do neural scaling laws presented in (Kaplan et al., 2020) hold for the foundation models for science?

Original:  Diversity Effect
Revised: (RQ3) Diversity Effect

Original: How does the depth ofscientific knowledge, e.g., from paper abstracts vs.full text, affect downstream performance?(RQ4)
Revised: How does the depth of scientific knowledge, e.g., from paper abstracts vs. full text, affect downstream performance?

Original:  Temporal Effect
Revised: (RQ4) Temporal Effect

Original: How does the recencyof scientific knowledge, e.g., when manipulatingthe temporal order of the documents processed bythe model, affect downstream performance?
Revised: How does the recency of scientific knowledge, e.g., when manipulating the temporal order of the documents processed by the model, affect downstream performance?

Original:  Related Work
Revised:  Related Work

Original: In this section we summarize previous efforts intwo categories: mixed-domain continual pretraining that continues pretraining of a base modelon domain data and in-domain pretraining fromscratch that pretrains a from scratch on domaindata.
Revised: In this section we summarize previous efforts in two categories: mixed-domain continual pretraining that continues pretraining of a base model on domain data and in-domain pretraining from scratch that pretrains a from scratch on domain data.

Original: We present a model summary in Table 1.
Revised: We present a model summary in Table 1.

Original: Mixed-Domain Continual Pretraining
Revised: Domain Continual Pretraining

Original: Manyefforts have focused on continual pretraining of a
Revised: Many efforts have focused on continual pretraining of a

Original: BERT (Devlin et al., 2018) base model.
Revised: BERT (Devlin et al., 2018) base model.

Original: Severalmodels have been developed for the biomedicaldomain and the most frequently used corpora fordomain-specific continual preraining are PubMedabstracts and PubMed Central full-text articles(PMC) (Lee et al., 2020; Peng et al., 2019; Phanet al., 2021).
Revised: Several models have been developed for the biomedical domain and the most frequently used corpora for domain-specific continual preraining are PubMed abstracts and PubMed Central full-text articles (PMC) (Lee et al., 2020; Peng et al., 2019; Phan et al., 2021).

Original: In the Chemistry domain, (Guo et al.,2021) performed continual pretraining of a base
Revised: In the Chemistry domain, Guo et al. (2021) performed continual pretraining of a base

Original: BERT model on 200K chemistry journal articlesfor product extraction(ChemBERT) and reactionrole labeling (ChemRxnBERT).
Revised: BERT model on 200K chemistry journal articles extraction

Original: In-Domain Pretraining from Scratch
Revised: In-Domain Pretraining from Scratch

Original: Previous work has shown that pretraining models from scratch on domain-specific data has a significant benefit over continual pretraining of generaldomain language models (Gu et al., 2021).
Revised: Previous work has shown that pretraining models from scratch on domain-specific data has a significant benefit over continual pretraining of generaldomain language models (Gu et al., 2021).

Original: This is mainly due to the availability of in-domain data for both generating the vocabulary and pretraining.
Revised: This is mainly due to the availability of in-domain data for both generating the vocabulary and pretraining.

Original: SciBERT (Beltagy et al., 2019) is pretrained according to this procedure using the vocabulary generated from computer science and biomedical domains.
Revised: SciBERT (Beltagy et al., 2019) is pretrained according to this procedure using the vocabulary generated from computer science and biomedical domains.

Original: PubMedBERT (Gu et al., 2021) is another example of pretraining the base BERT model from scratch using PubMed.
Revised: PubMedBERT (Gu et al., 2021) is another example of pretraining the base BERT model from scratch using PubMed.

Original: Unlike any previous work, we use both continual and from scratch pretraining to build the largest foundation model for
Revised: Unlike any previous work, we use both continual and from scratch pretraining to build the largest foundation model for

Original: Chemistry (1.47B) on the largest (0.67TB) and the most diverse corpus (10+ sources) collected to date.
Revised: Chemistry (1.47B) on the largest (0.67TB) and the most diverse corpus (10+ sources) collected to date.

Original:  Model Pretraining
Revised:  Model Pretraining

Original: Unlike the majority of related models that rely on a base BERT (or variant) model, we adapt the OpenAI’s GPT-2 transformer decoder architecture (Radford et al., 2019) to train autoregressive language models for Chemistry.
Revised: Unlike the majority of related models that rely on a base BERT (or variant) model, we adapt the OpenAI’s GPT-2 transformer decoder architecture (Radford et al., 2019) to train autoregressive language models for Chemistry.

Original: To understand the impact of modelshape and size (RQ2), we experiment with four different Transformer sizes: small (S), medium (M), large (L), and extra-large (XL).
Revised: To understand the impact of model size (RQ2), we experiment with four different Transformer sizes: small (S), medium (M), large (L), and extra-large (XL).

Original: These models differ in the number of decoder layers, hidden size of the model, and the number of attention heads in transformer blocks as shown in Table 2.
Revised: These models differ in the number of decoder layers, hidden size of the model, and the number of attention heads in transformer blocks as shown in Table 2.

Original: Our experiments leverage the GPT-NeoX Python library (Andonian et al., 2021) developed with
Revised: Our experiments leverage the GPT-NeoX Python library (Andonian et al., 2021) developed with

Original: Megatron (Shoeybi et al., 2019) and DeepSpeed (Rasley et al., 2020).
Revised: Megatron (Shoeybi et al., 2019) and DeepSpeed (Rasley et al., 2020).

Original: We optimize the autoregressive log-likelihood ( i.e., cross-entropy loss) averaged over a 2048-token context.
Revised: We optimize the autoregressive log-likelihood ( i.e., cross-entropy loss) averaged over a 2048-token context.

Original: We control the micro batch size per GPU as 4, set the learning rate to 2 × 10 − 4 , and rely on the cosine decay.
Revised: We set the micro batch size per GPU as 4, and the learning rate to 2 × 10 − 4 , and rely on the cosine decay.

Original: We use an Adam optimizer with β 1 = 0 .
Revised: We use an Adam optimizer with β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 99 , and σ = 10 − 

Revised: 99 , and σ = 10 −

Original: and clip the gradient norm at 1.0.
Revised: and clip the gradient norm at 1.0.

Original: In addition, ZeRO optimizer (Rajbhandari et al., 2019) was used to reduce memory footprint by distributing optimizer states across several processes.
Revised: In addition, ZeRO optimizer (Rajbhandari et al., 2019) was used to reduce memory footprint by distributing optimizer states across several processes.

Original: To reduce memory and increase training throughput, we use mixed-precision training (Rasley et al., 2020) and the parallel attention and feed-forward implementations available in GPT-NeoX (Black et al., 2022).
Revised: To reduce memory and increase training throughput, we use mixed-precision training (Rasley et al., 2020) and the parallel attention and feed-forward implementations available in GPT-NeoX (Black et al., 2022).

Original: We also use the Rotary positional embeddings (Su et al., 2021) instead of the learned positional embeddings used in the GPT-2 model (Radford et al., 2019) because they offer performance
Revised: We also use the Rotary positional embeddings (Su et al., 2021) instead of the learned positional embeddings used in the GPT-2 model (Radford et al., 2019) because they offer performance

Original: [Table]
Revised: [Table]

Original: model configurations

Original: [Table]
Revised: [Table]

Original: advantages in tasks with longer texts by capturingrelative position dependency in self-attention.
Revised: advantages in tasks with longer texts by capturing relative position dependency in self-attention.

Original: Our models are pretrained across multiple workers with data parallelism.
Revised: Our models are pretrained across multiple workers with data parallelism.

Original: As the largest model inour experiments fit on a single GPU, we didn’t usethe model (tensor) or pipeline parallelism.
Revised: As the largest model in our experiments fit on a single GPU, we didn’t use the model (tensor) or pipeline parallelism.

Original: Modelsare pretrained fromthe scratch for a total of 320Ksteps.
Revised: Models are pretrained from scratch for a total of 320K steps.

Original: The original GPT-2 modelsthat are finetuned for 150K steps.
Revised: The original GPT-2 models are fine-tuned for 150K steps.

Original: We perform experiments in2 DGX-A100 machines with 16 80Gb GPUs.
Revised: We perform experiments in a single DGX-A100 machine with 8 80Gb GPUs.

Original:  Data Collection and Processing
Revised:  Data Collection and Processing

Original: We collected a large corpus of 53.45 millionchemistry-focused scientific articles and abstracts,resulting in 670GB of text data.
Revised: We collected a large corpus of 53.45 million chemistry-focused scientific articles and abstracts, resulting in 670GB of text data.

Original: As shown in Table 3, our corpus was collected from 10 differentdata sources: Arxiv, Aminer (AMiner), CORD19 (Wang et al., 2020b), CORE (Pontika et al.,2016), Microsoft Academic Graph (MAG)
Revised: As shown in Table 3, our corpus was collected from 10 different data sources: Arxiv, Aminer (AMiner), CORD19 (Wang et al., 2020b), CORE (Pontika et al., 2016), Microsoft Academic Graph (MAG)

Original: (Wanget al., 2020a), OSTI, PubMed (Gao et al., 2020)(abstracts and fulltexts), and the Web of Science (WoS).
Revised: (Wang et al., 2020a), OSTI, PubMed (Gao et al., 2020) (abstracts and fulltexts), and the Web of Science (WoS).

Original: See Appendix A for full data descriptions.
Revised: See Appendix A for full data descriptions.

Revised: duplication process.

Revised: We split datasets to those that include abstracts 〈 A 〉 vs. full texts 〈 FT 〉 .

Original: [Table]
Revised: [Table]

Original: Because the data sources we relied on comprise research publications from many science domains
Revised: Because the data sources we relied on comprise research publications from many science domains

Original: [Table]
Revised: [Table]

Original: we sampled articles using a list of domain-specific keywords for chemistry to create the dataset summarized in Table 3.
Revised: we sampled articles using a list of domain-specific keywords for chemistry to create the dataset summarized in Table 3.

Original: These keywords were extracted by using a Correlation Explanation (Gallagher et al., 2017) topic model followed by manual filtering by subject matter experts.
Revised: These keywords were extracted by using a Correlation Explanation (Gallagher et al., 2017) topic model followed by manual filtering by subject matter experts.

Original: This resulted in a list of more than 1K chemistry-related entities, ranging from compound names like ethyl acetate , methyl methacrylate , sulfoxide , etc. to experiment and procedures like tunneling microscopy , neutralization , enzymatic hydrolysis , etc.
Revised: This resulted in a list of more than 1K chemistry-related entities, ranging from compound names like ethyl acetate , methyl methacrylate , sulfoxide , etc. to experiment and procedures like tunneling microscopy , neutralization , enzymatic hydrolysis [Table] etc.

Original: [Figure]
Revised: [Figure]

Original: Data Cleaning
Revised: Data Cleaning

Original: Recent research has shown thatduplicates in training data can significantly impactthe downstream task performance of LLMs (Leeet al., 2021;
Revised: Recent research has shown that duplicates in training data can significantly impact the downstream task performance of LLMs (Lee et al., 2021;

Original: Carlini et al., 2022).
Revised: Carlini et al., 2022).

Original: To this end, weperformed deduplication of our corpus based onoverlap of titles within and across data sources.
Revised: To this end, we performed deduplication of our corpus based on overlap of titles within and across data sources.

Original: Weprocessed titles to strip punctuation and casefoldand considered two articles A 1 and A 2 to be duplicates if they had the same processed title.
Revised: We processed titles to strip punctuation and casefold and considered two articles A 1 and A 2 to be duplicates if they had the same processed title.

Original: Withthis technique, we were able to remove significantamounts of duplicate scientific articles both withinand across sources.
Revised: With this technique, we were able to remove significant amounts of duplicate scientific articles both within and across sources.

Original: The deduplication process reduced our corpus from 875GB to 670GB (67.8M to53.5M publications), removing 14.3M duplicates.
Revised: The deduplication process reduced our corpus from 875GB to 670GB (67.8M to 53.5M publications), removing 14.3M duplicates.

Original: Tokenization
Revised: Tokenization

Original: As used in GPT-2 model, we use a
Revised: As used in GPT-2 model, we use a

Original: Byte Pair Encoding (BPE) tokenizer.
Revised: Byte Pair Encoding (BPE) tokenizer.

Original: We train BPEtokenizers for each data sample with a vocabularysize of 64K as preliminary experiments varyingvocabulary sizes from 64K to 256K for smallerscale model pretraining did not show significantdifferences in performance.
Revised: We train BPE tokenizers for each data sample with a vocabulary size of 64K as preliminary experiments varying vocabulary sizes from 64K to 256K for smaller scale model pretraining did not show significant differences in performance.

Original: We compare the GPT2 vocabulary generated from the WebText and thein-domain vocabularies generated from our corpora and find that the in-domain vocabulary breakschemical entities into fewer tokens.
Revised: We compare the GPT2 vocabulary generated from the WebText and the in-domain vocabularies generated from our corpora and find that the in-domain vocabulary breaks chemical entities into fewer tokens.

Original: For example,dimethylnitroxide was tokenized into #dimethyl,#nitr, #oxide using the in-domain vocabulary and#dim, #ethyl, #nit, #rox, #ide using the GPT-2 vocabulary.
Revised: For example, dimethylnitroxide was tokenized into #dimethyl, #nitr, #oxide using the in-domain vocabulary and #dim, #ethyl, #nit, #rox, #ide using the GPT-2 vocabulary.

Original:  Analysis and Results
Revised:  Analysis and Results

Original: This section presents the analysis of 32 pretrained models evaluated on 15+ in-domain and out-ofdomain downstream tasks ( RQ1 , Section 5.1).
Revised: This section presents the analysis of 28 pretrained models evaluated on 15+ in-domain and out-ofdomain downstream tasks ( RQ1 , Section 5.1).

Original: We investigate the effects of model and data scaling ( RQ2 , Section 5.2), knowledge diversity ( RQ3 , Section 5.3), and temporal order ( RQ4 , Section 5.4) on the downstream performance.
Revised: We investigate the effects of model and data scaling ( RQ2 , Section 5.2), knowledge diversity ( RQ3 , Section 5.3), and temporal order ( RQ4 , Section 5.4) on the downstream performance.

Original: We also compare the results from continual vs. from scratch pretraining (Section 5.5) and present the analysis of large-scale training efficiency (Section 5.6).
Revised: We also compare the results from continual vs. from scratch pretraining (Section 5.5) and present the analysis of large-scale training efficiency (Section 5.6).

Original: Baseline Models
Revised: Baseline Models

Original: As we use a similar model architecture, we identify Open AI’s GPT-

Revised: As we use a similar model architecture, we identify Open AI’s GPT-

Original: (Radford et al., 2019) as a baseline comparison model.
Revised: (Radford et al., 2019) as a baseline comparison model.

Original: We compare our performance with four variants of the original GPT-2 models, corresponding to small (S), medium (M), large (L), and extra-large (XL) sized transformer architectures shown in Table 2.
Revised: We compare our performance with four variants of the original GPT-2 models, corresponding to small (S), medium (M), large (L), and extra-large (XL) sized transformer architectures shown in Table 2.

Original: We note that GPT-2 models were pretrained on WebText – 8 million web documents (40Gb).
Revised: We note that GPT-2 models were pretrained on WebText – 8 million web documents (40Gb).

Original: Thus, we also include a base GPT-2 model (medium) that has been updated with continual pretraining using our Combined 〈 A+FT 〉 dataset. 
Revised: Thus, we also include a base GPT-2 model (medium) that has been updated with continual pretraining using our Combined 〈 A+FT 〉 dataset. Our Models

Revised: We pretrained models with individual datasets (AMiner, CORE, MAG, PubMed,

Revised: S2ORC, WOS) and combined abstracts and fulltexts.

Revised: Our goal is to systematically study data biases in the model performance when pretraining models with individual datasets.

Revised: For example,

Original: 5. Zero-shot Performance
Revised: PubMed publications cover mostly bio-medicinal terms (Gu et al., 2021), while the majority of S2ORC publications are from medicine, biology, physics, and mathematics (Lo et al., 2020). We only use 4 GPUs for the models pretrained with individual datasets and 8 GPUs for the rest. This is to control the number of tokens seen during model pretraining (320,000 steps * 4 GPUs * 4 micro batch size * 2,048 context size = 10B tokens) relative to the maximum number of tokens available in the respective datasets (as reported in Table 3).

Revised: We also trained one XL (4x) model with 4x larger batch size than what used in XL model to evaluate the impact of the number of training tokens.

Revised: 5. Zero-shot Performance

Original: We evaluate our models using several benchmarks to assess the effectiveness in both in-domain and out-of-domain tasks.
Revised: We evaluate our models using several benchmarks to assess the effectiveness in both in-domain and out-of-domain tasks.

Original: The benchmarks we include are described in Appendix B.
Revised: The benchmarks we include are described in Appendix B.

Original: We use the lmevaluation-harness Python repository (Gao et al.,  2021) for the benchmark implementation.
Revised: We use the lmevaluation-harness Python repository (Gao et al., [Table] 2021) for the benchmark implementation.

Original: In-domain Evaluation
Revised: In-domain Evaluation

Original: We consider five existing chemistry benchmarks, specifically HendrycksTest (Hendrycks et al., 2020) for high school (HT-HC) and college (HT-CC) levels, and sciencefocused – ARC (Clark et al., 2018), SciQ (Welbl et al., 2017), OpenBookQA (Mihaylov et al., 2018),
Revised: We consider five existing chemistry benchmarks, specifically HendrycksTest (Hendrycks et al., 2020) for high school (HT-HC) and college (HT-CC) levels, and sciencefocused – ARC (Clark et al., 2018), SciQ (Welbl et al., 2017), OpenBookQA (Mihaylov et al., 2018),

Original: Pile-PubMed-Abstracts (Gao et al., 2020)).
Revised: Pile-PubMed-Abstracts (Gao et al., 2020)).

Original: As shown in Table 4, one or more of our models outperform baseline GPT-2 models for the two chemistry tasks and the science-focused language modelling, with matched best performance on SciQ. Of the remaining tasks, our models perform within 1-4% of GPT-2 baselines.
Revised: As shown in Table 4, one or more of our models outperform baseline GPT-2 models for the two chemistry tasks, general science QA (SciQ) and the sciencefocused language modelling. Of the remaining tasks, our models perform within 1-4% of GPTbaselines.

Original: Out-of-domain Evaluation
Revised: Out-of-domain Evaluation

Original: We evaluate outof-domain performance using 9 commonly used
Revised: We evaluate outof-domain performance using 9 commonly used

Original: LLM benchmarks: BoolQ (Clark et al., 2019),
Revised: LLM benchmarks: BoolQ (Clark et al., 2019),

Original: CB (De Marneffe et al., 2019), WIC (Pilehvar and
Revised: CB (De Marneffe et al., 2019), WIC (Pilehvar and

Original: Camacho-Collados, 2018), WSC (Levesque et al.,[Table] 2012), MathQA ( Amini et al., 2019), PIQA (Bisket al., 2020), PubMedQA (Jin et al., 2019), Lambada (Paperno et al., 2016) and WikiText (Merityet al., 2016).
Revised: Camacho-Collados, 2018), WSC (Levesque et al., 2012), MathQA (Amini et al., 2019), PIQA (Bisk et al., 2020), PubMedQA (Jin et al., 2019), Lambada (Paperno et al., 2016) and WikiText (Merity et al., 2016).

Original: As shown in Table 5, our models outperform baseline GPT-2 models for CB, WIC and
Revised: As shown in Table 5, our models outperform baseline GPT-2 models for CB, WIC and

Original: WSC and match the best accuracy for BoolQ butthe GPT-2 baselines outperform on the remainingtasks, particularly Lambada and Wikitext – the twogeneral language modeling tasks.
Revised: WSC and match the best accuracy for BoolQ but the GPT-2 baselines outperform on the remaining tasks, particularly Lambada and Wikitext – the two general language modeling tasks.

Original: 5. Scaling Effect
Revised: 5. Scaling Effect

Original: Previous work (Kaplan et al., 2020) has shown thatupstream cross entropy loss scales as a power-lawwith model size, dataset size, and the amount ofcompute.
Revised: Previous work (Kaplan et al., 2020) has shown that upstream cross entropy loss scales as a power-law with model size, dataset size, and the amount of compute.

Original: More recently, Tay et al.

Original: (Tay et al., 2021)reported a distinct discrepancy when consideringpractical downstream performance along with thereported model loss.

Original: In this section, we revisit theseclaims on scaling Transformer architectures.
Revised: In this section, we revisit these claims on scaling Transformer architectures.

Original: Analyzing upstream cross entropy loss
Revised: Analyzing upstream cross entropy loss

Original: During pretraining, we group each dataset into training/validation/test (949/50/1) splits.
Revised: During pretraining, we group each dataset into training/validation/test (949/50/1) splits.

Original: We report the model performance on validation data using cross entropy loss in nats.

Original: This measure will be averaged over the 2048-token context.

Original: We find that the cross entropy loss decreases as we increase the model size (as shown in Figure 2). Larger models reach a given loss value in a higher rate than the smaller models. This observation illustrates the relationship between model performance (as measured by the upstream cross entropy loss) and model size, confirming (Kaplan et al., 2020). Our results also illustrate that the model shape, as measured by the number of decoder layers (depth) and hidden size (width), also contributes to themodel performance (Tay et al., 2021).
Revised: We report the

Original: For example, when the hidden size increases from M to XL models (as reported in Table 2), the model performance increases (as shown in Figure 2).

Original: Analyzing downstream task performance

Original: Can we speculate downstream task performance of a model from the pretraining performance?

Original: We find

Original: [Table]
Revised: [Table]

Revised: 100.53 M 0.

Original: 77.96 L 0.
Revised: 77.96 L 0.

Original: 69.62 XL 0.
Revised: 69.62 XL 0.

Original: MAG
Revised: MAG

Original: S 0.
Revised: S 0.

Original: 118.40 M 0.
Revised: 118.40 M 0.

Original: 91.36 L 0.
Revised: 91.36 L 0.

Original: 67.74 XL 0.
Revised: 67.74 XL 0.

Original: PubMed-F
Revised: PubMed-F

Original: S 0.
Revised: S 0.

Original: 148.88 M 0.
Revised: 148.88 M 0.

Original: 119.74 L 0.
Revised: 119.74 L 0.

Original: 95.75 XL 0.
Revised: 95.75 XL 0.

Original: S2ORC
Revised: S2ORC

Original: S 0.
Revised: S 0.

Original: 403.48 M 0.
Revised: 403.48 M 0.

Original: 330.56 L 0.

Original: [Table]
Revised: [Table]

Revised: model performance on validation data using cross entropy loss in nats.

Revised: This measure will be averaged over the 2048-token context.

Revised: We find that the cross entropy loss decreases as we increase the model size (as shown in Figure 2).

Revised: Larger models reach a given loss value in a higher rate than the smaller models.

Revised: This observation illustrates the relationship between model performance (as measured by the upstream cross entropy loss) and model size, confirming (Kaplan et al., 2020).

Original:  that the models perform considerably well on PilePubMed-Abstract in comparison to the Lambadaor WikiText .
Revised: Can we speculate downstream task performance of a model from the pretraining performance? First, we find that the models perform considerably well on Pile in comparison to the Lambada or WikiText .

Revised: There is a 48% performance advantage in this task over the best performing baseline GPT-2 model.

Original: This may be due to  models capturingscientific language better than general language.
Revised: This may be due to the models capturing scientific language better than general language.

Revised: It is im- portant to note that we exclude PubMed Abstracts in the individual data collection to avoid potential contamination between the training and Pile testing data.

Original: As shown in Table 4, larger models perform wellon these language modeling tasks.
Revised: As shown in Table 4, larger models perform well on these language modeling tasks.

Revised: Second, we noticed that the XL (4x) model trained for more tokens performs significantly better than the similar sized XL model.

Revised: Specifically,

Revised: XL (4x) model was trained with 128 total batch size compared to the 32 total batch size used in XL model.

Revised: XL (4x) model achieves the lowest Lambada and WikiText perplexity values across all our models trained from scratch (as shown in Table 5).

Revised: The same model also achieves the best SciQ performance with 0.84 accuracy and comparable in other tasks performance with the XL model.

Original: Second, we find that zero-shot task performancein SciQ, HT-CC and ARC-E increases as we increase the model size (see Table 5).
Revised: This experiment highlights the importance of training models with larger batch size. We note that the baseline models (Radford et al., 2019) were trained with 4x larger batch size (total batch size 512) than what used in XL (4x) model. We believe that the XL (4x) model can reach the similar perplexity values when trained for this data scale. Third, we find that zero-shot task performance in SciQ, HT-CC and ARC-E increases as we increase the model size (see Table 5).

Original: However, thereis no clear relationship between the task perfor- mance and the model sizes in the rest of benchmark datasets.
Revised: However, there is no clear relationship between the task performance and the model sizes in the rest of benchmark datasets.

Original: We suggest that pretraining performance may not be the ideal indicator to speculate the overall downstream task performance, especially in the zero-shot setting.
Revised: We suggest that pretraining performance may not be the ideal indicator to speculate the overall downstream task performance, especially in the zero-shot setting.

Original: However, model size significantly contributes to the task performance.
Revised: However, model size significantly contributes to the task performance.

Original: 5. Diversity Effect
Revised: 5. Diversity Effect

Original: While abstracts often provide a summary of scientific publications, the full text contains more details.
Revised: While abstracts often provide a summary of scientific publications, the full text contains more details.

Original: In this section, we analyze the performance of models trained on paper abstracts versus full texts.
Revised: In this section, we analyze the performance of models trained on paper abstracts versus full texts.

Original: First, the XL models trained with the combined abstract dataset achieve the lowest perplexity score (22.77) in the Pilelanguage modeling task – a 45% performance advantage over the full text-based model.
Revised: First, the XL models trained with the combined abstract dataset achieve the lowest perplexity score (22.77) on the Pile – a 45% performance advantage over the full text version.

Original: There are might be several factors that contribute to this, but one may be the focused language in abstracts.
Revised: There are might be several factors that contribute to this, but one may be the focused language in abstracts.

Original: Second, the model trained with the combined abstracts achieves the  best accuracy (0.83 in comparison to 0.79 for the full text model) inSciQ. Some of the models pretrained on individ-
Revised: Second, the model trained with the combined abstracts achieves the second best accuracy (0. in comparison to 0.79 for the full text model) in

Original: [Figure]

Original: ual abstract data achieve comparable performancein SciQ, e.g., MAG and AMiner models achieve0.8 and 0.78 accuracy, respectively.
Revised: SciQ. Some of the models pretrained on individual abstract data achieve comparable performance in SciQ, e.g., MAG and AMiner models achieve 0. and 0.78 accuracy, respectively.

Original: We believe thediversity of scientific knowledge provided from theabstract data is useful since SciQ questions spanbiology, chemistry, earth science, and physics.
Revised: We believe the diversity of scientific knowledge provided from the abstract data is useful since SciQ questions span biology, chemistry, earth science, and physics.

Original: When we compare model performance trainedwith abstracts vs. full texts in the Hendrycks Test(HT), we see that the best accuracy is achievedusing the MAG and S2ORC datasets rather thanthe combined abstracts.
Revised: Third, we compare model performance trained with abstracts vs. full texts in the HT task and see that the best accuracy is achieved using the MAG and S2ORC datasets rather than the combined abstracts.

Revised: This suggests the importance of contextual knowledge provided by different data sources.

Original: Finally, combined fulltext model performs better than the model trainedwith the abstracts in all of the out-of-domain evaluation except PIQA.
Revised: Finally, combined full text model performs better than the model trained with the abstracts in all outof-domain tasks except PIQA.

Original: This performance differencemay be due to the more expressive and diverselanguage presented in the full texts than in the abstracts.
Revised: This performance difference may be due to the more expressive and diverse language presented in the full texts than in the abstracts.

Original: Thus, expanding full text coverage mayimprove out-of-domain task generalization.
Revised: Thus, expanding full text coverage may improve out-of-domain task generalization.

Original: 5. Temporal Effect
Revised: 5. Temporal Effect

Original: Scientific knowledge evolves over time reflectingnew research ideas, innovations, and findings.
Revised: Scientific knowledge evolves over time reflecting new research ideas, innovations, and findings.

Original: Inthis section, we test how continual pretraining ontemporal-aligned scientific publications impactsdownstream performance.
Revised: In this section, we test how continual pretraining on temporal-aligned scientific publications impacts downstream performance.

Original: For this experiment,we maintain two variants of the MAG datasetwith random-ordered and temporal-ordered articles,splitting each into ten equal subsets.
Revised: For this experiment, we maintain two variants of the MAG dataset with random-ordered and temporal-ordered articles, splitting each into ten equal subsets.

Original: We continuepretraining a base medium (M) sized model iteratively with the subsets in the order they appearedin the respective data variant.
Revised: We continue pretraining a base medium (M) sized model iteratively with the subsets in the order they appeared in the respective data variant.

Original: For example, in thetemporally-aligned experiments, we first pretrain amodel with 3.4M (10%) articles from before 1978,and then use it as the base model to continue pretraining with another 3.4M (10%) articles frombetween 1978 and 1989.
Revised: For example, in the temporally-aligned experiments, we first pretrain a model with 3.4M (10%) articles from before 1978, and then use it as the base model to continue pretraining with another 3.4M (10%) articles from between 1978 and 1989.

Original: We train the initial modelfor 150K steps and each subsequent model for 10K steps with additional data.
Revised: We train the initial model for 150K steps and each subsequent model for 10K steps with additional data.

Original: Figure 3 shows the performance of model checkpoints across in-domain and out-of-domain tasks.
Revised: Figure 3 shows the performance of model checkpoints across in-domain and out-of-domain tasks.

Original: There are two key findings.
Revised: There are two key findings.

Original: First, SciQ and ARCE zero-shot task performances improve over time with the models trained with temporally-ordered scientific texts (as shown in Figure 3b).
Revised: First, SciQ and ARCE zero-shot task performances improve over time with the models trained with temporally-ordered scientific texts (as shown in Figure 3b).

Original: For example, SciQ accuracy improves from 0.64 to 0.
Revised: For example, SciQ accuracy improves from 0.64 to 0.

Original: from the base model checkpoint to the final model checkpoint.
Revised: from the base model checkpoint to the final model checkpoint.

Original: Similarly, ARC-E accuracy improves from 0.43 to 0.45.
Revised: Similarly, ARC-E accuracy improves from 0.43 to 0.45.

Original: This is due to the temporal order of the knowledge acquired by the model.
Revised: This is due to the temporal order of the knowledge acquired by the model.

Original: When the model was pretrained with random-ordered data subsets, we observe only a slight ( < 1% ) performance increase (as shown in Figure 3a).
Revised: When the model was pretrained with random-ordered data subsets, we observe only a slight ( < 1% ) performance increase (as shown in Figure 3a).

Original: There are mixed patterns in performance across out-of-domain tasks.
Revised: There are mixed patterns in performance across out-of-domain tasks.

Original: For example, a slight performance increase in the PIQA, CB, PubMedQA, and WIC over time with the models trained with temporally-ordered scientific texts.
Revised: For example, a slight performance increase in the PIQA, CB, PubMedQA, and WIC over time with the models trained with temporally-ordered scientific texts.

Original: On the other hand, there is a performance drop in the BoolQ and WSC over time.
Revised: On the other hand, there is a performance drop in the BoolQ and WSC over time.

Original: This may be due to the catastrophic forgetting prevalent in continual learning (Ramasesh et al., 2021).
Revised: This may be due to the catastrophic forgetting prevalent in continual learning (Ramasesh et al., 2021).

Original: Future work will investigate other confounding factors that may contribute to this performance patterns.
Revised: Future work will in- [Figure] vestigate other confounding factors that may contribute to this performance patterns.

Original: 5. Continual vs. From Scratch Pretraining
Revised: 5. Continual vs. From Scratch Pretraining

Original: In this section, we test whether the continual pretraining of a base GPT model with additional domain-specific data is helpful in the downstream task performance.
Revised: In this section, we test whether the continual pretraining of a base GPT model with additional domain-specific data is helpful in the downstream task performance.

Original: We report the zero-shot performance of the tuned model across in-domain (Table 4) and out-of-domain (Table 5) tasks.
Revised: We report the zero-shot performance of the tuned model across in-domain (Table 4) and out-of-domain (Table 5) tasks.

Original: We have two main observations from this experiment.
Revised: We have two main observations from this experiment.

Original: [Figure]

Original: First, fine-tuned models fall behind other baselines in a majority of in-domain tasks.
Revised: First, fine-tuned models fall behind other baselines in a majority of in-domain tasks.

Original: HT-CC isthe only in-domain task that the tuned model outperforms the rest of models, yet fails to outperformthe best performing model trained from scratch.
Revised: HT-CC is the only in-domain task that the tuned model outperforms the rest of models, yet fails to outperform the best performing model trained from scratch.

Original: Second, fine-tuned models have a significant performance drop in the general language modelingtasks (Lambada and Wikitext).
Revised: Second, fine-tuned models have a significant performance drop in the general language modeling tasks (Lambada and Wikitext).

Original: For example, thetuned model records 6x performance drop in the
Revised: For example, the tuned model records 6x performance drop in the

Original: Wikitext compared to the best performing model.
Revised: Wikitext compared to the best performing model.

Original: There are several factors in the continual pretraining that may contribute to this.
Revised: There are several factors in the continual pretraining that may contribute to this.

Original: As the tuned modeluses the original GPT-2 vocabulary, it must use thefragmented general subwords to tokenize the chemistry terms available in our corpora.
Revised: As the tuned model uses the original GPT-2 vocabulary, it must use the fragmented general subwords to tokenize the chemistry terms available in our corpora.

Original: On the otherhand, the tuned model starts with the suboptimalinitialization from the general-domain languagemodel (Gu et al., 2021).
Revised: On the other hand, the tuned model starts with the suboptimal initialization from the general-domain language model (Gu et al., 2021).

Original: This initialization maydiverge the model in the optimization process thatmay not be recovered.
Revised: This initialization may diverge the model in the optimization process that may not be recovered.

Original: 5. Training Efficiency
Revised: 5. Training Efficiency

Original: We use several dimensions to describe the trainingefficiency, i.e., #FLOPs, throughput (speed), andmemory.
Revised: We use several dimensions to describe the training efficiency, i.e., #FLOPs, throughput (speed), and memory.

Original: We compare these compute dimensionsacross the four model sizes described in the Table 2.
Revised: We compare these compute dimensions [Figure] across the four model sizes described in the Table 2.

Original: The smallest (S) model has 59% FLOPs of thelargest (XL) model, twice the speed (steps/s), 32%per device GPU memory savings, and 76% total parameter savings (see Figure 4).
Revised: The smallest (S) model has 59% FLOPs of the largest (XL) model, twice the speed (steps/s), 32% per device GPU memory savings, and 76% total parameter savings (see Figure 4).

Original: With such computebudget, small (S) models only outperforms the XLmodel in 21% in-domain and 34% out-of-domainevaluation tasks.
Revised: With such compute budget, small (S) models only outperforms the XL model in 21% in-domain and 34% out-of-domain evaluation tasks.

Original: This suggests the importance ofscaling foundation models.[Figure]
Revised: This suggests the importance of compute budget required in scaling foundation models.

Original: The largest (XL) model has 92 TFLOPs in average and use 44% GPU memory.

Original: While other larger models (M and L) have 54 TFLOPs in average, their GPU memory allocations ( > 33% ) are quite comparable with the XL model.

Original: Model Parallelism

Original: We introduce model parallelism ( mp ) to scale our models by increasing the width (hidden size) and depth (#layers) of the model architecture.

Original: To this end, we test a model architecture with 32 layers, 32 attention heads, and 2,560 hidden size resulting in 2.7B total parameters.

Original: We trained two such models with ( mp = 8 ) and without ( mp = 1 ) model parallelism using the PubMed full text corpus and the same hyper parameters.

Original: While model parallelism reduces overall training time, it introduces a penalty in performance.

Original: The non-parallelized model achieves the lowest validation loss value (1.64) in the pretraining while the parallelized model achieves 1.

Original: validation loss after 320K training steps.

Original:  Conclusions
Revised:  Conclusions

Original: In this paper, we collected and released 0.67TB of research publication data collected across 10+ sources for chemistry.
Revised: In this paper, we collected and released 0.67TB of research publication data collected across 10+ sources for chemistry.

Original: We pretrained and released 25+ foundation models for chemistry.
Revised: We pretrained and released 25+ foundation models for chemistry.

Original: We rigorously analyzed model performance on 15+ indomain and out-of-domain tasks.
Revised: We rigorously analyzed model performance on 15+ indomain and out-of-domain tasks.

Original: A Data DescriptionsAMiner
Revised: A Data Descriptions AMiner

Original: ArnetMiner (AMiner) is a service thatcrawls research publications, performs profile extraction of scientists, models academic networksby integrating publication data from the existing libraries.
Revised: ArnetMiner (AMiner) is a service that crawls research publications, performs profile extraction of scientists, models academic networks by integrating publication data from the existing libraries.

Original: For the experiments described in this work,we use a sub-sampled version of the data presentedin the Open Academic Graph (OAG) version of the
Revised: For the experiments described in this work, we use a sub-sampled version of the data presented in the Open Academic Graph (OAG) version of the

Original: AMiner dataset, which originally consisted of morethat 172M articles, with 18.5M chemistry-relatedabstracts.
Revised: AMiner dataset, which originally consisted of more that 172M articles, with 18.5M chemistry-related abstracts.

Original: CORE
Revised: CORE

Original: COnnecting REpositories (CORE) (Pontika et al., 2016) is a large-scale aggregation system which provides an open access to the globalnetwork of scientific journals and publications.
Revised: COnnecting REpositories (CORE) (Pontika et al., 2016) is a large-scale aggregation system which provides an open access to the global network of scientific journals and publications.

Original: CORE currently contains more than 207M openaccess articles collected from over 10 thousand dataproviders, out of which more than 92M are open access full-text research papers.
Revised: CORE currently contains more than 207M openaccess articles collected from over 10 thousand data providers, out of which more than 92M are open access full-text research papers.

Original: We sub-sampled theoriginal collect into our chemistry-specific corpusconsisting of more than 7M full-text articles.
Revised: We sub-sampled the original collect into our chemistry-specific corpus consisting of more than 7M full-text articles.

Original: CORDCORD-19 corpus contains COVID19 (Cord19) and other coronavirus-related publications (e.g. SARS, MERS, etc.) from PubMed’s
Revised: CORD-19 corpus contains COVID19 (Cord19) and other coronavirus-related publications (e.g. SARS, MERS, etc.) from PubMed’s

Original: PMC open access corpus, bioRxiv, and medRxivpre-prints, in addition to COVID-19 articles maintained by the World Health Organization (WHO).
Revised: PMC open access corpus, bioRxiv, and medRxiv pre-prints, in addition to COVID-19 articles maintained by the World Health Organization (WHO).

Original: MAG
Revised: MAG

Original: Microsoft Academic Graph (MAG) is aheterogeneous graph created by extracting knowledge from scholarly publications on the web (Wanget al., 2020a).
Revised: Microsoft Academic Graph (MAG) is a heterogeneous graph created by extracting knowledge from scholarly publications on the web (Wang et al., 2020a).

Original: The data used in this work is a subsample from the OAG version of the MAG dataset,which originally consisted of > 208M articles, with34M chemistry-related articles with abstracts.
Revised: The data used in this work is a subsample from the OAG version of the MAG dataset, which originally consisted of > 208M articles, with 34M chemistry-related articles with abstracts.

Original: PubMed
Revised: PubMed

Original: PubMed is a domain-specific datasource that allows for search and retrieval of thebiomedical and life sciences literature.
Revised: PubMed is a domain-specific data source that allows for search and retrieval of the biomedical and life sciences literature.

Original: It is maintained by the National Centre for Biotechnology
Revised: It is maintained by the National Centre for Biotechnology

Original: Information (NCBI) at the U.S. National Libraryof Medicine (NLM).
Revised: Information (NCBI) at the U.S. National Library of Medicine (NLM).

Original: For this work we utilized the
Revised: For this work we utilized the

Original: PubMed Central data provided in the Pile corpus(Gao et al., 2020).
Revised: PubMed Central data provided in the Pile corpus (Gao et al., 2020).

Original: As presented in Table 3 thesub-sampled data consists of documents with morethan 280K abstracts and 700K full text articles.
Revised: As presented in Table 3 the sub-sampled data consists of documents with more than 280K abstracts and 700K full text articles.

Original: S2ORC
Revised: S2ORC

Original: The Semantic Scholar Open Research
Revised: The Semantic Scholar Open Research

Original: Corpus (S2ORC) (Lo et al., 2020) is a large academic corpus consisting of 81.1M documents.
Revised: Corpus (S2ORC) (Lo et al., 2020) is a large academic corpus consisting of 81.1M documents.

Original: Thedata includes the metadata, abstracts, bibliographical references and full-text publications for over8M open access research articles.
Revised: The data includes the metadata, abstracts, bibliographical references and full-text publications for over 8M open access research articles.

Original: In this work, we utilize the sub-sampled version of the original data specific to chemistry, which includes more than 10M abstracts
Revised: In this work, we utilize the sub-sampled version of the original data specific to chemistry, which includes more than 10M abstracts.

Original: WoS
Revised: WoS

Original: The Web of Science (WoS) is a multidiscipline citation database produced by the Institute of Scientific Information.
Revised: The Web of Science (WoS) is a multidiscipline citation database produced by the Institute of Scientific Information.

Original: The platform hosts over 171M records across various disciplines, which, when sub-sampled for our chemistry domain, rounded to more than 7M records with abstracts available.
Revised: The platform hosts over 171M records across various disciplines, which, when sub-sampled for our chemistry domain, rounded to more than 7M records with abstracts available.

Original: B Task Descriptions 
Revised: B Task Descriptions HendrycksTest-Chemistry

Original: HendrycksTest-Chemistry The Hendrycks
Revised: The Hendrycks

Original: Test (Hendrycks et al., 2020) is a large scale collection of multiple choice questions covering subjects.
Revised: Test (Hendrycks et al., 2020) is a large scale collection of multiple choice questions coveringsubjects.

Original: In our experiments, we subsampled college chemistry (HT-CC) and high school chemistry (HT-HC).
Revised: In our experiments, we subsampled college chemistry (HT-CC) and high school chemistry (HT-HC).

Original: HT-CC contains 100 questions related to analytical, organic, inorganic, physical, etc.
Revised: HT-CC contains 100 questions related to analytical, organic, inorganic, physical, etc.

Original: and
Revised: and

Original: HT-HC contains 203 questions related chemical reactions, ions, acids and bases, etc.
Revised: HT-HC contains 203 questions related chemical reactions, ions, acids and bases, etc.

Original: ARC
Revised: ARC

Original: The ARC dataset (Clark et al., 2018) contains 7,787 genuine grade-school level, science
Revised: The ARC dataset (Clark et al., 2018) contains 7,787 genuine grade-school level, science

Original: MCQs and is partitioned into a Challenge Set (ARC-C) and an Easy Set (ARC-E).
Revised: MCQs and is partitioned into a Challenge Set (ARC-C) and an Easy Set (ARC-E).

Original: Additionally, 14M science-related sentences are provided with relevant knowledge to answer the ARC questions.
Revised: Additionally, 14M science-related sentences are provided with relevant knowledge to answer the ARC questions.

Original: SciQ
Revised: SciQ

Original: The SciQ dataset (Welbl et al., 2017) contains 13,679 crowdsourced multiple-choice science exam questions about Physics, Chemistry and Biology, among others.
Revised: The SciQ dataset (Welbl et al., 2017) contains 13,679 crowdsourced multiple-choice science exam questions about Physics, Chemistry and Biology, among others.

Original: OpenBookQA
Revised: OpenBookQA

Original: The OpenBookQA (Mihaylov et al., 2018) dataset consists of 5,957 multiple choice questions and 1,326 elementary-level science facts.
Revised: The OpenBookQA (Mihaylov et al., 2018) dataset consists of 5,957 multiple choice questions and 1,326 elementary-level science facts.

Original: The facts alone do not contain enough information to correctly answer the multiple choice questions, therefore the task is designed to evaluate systems beyond paraphrase matching.
Revised: The facts alone do not contain enough information to correctly answer the multiple choice questions, therefore the task is designed to evaluate systems beyond paraphrase matching.

Original: Pile PubMed Abstracts
Revised: Pile PubMed Abstracts

Original: The Pile dataset (Gao et al., 2020) contains 800GB of diverse text sources for benchmarking language models.
Revised: The Pile dataset (Gao et al., 2020) contains 800GB of diverse text sources for benchmarking language models.

Original: We limit this task to only include abstracts from the Pile’s
Revised: We limit this task to only include abstracts from the Pile’s

Original: PubMed collection.
Revised: PubMed collection.

Original: As this is framed as a language modeling task, we report word level perplexity.
Revised: As this is framed as a language modeling task, we report word level perplexity.

Original: BoolQ 
Revised: BoolQ BoolQ

Original: BoolQ (Clark et al., 2019) is a reading comprehension dataset comprised of 16k real, naturally formed queries to the Google search engine with a yes or no answer.
Revised: (Clark et al., 2019) is a reading comprehension dataset comprised of 16k real, naturally formed queries to the Google search engine with a yes or no answer.

Original: Each question-answer pairis accompanied by a Wikipedia article providingevidence to support the correct answer.
Revised: Each question-answer pair is accompanied by a Wikipedia article providing evidence to support the correct answer.

Original: CB
Revised: CB

Original: Commitment Bank (CB) (De Marneffe et al. ,2019) is a 3-way classification of textual entailment (true, false, unknown) from 1,200 short textsegments where at least one sentence contains anembedded clause.
Revised: Commitment Bank (CB) (De Marneffe et al., 2019) is a 3-way classification of textual entailment (true, false, unknown) from 1,200 short text segments where at least one sentence contains an embedded clause.

Original: The dataset contains passagesfrom three sources: the Wall Street Journal, the
Revised: The dataset contains passages from three sources: the Wall Street Journal, the

Original: British National Corpus, and Switchboard.
Revised: British National Corpus, and Switchboard.

Original: WIC
Revised: WIC

Original: The Word-in-Context dataset (WIC) (Pilehvar and Camacho-Collados, 2018) is a benchmarkfor evaluating context-sensitive word embeddings.
Revised: The Word-in-Context dataset (WIC) (Pilehvar and Camacho-Collados, 2018) is a benchmark for evaluating context-sensitive word embeddings.

Original: The task is to classify if a target word has the samemeaning in two context sentence.
Revised: The task is to classify if a target word has the same meaning in two context sentence.

Original: WSC
Revised: WSC

Original: The
Revised: The

Original: Winograd
Revised: Winograd

Original: Schema
Revised: Schema

Original: Challenge(WSC) (Levesque et al., 2012) dataset is acollection of 804 sentences in which the task is toresolve coreferences.
Revised: Challenge (WSC) (Levesque et al., 2012) dataset is a collection of 804 sentences in which the task is to resolve coreferences.

Original: MathQA
Revised: MathQA

Original: MathQA (Amini et al., 2019) is adataset containing 37k multiple choice mathword problems built from the existing dataset,
Revised: MathQA (Amini et al., 2019) is a dataset containing 37k multiple choice math word problems built from the existing dataset,

Original: AQuA (Ling et al., 2017).
Revised: AQuA (Ling et al., 2017).

Original: PIQA
Revised: PIQA

Original: The Physical Interactions: Question Answering (PIQA) (Bisk et al., 2020) benchmarkdataset provides 21k questions about the physicalworld and plausible interactions encountered byhumans.
Revised: The Physical Interactions: Question Answering (PIQA) (Bisk et al., 2020) benchmark dataset provides 21k questions about the physical world and plausible interactions encountered by humans.

Original: Annotators provided correct and incorrect answers to questions extracted from instructables.com, a website of instructions for completingmany everyday tasks.
Revised: Annotators provided correct and incorrect answers to questions extracted from instructables.com, a website of instructions for completing many everyday tasks.

Original: PubMedQA
Revised: PubMedQA

Original: The PubMedQA dataset (Jin et al. ,2019) is a collection of 273.5k biomedical research questions and related PubMed articles withyes/no/maybe answers.
Revised: The PubMedQA dataset (Jin et al., 2019) is a collection of 273.5k biomedical research questions and related PubMed articles with yes/no/maybe answers.

Original: Lambada
Revised: Lambada

Original: Lambada (Paperno et al., 2016) contains passages and target sentences from 5,325 novels collected from Book Corpus (Zhu et al. , 2015),and the goal is to predict the last word of the targetsentence given the context passage.
Revised: Lambada (Paperno et al., 2016) contains passages and target sentences from 5,325 novels collected from Book Corpus (Zhu et al., 2015), and the goal is to predict the last word of the target sentence given the context passage.

Original: This task wasdesigned to test genuine language understandingsince accurate prediction of the final word wouldbe improbable without the context passage.
Revised: This task was designed to test genuine language understanding since accurate prediction of the final word would be improbable without the context passage.

Original: WikiText
Revised: WikiText

Original: The Wikitext benchmark (Merity et al.,2016) is a language modeling dataset of 29k articlesfrom Wikipedia.
Revised: The Wikitext benchmark (Merity et al., 2016) is a language modeling dataset of 29k articles from Wikipedia.

Original: Only articles classified as Good or
Revised: Only articles classified as Good or

Original: Featured by Wikipedia editors are included sincethey are considered to be well written and neutral in language.
Revised: Featured by Wikipedia editors are included since they are considered to be well written and neutral in language.

Original: All results are reported on Wikitext-2.
Revised: All results are reported on Wikitext-2.