GPT-NeoX-20B: An Open-Source Autoregressive Language Model
Abstract
We introduce GPT-NeoX-20B, a 20 billion parameter autoregressive language model trained on the Pile, whose weights will be made freely and openly available to the public through a permissive license.
It is, to the best of our knowledge, the largest dense autoregressive model that has publicly available weights at the time of submission.
In this work, we describe GPT-NeoX-20B’s architecture and training, and evaluate its performance on a range of language-understanding, mathematics and knowledge-based tasks.
We open-source the training and evaluation code, as well as the model weights, at https://github.com/
EleutherAI/gpt-neox .
Introduction
Over the past several years, there has been an explosion in research surrounding large language models (LLMs) for natural language processing, catalyzed largely by the impressive performance of
Transformer-based language models such as BERT (Devlin et al., 2019), GPT-2 (Radford et al., 2019),
GPT-3 (Brown et al., 2020), and T5 (Raffel et al., 2020).
One of the most impactful outcomes of this research has been the discovery that the performance of LLMs scales predictably as a power-law with the number of parameters, with architecture details such as width/depth ratio having a minimal impact on performance within a wide range (Kaplan et al., 2020).
A consequence of this has been an abundance of research focusing on scaling
Transformer models up to ever-larger scales, resulting in dense models that surpass 500B parameters (Smith et al., 2022; Chowdhery et al., 2022), a milestone that would have been almost unthinkable just a few years prior.
Today, there are dozens of publicly acknowledged LLMs in existence.
The largest have more than two orders of magnitude more parameters than
GPT-2, and even at that scale there are nearly a dozen different models.
However, these models are almost universally the protected intellectual property of large tech companies, and are gated behind a commercial API, available only upon request, or not available for outsider use at all.
To our knowledge, the only freely and publicly available dense autoregressive language models larger than GPT2 are GPT-Neo (2.7B parameters)
(Black et al., 2021), GPT-J-6B (Wang and Komatsuzaki, 2021),
Megatron-11B 1 , Pangu- α -13B
(Zeng et al., 2021), and the recently released FairSeq models (2.7B, 6.7B, and 13B parameters) (Artetxe et al., 2021).
In this paper we introduce GPT-NeoX-20B, abillion parameter open source autoregressive language model.
We make the models weights freely and openly available to the public through a permissive license, motivated by the belief that open access to LLMs is critical to advancing research in a wide range of areas—particularly in AI safety, mechanistic interpretability, and the study of how
LLM capabilities scale.
Many of the most interesting capabilities of LLMs only emerge above a certain number of parameters, and they have many properties that simply cannot be studied in smaller models.
Although safety is often cited as a justification for keeping model weights private, we believe this is insufficient to prevent misuse, and is largely a limitation on the ability to probe and study LLMs for researchers not based at the small number of organizations that have access to state of the art language models.
In the following sections, we give a broad overview of GPT-NeoX-20B’s architecture and training hyperparameters, detail the hardware and software setup used for training and evaluation, and elaborate on the choices made when designing the training dataset and tokenization.
We also address of some of the difficulties and unknowns we encountered in training such a large model.
We place significant importance on the broader impacts of the release GPT-NeoX-20B and other such LLMs, and have prepared a separate manuscript for dissecting these issues in greater detail.
In addition, we also make available the model weights at evenly spaced 1000 step intervals throughout the whole of training.
We hope that by making a wide range of checkpoints throughout training freely available, we will facilitate research on the training dynamics of LLMs, as well as the aforementioned areas of AI safety and interpretability.
Model Design and Implementation
GPT-NeoX-20B is an autoregressive transformer decoder model whose architecture largely follows that of GPT-3 (Brown et al., 2020), with a few notable deviations described below.
Our model has 20 billion parameters, of which 19.9 billion are “non-embedding” parameters that Kaplan et al.
(2020) identify as the proper number to use for scaling laws analysis.
Our model has 44 layers, a hidden dimension size of 6144, and 64 heads.
2. Model Architecture
Although our architecture is largely similar to GPT3, there are some notable differences.
In this section we give a high-level overview of those differences, but ask the reader to refer to (Brown et al., 2020) for full details of the model architecture.
Our model architecture is almost identical to that of
GPT-J (Wang and Komatsuzaki, 2021) 2 , however we choose to use GPT-3 as the point of reference because there is no canonical published reference on the design of GPT-J.
2.1. Rotary Positional Embeddings
We use rotary embeddings (Su et al., 2021) instead of the learned positional embeddings used in GPT models (Radford et al., 2018), based on our positive prior experiences using it in training LLMs.
Rotary embeddings are a form of static relative positional embeddings.
In brief, they twist the embedding space such that the attention of a token at position m to token at position n is linearly dependent on m − n .
More formally, they modify the standard multiheaded attention equations from
[Equation]
where x m , x n are (batched) embeddings of tokens at position m and n respectively and W Tq , W k are the query and key weights respectively to softmax
[Equation]
where R d Θ , x is a d × d block diagonal matrix with the block of index i being a 2D rotation by x θ i for hyperparameters
Θ = {
θ i = 10000 − 2 i / d | i ∈ { 0 , 1 , 2 ,..., ( d − 1 ) / 2 }} .
While Su et al.
(2021) apply rotary embeddings to every embedding vector, we follow Wang and
Komatsuzaki (2021) and instead apply it only to the first 25% of embedding vector dimensions.
Our initial experiments indicate that this strikes the best balance of performance and computational efficiency.
2.1. Parallel Attention + FF Layers
We compute the Attention and Feed-Forward (FF) layers in parallel 4 and sum the results, rather than running them in series.
This is primarily for efficiency purposes, as each residual addition with op-sharding requires one all-reduce in the forward pass and one in the backwards pass (Shoeybi et al., 2020).
By computing the Attention and FFs in parallel, the results can be reduced locally before performing a single all-reduce.
In Mesh Transformer
JAX (Wang, 2021), this led to a 15% throughput increase, while having comparable loss curves with running them in series during early training.
Due to an oversight in our code, we unintentionally apply two independent Layer Norms instead of using a tied layer norm the way Wang and Komatsuzaki (2021) does.
Instead of computing
[Equation]
as intended, our codebase unties the layer norms:
[Equation]
Unfortunately, this was only noticed after we were much too far into training to restart.
Subsequent experiments at small scales indicated that the untied layer norm makes no difference in performance, but we nevertheless wish to highlight this in the interest of transparency.
2.1. Initialization
For the Feed-Forward output layers before the residuals, we used the initialization scheme introduced in Wang (2021), 2 L √ d .
This prevents activations from growing with increasing depth and width, with the factor of 2 compensating for the fact that the parallel and feed-forward layers are organized in parallel.
For all other layers, we use the small init scheme from Nguyen and Salazar (2019), (cid:113) 2 d + 4 d
2.1. All Dense Layers
While GPT-3 uses alternating dense and sparse layers using the technique introduced in Child et al.
(2019), we instead opt to exclusively use dense layers to reduce implementation complexity.
2. Software Libraries
Our model is trained using a codebase that builds on Megatron (Shoeybi et al., 2020) and DeepSpeed (Rasley et al., 2020) to facilitate efficient and straightforward training of large language models with tens of billions of parameters.
We use the official PyTorch v1.10.0 release binary package compiled with CUDA 11.1.
This package is bundled with NCCL 2.10.3 for distributed communications.
2. Hardware
We trained GPT-NeoX-20B on twelve Supermicro AS-4124GO-NART servers, each with eight
NVIDIA A100-SXM4-40GB GPUs and configured with two AMD EPYC 7532 CPUs.
All GPUs can directly access the InfiniBand switched fabric through one of four ConnectX-6 HCAs for
GPUDirect RDMA.
Two NVIDIA MQM8700HS2R switches—connected by 16 links—compose the spine of this InfiniBand network, with one link per node CPU socket connected to each switch.
Figure 7 shows a simplified overview of a node as configured for training.
Training
Due to the intractability of performing a hyperparameter sweep for a 20 billion parameter model, we opted to use the values from Brown et al.
(2020) to guide our choice of hyperparameters.
As Brown et al.
(2020) did not train a model at our exact scale, we interpolate between the learning rates of their 13B and
175B models to arrive at a learning rate of 0 .
97 E − 5.
Based on the results of smaller scale experiments, we select a weight decay of 0.01.
To achieve a higher training throughput, we opt to use the same batch size as OpenAI’s 175B model–approximately 3.15M tokens, or 1538 contexts of 2048 tokens each, and train for a total of 150 , 000 steps, decaying the learning rate with a cosine schedule to 10% of its original value at the end of training.
We use the AdamW (Loshchilov and Hutter, 2019) optimizer, with beta values of 0 .
9 and 0 .respectively, and an epsilon of 1 .
E − 8.
We extend
AdamW with the ZeRO optimizer (Rajbhandari et al., 2020) to reduce memory consumption by distributing optimizer states across ranks.
Since the weights and optimizer states of a model at this scale do not fit on a single GPU, we use the tensor parallelism scheme introduced in Shoeybi et al.
(2020) in combination with pipeline parallelism (Harlap et al., 2018) to distribute the model across
GPUs.
To train GPT-NeoX-20B, we found that the most efficient way to distribute the model given our hardware setup was to set a tensor parallel size of 2, and a pipeline parallel size of 4.
This allows for the most communication intensive processes, tensor and pipeline parallelism, to occur within a node, and data parallel communication to occur across node boundaries.
In this fashion, we were able to achieve and maintain an efficiency ofteraFLOPS per GPU.
3. Training Data
GPT-NeoX-20B was trained on the Pile (Gao et al., 2020), a massive curated dataset designed specifically for training large language models.
It consists of data from 22 data sources, coarsely broken down into 5 categories:
• Academic Writing : Pubmed Abstracts and PubMed Central, arXiv, FreeLaw, 5 USPTO Backgrounds, 6 PhilPapers, 7 NIH Exporter• Web-scrapes and Internet Resources : CommonCrawl, OpenWebText2, StackExchange,Wikipedia (English)
com
• Prose : BookCorpus2, Bibliotik, Project Gutenberg (PG-19; Rae
et al
) • Dialogue : Youtube subtitles, Ubuntu IRC,OpenSubtitles (Lison and Tiedemann, 2016), Hacker News, 11 EuroParl (Koehn, 2005)
• Miscellaneous : GitHub, the DeepMind Mathematics dataset (Saxton et al., 2019), Enron Emails (Klimt and Yang, 2004)
In aggregate, the Pile consists of over 825GiB of raw text data.
The diverse data sources reflects our desire for a general-purpose language model.
Certain components are up-sampled to obtain a more balanced data distribution.
In contrast, GPT3’s training data consists of web-scrapes, books datasets, and Wikipedia.
When comparing results in this work to GPT-3, the training data is almost certainly the biggest known unknown factor.
Full details of the Pile can be found in the technical report (Gao et al., 2020) and the associated datasheet (Biderman et al., 2022).
It is particularly notable that the Pile contains a scrape of StackExchange preprocessed into a
Q/A form.
There is a significant and growing body of work on the influence of the syntactic structure of finetuning data on downstream performance (Zhong et al., 2021; Tan et al., 2021;
Sanh et al., 2021; Wei et al., 2021).
While so far there has been no systematic work that focuses on prompted pretraining , recent work (Biderman and
Raff, 2022) observed that the formulation of the
StackExchange component of the Pile appears to heavily influences code generation.
3. Tokenization
For GPT-NeoX-20B, we use a BPE-based tokenizer similar to that used in GPT-2, with the same total vocabulary size of 50257, with three major changes to the tokenizer.
First, we train a new BPE tokenizer based on the Pile, taking advantage of its diverse text sources to construct a more generalpurpose tokenizer.
Second, in contrast to the GPTtokenizer which treats tokenization at the start of a string as a non-space-delimited token, the GPTNeoX-20B tokenizer applies consistent space delimitation regardless.
This resolves an inconsistency regarding the presence of prefix spaces to a tokenization input.
An example can be seen in
Figure 1.
Third, our tokenizer contains tokens for repeated space tokens (all positive integer amounts of repeated spaces up to and including 24).
This allows the GPT-NeoX-20B tokenizer to tokenize text with large amounts of whitespace using fewer tokens; for instance, program source code or arXiv
L A TEX source files.
See Appendix F for an analysis of the tokenizer.
[Figure]
if n < 2: ← (cid:45) return n ← (cid:45) else: ← (cid:45) return fibRec(n
[Figure]
1) + fibRec(n
[Figure]
[Figure] [Figure]
n ← (cid:45)
[Figure] [Figure]
[Figure]
3. Data Duplication
In the past two years, the standard practice when training autoregressive language models has become to train for only one epoch (Komatsuzaki, 2019; Kaplan et al., 2020; Henighan et al., 2020).
Recent research has claimed to see significant benefits from going even further and deduplicating training data (Lee et al., 2021;
Kandpal et al., 2022; Roberts et al., 2022).
In particular, every publicly known larger language model other than
GPT-3 (Brown et al., 2020) and Jurassic-1 13 either uses some form of deduplication (Rae et al., 2022;
Askell et al., 2021; Zeng et al., 2021; Sun et al., 2021; Smith et al., 2022; Hoffmann et al., 2022;
Chowdhery et al., 2022) or does not discuss the training data in sufficient detail to determine what was done (Kim et al., 2021).
When the Pile was originally made, the only language model larger than GPT-NeoX-20B that existed was GPT-3, which upsampled high quality subsets of its training data.
The Pile followed suit, and due to a combination of a lack of resources for large scale ablations and a lack of noticeable impact at smaller scales, we opt to use the Pile as-is.
As shown in fig.
2, even at the 20B parameter scale we see no drop in test validation loss after crossing the 1 epoch boundary.
Unfortunately, none of the papers that have claimed to see an improvement from deduplication have released trained models that demonstrate this, making replication and confirmation of their results difficult.
Lee et al.
(2021) releases the deduplication code that they used, which we intend to use to explore this question in more detail in the future.
It is important to note that even if there is not an improvement in loss or on task evaluations there are nevertheless compelling reasons to deduplicate training data for any model put into production.
In particular, systematic analysis has shown significant benefits in terms of reducing the leakage of training data (Lee et al., 2021;
Zhang et al., 2021;
Carlini et al., 2022; Kandpal et al., 2022).
Performance Evaluations
To evaluate our model we use the EleutherAI
Language Model Evaluation Harness (Gao et al., 2021b), an open source codebase for language model evaluation that supports a number of model
APIs.
As our goal is to make a powerful model publicly accessible, we compare with English language models with at least 10B parameter that are publicly accessible.
We compare with the GPTmodels on the OpenAI API(Brown et al., 2020), the open source FairSeq dense models (Artetxe et al., 2021), and GPT-J-6B (Wang and Komatsuzaki, 2021).
We do not compare against T5 (Raffel et al., 2020) or its derivatives as our evaluation methodology assumes that the models are autoregressive.
While there is a Megatron 11B checkpoint that has been publicly released, the released code is non-functional and we have not been able to get the model to work.
We do not compare against any mixture-of-experts models as no public MoE model achieves performance comparable to a 10B parameter dense model.
While it is common to display “scaling laws” curves of best fit, we opt to not do so as the small number of OpenAI API models give DaVinci an outsized influence on the slope of the curve.
In many of the examples we study, including DaVinci in the scaling laws calculation moves the line of best fit so far as to entirely change the conclusions.
Instead, we connect the points with lines directly.
We categorize both GPT-J-6B and GPT-NeoX-20B under the umbrella of GPT-NeoX models, as both models are trained with the same architecture (except for the negligible differences described in Section 2.1.2) and were trained on the same dataset.
However, we connect them using a dashed line to reflect the fact that these two models are not the same model trained at two different scales the way the FairSeq and OpenAI models are, having been trained using different codebases, different tokenizers, and for different numbers of tokens.
Where we were able to obtain the relevant information, we report two baselines: human-level performance and random performance.
All plots contain error bars representing two standard errors, indicating the 95% confidence interval around each point.
For some plots, the standard error is so small that the interval is not visible.
4. Tasks Evaluated
We evaluate our model on a diverse collection of standard language model evaluation datasets that we divide into three main categories: natural language tasks, Advanced Knowledge-Based Tasks, and Mathematical Tasks.
Due to space constraints a representative subset of the results are shown here, with the rest in Appendix E.
Natural Language Tasks
We evaluate our model on a diverse collection of standard language model evaluation datasets: ANLI (Nie et al., 2020), ARC (Clark et al., 2018), HeadQA (English) (Vilares and Gómez-Rodríguez, 2019), HellaSwag (Zellers et al., 2019), LAMBDADA (Paperno et al., 2016),
LogiQA (Liu et al., 2020), OpenBookQA (Mihaylov et al., 2018), PiQA (Bisk et al., 2020),
PROST (Aroca-Ouellette et al., 2021), QA4MRE (Peñas et al., 2013) (2013), SciQ (Welbl et al., 2017), TriviaQA (Joshi et al., 2017), Winogrande (Sakaguchi et al., 2021), and the SuperGlue version of the Winograd Schemas Challenge (WSC) (Wang et al., 2019).
Mathematical Tasks
The solving of mathematical problem solving is an area that has had a long history of study in AI research, despite the fact that large language models tend to perform quite poorly on both arithmetic tasks and mathematical problems phrased in natural language.
We evaluate on the MATH test dataset (Hendrycks et al., 2021b) as well as on the numerical arithmetic problems introduced by Brown et al.
Note that the MATH test dataset is an evaluation metric that is generally finetuned on, but due to computational limitations we only evaluate models zero- and five-shot here.
Advanced Knowledge-Based Tasks
We are also interested in the ability of our models to answer factual questions that (for humans) require advanced knowledge.
To do this, we use a dataset of multiple choice questions in a variety of diverse domains developed by Hendrycks et al.
(2021a).
Following common practice on this dataset, we focus on results aggregated by subject area: Humanities,
Social Sciences, STEM, and Miscellaneous as presented in Figure 6.
We report five-shot performance to be comparable to previous work.
tasks we were unable
Discussion 5. Performance Results Natural Language Tasks
While GPT-NeoX20B outperforms FairSeq 13B on some tasks (e.g.
ARC, LAMBADA, PIQA, PROST), it underperforms on others (e.g. HellaSwag, LogiQA zeroshot).
In total, across the 32 evaluations we did we outpreform on 22 tasks, underpreform on four tasks, and fall within the margin of error on six tasks.
By far our weakest performance is on HellaSwag, where we score four standard deviations below FairSeq 13B in both zero- and five-shot evaluations.
Similarly, GPT-J underperforms FairSeq 6.7B by three standard deviations zero-shot and six standard deviations five-shot on HellaSwag.
We find this massive performance loss largely inexplicable; while we originally assumed that the substantial non-prose components of the Pile were to blame, we note that GPT-J and GPT-NeoX overpreform FairSeq models on the very similar Lambada task by roughly the same amount.
Mathematics
While GPT-3 and FairSeq models are generally quite close on arithmetic tasks, they are consistently out-preformed by GPT-J and GPTNeoX.
We conjecture that this is traceable to the prevalence of mathematics equations in the training data, but warn that people should not assume that this means that training on the Pile produces better out-of-distribution arithmetic reasoning.
Razeghi et al.
(2022) show that there is a strong correlation between the frequency of a numerical equation in the Pile and GPT-J’s performance on that equation, and we see no reason this would not hold in GPT-NeoX 20B, FairSeq, and GPT-3.
We are unfortunately unable to investigate this effect in
FairSeq and GPT-3 models because the authors do not release their training data.
Advanced Knowledge-Based Tasks
While GPTNeoX and FairSeq both exhibit dominant performance on MMMLU compared to GPT-3 in the five-shot setting (Figures 6 and 11), their performance is much closer in the zero-shot setting (Figure 10).
Hendrycks et al.
(2021b) find that few-shot evaluation does not improve performance, but that appears to be only the case for GPT-3.
We view this as a warning against drawing strong conclusions about evaluation metrics based only on one model, and encourage researchers developing new evaluation benchmarks to leverage multiple different classes of models to avoid overfitting their conclusions to a specific model.
5. Powerful Few-Shot Learning
Our experiments indicate that GPT-J-6B and GPTNeoX-20B benefit substantially more from fewshot evaluations than the FairSeq models do.
When going from 0-shot to 5-shot evaluations, GPT-J-6B improves by 0 .
{+0526 and GPT-NeoX-20B improves by 0 . 
+}
while the FairSeq 6.7B and 13B models improve by 0 . 0051 and 0 . 0183 respectively.
This result is statistically significant and robust to purturbations of prompting.
While we do not have a particular explanation for this currently, we view this as a strong recommendation for our models.
5. Limitations Optimal Training
Hyperparameter tuning is an expensive process, and is often infeasible to do at full scale for multi-billion parameter models.
Due to the aforementioned limitations, we opted to choose hyperparameters based on a mixture of experiments at smaller scales and by interpolating parameters appropriate for our model size based on previously published work (Brown et al., 2020).
However, several aspects of both our model architecture [Section 2.1] and training setup, including the data [Section 3.1] and the tokenizer [Section 3.2], diverge significantly from Brown et al.
As such, it is almost certainly the case that the hyperparameters used for our model are no longer optimal, and potentially never were.
Lack of Coding Evaluations
Many of the design choices we made during the development of this model were oriented towards improving performance on coding tasks.
However, we underestimated the difficulty and cost of existing coding benchmarks (Chen et al., 2021), and so were unable to evaluate out model in that domain.
We hope to do so in the future.
Data Duplication
Finally, the lack of dataset deduplication could also have had an impact on downstream performance.
Recent work has shown that deduplicating training data can have a large effect on perplexity (Lee et al., 2021).
While our experiments show no sign of this, it is hard to dismiss it due to the number of researchers who have found the opposite result.
5. Releasing a 20B Parameter LLM
The current status quo in research is that large language models are things people train and publish about, but do not actually release.
To the best of our knowledge, GPT-NeoX-20B is the largest and most performant dense language model to ever be publicly released.
A variety of reasons for the nonrelease of large language models are given by various groups, but the primary one is the harms that public access to LLMs would purportedly cause.
We take these concerns quite seriously.
However, having taken them quite seriously, we feel that they are flawed in several respects.
While a thorough analysis of these issues is beyond the scope of this paper, the public release of our model is the most important contribution of this paper and so an explanation of why we disagree with the prevailing wisdom is important.
Providing access to ethics and alignment researchers will prevent harm.
The open-source release of this model is motivated by the hope that it will allow researchers who would not otherwise have access to LLMs to use them.
While there are negative risks due to the potential acceleration of capabilities research, we believe the benefits of this release outweigh the risks.
We also note that these benefits are not hypothetical, as a number of papers about the limits and ethics of LLMs has been explicitly enabled by the public release of previous models (Zhang et al., 2021; Kandpal et al., 2022;
Carlini et al., 2022; Birhane et al., 2021; nostalgebraist, 2020; Meng et al., 2022; Lin et al., 2021).
Limiting access to governments and corporations will not prevent harm.
Perhaps the most curious aspect of the argument that LLMs should not be released is that the people making such arguments are not arguing they they should not use
LLMs.
Rather, they are claiming that other people should not use them.
We do not believe that this is a position that should be taken seriously.
The companies and governments that have the financial resources to train LLMs are overwhelmingly more likely to do large scale harm using a LLM than a random individual.
Releasing this model is the beginning, not the end, of our work to make GPT-NeoX-20B widely accessible to researchers.
Due to the size of the model, inference is most economical on a pair of
RTX 3090 Tis or a single A6000 GPU and finetuning requires significantly more compute.
Truly promoting widespread access to LLMs means promoting widespread access to computing infrastructure in addition to the models themselves.
We plan to make progress on this issue going forward by continuing to work on reducing the inference costs of our model, and by working with researchers to provide access to the computing infrastructure they need to carry out experiments on our models.
We strongly encourage researchers who are interested in studying GPT-NeoX-20B but lack the necessary infrastructure to reach out to discuss how we can help empower you.
Summary
We introduce GPT-NeoX-20B, a 20 billion parameter autoregressive Transformer language model trained on the Pile (Gao et al., 2020) dataset, and detail the main architectural differences between GPTNeoX-20B and GPT-3—most notably the change in tokenizer, the addition of Rotary embeddings, the parallel computation of attention and feed-forward layers, and a different initialization scheme and hyperparameters.
We run extensive evaluations of GPT-NeoX-20B on natural language and factual knowledge tasks, and compare it with other publicly available models, finding it performed particularly well on knowledge-based and mathematical tasks.
Finally, we are open sourcing the training and evaluation code at https://github. com/EleutherAI/gpt-neox , where readers can find a link to download the model weights across the whole training run.
Acknowledgments
We thank staff at CoreWeave—in particular
Max Hjelm, Brannin McBee, Peter Salanki, and
Brian Venturo—for providing the GPUs and computing infrastructure that made this project possible.
We would also like to acknowledge Eren Do˘gan and Wesley Brown for feedback and technical support throughout the project, and John Schulman,
Evan Hubinger, Victor Sanh, Jacob Hilton, and Siddharth Karamcheti for providing feedback on drafts of the paper.
Finally, we thank Anthony DiPofi, Charles Foster,
Jeffrey Hsu,
Eric Tang,
Anish Thite,
Kevin Wang, and Andy Zou for their contributions to the EleutherAI Language Modeling Evaluation
Harness we used to evaluate GPT-NeoX-20B.
Large knowledge enhanced pre-training for language understanding and generation.
Computing Research Repository , arXiv:2107.02137.
Version
Zeerak Talat, Aurélie Névéol, Stella Biderman, Miruna
Clinciu, Manan Dey, Shayne Longpre, Alexandra Sasha Luccioni, Maraim Masoud, Margaret
Mitchell, Dragomir Radev, Shanya Sharma, Arjun
Subramonian, Jaesung Tae, Samson Tan, Deepak
Tunuguntla, and Oskar van der Wal. 2022.
You reap what you sow:
On the challenges of bias evaluation under multilingual settings.
In Proceedings of the 1st
Workshop
Large tional
Jie Tang. 2021.
WuDao
Pretrain
Yada Pruksachatkun, Nikita Nangia, Aman
Ben Wang and Aran Komatsuzaki
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin
Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le guage
Computing
Research Repository , arXiv:
Johannes Welbl, Nelson F. Liu, and Matt Gardner.
In Proceedings 3rd Noisy Usergenerated Text , pages 94–106, Copenhagen, Denmark.
Association for Computational Linguistics.
John Wentworth ment
Linting Xue, Aditya Barua Noah Constant Rami Al
Rfou, Sharan Narang
Mihir Kale
Roberts,
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale,
Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and
Colin Raffel. 2020.
: A massively multilingual pre-trained text text
Repository
Version 1.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali
Farhadi, and Yejin Choi. 2019.
HellaSwag:
Can a machine really finish your sentence?
In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 4791–4800, Florence,
Italy.
Association for Computational Linguistics.
Wei Zeng, Xiaozhe Ren, Teng Su, Hui Wang, Yi Liao,
Zhiwei Wang, Xin Jiang, ZhenZhang Yang, Kaisheng
Wang, Xiaoda Zhang, Chen Li, Ziyan Gong, Yifan Yao, Xinjing Huang, Jun Wang, Jianfeng Yu,
Qi Guo, Yue Yu, Yan Zhang, Jin Wang, Hengtao
Tao, Dasen Yan, Zexuan Yi, Fang Peng, Fangqing
Jiang, Han Zhang, Lingfeng Deng, Yehong Zhang,
Zhe Lin, Chao Zhang, Shaojie Zhang, Mingyue Guo,
Shanzhi Gu, Gaojun Fan, Yaowei Wang, Xuefeng
Jin, Qun Liu, and Yonghong Tian
Chiyuan Zhang, Daphne Ippolito, Katherine Lee,
Matthew Jagielski, Tramèr, and Car
Zhong
A Individual Contributions
Sid Black was the lead developer and overall point person for the project.
Stella Biderman was the lead scientist and project manager.
Implementation and Engineering
Implementation of training infrastructure:
Sid Black,
Stella Biderman,
Eric Hallahan,
Quentin Anthony, Samuel Weinbach
Scaling experiments and optimization:
Sid Black, Stella Biderman, Quentin Anthony,
Samuel Weinbach
Positional Embeddings:
Sid Black, Eric Hallahan, Michael Pieler
Tokenizer:
Sid Black
Miscellaneous:
USVSN Sai Prashanth, Ben Wang
Scientific
Experimentation
Evaluations:
Stella Biderman,
Leo Gao,
Jonathan Tow,
Sid Black,
Shivanshu Purohit,
Horace He,
Laurence Golding
Positional Embeddings:
Stella Biderman,
Laurence Golding,
Michael Pieler
Tokenizer:
Stella Biderman, Jason Phang, Leo Gao
Broader Impacts
Alignment Implications:
Leo Gao,
Connor Leahy,
Laria Reynolds,
Kyle McDonell
Environmental Impact:
Stella Biderman, Eric Hallahan
B Full Configuration Details
In Table 1 we attach the full configuration details used to train GPT-NeoX-20B.
The file is available in .yaml format usable in gpt-neox at https:// github.com/EleutherAI/gpt-neox , where we also provide documentation describing the role of each parameter.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
C Broader Impacts
The current status quo in research is that large language models are things people train and publish about, but do not actually release.
To the best of our knowledge, GPT-NeoX-20B is the largest dense language model to ever be publicly released with a several-way tie for second place at 13 billion parameters (Artetxe et al., 2021; Xue et al., 2020, 2021) and many more models at the 10-11B parameter scale.
A variety of reasons for the non-release of large language models are given by various groups, but the primary one is the harms that public access to LLMs would purportedly cause.
We take these concerns quite seriously.
However, having taken them quite seriously, we feel that they are flawed in several respects.
While a thorough analysis of these issues is beyond the scope of this paper, the public release of our model is the most important contribution of this paper and so an explanation of why we disagree with the prevailing wisdom is important.
Providing access to ethics and alignment researchers will prevent harm.
The open-source release of this model is motivated by the hope that it will allow researchers who would not otherwise have access to LLMs to use them.
While there are negative risks due to the potential acceleration of capabilities research, we believe the benefits of this release outweigh the risks.
We also note that these benefits are not hypothetical, as a number of papers about the limits and ethics of LLMs has been explicitly enabled by the public release of previous models (Zhang et al., 2021; Kandpal et al., 2022;
Carlini et al., 2022; Birhane et al., 2021; nostalgebraist, 2020; Meng et al., 2022; Lin et al., 2021).
Limiting access to governments and corporations will not prevent harm.
Perhaps the most curious aspect of the argument that LLMs should not be released is that the people making such arguments are not arguing they they should not use
LLMs.
Rather, they are claiming that other people should not use them.
We do not believe that this is a position that should be taken seriously.
The companies and governments that have the financial resources to train LLMs are overwhelmingly more likely to do large scale harm using a LLM than a random individual.
The open-source release of this model is motivated by the hope that it will allow ethics and alignment researchers who would not otherwise have access to LLMs to use them.
While there are negative risks due to the potential acceleration of capabilities research, we believe the benefits of this release outweigh the risks of accelerating capabilities research.
C. Impact on Capabilities Research and Products
When discussing the impact of access to technology, it is important to distinguish between capacities research which seeks to push the current stateof-the-art and research on
We feel the risk of releasing GPT-NeoX-20B is acceptable, as the contribution of the model to capabilities research is likely to be limited, for two reasons.
We ultimately believe that the benefits of releasing this model outweigh the risks, but this argument hinges crucially on the particular circumstances of this release.
All actors considering releasing powerful AI models or advancing the frontier of capabilities should think carefully about what they release, in what way, and when.
C. Impact on Ethics and Alignment Research
To oversimplify a complex debate, there are broadly speaking two schools of thought regarding the mitigation of harm that is done by AI algorithms: AI Ethics and AI Alignement .
AI Ethics researchers are primarily concerned with the impact of current technologies or technologies very similar to current technologies, while AI Alignment is primarily concerned with future “generally intelligent” systems whose capacities greatly outclass currently existing systems and possess human and superhuman levels of intelligence.
While the tools, methods, and ideas of these camps are very different, we believe that increasing access to these technologies will empower and advance the goals of researchers in both schools.
C.2. The Necessity of Model Access for AI Ethics
Analyzing and documenting the limitations of models is an essential aspect of AI ethics research (Matias, 2020).
Work examining and criticizing datasets (Kreutzer et al., 2022; Dodge et al., 2021;
Birhane et al., 2021), functionality (Smart, 2021;
Zhang et al., 2021; Carlini et al., 2022; Biderman and Raff, 2022), evaluation and deployment procedures (Biderman and Scheirer, 2020;
Talat et al.,
, and more are essential to well-rounded and informed debate on the value and application of technology.
However the current centralization of LLM training also creates a centralization of control of technology (Sadowski et al., 2021; Whittaker, 2021) that makes meaningful independent evaluation impossible.
This means that it is often not possible to do this kind of work in practice because of the severe access restrictions companies that own large language models put on them.
While GPT-NeoX is the 13th largest dense language model at time of writing only model larger than GPT-NeoX 20B that is publicly accessible is GPT-3.
There are significant limitations on people’s ability to do research on GPT-3 though, as it is not free to use and its training data is private.
C.2. The Usefulness of Large Language Models in Alignment
LLMs represent a different paradigm than the AI systems generally studied by alignment researchers because they are not well-described as coherent agents or expected utility maximizers.
Though trained to optimize a log-likelihood loss function, at a high level the goals a LLM pursues are varied and contradictory, depending on the way it is prompted.
This introduces additional challenges, but may also enable new approaches to alignment.
GPT-NeoX-20B itself is not the system we need to align, but we hope it can serve as a publicly available platform for experiments whose results might generalize to crucial future work.
The following is a non-exhaustive list of potential approaches we consider promising for further investigation.
Mechanistic interpretability.
Mechanistic interpretability research (Cammarata et al., 2020) hopes to gain an understanding into how models accomplish the tasks they do, in part in the hopes of detecting problematic or deceptive algorithms implemented by models before these failures manifest in the real world.
Being able to interpret and inspect the detailed inner workings of trained models would be a powerful tool to ensure models are optimizing for the goals we intended (Hubinger et al., 2021;
Koch et al., 2021).
Reverse engineering transformer language models has already yielded insights about the inner functioning of LMs (Elhage et al., 2021; nostalgebraist, 2020;
Meng et al., 2022; Dai et al., 2021).
Using a LLM as a reward model.
Because they are trained to predict human writing, LLMs also appear to develop a useful representation of human values at the semantic level.
Finding a way to utilise these representations could be a possible path toward solving the problem of reward robustness in RL and other algorithms which require a proxy of human judgment (Stiennon et al., 2022;
Wentworth, 2020).
Despite fundamental theoretical limitations on learning human values (Armstrong and Mindermann, 2018; Kosoy, 2016), value learning may still be robust enough to align weaker superhuman AIs.
Future experiments could explore the extent to which LLM pretraining improves downstream reward model robustness and generalization.
Natural language transparency.
Since LLM prompts are in a human-readable form, it can provide insight on the LLM’s expected behavior.
Prompt programming or finetuning can be used to leverage this fact and force a LLM to execute more transparent algorithms, such as splitting problems into steps or explicitly writing an “internal monologue” (Soares, 2021; Gao et al., 2021a; Nye et al., 2021).
Reliability and trustworthiness can present significant challenges for these approaches.
However, this form of transparency also has its limits.
In particular, models can often respond unpredictably to prompts, and internal monologues may become completely detached from the model’s decision making process if translating between the model’s ontology and the human ontology is more complex than simply modeling human monologues (Christiano et al., 2021).
Simulating agents at runtime.
Although LLMs are not well-described as coherent agents, they can still be used to generate goal-directed processes.
Given an appropriate prompt (such as a story of a character working to achieve a goal), LLMs can predict and thus simulate an agent (Huang et al., 2022).
Simulated agents take representative actions according to the patterns present in the training data, similar to behavior cloning.
One potential future research direction is testing whether they are less susceptible to failure modes that follow from expected utility maximization, such as Goodhart failures and power-seeking behavior.
However, other failure modes can be introduced by the LM training procedure, such as “delusions” or “hallucinations” (Ortega et al., 2021; Gao, 2021; Maynez et al., 2020).
Additionally, simulated agents may be uncompetitive with optimal agents like those produced by Reinforcement Learning.
An important research direction is to explore how the beneficial properties of simulated agents can be maintained while making them competitive with RL based approaches.
Tool AI and automated alignment research.
LMs can be used as relatively unagentic tools, such as OpenAI’s Codex model (Chen et al., 2021) acting as a coding assistant.
Because pretrained LLMs are not directly optimized for the factual accuracy of their predictions, it is possible they avoid some of the traditional problems with tool or oracle AI (Armstrong et al., 2012), such as the incentive to produce manipulative answers (Demski, 2019).
Tool AI is not a long-term solution to the problem of alignment, but it could be used to assist alignment research or even automate large parts of it.
For example, language models could be used to help brainstorm alignment ideas more quickly, act as a writing assistant, or directly generate alignment research papers for humans to review.
This line of research also risks accelerating capabilities research, a concern we discuss more below.
C. Differential Impact on Access
Because training large models requires a significant engineering and capital investment, such models are often out of reach for small labs and independent researchers.
As it stands, only large organizations have access to the latest generation of powerful language models (Brown et al., 2020; Rae et al., 2022; Fedus et al., 2021; Lieber et al., 2021; Tang, 2021).
The number of researchers focused primarily on ethics and alignment working at these labs is much lower than those working on developing new capabilities.
We feel the risk of releasing GPT-NeoX-20B is acceptable, as the contribution of the model to capabilities research is likely to be limited, for two reasons.
Firstly, the organizations pursuing capabilities research most aggressively are unlikely to benefit from our open-source release of this model as they have already developed more powerful models of their own.
Secondly, we believe the single most important piece of knowledge that drives advancing capabilities research is the knowledge that scaling LLMs was possible in the first place (Leahy, 2021; Leahy and Biderman, 2021).
Whereas the actual implementation is very fungible (as evidenced by the large number of parties who have succeeded in creating their own LLMs in the past two years).
This differential impact, wherein our release is expected to benefit primarily people who have less funding and infrastructure, is a key factor in our decision to release this model publicly.
We ultimately believe that the benefits of releasing this model outweigh the risks, but this argument hinges crucially on the particular circumstances of this release.
All actors considering releasing powerful AI models or advancing the frontier of capabilities should think carefully about what they release, in what way, and when.
C. Environmental Impact
A significant point of concern in some recent work is the energy usage and carbon emissions associated with training large language models (Strubell et al., 2019; Schwartz et al., 2020; Lacoste et al., 2019; Bender et al., 2021).
In particular, Strubell et al.
(2019) estimate that a then-recent paper by the authors released 626 , 155 lbs or 284 .
01 metric tons 14 of CO 2 (t CO 2 ).
As Strubell et al.
(2019) has been widely cited and quoted in the media as representative of large-scale language models, we decided to explicitly and carefully track our energy usage and carbon emissions to see if this is truly a representative account of NLP emissions.
Throughout the development and training of our model, we tracked our energy usage and carbon emissions.
We found that the process of developing and training GPT-NeoX-20B emitted almost exactly 10% of Strubell et al.
(2019)’s estimate, coming in at a total of 69957 lbs or 31 .
73 metric tons of CO 2 .
This is roughly the equivalent of the yearly emissions of the average American or 35 round-trip flights between New York City and
San Francisco.
Our systems were based in Illinois,
USA, and consumed energy sourced from the mix as follows
40% Coal (0 .
95t CO 2 /MWh)
30% Gas (0 .
6078t CO 2 /MWh)
30% Hydroelectric (0t CO 2 /MWh)
40% Nuclear (0t CO 2 /MWh) • 0 .
30% Solar (0t CO 2 /MWh) • 18 .
10% Wind (0t CO 2 /MWh)
• 1 . 30% Other Renewables (0t CO 2 /MWh)
This
mixture
produces an
average
of 0 .
t CO 2 /MWh,
and we consumed a total of 43 .
92 MWh of electricity over the course ofhours of training.
Scaling, testing, and evaluation were responsible for the equivalent of anotherhours on our systems, for a total energy consumption 66 .
24 MWh and thus the production of just under 35 metric tons of CO 2 .
It is noteworthy that Strubell et al.
(2019) are estimating emissions from a neural architecture search paper, and is therefore not directly comparable to ours.
The primary motivation for our comparison is that their number has attracted a lot of attention and is often taken to be respresentative of NLP research.
In general, we advocate for more systematic and comprehensive reporting to improve transparency surrounding this important topic.
D Architecture Diagram E Full Evaluation Results
Results for natural language understanding tasks are shown in Tables 2 and 3, while results for
Hendrycks tasks are found in Tables 10 to 13.
All evaluations had version 0 in the Evaluation
Harness.
This information is reported in the output of the Evaluation Harness and should be used for ensuring reproducibility of these results, even as the task implementations themselves may change to fix bugs.
CPU
[Figure]
PLX PLX
CPU
PLX PLX
HCA
[Figure]
HCA
[Figure]
HCA
[Figure]
HCA
[Figure]
GPU
GPU GPU GPU
[Figure]
GPU
[Figure]
GPU
GPU GPU
NVSwitch
NVSwitch NVSwitch
NVSwitch NVSwitch
NVSwitch2x
[Figure]
[Table]
SciQ
[Table] [Table]
HellaSwag
[Table]
SciQ
[Table]
[Table] [Table]
SciQ
[Table]
Models)
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
MATH
[Table]
Number Theory) 0 .
[Table] [Table]
[Table] [Table]
007 ±
[Table] [Table]
[Table] [Table]
[Table] [Table]
MATH
[Table] [Table]
[Table] [Table]
GPT
[Table]
6B 20B
Ada
Babbage
Curie
DaVinci
[Table]
History
[Table]
Geography
[Table]
Government
[Table]
Politics
[Table]
Macroeconomics
[Table]
Mathematics
[Table] [Table]
FairSeq
[Table]
125M 355M 1.3B 2.7B
[Table]
13B
[Table]
Engineering
[Table]
Biology
[Table]
Chemistry
[Table]
Science
[Table]
History
[Table]
Geography
[Table]
Politics
[Table] [Table]
[Table]
F Tokenizer Analysis
Both tokenizers share 36938 out of 50257 tokens, a ∼ 73.5% overlap in tokens.
In this section, we perform comparison between the GPT-NeoX-20B tokenizer to the GPT-2 tokenizer using the validation set of the
Pile.
In Table 15, we show the resulting number of tokens from tokenizing each component of the Pile’s validation set with both tokenizers, and the ratio of GPT-NeoX-20B tokens to GPT-2 tokens.
We observe that the GPT-NeoX-20B tokenizer represents all Pile components using fewer or very closely comparable numbers of tokens.
The largest percentage improvement in token counts are in the
EuroParl, GitHub, and PubMed Central components, with a more than 20% savings in the number of tokens needed to represent that component.
We highlight that arXiv, GitHub, and StackExchange—subsets with large code components—can be represented with meaningfully fewer tokens with the GPT-NeoX-20B tokenizer compared to the GPT-2 tokenizer.
Overall, the GPT-NeoX-20B tokenizer represents the Pile validation set with approximately 10% fewer tokens compared to the GPT-2 tokenizer.
Given that the GPT-NeoX-20B tokenizer is tweaked to better tokenize whitespace, we also perform a comparison between the two tokenizers excluding whitespace.
We perform the same analysis as the above, but exclude all whitespace tokens from our computations, only counting the non-whitespace tokens.
A token is considered a whitespace token if it consists only of whitespace characters.
The results are shown in Table 16 in the Appendix.
We observe that the GPT-NeoX-20B tokenizer still uses 5% fewer tokens to represent the Pile validation set compared to the GPT-2 tokenizer.
As expected, the token ratios for certain components such as GitHub and StackExchange become closer to even once the whitespace characters are excluded.
[Figure]
NeoX
[Figure]
20B
[Figure]
-NeoX-20BGPTPile (val) 383,111, 342,
[Table]
While we evaluated our tokenizer using the validation set for the Pile, the Pile components would still be considered in-domain for the tokenizer and may not provide the most informative comparison point.
To perform an out-of-domain comparison, we perform the same analysis using the AllenAI replication of
C4, 15 , another popular pretraining corpus for large language models.
As above, we use the validation set for our analysis.
Our results are shown in Table 14.
We find that the GPT-NeoX-20B tokenizer tokenizes the C4 validation set to approximately the same number of tokens as the GPT-2 tokenizer.
When excluding all whitespace tokens, the GPT-NeoX-20B requires approximately 1% more tokens to represent the corpus compared to the GPT-2 tokenizer.
F. Tokenizer Comparisons F.1. Longest Tokens
We show in Table 17 the 10 longest tokens in each tokenizer vocabulary.
We exclude consideration of tokens that comprise only symbols or whitespace characters.
We observe that for the GPT-2 tokenizer, many of the longest tokens appear to reflect artifacts in the tokenizer training data, likely with certain websites or web-scrapes being overrepresented in the training data.
For the GPT-NeoX-20B tokenizer, we observe that most of the longest tokens are scientific terms, likely arising from the PubMed components of the Pile
. F.1. Worst Case Word Tokenization Comparison
We consider the words for which there is the greatest discrepancy in the resulting token length between the two tokenizers, where one tokenizer needs many tokens to represent while the other tokenizer uses
[Table]
Mathematics
[Table] [Table]
OpenSubtitles
[Table]
OpenWebText
[Table]
PhilPapers
[Table] [Table]
Pile-CC
[Table]
PubMed Abstracts
[Table] [Table]
Central
[Table] [Table]
[Table]
StackExchange
[Table] [Table]
[Table]
USPTO Backgrounds
[Table]
[Table] [Table]
Ubuntu IRC
[Table] [Table]
[Table] [Table]
relatively few tokens.
We define a word as a contiguous string delimited by whitespace or punctuation (as defined by strings.punctuation in Python).
We perform this analysis at the component level.
We only consider words that occur at least 10 times within the given component.
We show in Table 18 a representative example from the Pile-CC corpus.
G Tokenization Examples
In Figures 12 and 17, we show examples of tokenized documents from the Pile, comparing the GPTtokenizer to ours.
GPTGPT-NeoX-20B rawdownloadcloneembedreportprint ˙Gimmunohistochemistry
BuyableInstoreAndOnline ˙Gimmunohistochemical cloneembedreportprint ˙Gtelecommunications ˙GRandomRedditorWithNo
˙Gimmunofluorescence ˙Gtelecommunications ˙Gimmunosuppressive
channelAvailability ˙GBytePtrFromString ˙Gdisproportionately ˙Gmultidisciplinary
˙GTelecommunications ˙Ghistopathological
˙GguiActiveUnfocused ˙Gneurodegenerative
[Table]
[Table] [Table]
abstract: ’The maximal minors of a $p\times (m + p)$-matrix of univariate polynomials of degree $n$ with indeterminate coefficients are themselves polynomials of degree $np$. The subalgebra generated by their coefficients is the coordinate ring of the quantum Grassmannian, a singular compactification of the space of rational curves of degree $np$ in the Grassmannian of $p$- planes in ($m + p$)-space.
These subalgebra generators are shown to form a sagbi basis.
The resulting flat deformation from the quantum Grassmannian to a toric variety gives a new “Grö bner basis style” proof of the Ravi-Rosenthal-Wang formulas in quantum Schubert calculus.
The coordinate ring of the quantum Grassmannian is an algebra with straightening law, which is normal, Cohen-Macaulay, Gorenstein and Koszul, and the ideal of quantum Plücker relations has a quadratic Gröbner basis.
This holds more generally for skew quantum Schubert varieties.
These results are well-known for the classical Schubert varietie
[Table]
abstract: ’The maximal minors of a $p\times (m + p)$-matrix of univariate polynomials of degree $n$ with indeterminate coefficients are themselves polynomials of degree $np$. The subalgebra generated by their coefficients is the coordinate ring of the quantum Grassmannian, a singular compactification of the space of rational curves of degree $np$ in the Grassmannian of $p$- planes in ($m + p$)-space.
These subalgebra generators are shown to form a sagbi basis.
The resulting flat deformation from the quantum Grassmannian to a toric variety gives a new “Grö bner basis style” proof of the Ravi-Rosenthal-Wang formulas in quantum Schubert calculus.
The coordinate ring of the quantum Grassmannian is an algebra with straightening law, which is normal, Cohen-Macaulay, Gorenstein and Koszul, and the ideal of quantum Plücker relations has a quadratic Gröbner basis.
This holds more generally for skew quantum Schubert varieties.
These results are well-known for the classical Schubert varietie
[Table]
This is a work of fiction.
Names, characters, places and incidents are products of the author ’s imagination or are used fictitiously and are not to be construed as real.
Any resemblance to actual events, locales, organizations, or persons, living or dead, is completely coincidental.
[Table]
www
[Table]
beverleykendall
[Table]
All rights reserved.
Except as permitted under the U.S. Copyright Act of 1976, no part of this publication may be reproduced, distributed or transmitted in any form or by any means, or stored in a database or retrieval system, without the prior written permission of the author .
←
[Table]
←
[Table]
This ebook is licensed for your personal enjoyment only.
This ebook may not be re-sold or given away to other people.
If you would like to share this book with another person, please purchase an additional copy for each reader.
If
[Table]
(cid:45) Published by Season Publishing LLC ← (cid:45) ← (cid:45)
This is a work of fiction.
Names, characters, places and incidents are products of the author ’s imagination or are used fictitiously and are not to be construed as real.
Any resemblance to actual events, locales, organizations, or persons, living or dead, is completely coincidental.
← (cid:45) ← (cid:45) www.beverleykendall.com
← (cid:45) ← (cid:45)
Cover Design ©
Okay Creations, Sarah Hansen ← (cid:45) ← (cid:45) All rights reserved.
Except as permitted under the U.S. Copyright Act of 1976, no part of this publication may be reproduced, distributed or transmitted in any form or by any means, or stored in a database or retrieval system, without the prior written permission of the author .
← (cid:45) ←
[Table]
** License Statement ** ← (cid:45) ← (cid:45)
This ebook is licensed for your personal enjoyment only.
This ebook may not be re-sold or given away to other people.
If you would like to share this book with another person, please purchase an additional copy for each reader.
If
477 tokens o?
← (cid:45)
True ← (cid:45) Suppose -3*t = 1 + 8.
Let s(d) =
d**
+ 6*d**2 + 2*d + 1.
Let u be s(t).
Suppose 10 =
5*z, 5*a + 0*z = -z
+ u. Is 4 a factor of a?
← (cid:45) True ← (cid:45) Suppose 5*l = r - 35, -2*r + 5*l -
= -70.
Is r a multiple of 4?
← (cid:45)
True ← (cid:45) Suppose 2*l + 11 - 1 = 0.
Does 15 divide (-2)/l - 118/(-5)?
← (cid:45)
False ← (cid:45) Suppose 3*k - 3*f + 0*f - 72 = 0, -
= -5*f.
Is 9 a factor of 2/(-4) + k/2?
← (cid:45)
False ← (cid:45) Suppose 6*w + 25 = w. Let t(c)
=
c + 9.
Let u be t(w).
Suppose -u*z = -3*z - 10.
Is z a multiple of 5?
← (cid:45) True ← (cid:45)
Let j = 81 + -139.
Let i =
j
+ 101.
Is 11 a factor of i?
← (cid:45) False ← (cid:45) Let q(s) =
s**3 + 4*s**2 - s + 2.
Let u be q(-4).
Let o(w) =
w**2 + w - 6.
Let t be o(u).
Suppose -3*l -
= -3*d - 2*l, 0 =
3*d - 2*l - t. Does 9 divide d?
← (cid:45) False ← (cid:45) Suppose -2*b + 39 + 13 = 0.
Is b a multiple of 14?
← (cid:45)
False ← (cid:45) Let q = -7 + 12.
Suppose 8*l
=
q*l + 81.
Suppose 129 = 4*f - l. Is 13 a factor of f? ← (cid:45)
True ← (cid:45) Suppose
= -4*n + j + 33, 4*n - n + 4*j = 20.
Let c = 5 - n. Is 35*1 - (-6)/c a multiple of 11?
← (cid:45) True ← (cid:45)
Let g(m)
=
m**2 - 2*m - 3.
Let k be g(3).
Let j be
GPT-NeoX-20B Tokenization 468 tokens o?
← (cid:45)
True ← (cid:45) Suppose -3*t = 1 + 8.
Let s(d) =
d**
+ 6*d**2 + 2*d + 1.
Let u be s(t).
Suppose 10 =
5*z, 5*a + 0*z = -z
+ u. Is 4 a factor of a?
← (cid:45) True ← (cid:45) Suppose 5*l = r - 35, -2*r + 5*l -
= -70.
Is r a multiple of 4?
← (cid:45)
True ← (cid:45) Suppose 2*l + 11 - 1 = 0.
Does 15 divide (-2)/l - 118/(-5)?
← (cid:45)
False ← (cid:45) Suppose 3*k - 3*f + 0*f - 72 = 0, -
= -5*f.
Is 9 a factor of 2/(-4) + k/2?
← (cid:45)
False ← (cid:45) Suppose 6*w + 25 = w. Let t(c)
=
c + 9.
Let u be t(w).
Suppose -u*z = -3*z - 10.
Is z a multiple of 5?
← (cid:45) True ← (cid:45)
Let j = 81 + -139.
Let i =
j
+ 101.
Is 11 a factor of i?
← (cid:45) False ← (cid:45) Let q(s) =
s**3 + 4*s**2 - s + 2.
Let u be q(-4).
Let o(w) =
w**2 + w - 6.
Let t be o(u).
Suppose -3*l -
= -3*d - 2*l, 0 =
3*d - 2*l - t. Does 9 divide d?
← (cid:45) False ← (cid:45) Suppose -2*b + 39 + 13 = 0.
Is b a multiple of 14?
← (cid:45)
False ← (cid:45) Let q = -7 + 12.
Suppose 8*l
=
q*l + 81.
Suppose 129 = 4*f - l. Is 13 a factor of f? ← (cid:45)
True ← (cid:45) Suppose
= -4*n + j + 33, 4*n - n + 4*j = 20.
Let c = 5 - n. Is 35*1 - (-6)/c a multiple of 11?
← (cid:45) True ← (cid:45)
Let g(m)
=
m**2 - 2*m - 3.
Let k be g(3).
Let j be
[Table]
[Figure] [Figure]
[Figure]
178 tokens Theresa May is expected to appoint an EU ambassador who “believes in Brexit” in the wake of the current Brussels representative’s decision to quit after being cut adrift by Downing Street.
← (cid:45) ← (cid:45)
Sir Ivan Rogers on Tuesday announced his resignation as Britain’s ambassador in Brussels after it was made clear Mrs May and her senior team had “lost confidence” in him over his “pessim istic” view of Brexit.
← (cid:45) ← (cid:45) Government sources made clear that Sir Ivan had “jumped before he was pushed” and that Number 10 believed his negative view of Brexit meant that he could not lead the negotiations after the Prime Minister triggers Article 50.
← (cid:45) ← (cid:45)
In a 1,400-word resignation letter to his staff leaked on Tuesday night, Sir Ivan launched a thinly-veiled attack on the "muddled thinking" in Mrs May’s Government.
[Figure]
170 tokens Theresa May is expected to appoint an EU ambassador who “believes in Brexit” in the wake of the current Brussels representative’s decision to quit after being cut adrift by Downing Street.
← (cid:45) ← (cid:45)
Sir Ivan Rogers on Tuesday announced his resignation as Britain’s ambassador in Brussels after it was made clear Mrs May and her senior team had “lost confidence” in him over his “pessim istic” view of Brexit.
← (cid:45) ← (cid:45) Government sources made clear that Sir Ivan had “jumped before he was pushed” and that Number 10 believed his negative view of Brexit meant that he could not lead the negotiations after the Prime Minister triggers Article 50.
← (cid:45) ← (cid:45)
In a 1,400-word resignation letter to his staff leaked on Tuesday night, Sir Ivan launched a thinly-veiled attack on the "muddled thinking" in Mrs May’s Government.
[Figure] [Figure]
Carotid endarterectomy: operative recurrent stenosis, and long-term stroke rates in a modern series.
← (cid:45) To determine whether carotid endarterectomy (CEA) safely and effectively maintained a durable reduction in stroke complications over an extended period, we reviewed our data on 478 consecutive patients who underwent 544 CEA’s since 1976.
Follow-up was complete in 83% of patients (mean 44 months).
There were 7 early deaths (1.3%), only 1 stroke related (0.2%).
Peri operative stroke rates (overall 2.9%) varied according to operative indications: asymptomatic,.4%; transient ischemic attacks (TIA)/amaurosis fugax (AF), 1.3%; nonhemispheric symptoms (NH), 4.9%; and prior stroke (CVA), 7.1%.
Five and 10-year stroke-free rates were 96% and 92% in the asymptomatic group, 93% and 87% in the TIA/AF group, 92% and 92% in the NH group, and 80% and 73% in the CVA group.
Late ipsilateral strokes occurred infrequently (8 patients, 1.7%).
Late deaths were primarily cardiac related (51.3%).
Stro
[Table]
250 tokens Carotid endarterectomy: operative risks, recurrent stenosis, and long-term stroke rates in a modern series.
← (cid:45) To determine whether carotid endarterectomy (CEA) safely and effectively maintained a durable reduction in stroke complications over an extended period, we reviewed our data on 478 consecutive patients who underwent 544 CEA’s since 1976.
Follow-up was complete in 83% of patients (mean 44 months).
There were 7 early deaths (1.3%), only 1 stroke related (0.2%).
Peri operative stroke rates (overall 2.9%) varied according to operative indications: asymptomatic,.4%; transient ischemic attacks (TIA)/amaurosis fugax (AF), 1.3%; nonhemispheric symptoms (NH), 4.9%; and prior stroke (CVA), 7.1%.
Five and 10-year stroke-free rates were 96% and 92% in the asymptomatic group, 93% and 87% in the TIA/AF group, 92% and 92% in the NH group, and 80% and 73% in the CVA group.
Late ipsilateral strokes occurred infrequently (8 patients, 1.7%).
Late deaths were primarily cardiac related (51.3%).
Stro