GPT - N eo X -20 B : An Open-Source Autoregressive Language Model (2022)

Abstract: We introduce GPT-NeoX-20B, a 20 billion parameter autoregressive language model trained on the Pile, whose weights will be made freely and openly available to the public through a permissive license. It is, to the best of our knowledge, the largest dense autoregressive model that has publicly available weights at the time of submission. In this work, we describe GPT-NeoX-20B’s architecture and training, and evaluate its performance. We open-source the training and evaluation code, as well as the model weights, at https://github.com/EleutherAI/gpt-neox .

## introduction over the past several years, there has been an explosion in research surrounding large language models (llms) for natural language processing, catalyzed largely by the impressive performance of transformer-based language models such as bert @xcite , @xcite , @xcite , and t5 @xcite . one of the most impactful outcomes of this research has been the discovery that the performance of llms scales predictably as a power-law with the number of parameters, with architecture details such as width/depth ratio having a minimal impact on performance within a wide range @xcite . a consequence of this has been an abundance of research focusing on scaling transformer models up to ever-larger scales, resulting in dense models that surpass 500b parameters @xcite , a milestone that would have been almost unthinkable just a few years prior. today, there are dozens of publicly acknowledged llms in existence. the largest have more than two orders of magnitude more parameters than gpt-2, and even at that scale there are nearly a dozen different models. however, these models are almost universally the protected intellectual property of large tech companies, and are gated behind a commercial api, available only upon request, or not available for outsider use at all. to our knowledge, the only freely and publicly available dense autoregressive language models larger than gpt-2 are gpt-neo (2.7b parameters) @xcite , gpt-j-6b @xcite , megatron-11b foot_1 , pangu-α-13b @xcite , and the recently released fairseq models (2.7b, 6.7b, and 13b parameters) @xcite . in this paper we introduce gpt-neox-20b, a 20 billion parameter open source autoregressive language model. we make the models weights freely and openly available to the public through a permissive license, motivated by the belief that open access to llms is critical to advancing research in a wide range of areas-particularly in ai safety, mechanistic interpretability, and the study of how llm capabilities scale. many of the most interesting capabilities of llms only emerge above a certain number of parameters, and they have many properties that simply cannot be studied in smaller models. although safety is often cited as a justification for keeping model weights private, we believe this is insufficient to prevent misuse, and is largely a limitation on the ability to probe and study llms for researchers not based at the small number of organizations that have access to state of the art language models. in the following sections, we give a broad overview of gpt-neox-20b's architecture and training hyperparameters, detail the hardware and software setup used for training and evaluation, and elaborate on the choices made when designing the training dataset and tokenization. we also address of some of the difficulties and unknowns we encountered in training such a large model. we place significant importance on the broader impacts of the release gpt-neox-20b and other such llms, and have prepared a separate manuscript for dissecting these issues in greater detail. in addition, we also make available the model weights at evenly spaced 1000 step intervals throughout the whole of training. we hope that by making a wide range of checkpoints throughout training freely available, we will facilitate research on the training dynamics of llms, as well as the aforementioned areas of ai safety and interpretability. ## model design and implementation gpt-neox-20b is an autoregressive transformer decoder model whose architecture largely follows that of @xcite , with a few notable deviations described below. our model has 20 billion parameters, of which 19.9 billion are "non-embedding" parameters that @xcite identify as the proper number to use for scaling laws analysis. our model has 44 layers, a hidden dimension size of 6144, and 64 heads. ## training due to the intractability of performing a hyperparameter sweep for a 20 billion parameter model, we opted to use the values from @xcite to guide our choice of hyperparameters. as brown et al. (2020) did not train a model at our exact scale, we interpolate between the learning rates of their 13b and 175b models to arrive at a learning rate of 0.97e-5. based on the results of smaller scale experiments, we select a weight decay of 0.01. to achieve a higher training throughput, we opt to use the same batch size as openai's 175b model-approximately 3.15m tokens, or 1538 contexts of 2048 tokens each, and train for a total of 150, 000 steps, decaying the learning rate with a cosine schedule to 10% of its original value at the end of training. we use the adamw (loshchilov and hutter, 2019) optimizer, with beta values of 0.9 and 0.95 respectively, and an epsilon of 1.0e-8. we extend adamw with the zero optimizer @xcite to reduce memory consumption by distributing optimizer states across ranks. since the weights and optimizer states of a model at this scale do not fit on a single gpu, we use the tensor parallelism scheme introduced in shoeybi et al. (2020) in combination with pipeline parallelism @xcite to distribute the model across gpus. to train gpt-neox-20b, we found that the most efficient way to distribute the model given our hardware setup was to set a tensor parallel size of 2, and a pipeline parallel size of 4. this allows for the most communication intensive processes, tensor and pipeline parallelism, to occur within a node, and data parallel communication to occur across node boundaries. in this fashion, we were able to achieve and maintain an efficiency of 117 teraflops per gpu. ## performance evaluations to evaluate our model we use the eleutherai language model evaluation harness @xcite , an open source codebase for language model evaluation that supports a number of model apis. as our goal is to make a powerful model publicly accessible, we compare with english language models with at least 10b parameter that are publicly accessible. we compare with the gpt-3 models on the openai api @xcite , the open source fairseq dense models @xcite , and gpt-j-6b @xcite . we do not compare against t5 @xcite or its derivatives as our evaluation methodology assumes that the models are autoregressive. while there is a megatron 11b checkpoint that has been publicly released, the released code is non-functional and we have not been able to get the model to work. we do not compare against any mixture-of-experts models as no public moe model achieves performance comparable to a 10b parameter dense model. while it is common to display "scaling laws" curves of best fit, we opt to not do so as the small number of openai api models give davinci an outsized influence on the slope of the curve. in many of the examples we study, including davinci in the scaling laws calculation moves the line of best fit so far as to entirely change the conclusions. instead, we connect the points with lines directly. we categorize both gpt-j-6b and gpt-neox-20b under the umbrella of gpt-neox models, as both models are trained with the same architecture (except for the negligible differences described in section 2.1.2) and were trained on the same dataset. however, we connect them using a dashed line to reflect the fact that these two models are not the same model trained at two different scales the way the fairseq and openai models are, having been trained using different codebases, different tokenizers, and for different numbers of tokens. where we were able to obtain the relevant information, we report two baselines: human-level performance and random performance. all plots contain error bars representing two standard errors, indicating the 95% confidence interval around each point. for some plots, the standard error is so small that the interval is not visible. ## summary we introduce gpt-neox-20b, a 20 billion parameter autoregressive transformer language model trained on the pile (gao et al., 2020) dataset, and detail the main architectural differences between gpt-neox-20b and gpt-3-most notably the change in tokenizer, the addition of rotary embeddings, the parallel computation of attention and feed-forward layers, and a different initialization scheme and hyperparameters. we run extensive evaluations of gpt-neox-20b on natural language and factual knowledge tasks, and compare it with other publicly available models, finding it performed particularly well on knowledge-based and mathematical tasks. finally, we are open sourcing the training and evaluation code at https://github. com/eleutherai/gpt-neox , where readers can find a link to download the model weights across the whole training run. wenlong huang, pieter abbeel, deepak pathak, and igor mordatch. 2022. language models as zeroshot planners: extracting actionable knowledge for embodied agents. computing research repository, arxiv:2201.07207. version 1. evan hubinger, chris van merwijk, vladimir mikulik, joar skalse, and scott garrabrant. 2021. risks from learned optimization in advanced machine learning systems. computing research repository, arxiv:1906.01820. version 3. mandar joshi, eunsol choi, daniel weld, and luke zettlemoyer. 2017. triviaqa: a large scale distantly supervised challenge dataset for reading comprehension. in proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1601-1611, vancouver, canada. association for computational linguistics. nikhil kandpal, eric wallace, and colin raffel. 2022. deduplicating training data mitigates privacy risks in language models. computing research repository, arxiv:2202.06539. version 2. jared kaplan, sam mccandlish, tom henighan, tom b brown, benjamin chess, rewon child, scott gray, alec radford, jeffrey wu, and dario amodei. 2020. scaling laws for neural language models. computing research repository, arxiv:2001.08361. version 1. boseop kim, hyoungseok kim, sang-woo lee, gichang lee, donghyun kwak, jeon dong hyeon, sunghyun park, sungju kim, seonhoon kim, dongpil seo, heungsub lee, minyoung jeong, sungjae lee, minsub kim, suk hyun ko, seokhun kim, taeyong park, jinuk kim, soyoung kang, na-hyeon ryu, kang min yoo, minsuk chang, soobin suh, sookyo in, jinseong park, kyungduk kim, hiun kim, jisu jeong, yong goo yeo, donghoon ham, dongju park, min young lee, jaewook kang, inho kang, jung-woo ha, woomyoung park, and nako sung. 2021. what changes can large-scale language models bring? intensive study on hyperclova: billions-scale korean generative pretrained transformers. in proceedings of the 2021 conference on empirical methods in natural language processing, pages 3405-3424, online and punta cana, dominican republic. association for computational linguistics. bryan klimt and yiming yang. 2004. the enron corpus: a new dataset for email classification research. in proceedings of the 15th european conference on machine learning, ecml'04, page 217-226, berlin, heidelberg. springer-verlag. jack koch, lauro langosco, jacob pfau, james le, and lee sharkey. 2021. objective robustness in deep reinforcement learning. computing research repository, arxiv:2105.14111. version 2. philipp koehn. 2005. europarl: a parallel corpus for statistical machine translation. in proceedings of machine translation summit x: papers, pages 79-86, phuket, thailand. aran komatsuzaki. 2019. one epoch is all you need. computing research repository, arxiv:1906.06669. version 1. vanessa kosoy. 2016. irl is hard. ai alignment forum. julia kreutzer, isaac caswell, lisa wang, ahsan wahab, daan van esch, nasanbayar ulzii-orshikh, allahsera tapo, nishant subramani, artem sokolov, claytone sikasote, monang setyawan, supheakmungkol sarin, sokhar samb, benoît sagot, clara rivera, annette rios, isabel papadimitriou, salomey osei, pedro ortiz suarez, iroro orife, kelechi ogueji, andre niyongabo rubungo, toan q. nguyen, mathias müller, andré müller, shamsuddeen hassan muhammad, nanda muhammad, ayanda mnyakeni, jamshidbek mirzakhalov, tapiwanashe matangira, colin leong, nze lawson, sneha kudugunta, yacine jernite, mathias jenny, orhan firat, bonaventure f. p. dossou, sakhile dlamini, nisansa de silva, sakine çabuk ballı, stella biderman, alessia battisti, ahmed baruwa, ankur bapna, pallavi baljekar, israel abebe azime, ayodele awokoya, duygu ataman, orevaoghene ahia, oghenefego ahia, sweta agrawal, and mofetoluwa adeyemi. 2022. quality at a glance: an audit of web-crawled multilingual datasets. transactions of the association for computational linguistics, 10:50-72. alexandre lacoste, alexandra luccioni, victor schmidt, and thomas dandres. 2019. quantifying the carbon emissions of machine learning. computing research repository, arxiv:1910.09700. version 2. connor leahy. 2021. why release a large language model? eleutherai blog. connor leahy and stella biderman. 2021. the hard problem of aligning ai to human values. in the state of ai ethics report, volume 4, pages 180-183. the montreal ai ethics institute. katherine lee, daphne ippolito, andrew nystrom, chiyuan zhang, douglas eck, chris callison-burch, and nicholas carlini. 2021. deduplicating training data makes language models better. computing research repository, arxiv:2107.06499. version 1. opher lieber, or sharir, barak lenz, and yoav shoham. 2021. jurassic-1: technical details and evaluation. technical report, ai21 labs. stephanie lin, jacob hilton, and owain evans. 2021. truthfulqa: measuring how models mimic human falsehoods. computing research repository, arxiv:2109.07958. version 1. pierre lison and jörg tiedemann. 2016. opensub-titles2016: extracting large parallel corpora from movie and tv subtitles. in proceedings of the tenth international conference on language resources and evaluation (lrec'16), pages 923-929, portorož, slovenia. european language resources association (elra). jian liu, leyang cui, hanmeng liu, dandan huang, yile wang, and yue zhang. 2020. logiqa: a challenge dataset for machine reading comprehension with logical reasoning. in proceedings of the twenty-ninth international joint conference on artificial intelligence, ijcai-20, pages 3622-3628. international joint conferences on artificial intelligence organization. ilya loshchilov and frank hutter. 2019. decoupled weight decay regularization. computing research