What Language Model to Train if You Have One Million GPU Hours? (2022)

Abstract: The crystallization of modeling methods around the Transformer architecture has been a boon for practitioners. Simple, well-motivated architectural variations can transfer across tasks and scale, increasing the impact of modeling research. However, with the emergence of state-of-the-art 100B+ parameters models, large language models are increasingly expensive to accurately design and train. Notably, it can be difficult to evaluate how modeling decisions may impact emergent capabilities, given that these capabilities arise mainly from sheer scale alone.In the process of building BLOOM–the Big Science Large Open-science Open-access Multilingual language model–our goal is to identify an architecture and training setup that makes the best use of our 1,000,000 A100-GPU-hours budget.Specifically, we perform an ablation study at the billion-parameter scale comparing different modeling practices and their impact on zero-shot generalization.In addition, we study the impact of various popular pre-training corpora on zero-shot generalization. We also study the performance of a multilingual model and how it compares to the English-only one. Finally, we consider the scaling behaviour of Transformers to choose the target model size, shape, and training setup. All our models and code are open-sourced at https://huggingface.co/bigscience.

## introduction recent years have seen the advent of large language models characterized by emergent capabilities (e.g., zero-shot generalization) arising from sheer scale alone @xcite . scaling llms results in a predictable increase in performance: simple scaling laws connect the number of parameters, pretraining dataset size, and compute budget @xcite @xcite , providing a clear figure 1 : smooth scaling of language modeling loss as compute budget and model size increase. we observe a power-law coefficient α c ∼ 0.046, in-line with @xcite . we use this fit to estimate the optimal size and number of tokens to train on for the final model given the available budget. path towards more capable models. this paradigm shift has been fueled by the wide adoption of the transformer @xcite , providing a scalable basis for practitioners to build upon. in this paper, we design an architecture and training setup for a multilingual 100b+ parameters model @xcite ), seeking to best use a fixed 1,000,000 a100-hours budget. because of the costs involved with training large language models, we cannot exhaustively explore the landscape of possible models. instead, we position ourselves as practitioners exploring "offthe-shelf" solutions. we thus test promising additions to the transformer to attempt to reproduce their findings in a controlled, large-scale setting. although our main goal was to prepare the architecture and training setup of bloom, our findings are also valuable for practitioners building models in the 1-10b range, as they equally improve the performance of such smaller models. at variance with major works on large language models, we also make a significant effort towards reproducibility and openness: all of our pretrained models, code, and notes from our weekly meetings are made available. see appendix a for the relevant links. contributions. we first study the impact of pretraining corpora, positional embeddings, activation functions, and embedding norm on zero-shot generalization. we base our study on the popular gpt-2 architecture @xcite , with experiments at the 1.3b parameters scale. we then consider the impact of massive multilinguality, showing language-specific scaling laws in a multilingual setting for the first time. finally, we describe our approach to drafting an architecture for the final 176b parameters bloom model. ## methods we first justify our choice to base our model on the popular recipe of combining a decoder-only model with an autoregressive language modeling objective, and introduce our experimental setup. we then discuss our evaluation benchmarks, and motivate our choice of zero-shot generalization as our key metric. finally, we introduce the baselines we compare to throughout the paper. ## impact of pretraining data we first study the impact of pretraining data on zero-shot generalization. more diverse pretraining data, ideally curated from a cross-domain collection of high-quality datasets, has been suggested to help with downstream task performance and zeroshot generalization @xcite . 1 these models are now referred to as text-babbage-001 and text-curie-001. ## architecture ablations we now consider ablation studies to better identify the best positional embedding, activation function, and embedding normalization placement. ## multilinguality the majority of 100b+ language models have been trained in english, with notable exceptions in chinese @xcite and korean @xcite models. smaller massively multilingual models have seen wider adoption @xcite , but these models are not suitable for zero-shot. recent results on large gpt-like multilingual models show that english-only performance is usually disappointing @xcite . training data. we train a multilingual model to evaluate the effectiveness and potential impacts of this practice. we use the oscar dataset @xcite , but here we include multiple languages, not only english as in the earlier experiments. the languages we include are arabic, basque, bengali, chinese, catalan, english, french, hindi, indonesian, portuguese, spanish, urdu, and vietnamese. we sample each language with a different probability that downsamples the most frequent languages and upsamples the least frequent ones, so that all languages are represented. we estimate the sampling probabilities similar to @xcite . ## scaling to 176b parameters we now detail how our previous findings influence our architecture and scaling decisions for the final 176b bloom model. compute allocation. we have been allocated 18 weeks of dedicated use of partition with 52 nodes of 8x 80gb a100 gpus on the jean zay supercomputer. we set four nodes aside as spare, so that our compute budget amounts to 1,161,216 a100-hours in total. assuming a throughput of 100 model tflops, approximately corresponding to state-ofthe-art hardware flops of 150 @xcite , we have a compute budget of 4,838 pf-days for the model training. we round this down to 4,500 pf-days, this ∼ 10% safety margin accounting for potential downtime and inefficiencies (e.g., batch size ramp-up) during training. to put this number in perspective, this is ∼ 23% more than the training budget of gpt-3. given this compute budget, our english-only scaling laws in 1 predict an optimal allocation for training a 392b parameter model for 165b tokens. we will use these as an upper bound in size: the largest model we can afford is 392b parameters, and the minimum number of tokens to train on is 165b tokens. model shape. @xcite studied the dependence of the loss with model shape, and found only a limited impact within a wide range of feed-forward ratios d f f /d model , aspect ratios d model /n layer , and attention head dimensions. @xcite proposed a theoretically motivated and empirically backed law describing the optimal compromise between width and depth. they predict that 100b+ parameters models such as gpt-3 are too deep, while models in the 10b or smaller range are usually too shallow. for a gpt-3-sized model with 175b parameters, they predict an ideal depth of 80 layers. ## conclusion seeking to establish the best possible model architecture that can be accommodated within a fixed 1,000,000 gpu-hours compute budget, we have presented an extensive study on principled modeling decisions for large language models. first, we have found that complimenting common crawl data with high-quality cross-domain curated data can boost zero-shot generalization, validating previous suggestions @xcite . through an ablation study, we have identified alibi as the position embedding of choice, confirmed the potential of swiglu, and highlighted that stabilizing techniques such as embedding normalization sometimes come at the expense of zero-shot generalization. exploring multilinguality, we have found that multilingual models significantly underperform their monolingual counterparts on english zero-shot benchmarks, but that they can learn under-resourced languages along with larger ones if given enough scale. finally, we identified a candidate architecture for bloom 176b, outlining the full reasoning behind every architectural parameter, including model shape. at variance with previous 100b+ models, such as gpt-3 @xcite or @xcite , this project was conducted in the open, and resulted in a number of open-access artefacts. notable similar projects conducted in parallel to this one include opt @xcite and glm @xcite , although they lacked the collaborative and massively multilingual components of this project. we hope our work can help practitioners better understand modeling decisions, leading to better language models, and that this transparency will accelerate future similar work.