OLM o: Accelerating the Science of Language Models (2024)

Abstract: Language models (LMs) have become ubiquitous in both NLP research and in commercial product offerings. As their commercial importance has surged, the most powerful models have become closed off, gated behind proprietary interfaces, with important details of their training data, architectures, and development undisclosed. Given the importance of these details in scientifically studying these models, including their biases and potential risks, we believe it is essential for the research community to have access to powerful, truly open LMs. To this end, we have built OLMo, a competitive, truly Open Language Model, to enable the scientific study of language models. Unlike most prior efforts that have only released model weights and inference code, we release OLMo alongside open training data and training and evaluation code. We hope this release will empower the open research community and inspire a new wave of innovation.

## introduction language models have been at the center of nlp technologies for many years @xcite @xcite @xcite . recently, due to largescale pretraining and human annotation for alignment, they have become commercially valuable @xcite . however, as their commercial value has increased, the largest models have become gated behind proprietary interfaces, with important details left undisclosed. we believe that full access to open language models for the research community is critical to the scientific study of these models, their strengths and weaknesses, and their biases and risks. accordingly, we introduce olmo, a powerful, truly open language model alongside open training data, training and evaluation code, intermediate model checkpoints, and training logs. recent lm releases have varied in their degree of openness. for example, mixtral 8x7b provided model weights and a brief report @xcite , while llama came with in-depth adaptation training instructions @xcite , and mosaic pretrained transformer came with many details, including the dataset distribution, though not the data itself @xcite ). falcon's pretraining data was partially released @xcite , and the most open models-the pythia suite @xcite and bloom @xcite -released training code, model checkpoints, data, and more. with olmo, we release the whole framework from data to training to evaluation tools: multiple training checkpoints across multiple hardware types, training logs, and exact datasets used, with a permissive license. we are not the only team to do this; recent work from llm360 targets similar goals @xcite . olmo narrows the gap from their models to state-of-the-art capabilities of models like llama 2. this project has benefited from lessons learned from all of these previous efforts with their varying degrees of openness, and we believe that a large, diverse population of open models is the best hope for scientific progress on understanding language models and engineering progress on improving their utility. the olmo framework encompasses the tools and resources required for building and researching language models. for training and modeling, it includes full model weights, training code, training logs, and inference code. the released model includes four variants of our language model at the 7b scale corresponding to different architectures, optimizers, and training hardware, and one model at the 1b scale, all trained on at least 2t tokens. we also release hundreds of intermediate checkpoints available as revisions on huggingface. for dataset building and analysis, the full training data used for these models is openly available (dolma; @xcite , including code that produces the training data, and tools for analyzing pretraining data @xcite . for evaluation, we build on catwalk @xcite for downstream evaluation and paloma @xcite for perplexity-based evaluation. for adaptation, we use open instruct @xcite to train with instruction and feedback data. finally, all code and weights are released under the apache 2.0 license. 1 with this release, we hope to catalyze research into as-yet poorly understood aspects of these models, for example, the relationship between pretraining data and model capabilities, the impact of design and hyperparameter choices, and various optimization methods and their impact on model training. in addition, we report on the lessons learned 1 https://allenai.org/olmo and important details necessary to successfully train language models at this scale. ## olmo framework this section describes the olmo framework, consisting of the olmo models (section 2.1), our pretraining dataset, dolma (section 2.2), and our evaluation framework (section 2.4). ## training olmo this section describes our pretraining setup, including our distributed training framework (section 3.1), optimizer (section 3.2), data preparation (section 3.3), and hardware (section 3.4). ## results the checkpoint used for evaluating olmo-7b is trained until 2.46t tokens on the dolma @xcite dataset with a linear learning rate decay schedule mentioned in section 3.2. in our experiments, we find that tuning this checkpoint further on the dolma dataset for 1000 steps with the learning rate linearly decayed to 0 boosts model performance on perplexity and end-task evaluation suites described in section 2.4. we compare olmo with other publicly available models including llama-7b @xcite , llama-2-7b @xcite , mpt-7b (mosaicml nlp team, 2023), pythia-6.9b @xcite , falcon-7b @xcite and rpj-incite-7b (together computer, 2023). ## conclusion and future work this paper presents our first release of olmo, a state-of-the-art, truly open language model and its framework to build and study the science of language modeling. unlike most prior efforts that have only released model weights and inference code, we release olmo and the whole framework, including training data, training and evaluation code, and detailed metrics collected during the training runs. additionally, we released adapted models, as well as all of our model adaptation code and data. we intend to continuously support and extend olmo and its framework, and continue to push the boundaries of open lms to empower the open research community. since the original release of olmo described here, we improved our data and training setup to significantly improve results. for example, mmlu scores have improved by 24 points to 52%. 9 we look forward to bringing different model sizes, modalities, datasets, safety measures, and evaluations into the olmo family. we hope this and future releases will empower and strengthen the open research community and inspire a new wave of innovation.