A Holistic Assessment of the Carbon Footprint of Noor, a Very Large A rabic Language Model (2022)

Abstract: As ever larger language models grow more ubiquitous, it is crucial to consider their environmental impact. Characterised by extreme size and resource use, recent generations of models have been criticised for their voracious appetite for compute, and thus significant carbon footprint. Although reporting of carbon impact has grown more common in machine learning papers, this reporting is usually limited to compute resources used strictly for training. In this work, we propose a holistic assessment of the footprint of an extreme-scale language model, Noor. Noor is an ongoing project aiming to develop the largest multi-task Arabic language models–with up to 13B parameters–leveraging zero-shot generalisation to enable a wide range of downstream tasks via natural language instructions. We assess the total carbon bill of the entire project: starting with data collection and storage costs, including research and development budgets, pretraining costs, future serving estimates, and other exogenous costs necessary for this international cooperation. Notably, we find that inference costs and exogenous factors can have a significant impact on total budget. Finally, we discuss pathways to reduce the carbon footprint of extreme-scale models.

## introduction recent progress in natural language processing (nlp) has been driven by the emergence of socalled foundation models @xcite . this paradigm shift is characterised by a homogenisation of modelling methods-crystallising around the transformer architecture @xcite and by emergent capabilities (e.g. zero-shot generalisation) predominantly arising from sheer scale alone @xcite . nlp models are now experiencing a 3-4 months doubling time in size, as outlined by figure 1 . most recent large language models such as mt-nlg 530b @xcite , gopher 280b @xcite , or jurassic-1 178b @xcite , all report training budgets in the thousands of pf-days foot_0 range. because ai accelerators performance per watt has plateaued compared to deep learning budgets @xcite , practitioners have had to scale-out training over an increasingly large number of accelerators @xcite . accordingly, the energy cost of training state-of-theart models has grown significantly: increase in compute is no longer fuelled by improvements in hardware efficiency, but in hardware scale. although this increase in size and compute budget is backed by empirical scaling laws drawing a clear link between compute spent and model performance @xcite , the societal benefits of larger models have been questioned @xcite . specifically to environmental concerns, in a time of climate crisis when carbon emissions must be drastically cut @xcite , one may question whether these large compute budgets are justified. a crucial step towards answering this question is an in-depth evaluation of the footprint of large models. existing assessments of the environmental impacts of large models are usually focused on hyperparameter tuning and pretraining costs @xcite . this trend is reflected by the growing number of tools available to help practitioners quantify the impact of machine learning computations @xcite . if some studies have also endeavoured to quantify select aspects of the machine learning pipeline (e.g. conference attendance @xcite , hardware lifecycle @xcite , etc.), end-to-end evaluations of machine learning projects life cycle emissions remain rare @xcite . to fill this gap, we produce an end-to-end assessment of the carbon footprint of noor, a project seeking to train a very large arabic language model. our contributions are the following: holistic assessment. we evaluate the total carbon bill of the entire project: starting with data collection, curation, and storage, including research and development and hyper-parameters tuning budgets, pretraining costs, future serving estimates, and other exogenous impacts sparked by this international cooperation (e.g. flights, personnel, etc.) beyond pretraining. we identify pretraining compute as driving more than half of the emissions of the project. however, all combined, other r&d, storage, and personnel counts still amount for 35% of the carbon footprint. we also identify downstream use in the wild as potentially significant. this leads us to recommend for the end-to-end footprint to be systematically assessed on a per-project basis. notably, in scenarios with a low-impact training electric mix, costs beyond pretraining may become the main sources of emissions. pathways to lower footprints. finally, we discuss ways to reduce the environmental footprints of projects involving large models, and put in perspective the footprint of similar projects. ## related work in light of ever increasing computational budgets @xcite and of the need to cut on emissions to abate global warming @xcite , the environmental impact of deep learning has drawn significant interest. @xcite notably highlighted the potential high environmental costs of deep learning. however, its headline figures were produced in the specific context of neural architecture search, a relatively rare practice for extreme-scale models nowadays. @xcite @xcite subsequently called for ai research to be more aware of its environmental cost. an increasing number of tools, such as codecarbon @xcite , have been developed to help with tracking the impact of deep learning experiments @xcite . all of these lines of research share similar recommendations: the carbon footprint of deep learning is a direct consequence of the electricity mix and efficiency of the data center, suggesting that picking an appropriate provider is the most straightforward way to reduce environmental impact. specifically to extreme-scale models, patterson et al., 2021 estimated the energy consumption of five large nlp models, including gpt-3. they identified that a judicious choice of neural architecture, datacenter and accelerator can help reduce considerably carbon budgets. thompson et al., 2020 identified a clear relationship between large models performance and their carbon impact, building upon work on neural scaling laws @xcite . @xcite estimated the cost of training gpt-3 in different data centers across the worldwide, highlighting again the high dependency on the local energy mix and specific infrastructure. two recent studies have provided insights into the end-to-end carbon footprint of deployed models in the industry. @xcite studied the impact of the increasingly large recommender systems leveraged at meta, while @xcite provided an assessment of the costs (including inference) of large models at google. they expect the carbon footprint of training to plateau in coming years, and then to shrink-owing to more efficient high performance computing platforms. they also assert that current studies are overestimating the real environmental costs of large models, in light of the wide availability of "clean" compute platforms. in the field of astrophysics, @xcite did an extensive study to estimate the carbon footprint of the giant array for neutrino detection (grand) project, a multi-decade worldwide experiment. inspired by their holistic methodology, we seek to establish the first end-to-end assessment of an extreme-scale nlp project. ## the noor project the current state-of-the-art generative language model in modern standard arabic is aragpt @xcite , a 1.5b parameters model. the noor project seeks to expand upon this model, introducing a 1.5b, 2.7b, 6.7b, and 13b arabic models, trained a custom curated dataset of 150b tokens, inspired by the pile @xcite . these larger scales are expected to make the model able to tackle novel tasks through zero-shot generalization, as exhibited by @xcite or gpt-j @xcite . noor is an on-going international cooperation between the technology innovation institute in the united arab emirates and lighton in france. the noor project can be split in four parts: ## factors influencing the carbon footprint of large models before beginning our assessment, we propose to identify some of the key influencing factors on the potential carbon footprint of large models, focusing first on factors directly related to the models themselves and not to the project producing them. ## best practices and recommendations from our experience with noor, we highlight some recommendations for future projects to minimise their carbon footprint. ## discussion and conclusion we undertook an end-to-end assessment of the carbon footprint associated with the development of an extreme-scale language model. we took into account data collection and storage, research and development, pretraining, and included estimates for future serving and inference. we also added personnel costs, such as international flights to run training workshops and brainstorming sessions. in total, we estimate the development of the suite of the four noor models to have emitted 36.5 tons of co2, 65% of which for training the models, 18% for the international flights, 12% for data storage, and 4% for small-scale research and development experiments. to put this in perspective, the average carbon footprint per individual in the us is around 20 tons, so our project generated a little over two years of individual us emissions. we find that the main driver of this carbon footprint is the carbon intensity of the mix used for model training. appropriately selecting the location of calculations can significantly reduce the environmental impact of a project. for instance, in this project, running all computations in france would have reduced the total footprint to 14.9 tco2e, 42% of which from the international flights. as the impact of the computations themselves become smaller, it is important for practitionners to more carefully weigh in exogenous contributions. all-in-all, with careful considerations around data center choice, it is possible to run extremescale nlp projects with a low carbon impact. finally, we also identified that large-scale inference could also rapidly outtake pretraining costs in terms of carbon impact. inference, if not centrally managed, is harder to control: with a publicly available model, it will happen on hardware decided by the end user. we thus think its equally important for practitioners to alert users regarding best efficient inference practices, and regarding best practices to limit the environmental cost of computations (e.g. choosing an efficient data center, running inference in a country with a low-impact mix, etc.)