Evaluating the carbon footprint of NLP methods: a survey and analysis of existing tools (2021)

Abstract: Modern Natural Language Processing (NLP) makes intensive use of deep learning methods because of the accuracy they offer for a variety of applications. Due to the significant environmental impact of deep learning, cost-benefit analysis including carbon footprint as well as accuracy measures has been suggested to better document the use of NLP methods for research or deployment. In this paper, we review the tools that are available to measure energy use and CO2 emissions of NLP methods. We describe the scope of the measures provided and compare the use of six tools (carbon tracker, experiment impact tracker, green algorithms, ML CO2 impact, energy usage and cumulator) on named entity recognition experiments performed on different computational set-ups (local server vs. computing facility). Based on these findings, we propose actionable recommendations to accurately measure the environmental impact of NLP experiments.

## introduction modern natural language processing (nlp) makes intensive use of deep learning methods because of the functional performance they offer for a variety of tasks, including text classification or named entity recognition @xcite . deep learning programs can have a high environmental impact in terms of greenhouse gas (ghg) emissions due in particular to the energy consumption of the computational facilities used to run them @xcite . the impact has been increasing over the years @xcite and is affecting populations that can be different from those generating the impact @xcite . in a recent medical imaging study, @xcite suggests that the increase of large model carbon footprint does not translate into proportional accuracy gains. measuring this impact is a first step for raising awareness and controlling the impact of nlp experiments and operations. some guidelines were offered in the sustainlp workshop foot_0 to measure impact, and it was suggested that different methods for measuring environmental impact can lead to different conclusions in terms of algorithm efficiency @xcite . our goal is to conduct a systematic review of tools available for measuring the impact of nlp tools and to offer a comparative analysis from the perspective of calculated impact measures and usability. we seek to understand the methods implemented by the tools as well the criteria used to assess impact. the contributions of this study are threefold: ## environmental impact due to deep learning programs as for any information and communication technology (ict) service, a deep learning program impacts several environmental indicators, among which the global warming potential, expressed in terms of greenhouse gases (ghgs) emissions, co 2 equivalent emissions, or carbon footprint. other indicators include: abiotic resource depletion, blue water shortage, human toxicity... existing tools for deep learning programs focus on the carbon footprint only. the environmental impact in terms of carbon footprint needs to account for the entire lifecycle of ict equipment from production, through use and finally end of life. life cycle analysis usually allocates part of the ghg emitted during the production of equipment to the use e.g. when running computer programs, since the equipment was partly produced for the purpose of running programs. for ict equipment, this phase is difficult to assess because data on ghg emissions during production are not always easily available. in the case of graphical processing unit (gpus) (or tensor processing unit (tpus) or equivalents), we could not find any publication or site giving an estimate of the ghg emissions due to the production phase. however, it has to be noted that production can account for a significant part of the total ghg emissions: a french study @xcite on a data center (with central processing unit (cpu) servers only) in grenoble found that around 40% of the total emissions released during one hour of cpu use were due to the production phase (including emissions due to the equipment alone). similarly, another recent study @xcite reports that the hardware manufacturing and infrastructure accounts for the bulk of the environmental impact of mobile and data center computing equipment, while the impact of operational energy consumption is diminishing. the end-of-life phase is very difficult to assess for ict due to lack of data. to summarize, there are at least four sources of co 2 equivalent emission sources that should be taken into account to assess the environmental impact of computational experiments: 1/ production of hardware equipment: router, pc, server; 2/ idle use of the hardware; 3/ dynamic use of the hardware; and 4/ end of life of equipment. ## tools our method for identifying tools and characterizing them relied on a recent study conducting a similar review of annotation tools @xcite . in our study, four authors of this paper contributed to the definition of the criteria and evaluation of the tools. they all have a computer science background with programming experience, and were not involved in the development of any of the tools reviewed or selected. ## use case: measuring the impact of named entity recognition in this section, we present experiments for a typical nlp task, named entity recognition (ner). experiments are conducted on two computational set-ups: (1) the use of a server within the laboratory (equipped with 2 gpus nvidia geforce gtx 1080 ti) and ( 2 ) the use of an external shared computer facility (equipped with 43 gpus including gpu nvidia tesla v100). we hypothesize that this type of set-up can be available to nlp researchers, and that it is relevant to document the implications of choosing one or the other for a set of experiments. we measure the impact of the experiments using the selected tools and compare the carbon footprint across tools and computational set-ups. ## discussion measuring carbon footprint is a novel undertaking for measuring carbon impact of experiments is quite recent. furthermore, these tools have been moderately used in the field of nlp (we identified 9 studies in total). however, we can note that the interest in producing impact measures is widespread as the six tools reviewed were developed by re-searchers in different countries across europe and north america. differences in carbon footprint measurements @xcite found the emissions computed by green algorithms to be higher than those computed by ml co2 impact. this is explained by the details of the functional criteria reported in the measures obtained from energy usage are much lower than those obtained from the other tools. after investigation, it seems that these results do not take into account the gpu consumption, due to a possible bug in the tool, which was reported to the code authors. carbon footprint is underestimated all the tools reviewed in this study only evaluate the carbon footprint of nlp experiments based on energy consumption during the dynamic use phase of equipment. emissions resulting from the production and end-of-life phases are unaccounted or partially accounted. in addition, we can note that setting up experiments also requires some upstream testing of configurations and model parameters that are often not accounted in the measures. green algorithm is the only tool in our selection that brings the users attention to this source of emission with the "pragmatic scaling factor". for these reasons, the measures of co 2 equivalent obtained with these tools underestimate the actual carbon footprint. ## conclusions in this paper, we have conducted a survey of the literature to identify tools for measuring the environmental impact of nlp experiments. we characterized six tools and evaluated them on a sample named entity recognition task. the measures obtained vary significantly and only account for one in four sources of carbon emissions. more work is needed to better understand the differences in measurement between the tools and to account for sources of carbon emissions other than the dynamic use of hardware equipment: production, idle use, end of life.