Towards Climate Awareness in NLP Research (2022)

Abstract: The climate impact of AI, and NLP research in particular, has become a serious issue given the enormous amount of energy that is increasingly being used for training and running computational models. Consequently, increasing focus is placed on efficient NLP. However, this important initiative lacks simple guidelines that would allow for systematic climate reporting of NLP research. We argue that this deficiency is one of the reasons why very few publications in NLP report key figures that would allow a more thorough examination of environmental impact, and present a quantitative survey to demonstrate this. As a remedy, we propose a climate performance model card with the primary purpose of being practically usable with only limited information about experiments and the underlying computer hardware. We describe why this step is essential to increase awareness about the environmental impact of NLP research and, thereby, paving the way for more thorough discussions.

## introduction as artificial intelligence (ai), and specifically natural language processing (nlp), scale up to require more computational resources and thereby more energy, there is an increasing focus on efficiency and sustainability @xcite . for example, training a single bert base model @xcite requires as much energy as a trans-american flight @xcite . while newer models are arguably more efficient @xcite @xcite , they are also an order of magnitude larger, raising environmental concerns @xcite . the problem will only worsen with time, as compute requirements double every 10 months @xcite . this problem has been recognized by the nlp community, and a group of nlp researchers has recently proposed a policy document foot_2 of recommendations for efficient nlp, aiming to minimize the greenhouse gas (ghg) emissions 4 resulting from experiments done as part of the research. this proposal is part of a research stream aiming towards green nlp and green ai @xcite , which refers to "ai research that yields novel results while taking into account the computational cost, encouraging a reduction in resources spent." while the branding of nlp and ai research as green has raised some awareness of the environmental impact, the large majority of nlp researchers are still not aware of their environmental impact resulting from training and running of large computational models. this also explains why a research stream in a similar direction (see §2.1), in which software tools are proposed to measure carbon footprint while training models @xcite @xcite , have not been adopted by the community to a large extent (see §3). however, we claim that climate awareness is essential enough to be promoted in mainstream nlp (rather than only as a niche field) and that positive impact must be an inherent part of the discussion @xcite . ideally environmental impact should always be taken into consideration, when deciding on which experiments to carry out. we aim to simplify climate performance reporting in nlp while at the same time increasing awareness to its intricacies. our contributions are: ## towards actionable awareness efficiency (alongside accuracy) has been one of the main objectives in nlp (and computer science in general) long before its environmental aspects have been widely considered. in general, it refers to the amount of resources consumed (input) in order to achieve a given goal, such as a specific computation or accuracy in a task (output). different definitions of efficiency correspond to different concepts of input and output. it is crucial to (1) understand the different concepts, (2) be aware of their differences and consequently their climate impact, and (3) converge towards a set of efficiency measures that will be applied for comparable climate performance evaluation in nlp research. @xcite quantify the financial and environmental cost of various nlp models, exposing substantial costs from model development and not just final model training. they recommend reporting training time and hyperparameter sensitivity, and prioritizing efficient hardware and algorithms. @xcite compare several efficiency measures, focusing on input or resource consumption: co 2 eq emission, electricity usage, elapsed real time, number of parameters, and fpo (floatingpoint operations). they suggest fpo as a concrete, reliable measure for climate-related efficiency that does not depend on the underlying hardware, local electricity infrastructure, or algorithmic details. they suggest measuring efficiency as a trade-off between performance and training set size to enable comparisons with small training budgets. @xcite show that fpos "are not adequate on their own to measure energy or even runtime efficiency." they recommend reporting various key figures, providing an automatic tool. ## climate performance model cards since 2020, neurips requires all research papers to submit broader impact statements @xcite . nlp conferences followed suit and introduced optional ethical and impact statements, starting with acl in 2021. 17 leins et al. ( 2020 ) discuss what an ethics assessment for acl should look like but focus solely on political and societal issues. tucker et al. (2020) analyze the implications of improved data efficiency in ai but only discuss the societal aspect of access in research and industry, leaving environmental issues unexplored. @xcite introduced model cards to increase transparency about data use in ai, similarly due to societal issues. we propose extending impact statements and model cards to include information about the climate-related performance of the development and training of the model, improvements compared to alternative solutions, measures undertaken to mitigate negative impact, and importantly, about the expected climate-related performance of reusing the model for research and deployment. 18 our proposed model card also includes any positive impact on the environment. a large direct negative impact does not rule out net positive impact due to contribution to downstream environmental efforts. while net impact cannot be measured objectively, since it depends on priorities and projections on the future use of the technology, we can set a framework for discussing this complex issue, providing researchers with the best practices to inform future researchers and practitioners. ## discussion ai and nlp research are behind in incorporating sustainability discourse in the discussion. in the field of finance, an increasing amount of companies worldwide are soon required to state their environmental and broader sustainability-related impacts and/or commitments in their annual reports, mostly following the recommendations laid out by the task force on climate-related financial disclosures (tcfd; financial stability board, 2017). 21 responsibility and accountability. significant differences exist between annual reports and research papers: companies are increasingly asked to take responsibility for their actions and are held accountable to their commitments by stakeholders, while researchers can shake off responsibility by transferring it to practitioners who use technology based on their research. researchers are thus never held responsible for committing to reducing negative environmental impact unless they choose to submit their work to specific workshops or conference tracks on sustainable and efficient nlp. however, there are no best practices on what they can do to help those who are responsible for committing to sustainability-what information is necessary for accurate reporting and informed decision making? extrapolation to indirect impact. the quantification of indirect impact during reuse and deployment of artifacts developed in research is complex and can only be estimated. we, therefore, expect that this discussion in environmental impact statements will be more abstract and harder to assess. as a framework, we propose borrowing the notion of scopes from corporate ghg accounting @xcite , where scopes 1, 2 and 3 correspond, respectively, to direct emissions (not applicable to nlp research); indirect emissions from operations, e.g., due to energy consumption (very common in nlp research); and indirect emissions upstream or downstream the value chain. for our case, we suggest the following scopes: 1. emissions generated during experiments for the paper itself, usually electricity consumption-related. 2. impact on other researchers and practitioners in reducing emissions using the technology. 3. the use of the technology for reducing emissions or other positive impact. note that these correspond, respectively, to scopes 2, 3 and 3 in the ghg protocol mentioned above. ## conclusion we argued that branding efficient methods in nlp as green or sustainable is insufficient and that due to the importance of the issue, climate awareness must be promoted in mainstream nlp rather than only in niche areas. we conducted a survey of climate discussion in nlp papers and found that climate-related issues are increasingly being discussed but are still uncommon. we proposed actionable measures to increase climate awareness based on experience from the finance domain and finally proposed a model card focusing on reporting climate performance transparently, which we encourage nlp researchers to use in any paper. while our discussion, survey, and recommendations are aimed towards the nlp community, much is applicable to other ai fields. indeed, specific recommendations have been made for machine learning @xcite and medical image analysis @xcite , for example. concurrently, @xcite propose a system-level roadmap addressing both ghg emissions and use of ai for climate change mitigation holistically. our focus on nlp enabled us to be more specific about relevant modeling components in our model card, as they are commonly used in nlp work. furthermore, framing our arguments within the discourse initiated in the nlp community allowed us to address the specific points raised in this discussion so far, and highlight specific avenues for positive impact.