C limate BERT - N et Z ero: Detecting and Assessing Net Zero and Reduction Targets (2023)

Abstract: Public and private actors struggle to assess the vast amounts of information about sustainability commitments made by various institutions. To address this problem, we create a novel tool for automatically detecting corporate and national net zero and reduction targets in three steps. First, we introduce an expert-annotated data set with 3.5K text samples. Second, we train and release ClimateBERT-NetZero, a natural language classifier to detect whether a text contains a net zero or reduction target. Third, we showcase its analysis potential with two use cases: We first demonstrate how ClimateBERT-NetZero can be combined with conventional question-answering (Q&A) models to analyze the ambitions displayed in net zero and reduction targets. Furthermore, we employ the ClimateBERT-NetZero model on quarterly earning call transcripts and outline how communication patterns evolve over time. Our experiments demonstrate promising pathways for extracting and analyzing net zero and emission reduction targets at scale.

## introduction motivation. "zero tolerance for net zero greenwashing" is the central aim of un general secretary antonio guterres at the gop27 (un, 2022). to limit global warming to 1.5°c, massive efforts in emission reduction are necessary @xcite . consequently, an increasing amount of net zero and emission reduction targets are introduced, communicated, or updated by private and public institutions @xcite . with the help of natural language processing (nlp) methods, it is possible to automatically assess large chunks of textual data and gain structured insights about climate-related information (e.g., @xcite @xcite . however, none of the previous works have studied the arguably most relevant and best tractable objective for institutions -the extraction of information about net zero and reduction targets. contribution. as a remedy, this study delivers a threefold contribution at the intersection of climate change and nlp by creating a tool that automatically extracts and assesses net zero and reduction target information from various sources. our first contribution is introducing an expert-annotated data set with 3.5k text samples, which builds on the net zero tracker project @xcite . second, we develop and publish climatebert-netzero (based on climate-bert @xcite ), a comparatively lightweight and, therefore, less energy-intensive nlp model that can effectively classify net zero and reduction targets in text samples. 1 third, we provide two real-world use cases: we demonstrate how to extend the model capabilities and analyze the ambitions of the net zero and reduction targets. finally, we utilize climatebert-netzero to analyze the appearance of net zero and reduction claims in companies' earning calls transcripts from 2003 to 2022. ## background large language models. in previous years, large language models (llms) continuously gained importance. llms have proved to possess humanlike abilities in a variety of domains @xcite @xcite . these models are primarily based on subtypes of the transformer architecture and are trained on massive amounts of data to learn patterns within the training data @xcite . as a result of the training process, the models can solve a multitude of downstream tasks. one particular downstream task represents text classification. for these tasks, commonly used models are based on the bidirectional encoder representations from transformers (bert) architecture @xcite . nlp in the climate change domain. our work contributes to an ongoing effort to employ llms to support climate action by developing targeted datasets and models. previously released work includes the verification of environmental claims @xcite , climate change topic detection @xcite , and the analysis of climate risks in regulatory disclosures @xcite . a particular way to improve the performance of models on downstream tasks is the creation of domain-specific models, such as cli-matebert. with further pre-training on climaterelated text samples, this model outperforms its peers trained on general domain language @xcite . climatebert has already been applied in recent studies that analyze corporate disclosures to investigate patterns around companies' cheap talk @xcite . apart from model-centric considerations and innovations, previous research and public attention have also shifted to the climate impact of training and deploying llms @xcite . with the ongoing development of new models, the models' number of parameters steadily rises. consequently, this increases electricity requirements and, therefore, the environmental footprint of training and using the models. for instance, while climatebert possesses 82 million parameters @xcite , chatgpt has up to 175 billion parameters (openai, 2023). hence, climate-aware decisions for model training and deployment must be carefully considered. ## data the raw data for this study originates from a collaboration with the net zero tracker project @xcite . the net zero tracker assesses targets for reduction and net zero emissions or similar aims (e.g., zero carbon, climate neutral, or net negative). reduction targets are claims that refer to an absolute or relative reduction of emissions, often accompanied by a baseline year to which the reduction target is compared. net zero targets represent a special case of reduction targets where an institution states to bring its emissions balance down to no additional net emissions by a certain year. the net zero tracker assesses claims of over 1,500 countries, regions, cities, or companies. 2 for this project, we created an expert-annotated 3.5k data set. the data set differentiates between three classes: net zero, reduction, and no targets. for the reduction and net zero targets, we further process the claims gathered by the net zero tracker. if necessary, we clean and hand-check the text samples and correct the pre-assigned labels by the net zero tracker. moreover, we enhance the data with a variety of non-target text samples. all of these samples originate from annotated and reviewed datasets of previous projects in the climate domain @xcite ## climatebert-netzero we use the data set to train and test the capabilities of several classifier models. we particularly want to focus on training a model that is both effective in performance and efficient in the use of resources. thus, as a base model for fine-tuning, we utilize cli-matebert @xcite . this model is based on distilroberta which is a smaller and thus less resource-demanding version of roberta @xcite . climatebert is further pretrained on a large corpus of climate-related language and thus outperforms conventional models on a variety of downstream tasks @xcite . we further fine-tune climatebert to classify whether a text contains a net zero target, a reduction target, or none of them. we christen this model climatebert-netzero. to set the results into context, we rerun the fine-tuning with its base architecture distilroberta as well as with the larger model roberta @xcite . furthermore, we also test how generative, much more resource-intensive, but also more general models perform in comparison to climatebert-netzero. to create this comparison, we prompt gpt-3.5turbo to classify the texts (see appendix b for the detailed prompt). to evaluate the performance of the bert-family models, we run a five-fold cross-validation and consult the performance measures accuracy, f1-score, precision, and recall (see appendix c for all explored variations of hyperparameters and appendix d for full results and description of the validation set). as climatebert-netzero also outperforms gpt-3.5turbo on this task. 3 to further evaluate the performance of the climatebert-netzero in a real-world setting, we conduct a hand-check on the output of the model on 300 sentences sampled from sustainability reports. we find that the model classifies 98% of the sentences right. this solidifies the robustness of the approach (see appendix e for more details). since net zero targets are a special case of reduction targets, we also create a model called climatebert-reduction. this model simply differentiates between general reduction targets (including net zero targets) and no targets. we repeat the previous setup to train and test the model. climatebert-reduction consistently reaches performance measures of ca. 98% (for more details, see appendix f). ## use cases to showcase further capabilities of climatebert-netzero, we develop two potential use cases. first, we showcase how the model can be used to perform a deeper analysis of net zero and reduction targets by analyzing the underlying ambitions of the statements. second, we demonstrate how climatebert-netzero can detect patterns in company communication by analyzing earning call transcripts. measure ambitions with q&a models. in this use case, we study methods to assess the ambitions of institutions further. for net zero claims, the primary measure of ambition is given by the year the institution wants to achieve net zero. for reduction claims, we use the target year of the reduction, the percentage of how much the institution wants to decrease its emissions, and the baseline year against which the reduction is measured. for evaluating the ambitions in the net zero and reduction claims, we employ the question answering (q&a) model roberta-base-squad2 @xcite . this model delivers concise answers to questions posed to a text sample. therefore, we develop questions to extract the relevant information for each dimension of ambition in the net zero and reduction claims (see appendix g for detailed questions). to evaluate this approach, we again use the data compiled by the net zero tracker project @xcite . the data set contains human-extracted data for each of the researched dimensions of ambitions. to test the ambitions, we preprocess and hand-check 750 text samples. finally, we run the q&a model on the text samples. the model typically outputs a single number. we assess the accuracy of the model by checking how many answers contain the correct number. detection net zero and reduction targets in earning call transcripts. in this use case, we employ climatebert-netzero on transcripts of quarterly earning calls from publicly-listed us companies from 2003-2022 to demonstrate its large-scale analysis potential. we create a yearly index by building an average of all quarters of a year. as figure 1 indicates, the net zero target discussion has experienced a sharp increase since 2019. 4 while the same observation is true for reduction targets, in conclusion, this paper demonstrates the development and exemplary employment of climatebert-netzero, a model that automatically detects net zero and reduction targets in textual data. we show that the model can effectively classify texts and even outperforms larger, more energy-consuming architectures. we further demonstrate a more finegrained analysis method by assessing the ambitions of the targets as well as demonstrating the largescale analysis potentials by classifying earning call transcripts. by releasing the dataset and model, we deliver an important contribution to the intersection of climate change and nlp research. identifying and analyzing reduction and net zero targets represent an important first step to controlling set commitments and further comparing them with other real-world indicators of corporate, national, or regional actors. especially in the realm of greenwashing, the abundance or failure to achieve set targets while benefitting from a differently communicated public image are key factors to investigate. ## limitations like all studies, ours also has limitations. first, the dataset size is limited to 3.5k samples. while this could entail lower generalizability, we show that the model performance is already at a very high level. the data size limitation also applies to the first use case, where further data could increase the validity. second, this study predominately focuses on bert-family models. in a strive to employ accessible models of smaller size, we do not employ larger, more recent architectures such as llama @xcite . while the impact on the classification task might be limited, as shown with gpt-3.5-turbo, especially for detecting ambitions, instruction-finetuned models could improve the results. the text is decisive. for instance, most reduction targets serve as intermediary steps for the final goal of net zero. thus, the focus lies on net zero. the following examples provide an impression of a reduction and net zero target: reduction target : 'we will reduce our absolute scope 1 and 2 ghg emissions from nonlocomotive operations ( including emissions associated with our buildings and facilities ) by 27.5% by 2030. ' net zero target : ' we are committed to net -zero energy across our businesses and achieved 100% carbon neutrality for our operations by purchasing carbon offsets with renewable energy certificates ( rec ).' third, we use a human-in-the-loop validation process to enhance the validity of the data further. we use five-fold-crossvalidation with the baseline climatebert model on the entirety of the dataset and save the missclassifications of each fold. this gives us the opportunity to train the model with the data and, at the same time, store the wrongly identified samples within the training process. we then involve a human annotator to check the misclassifications and readjust labels if necessary. this way, we use the machine to identify edge cases and the human to interfere if needed. we repeat this human-in-the-loop validation three times. non-target samples. for non-target text samples, we use publicly available datasets in the climate domain @xcite . we aim to increase the generalizability of the models by creating a well-balanced dataset. including climate-related samples that are not targets is crucial for the models to learn the specific differentiation between targets and general sentences in the climate domain. furthermore, the datasets enable us also to include generic, non-climate-related sentences. we follow the same procedure described above to improve the validity of the dataset. ultimately, we create a dataset with 3.5k text samples. table a .1 gives an overview of the descriptive statistics of the text length.