Environmental Claim Detection (2023)

Abstract: To transition to a green economy, environmental claims made by companies must be reliable, comparable, and verifiable. To analyze such claims at scale, automated methods are needed to detect them in the first place. However, there exist no datasets or models for this. Thus, this paper introduces the task of environmental claim detection. To accompany the task, we release an expert-annotated dataset and models trained on this dataset. We preview one potential application of such models: We detect environmental claims made in quarterly earning calls and find that the number of environmental claims has steadily increased since the Paris Agreement in 2015.

## introduction in the face of climate change, we witness a transition towards a more sustainable and green economy. this change is driven by changes in regulation, public opinion, and investor attitudes. for example, global assets managed under a sustainability label are on track to exceed $53 trillion by 2025, more than a third of total assets under management. however, unfortunately, the boom has been accompanied by rampant greenwashing, with companies boasting about their environmental credentials. 1 because of this surge in environmental claims and to protect consumers, initiatives on substantiating green claims are developed. 2 due to an ever-growing amount of text, there is a need for automated methods to detect environmental claims. detecting such claims at scale can assist policy-makers, regulators, journalists, activists, the research community, and an informed public in analyzing and scrutinizing environmental claims made by companies and facilitating the transition to a green economy. environmental claim: a total population of 6148 is getting the benefit of safe potable drinking water due to this initiative. environmental claim: hydro has also started working on several initiatives to reduce direct co2 emission in primary aluminium production. negative example: generally, first of all, our transmission department is very busy, both gas and electric transmission, i should say, meeting the needs of our on-network customers. negative example: teams are thus focused on a shared objective in terms of growth and value creation. thus, we introduce the task of environmental claim detection. environmental claim detection is a sentence-level classification task with the goal of predicting whether a sentence contains an environmental claim or not. often, environmental claims are made in a clear and concise matter on a sentence level, with the intention to convey to a consumer or stakeholder that a company or product is environmentally friendly. to facilitate future research on environmental claim detection, we release an expert-annotated dataset containing real-world environmental claims and models which can be used by practitioners. for constructing the dataset, we were inspired by the european commission (ec), which defines such claims as follows: environmental claims refer to the practice of suggesting or otherwise creating the impression (in the context of a commercial communication, marketing or advertising) that a product or a service is environmentally friendly (i.e., it has a positive impact on the environment) or is less damaging to the environment than competing goods or services. 3 while such claims can be truthful and made in good faith, boasting about environmental credentials can also be monetized @xcite . for example, consumers are willing to spend more money on environmentally friendly products @xcite . the commission states if environmental claims are too vague, unclear, or misleading, we are confronted with an instance of "greenwashing" (this definition is given in the same commission staff working document). we situate environmental claim detection at the intersection of claim detection (e.g., arslan et al., 2020) and pledge detection @xcite . an environmental claim is typically made to increase the environmental reputation of a firm or a product. we show that models trained on the current claim and pledge detection datasets perform poorly at detecting environmental claims, hence the need for this new dataset. we make our dataset, code and models publicly available. 4 lastly, we envision computerassisted detection of greenwashing in future work, i.e., the automatic determination if an environmental claim is false, too vague, non-verifiable, or misleading. to make progress on automated greenwashing detection, it is mandatory to first detect environmental claims at scale. ## related work this work is part of an ongoing effort at the intersection of environmental and climate changerelated topics and natural language processing @xcite . resulting datasets and methods can help regulators, policy-makers, journalists, the research community, activists, and an informed public investigate such topics at scale with the help of computer assistance. methods include climatebert @xcite , and climategpt @xcite , two language models pre-trained on climate-related text. nlp tasks and datasets include climate change topic detection @xcite and detecting media stance on global warming @xcite further, there exists work about claim verification of climate change related claims @xcite , detecting media stance on global warming @xcite , collecting climate change opinions at scale from social platforms @xcite , and finally, the analysis of regulatory disclosures @xcite . in this broader context of applying nlp methods for climate change-related topics, we situate environmental claim detection at the intersection of claim spotting and pledge detection, covering the domain of text produced by companies with the goal of boosting their environmental credentials. claim spotting is the task of finding fact-check worthy claims @xcite @xcite . pledge detection aims to detect pledges made in, for example, political campaigns @xcite . environmental claims state an environmental benefit (claim) or convey the intention (pledge) for a material impact, i.e., some environmental benefit, which pleases the audience (consumers or stakeholders) of the claim. ## dataset our dataset contains environmental claims made by listed companies. we collected text from sustainability reports, earning calls, and annual reports of listed companies and annotated 3'000 sentences. after discarding tied annotations, our resulting dataset contains 2'647 examples. 5 we provide dataset statistics in the authors drafted annotation guidelines in an iterative process and added examples of clear and borderline environmental claims to the guidelines. in appendix b, we list the complete guidelines available to the annotators, along with examples and rationales that the authors discussed in pilot annotation rounds. to extract the sentences annotated in our dataset, we use a preliminary model to sample candidate sentences from various text sources produced by firms. furthermore, we randomly sample sentences from different clusters obtained with k-means to increase the coverage of the domain. we describe the sampling process of the dataset in detail in appendix a and provide further information on the data sources in appendix c. while we do not release a large-scale dataset, this is the result of a conscious decision to prioritize quality over quantity. we employed domain experts to annotate the data, which results in costly annotations. in appendix d, we show that the performance of models converges after being trained on more than 60% of the training set, and we find diminishing marginal utility of including more sentences. hence our decision to stop annotation here and release an annotated dataset with 2'647 examples. we assigned each sentence to four annotators. the annotations are aggregated by majority vote. 60% of the 3'000 samples was decided unanimously by the annotators, and 88.3% of the annotations made were part of a majority decision. 353 sentences received tied annotations (11.7% of the samples), and we discarded these examples from the dataset.the overall inter-annotator agreement measured in krippendorff's alpha is 0.47, indicating moderate agreement. ## experiments we conduct two types of experiments: (1) we examine the performance of various models on our dataset, among them pre-trained claim and pledge detection models and fine-tuned environmental claim detection transformer models (such as, e.g. @xcite @xcite . (2) we apply our models to the text produced by listed companies, which leads to a small case study demonstrating one of the intended use cases of the dataset. ## conclusion the vast and ever-growing volume of corporate disclosures, regulatory filings, and statements in the news calls for an algorithmic approach to detect environmental claims made by companies at scale. thus, we introduce the nlp task of detecting environmental claims, a dataset containing such claims and associated models which can detect these claims in the wild. our dataset is annotated by domain experts and thus of high quality. we describe the dataset and its construction process and present various models for detecting environmental claims in our dataset and a small case study. we envision several directions for future work. first, we plan to investigate "greenwashing", the practice of making a false, vague, unclear, or mis-leading environmental claim. to make progress on this front, it is mandatory that we can detect environmental claims in the first place. second, models trained on detecting environmental claims have merits of their own, as previewed in our case study. we plan to explore more such applications in detail, e.g., analyzing annual reports and tcfd foot_8 reports at scale. for example, it would be interesting to see in which sections of tcfd reports firms make environmental claims. lastly, we expect an increase of contributions at the intersection of environmental topics, climate change, and nlp in the near future. this work contributes to such efforts.