Informing climate risk analysis using textual information - A research agenda (2024)

Abstract: We present a research agenda focused on efficiently extracting, assuring quality, and consolidating textual company sustainability information to address urgent climate change decision-making needs. Starting from the goal to create integrated FAIR (Findable, Accessible, Interoperable, Reusable) climate-related data, we identify research needs pertaining to the technical aspects of information extraction as well as to the design of the integrated sustainability datasets that we seek to compile. Regarding extraction, we leverage technological advancements, particularly in large language models (LLMs) and Retrieval-Augmented Generation (RAG) pipelines, to unlock the underutilized potential of unstructured textual information contained in corporate sustainability reports. In applying these techniques, we review key challenges, which include the retrieval and extraction of CO _2 emission values from PDF documents, especially from unstructured tables and graphs therein, and the validation of automatically extracted data through comparisons with human-annotated values. We also review how existing use cases and practices in climate risk analytics relate to choices of what textual information should be extracted and how it could be linked to existing structured data.

## introduction in light of the climate crisis, there is an increasing call to integrate climate risk with the decisionmaking of companies, banks and regulators. climate risks for companies and, by extension, financial institutions have been grouped into two types: transition risks and physical risks @xcite . transition risks arise from the transition of the economy towards carbon neutrality and can materialize, e.g., in the form of higher-than-expected carbon prices, stricter regulation, or changes in technology and consumer preferences. these risks affect companies and sectors with high (expected) carbon emissions. physical risks, on the other hand, denote the direct adverse effects of a changing global climate, such as sea level rise or increases in storms and floods, droughts, and other natural disasters @xcite . unlike transition risks, physical risks do not depend primarily on companies' carbon footprint, but on the vulnerability of their assets and business operations to physical damage based on their geographic location. besides the companies themselves, climate risks are relevant to the financial institutions which are exposed to the affected companies through financial instruments such as loans or bonds. a bottleneck in climate risk analysis is the availability of reliable data @xcite . items that can help measure companies' physical or transition risk profiles, such as carbon emissions and transition plans, are scarcely available. as a consequence, institutions like the european system of central banks (escb) have thus far relied on proprietary datasets from private data providers @xcite . these commercial providers often source their climate risk data from corporate (sustainability) reports through manual annotation. whenever reported data is not available or deemed insufficiently reliable, these data providers estimate numbers. often, however, neither the reported nor the estimated data is replicable, since the providers do not disclose their estimation methods, and human annotators can be prone to errors. despite recent regulatory efforts which have led to an uptick in company sustainability disclosures, the data is most often provided in relatively unstructured sustainability reports. within these reports, important information is not usually presented in consistent and numeric formats (e.g., in structured tables), but can be presented in any form of text and even graphics. beyond corporate sustainability reports, unstructured textual sustainability information on climate risks is also available in the form of newspaper articles, social media comments, and other dispersed sources. the left panel of figure 1 presents an overview of existing structured and unstructured sources of climate information. in this landscape, recent technological progress in natural language processing (nlp) opens up a range of new opportunities in efficiently extracting relevant data from unstructured textual information, which then can be linked to other data sources. within the possible sources of textual information, companies' sustainability reports are arguably the most relevant document type for climate risk analysis since some form of sustainability disclosure tends to be mandatory. the information contained in such reports is mostly related to transition risks. this stems from the fact that, while sustainability reports could conceivably also include information on physical risks, the focus (beyond marketing considerations) usually lies on the companies' ecological footprint. therefore, when referring to climate risks in the context of this paper, we focus on extracting information related to transition risks unless explicitly stated otherwise. for physical risks, unstructured information also exists largely in the form of images, e.g., satellite imagery or street view. in this domain, recent research also aims to convert unstructured information from images into usable data @xcite . our goal is thus to leverage sustainability reports in order to validate existing data sets, close data gaps by making new variables available, increase the coverage of company-level data, and improve the accessibil-ity of information. the remainder of this paper develops a research agenda that leverages nlp methods to condense the disparate sources of unstructured information into a structured, comprehensive, accessible, and trustworthy database. we develop this proposal across three sections: the first section discusses the latest research and use cases of nlp in the context of textual sustainability information in general and corporate sustainability reports in particular. the second section further explores the specifications and challenges related to llm-based extraction pipelines by reporting the results from three initial experiments aimed at extracting emission values from 39 sustainability reports. the third section addresses the questions of (i) how data extraction should be organized, (ii) what information should be prioritized for extraction, and (iii) how data linkage and post-processing should be undertaken in order to create an integrated data infrastructure. the fourth section concludes the paper. ## background on nlp for sustainability data recent innovations in nlp, especially llms, such as bidirectional encoder representations from transformers (bert) and generative pre-trained transformers (gpt), have enabled major advances in the availability of research and web-based tools for analyzing documents. company sustainability and financial reports contain a wealth of data in unstructured, multi-modal (e.g., as tables, graphs, and text), and only partially standardized formats. as such, they provide a strong use case for the application of this new generation of nlp approaches. their potential is illustrated by the fact that freely available online tools for the analysis of texts have mushroomed recently. next to general-purpose chat bots including openai's chatgpt and similar (at times derivative) products such as chatpdf and pdf.ai, there are also products with an exclusive focus on sustainability. examples of these tools include the sustainable development goals (sdg) prospector @xcite , which highlights all sdg-related paragraphs in the uploaded documents, or chatclimate, which targets the analysis corporate sustainability reports. these solutions, however, generally focus on interactive chat bots with graphical user interfaces (guis). similar in design and usability to openai's chatgpt, they target human, ad-hoc, infrequent users who can profit from a more time-efficient extraction of specific relevant information from sustainability disclosure -essentially, users who do not wish to read complete documents to find specific types or single pieces of information. apart from chat bots, academics from a variety of disciplines leverage nlp methods to systematically gather and evaluate sustainability information from large text corpora. in the field of corporate sustainability research, earlier bag-of-words approaches that relied on word-frequency have been increasingly replaced by more sophisticated methods that take the context of textual documents into account and can be leveraged for the extraction and analysis of various types of information. in this context, one strand of research has developed different extensions to bert models to perform text classification of sustainability-related information, such as finbert-esg @xcite , cli-matebert @xcite , and climateqa @xcite . this class of domain-specific language models expands the general bert model through a pre-training and a fine-tuning stage: during pretraining, the model is augmented with domainspecific texts. in the context of corporate sustainability research, corporate financial and sustainability reports, financial analyst reports, earning call transcripts, (keyword-filtered) news, and scientific abstracts have been used as pre-training data (cf. @xcite @xcite ). in the fine-tuning stage, the model is provided with a set of human-annotated texts which have been assigned to a specific outcome category. such annotation efforts have been undertaken inter alia to the concept of environmental, social and governance (esg) issues @xcite , each of its subdomains or pillars (i.e., e, s and g separately; cf. @xcite ), companies' "environmental claims" @xcite , and particular sustainability disclosure frameworks, i.e., the taskforce on climate-related financial disclosures (tcfd, @xcite . domain-specific models have been applied to a variety of tasks including text classification, sentiment analysis, and "fact-checking". these models have also been found to outperform generic language models with regards to the accuracy of text classification @xcite @xcite @xcite . in addition, first proposals suggest that these models could be applied for text classification tasks related to the identification of "greenwashing" @xcite @xcite , i.e., the promulgation of unsubstantiated environmental claims @xcite . while domain-specific models have generally focused on the classification of textual data, a second strand of research has applied language models to find and extract numerical as well as textual data. to this end researchers have deployed so called retrieval-augmented generation (rag) pipelines that add domain-specific context to an llm prompt. in the field of sustainability research, applications of rag include the gpt-4 based chat-climate @xcite that extracts information from the intergovernmental panel on climate change (ipcc) ar 6 based on user prompts and chatreport @xcite , which extracts information from corporate sustainability reports and checks the alignment of the extracted information with tcfd disclosure rules. another rag application for extracting sustainability data from companies' sustainability reports is explored by @xcite , who use a llama-2 model for a finegrained assessment of companies' sustainabilitylinked topics and actions. in addition, @xcite have tested the performance of different language models in processing sustainability reports, by adopting a rag pipeline that extracts the numerical and textual indicators that are defined in the global reporting initiative (gri) and sustainability accounting standards board (sasb) disclosure standards from pre-processed company reports. more recently, a similar workflow has been adopted by the innovation hub of the bank for international settlement's (bisih) "project gaia" (bis innovation hub, 2024), which develops an application that uses gpt-4 in a rag setting and a module that integrates indicator definitions from legislative texts to extract numerical and categorical key performance indicators (kpis) from sustainability reports. these examples from the prior literature underscore the immense potential of novel nlp methods to facilitate the efficient extraction of sustainabilityrelated information from corporate disclosure documents and -once implemented -to do so at a relatively low cost. their achievements notwithstanding, there are arguably still important challenges that limit the usefulness of such methods for systematic analysis of climate risks and related issues. first, concerning the technical specifications there remain open questions with respect to the validation of the extracted values as well as to cost and time-efficient set-ups of the extraction pipelines. second, so far there has been comparatively little discussion on how the obtained values can be meaningfully integrated into existing practices of data analysis in the context of climate risk assessments. in light of these challenges, in the following sections we delve further into both technical and user-related issues and propose first steps in a research agenda that tackles these various challenges together. ## preliminary results from a technical point of view, the automatized extraction of information from sustainability reports faces various challenges. with ragbased pipelines these challenges include the preprocessing of pdfs and the text therein (i.e., the conversion of pdf files into a machine-readable format), cost-efficient procedures for large numbers of pdf documents, and the validation of the extracted values against benchmarks (bis innovation hub 2024, @xcite . especially with regards to validation, the absence of gold-standard benchmarking data has proved to be challenging as existing datasets on corporate sustainability indicators tend to be proprietary, intransparent, and values vary substantially among commercial providers (cf. @xcite . to get a clear overview of the challenges and potential trade-offs along the extraction pipeline, we set up a first experiment that enables us to compare different technical specifications of the model but also focuses on the potential pitfalls of human labelled benchmark data. the first step in this experiment was to annotate 39 sustainability reports from large companies from the years 2010 to 2021. these are randomly sampled from the universe of msci world firms that published english language reports. the list of selected reports is presented in we chose to extract the values for greenhouse gas (ghg) emissions in our experiment. compared to other indicators, ghg emissions disclosures are more frequent and less variable as most companies report according to the ghg protocol (ghgp) standard @xcite . first introduced in 2004 by the world resource institute and the world business council for sustainable development, the ghgp has since been adopted by most large companies and been integrated into regulatory requirements across the world @xcite . the ghgp standardizes emission disclosures through three categories of emissions, so called "scopes", that reflect the operational control of the company over the released ghg. accordingly, "scope 1" emissions denote ghg releases from sources that are directly controlled and operated by the company. "scope 2" emissions, meanwhile, refer to emissions from that were generated from the generation of electricity that the company purchased. finally, "scope 3" emissions refer to other indirect emissions that occur in the company's value chain such as the extraction and production of purchased materials or the use of sold products and services. five human annotators extracted scope 1, 2, and 3 ghg emissions. annotators were asked to open the .pdf file, search for the term "scope 1" (respectively "scope 2" or "scope 3") and a predefined list of synonyms including "direct / indirect emissions" and extract (if found) the resulting value, unit, variable name, year, page number, and origin (one of "table ", "text", or "graphic") into a spreadsheet (see appendix c). among the pitfalls that were encountered by human annotators, missing information is among the most prominent. we found that eleven reports, or 28 percent of the sample, do not report any emission values. the problem of missing information becomes even more accentuated for scope 2 and 3 emissions, which are often not contained in older reports. a second pitfall concerns unclear and varying concept definitions. for instance, some reports only report employee travel under their scope 3 ghg emissions, whereas others use this concept to refer to total upstream and downstream emissions. thirdly, we encountered different ways of disseminating information including text, tables and infographics. a final pitfall is the presence of different measurement units for ghg emissions. while some of these are easy to convert (e.g., tco 2 eq vs kgco 2 eq), other units such as emission intensities as opposed to absolute emissions, or co 2 equivalents as opposed to separate depiction of single greenhouse gases are more problematic in this regard. the next step was to set up an automatic data extraction pipeline. we use an llm to convert raw text from pdfs into a structured, tabular format. since sustainability reports can be rather long, we first need to search for the most relevant content (e.g., pages, tables) before passing it to the llm. this coupling of search, typically done via embeddings, with llms is a common architectural pattern to enhance llm capabilities, known as naive retrieval augmented generation (naive rag) @xcite . three approaches were tried to extract all scope 1/2/3 ghg emissions for each year from each report: first, we search for relevant pages and pass the raw text of the so-found pages to an llm. specifically, we embed the search query "what are the total co 2 emissions in different years? include scope 1, scope 2, and scope 3 emissions if available." using openai's text embedding model ada-002 and compare it with the embedding of each page from the pdf report. the two most relevant pages from this search are kept, concatenated, and submitted in a single query to openai's flagship llm, gpt-4-turbo. based on the raw text from these two pages, the llm is prompted to answer a list of 48 questions (16 years × 3 scopes): "1. what are the scope 1 emissions in 2010: <@xmath0. what are the scope 1 emissions in 2011: <@xmath1, and so on, for all possible combinations of year (2010 -2025) and scope ( 1 2 3 ). the search query and the complete llm prompt are provided in appendix b. the output from the llm is typically well structured, meaning that it can be parsed using regular expressions to insert the extracted (value, unit)-tuples into a data frame. the second approach is very similar: the general pipeline, the models, and the queries remain the same. we only change the selection of pages and their handling. we now keep the three most relevant pages from the search, along with each page's preceding and subsequent page. this gives us at most nine pages per report in total. we do not concatenate the pages as in approach 1, but send each page in separate queries to the llm, because we found during preliminary testing that gpt-4-turbo overlooks relevant values more often if pages were concatenated. the output from each query gets parsed separately implying that for a single scope-year combination from a single report we may extract more than one value as the llm may extract different values from different pages. third, again following the same pipeline, we adopt a table-only approach. since the co 2 emission scopes are predominantly presented in tables within sustainability reports, we leverage the python package pdfplumber foot_0 , which enables table extraction from pdf files. after extracting the tables, we apply a similar pipeline as in our second experiment keeping the ten most relevant tables from the search and feeding them into the llm. we present the results of the three preliminary experiments in the main drawback of e1 is, however, its retrieval strategy: for ten reports we would have liked the algorithm to extract values from specific pages that were not found during our search and were therefore not passed to the llm. our second approach (e2) was designed to alleviate this problem: as we widen the search, we reduce the tally of retrieval failures to just four. this success, unfortunately, is not reflected in the number of correctly extracted values (1+3 reports), because the llm frequently extracts wrong values (five reports) or, in reverse, does not extract values that should have been extracted (3+12 reports). while the performance is not yet satisfactory, these results suggest that future work is needed in three areas: retrieval, usage of llms for extraction tasks, and unit harmonization. the third approach (e3) yields even poorer results for the task. this indicates the inadequacy of only relying on tables for content extraction, even though based on human annotation we would expect that emission values are usually summarized in tables in the reports. ## discussion and research agenda as outlined in section 1, the goal of applying nlp techniques to unstructured corporate sustainability information is to extract high-quality data. notably, this includes a large coverage to enable comparative assessments of transition risks and related use cases by academics, financial supervisors and other public and private institutions. based on our analysis of the literature and first findings from experiments with a rag pipeline, we segment the challenges and research gaps for creating a highquality, accessible database on corporate sustainability into two how and one what questions. the first how question relates to the design of the rag pipeline and covers issues like the the set-up of human annotation, prompt engineering, and the extraction of different presentation formats within the sustainability reports (e.g., tables, graphs). the what question, in turn, asks which variables should be contained in the structured database. answering this question, notably, requires domain-specific expertise as it not only relates to the indicators such as ghg emissions that should be extracted, but also to contextual information that could help users to assess the credibility of the reported data. the second how question, finally, refers to the post-processing of the extracted values through data science techniques. these operations can include the creation of new indicators pertaining to the reliability of the company disclosed data as well as to the linkage of the extracted values with other datasets. ## outlook and conclusion as companies and other stakeholders produce an ever increasing volume of climate and sustainability information, we are confronted with the paradoxical situation, where a wealth of data is freely available, while climate risk analysts simultaneously point to data gaps. technological progress in llms offers an opportunity to overcome this apparent gulf, by turning dispersed unstructured information into fair data. creating integrated fair data, however, comes with technical challenges and domain-specific choices regarding the data infrastructure, both of which should be addressed systematically and transparently as part of an integrated research agenda.