Structuring Sustainability Reports for Environmental Standards with LLM s guided by Ontology (2024)

Abstract: Following the introduction of the European Sustainability Reporting Standard (ESRS), companies will have to adapt to a new policy and provide mandatory sustainability reports. However, implementing such reports entails a challenge, such as the comprehension of a large number of textual information from various sources. This task can be accelerated by employing Large Language Models (LLMs) and ontologies to effectively model the domain knowledge. In this study, we extended an existing ontology to model ESRS Topical Standard for disclosure. The developed ontology would enable automated reasoning over the data and assist in constructing Knowledge Graphs (KGs). Moreover, the proposed ontology extension would also help to identify gaps in companies’ sustainability reports with regard to the ESRS requirements.Additionally, we extracted knowledge from corporate sustainability reports via LLMs guided with a proposed ontology and developed their KG representation.

## introduction presently, environmental and social justice are the main challenges demanding our attention, with a focus on transparency. as a result, there is an increasing demand for organizations to disclose nonfinancial information, particularly in sustainability reports. to regulate the disclosed information, companies can adhere to environmental, social and governance (esg) regulations. while there exist multiple esg frameworks, such as the widely adopted global reporting initiative (gri) and sustainability accounting standards board (sasb), effectively addressing all policy requirements and keeping track of all standards presents a challenging task. the most recent standard, released by the european union corporate sustainability reporting directive (csrd), is the european sustainability reporting standard (esrs). esrs aims to disclose the company's strategy to mitigate negative impact and to align with the paris agreement. incorporating esrs standards in annual reports will become mandatory in the european union, starting from 2025. the transition from well-known frameworks like gri to esrs involves several modifications. while the majority of gri disclosures have a corresponding esrs equivalent, the latter demands more data points and detailed information. specifically, financial materiality analysis in terms of risks and opportunities the company's environmental impact entails. advancements in large language models (llms) enable fast and effective processing and extraction of relevant information from the textual sources @xcite . coupling llms with knowledge graphs (kg) allows us to conveniently represent unstructured reports in a structured format automatically. since reading and analysing non-financial disclosure reports can become a long and cumbersome task, leveraging deep learning models provides a means for fast automatic analysis of large numbers of information. making it feasible to transition the existing gri and esg reports into esrs format, as well as pinpoint requirements that need to be examined and addressed in a more detailed fashion. the primary objective of this study is to automate the extraction of climate change-relevant information disclosed by companies in their nonfinancial reports and investigate the seamless transition to esrs reporting standards. large unstructured disclosure reports are transformed into structured graph-based representations for further analysis. the proposed approach is based on an extension of the text2kgbenchmark @xcite knowledge graph generation from text guided by an ontology. therefore, we extended the existing ontosustain @xcite ontology to include the esrs topical en-vironmental standards. we evaluate our approach through human annotation and see satisfying results. a link to the source code will be added after the review phase. ## related work developing a non-financial disclosure report analysis involves collecting information from multiple textual sources, which may lead to confusion due to the number of reporting requirements. ontologies are capable of modelling complex domain knowledge and mitigating natural language ambiguities @xcite , hence, they are perfectly suitable for the task. ontologies are defined as "a means to formally model the structure of a system, i.e., the relevant entities and relations that emerge from its observation, and which are useful to our purposes" @xcite . @xcite developed an ontology -ontosustain -that models sustainability domain knowledge and offers a platform for companies data collection process. knowledge graphs (kgs) offer a structured database for storing and representing the information from multiple sources, which facilitates a wide range of tasks, like semantic search, explainable ai, question answering, information retrieval @xcite . kgs contain the instance data for information modelled according to a specific ontology. llms trained on large corpora reach state-ofthe-art performance across multiple nlp tasks due to prompt engineering @xcite . llms' information extraction capabilities have shown to provide relevant structured information for kg construction @xcite @xcite @xcite . the resulting kg represents extracted information as concepts and the relations between them as edges @xcite . @xcite proposed an approach for extracting structural insights related to esg aspects from sustainability reports by leveraging llms, in-context learning and retrieval augmented generation (rag). the statistical analysis proved disclosure similarities between companies within a sector and region. the study also analysed the impact of esg ratings on companies. unlike @xcite study, our focus is on structuring non-quantifiable report aspects. to the best of our knowledge, to date, there have not been any studies conducted to represent and analyse esrs standards for disclosure reports, particularly esrs 2 topical standard, as kgs. hence, our work aims to address this challenge and offers an ontology and a kg construction method for representing esrs 2 topical standard. ## methods this section outlines the ontology design, dataset selection and kg construction procedures. ## evaluation since there are no human-labelled or classifiertrained esrs topical standard examples or training data, we asked three annotators to label the extracted topical standard aspects. firstly, annotators were given a brief explanation of the conducted study and topical standard aspects definitions. aspects extracted from the "siemens sustainability report 2022" were given as examples. later, annotators were asked to label 6 aspects from 14 reports, a total of 84 entries, based on how well the extracted descriptions match the definition of the respective aspect from the climate change topical standard. we also asked annotators to indicate if they think the aspect description is too general or vaguely written. our annotation guide will be published after review, too. all three annotators completed the informed consent procedure, demonstrating their understanding and willingness to participate in the study. annotators provided scores individually and did not interact with each other. figure 6 shows a screenshot of the spreadsheets form where annotators were asked to provide feedback, see appendix a section. inter-annotator agreement (iaa) @xcite assesses the level of agreement between multiple annotators in their evaluations of topical match. specifically, we use cohen's kappa (κ) to quantify the degree of agreement among the annotators. the formula for cohen's kappa is given by: we calculated the pairwise cohen's kappa, then averaged these values across all pairs of annotations to obtain an overall agreement measure. the value of κ ranges from -1 (complete disagreement) to 1 (complete agreement), with a value of 0 indicating agreement by chance. in our case, the average cohen's kappa score across all pairs of annotators is 0.512, which suggests a moderate level of agreement among the annotators. annotation results per aspect and annotator are presented in ## experimental results as mentioned above, the calculated cohen's kappa resulted in 0.512, a moderate agreement. factors such as task complexity, ambiguity in annotation guidelines or differences in annotators' expertise could have influenced the outcome. the iia results could potentially change when having more participants for annotation, however, our study had limited resources and missed more experts from the field of economics. the llm was able to accurately extract organization name, reporting name and its release year for all 14 reports. as seen from in most cases where impact was classified as not match, the aspect summary was reporting either the impact of climate change on the organization, which is more related to risks aspect, or actions the organization takes to be more sustainable, which belongs to the actions aspect. opportunities refers to significant financial impacts that climate change brings into the organization structure and economic decisions. hence, many mismatched summaries were a mix of actions and risks. figure 3 depicts the number of topical match and vagueness results per aspect. aspects that have the least topical match are also the ones containing the most vague descriptions. targets were commonly labelled as vague due to the absence of quantifiable and measurable goals. many impact entries were considered vague because the descriptions included the impact of climate change on the organization, as well as the impact of the organization on climate change. on the contrary, opportunities provided the definition of financial materiality in multiple entries, which was considered too general by annotators. clearly, both aspects require a much more descriptive demonstration example during the knowledge extraction phase. overall, the suggested ontology extension and the ontology-guided prompting technique demonstrated reasonable results. an example kg extracted from the airbus report is shown in figure 4 . ## conclusion we present a novel method for extracting and structuring information from disclosure reports. representing knowledge in such a way would offer more transparency and reusability of the data for further analysis. in this study, first, we designed an extension for existing ontology to represent esrs 2 topical standard information. second, we also tested the proposed extension by prompting gpt-4 with ontology guidance and expert prompt engineering. the results demonstrated that the proposed extension adapts well to many aspects, namely strategy, risks, actions and targets. while impact and opportunities require better description and representation.