CHATREPORT : Democratizing Sustainability Disclosure Analysis through LLM -based Tools (2023)

Abstract: In the face of climate change, are companies really taking substantial steps toward more sustainable operations? A comprehensive answer lies in the dense, information-rich landscape of corporate sustainability reports. However, the sheer volume and complexity of these reports make human analysis very costly. Therefore, only a few entities worldwide have the resources to analyze these reports at scale, which leads to a lack of transparency in sustainability reporting. Empowering stakeholders with LLM-based automatic analysis tools can be a promising way to democratize sustainability report analysis. However, developing such tools is challenging due to (1) the hallucination of LLMs and (2) the inefficiency of bringing domain experts into the AI development loop. In this paper, we introduce ChatReport, a novel LLM-based system to automate the analysis of corporate sustainability reports, addressing existing challenges by (1) making the answers traceable to reduce the harm of hallucination and (2) actively involving domain experts in the development loop. We make our methodology, annotated datasets, and generated analyses of 1015 reports publicly available. Video Introduction: https://www.youtube.com/watch?v=Q5AzaKzPE4M Github: https://github.com/EdisonNi-hku/chatreport Live web app: reports.chatclimate.ai

## introduction as climate change becomes an increasingly urgent issue, sustainability is becoming a key global concern, necessitating transparent public oversight of corporate sustainability practices. however, the substantial length of sustainability reports (often more than 70 pages) makes it challenging for the majority of stakeholders (including investors, policymakers, and the general public) to digest and analyze them. at the same time, relying on thirdparty rating agencies is not always a solution, as their services can be expensive, lack transparency, and vary due to differing criteria for evaluating sustainability performance @xcite . in light of these challenges, automated and transparent approaches are essential to improving accessibility, efficiency, and accuracy when analyzing corporate sustainability reports. large language models (llms) @xcite @xcite @xcite have revolutionized natural language processing (nlp), enabling advancements in automated reasoning, understanding, and generation of text. such advances can assist in conducting comprehensive analyses of corporate sustainability reports automatically. however, to develop such an llm-based system, there are two major challenges: llms (1) may hallucinate in their outputs @xcite , and (2) have no expertise in sustainability report analysis. furthermore, there exists no framework which would actively involve domain experts in the prompt development loop, injecting domain expertise into the prompts. in this paper, we propose chatreport, a system that automatically analyzes sustainability reports based on the tcfd foot_2 (task force on climaterelated financial disclosures) recommendations. it computes the reports' conformity score to tcfd guidelines, proposing the first automatic metric for disclosure quality benchmarking. chatreport also supports customized analysis with user question answering. to reduce hallucination, we ground the analytical prompts with retrieved information from the target report, and further make the answers traceable to help users identify hallucinations. to actively bring domain experts into the development loop, we design an automatic prompt engineering algorithm that transfers experts' feedback on specific outputs to general analysis guidelines, which can be injected into our prompt template for future analysis. furthermore, we conduct a rigorous human evaluation to analyze the system's hallucination rate quantitatively. we find that the system achieves an admirable hallucination-free rate. for those hallucinated cases, it is easy for users to identify them because the system always (1) refers to relevant sources and pages; and (2) answers questions in an extractive manner, making it convenient to identify evidence sentences by keyword search. moreover, we achieve a moderate inter-annotator agreement on annotating hallucination (cohen's kappa of 0.54), further illustrating that the discrepancies between answers and references are easy to identify. our human evaluation results in an annotated dataset of llm outputs with attributions, which may contribute to other domains (e.g., llm attribution verification @xcite to check the supportiveness of cited sources for the answer.). our contributions include: more recently, vaghefi et al. ( 2023 ) introduced chatclimate, a chatbot based on the latest ipcc assessment report. by leveraging nlp, researchers aim to extract valuable insights from textual data related to climate change to advance research, decision-making, and public engagement. large language models llms have emerged as the de-facto standard in recent years @xcite @xcite @xcite @xcite . instruction finetuned models, such as chatgpt @xcite and @xcite , have showcased their potential on comprehensive prompt-based ai applications @xcite . some strong llms can even be a cheap and reliable proxy for human preference, evaluating the quality of generated texts @xcite @xcite . however, hallucination still remains a major limitation of the sota llms @xcite . related work has proposed initial efforts to (1) better align llms @xcite ; and (2) fight false attribution from llm-based search engine @xcite and llm-generated misinformation @xcite . these efforts suggest potential ways to mitigate llm hallucinations, but still left it as an open research question. utilizing experts' feedback involving a human in the loop has a long history in machine learning and nlp. however, previous work mainly focuses on active learning @xcite and using human feedback to improve spe-cific outputs @xcite . in this work, we propose a novel promptingbased approach to automatically improve general prompts using experts' feedback on specific outputs, which actively brings human experts into the prompt engineering loop. ## usages we collected 9781 sustainability reports spanning 2010 to 2022 (fiscal years). most of the reports are companies that are traded on the nasdaq and nyse. we find that the number of pages in corporate sustainability reports has slightly increased over recent years: in the fiscal year 2017, the mean length of the report has been at 59 pages. in 2021, this number increased to 70 pages, illustrating the increasing effort required by analyzing the reports manually. ## hallucination analysis we conduct a human evaluation to assess the frequency and degree of hallucinations in chatre-port's output when answering questions. 6 hallucination is evaluated along two dimensions: (1) content: an answer is not hallucinated if all its covered information is supported by the report. all answers that are not fully supported (e.g., extrapolation or partial support) are considered hallucinated on the content dimension. (2) source: an answer is not hallucinated on the source dimension only when the model honestly reports its references and the content is not hallucinated; otherwise, the answer is hallucinated on the source dimension (we use binary annotation inspired by @xcite ). we randomly sampled 10 sustainability reports (110 tcfd question-answering pairs in total) for human evaluation (sampling details in appendix f). we have two different annotators to annotate each answer. if there is a disagreement on labeling, we assign a third annotator to make the decision. we conduct human evaluations on both chatgpt and gpt-4 as the backbone llm. we surprisingly find that despite our strict annotation standard, chatreport reaches a satisfactory hallucination-free rate. with chatgpt, it honestly conveys information from the report 83.63% of the time, considering the 51.5% average hallucination-free rate of existing generative search engines reported by @xcite . 7 further findings and discussion are presented in the following subsections. ## conclusion and future work we propose chatreport for automatic sustainability report analysis and demonstrating its potential applications and implications. our prompt development loop and annotated datasets about hallucination could positively transfer to other nlp and interdisciplinary research. chatreport is an open-sourced ongoing project. our future work will focus on (1) enhancing the retrieval module to provide more accurate contexts for generation, (2) developing automatic attribution-checking tools to fight hallucination, and (3) migrating from openai models to local llm for more controllable output.