C lim R etrieve: A Benchmarking Dataset for Information Retrieval from Corporate Climate Disclosures (2024)

Abstract: To handle the vast amounts of qualitative data produced in corporate climate communication, stakeholders increasingly rely on Retrieval Augmented Generation (RAG) systems. However, a significant gap remains in evaluating domain-specific information retrieval – the basis for answer generation. To address this challenge, this work simulates the typical tasks of a sustainability analyst by examining 30 sustainability reports with 16 detailed climate-related questions. As a result, we obtain a dataset with over 8.5K unique question-source-answer pairs labeled by different levels of relevance. Furthermore, we develop a use case with the dataset to investigate the integration of expert knowledge into information retrieval with embeddings. Although we show that incorporating expert knowledge works, we also outline the critical limitations of embeddings in knowledge-intensive downstream domains like climate change communication.

## introduction motivation. climate change presents the most pressing challenge of our time. the underlying concepts and challenges generate a wealth of information with inherent complexity and interconnectedness. at the same time, most of the data on corporate climate disclosure is qualitative -hidden in textual statements @xcite . qualitative disclosures typically include narrative descriptions of climate-related risks, opportunities, strategies, and governance. these are crucial to understanding how a company perceives and manages climate-related issues and their potential impacts on business operations. advances in natural language processing (nlp) try to address data structuring and analysis challenges. specifically, retrieval-augmented-generation (rag) emerged as a method to address knowledge-intensive questions around climate change @xcite @xcite . despite the growing demand for more precise climate change data @xcite , a significant gap exists in evaluating rag systems. while researchers have developed methodologies for the automatic evaluation of generated content @xcite @xcite , the preceding crucial phase of information retrieval remains largely unexamined in the context of climate change. 4 contribution. therefore, this paper delivers two contributions. first, it introduces a comprehensive expert-annotated dataset for the retrieval and generation part of rag. the dataset emulates an analyst workflow to answer questions based on the provided documents. thus, the core data set comprises questions, the corresponding sources recovered from experts, their relevance, and an answer to the question (see figure 1 ). second, we design an ties, as well as the roles of the board and management in these processes. implications. the implications of our study are significant for both practice and research. knowledge-intensive downstream domains like climate change are nuanced, and details matter. this paper can significantly help researchers evaluate new rag systems (e.g., @xcite and corporate climate report analysts to obtain useful information for decision-making. ## data this project constructs a dataset comprising authentic questions, sources, and answers to benchmark information retrieval in rag systems in the use case of corporate climate disclosures. in this process, we simulate an analyst question-answering process based on documents. the dataset creation involves an iterative question definition and report span labeling process (see figure 2 ). it starts with 16 yes/no questions about climate change. the questions are inspired by the guidance of @xcite and analyze companies' climate change adaptation. thus, the question asks for details simulating an analyst's point of view on a company (see appendix c). these questions are distributed among three expert annotators (see appendix d). for each question, an annotator creates a definition and concepts of the information sought in the question. then, both are discussed in the expert group. this step is crucial to understanding the question in detail (see appendix b for details on the question definition and concepts). in the next step, the expert annotators create the dataset using a specific sustainability report. annotators search for relevant information in the report and annotate the sources from various perspectives. in this way, they replicate an analyst workflow in which the task is to read the document and search for relevant information to answer the question and rate its relevancy. then, they answer the question based on the information. ultimately, they create a dataset containing the following columns: climate change questions question understanding report labelling initial definition used for informs and adjusts definitions & concepts span labelling relevant information answering questions figure 2: labeling process to obtain the climretrieve dataset. 4. context: context of the question relevant information (extending the relevant information by a sentence before and afterward). 5. page: page of the relevant information. 6. source from: answers whether the relevant information is from text, table or graph. 7. source relevance score: classifies from 1-3 how relevant the information is for answering the question (see appendix e for details on the relevance classification). 8. unsure flag: flag whether it is unclear if this source is question-relevant. 9. addressed directly: flag whether the relevant information addresses the question directly or indirectly. this process is repeated for 30 sustainability reports. as a result, we obtain a base dataset with 743 entries of relevant question-source-answer pairs (see appendix f for details). furthermore, we can create a report-level dataset since we know which parts of the report are relevant. in this dataset, we split the reports into paragraphs of equal length and mark relevant vs. nonrelevant parts with the question-source-answer pairs. this results in a dataset with 8.628 paragraphs labeled with the question's relevance. since the questions are in semantic proximity, one paragraph can be relevant to multiple questions. for this reason, we ultimately create a dataset that contains unique reportparagraph-question pairs. for each question, the whole report is labeled. thus, a report's paragraphs are repeated for each question to create an easyto-assess dataset. in this way, we obtain a large report-level dataset with 43.445 entries (for details, see appendix g). ## investigating embedding search we construct a specific use case to demonstrate the report-level dataset's practical applicability. given the scarcity of research on information retrieval specific to climate-related corporate disclosures, this use case study is concentrated on this particular area. within the framework of a basic rag model, inquiries posed to the document are utilized to identify pertinent paragraphs. this information retrieval typically follows a two-step process. first, embedding models are used to create a vector representation of the questions and all the paragraphs in the report. second, the question vector is compared to all paragraph vectors to obtain the top k most similar paragraphs. however, as previous research has shown, llms are prone to be confused when presented with wrong or contradictory sources @xcite @xcite , and the relevancy of the question to the sources plays a significant role @xcite . thus, the retrieval process is central to creating the true output. as previously outlined, climate change is a complex downstream domain with knowledge-intensive questions (see section 2 and appendix b). an expert labeler will likely consider additional concepts and definitions when searching for relevant information in reports. thus, only using the question in the embedding search process might limit the results to semantically similar paragraphs to the question, not to all concepts embedded in the expert annotator's mind. therefore, we construct an experiment that gradually replaces question (question) in the topk search process with longer and more expertinformed question explanations. to obtain question explanations, we use two setups. first, we use the definitions and concepts the labelers used during their annotation (see appendix b for an example). second, we make use of the capabilities of the closed-source llm gpt-4. we proceed in two steps. an explanation for the question of (1) 60 words (short) and (2) 150 words (long). we further ask the model to include and exclude the question (e.g., short_q / short_noq). these definitions serve as generic base cases (generic). in the second step, we gradually create more example-informed question explanations. in this artificial setup, we allow information leakage from the labeled relevant information to inform the explanation creation process. we use the relevant information with a label of 2 or higher as examples that should inspire the explanation (see appendix e for justification of the threshold). we create two settings: randomly choosing labeled relevant information from three reports (inf _3), and using all labeled relevant information (inf _all). for more details, see appendix h. finally, we employ simple evaluation metrics to compare the approaches. we define our first evaluation metric as the ratio of relevant sources found among all annotated sources. this equals the recall@k. thus, we try to optimize the number of relevant sources obtained by the information retrieval. furthermore, we use the ratio of all relevant sources found in the retrieved sources, which equals the precision@k in a classification task. this also allows us to calculate the weighted average, i.e., the f1 score. we calculate these scores at the top k values of 5, 10, and 15. furthermore, we run the information retrieval with a variety of embedding models: a random baseline, bm25, dragon+, gte-base, colbertv2, openai's text-embedding-ada-002, text-embedding3-small, and text-embedding3-large (see appendix i for details on the experimental setup). we analyze all setups individually as well as aggregate the scores over all embeddings or top-k values. our first step is to compare the questions in the retrieval process with the definitions and concepts written by the annotators. as this trend changes when using exampleinformed question explanations. as ## conclusion in this work, we develop a unique dataset that simulates an expert analyst workflow to evaluate rag systems. we show its utility by analyzing the dominant embedding retrieval strategy with different search setups. we find that embeddings face major limitations in information retrieval for knowledgeintensive tasks. therefore, this work sets the path for including and evaluating the improvement of expert-integrated information retrieval for rag systems (see @xcite for a potential solution approach).