SDG target detection in environmental reports using Retrieval-augmented Generation with LLM s (2024)

Abstract: With the consolidation of Large Language Models (LLM) as a dominant component in approaches for multiple linguistic tasks, the interest in these technologies has greatly increased within a variety of areas and domains. A particular scenario of information needs where to exploit these approaches is climate-aware NLP. Paradigmatically, the vast manual labour of inspecting long, heterogeneous documents to find environment-relevant expressions and claims suits well within a recently established Retrieval-augmented Generation (RAG) framework. In this paper, we tackle two dual problems within environment analysis dealing with the common goal of detecting a Sustainable Developmental Goal (SDG) target being addressed in a textual passage of an environmental assessment report.We develop relevant test collections, and propose and evaluate a series of methods within the general RAG pipeline, in order to assess the current capabilities of LLMs for the tasks of SDG target evidence identification and SDG target detection.

## introduction a series of sustainable development goals (sdgs) were established by experts in the united nations, as a reference framework with respect to which guide the progress of human activities, altogether oriented to the common good @xcite . according to their respective legal requirements, practitioners in the area of environmental assessment (e.g., professional assessors, developers, authorities) have to incorporate this framework in multiple spheres. in particular, the activities, impacts and mitigation measures described in environmental assessment documents are increasingly required to report how they address one or more sdgs; especially sdg targets, these being focused, actionable subgoals within a given sdg. identifying textual passages relevant in addressing an sdg target of interest then becomes a fundamental problem in the practice of environmental assessment. this problem naturally lends itself within an approach based on obtaining an initial selection of passage candidates, and then apply onto this a more advanced detection technique to refine the decisions about relevance to the target. the recently established retrieval-augmented generation (rag) framework @xcite embraces this basic approach and couples it with a text generation component powered by large language models (llms), the dominant technology in nlp that shows state-of-the-art performance in multiple tasks @xcite . figure 1 depicts an overview of the two tasks that we address in this work, with examples of sdg targets and excerpts of environmental impact assessment (eia) reports. specifically, these dual tasks couple with each other in the need of practitioners in eia for finding instances of correspondence between the information spaces of sdg targets and textual evidence in specialized reports. we contribute by developing a test collection for each of these two tasks. we also propose and evaluate a series of experimental configurations for each of the rag components, in order to assess the current capabilities of llms for these paradigmatic tasks in climate-aware nlp. our test collection and related resources developed in this study are made publicly available in a repository at https://bit.ly/ recently, the area of climate-aware natural language processing has decisively emerged led by a broad interest of developing information access methods to strengthen awareness in phenomena within climate change, as well as to process climate-related data in specific tasks within this domain. for example, @xcite introduce a climate performance model card to summarize the impact of the experimentation corresponding to a scientific work in nlp research. some of the related literature focuses on designing and analyzing methods for extracting climatecentric information, for example, to answer questionnaires @xcite and to detect climate-relevant claims in documents @xcite . works like the ones by @xcite and @xcite reveal the capabilities of well-established language models in communicating around climate awareness. other lines of research in this area have produced fundamental resources, including language models such as climatebert @xcite and climategpt @xcite , as well as systems like chatclimate @xcite and resources like eia-centric ontologies @xcite , to power approaches for a variety of tasks. an ever-increasing dominant technology in nlp, llms store vast amounts of information implicitly in their of billions of parameters trained on large general-purpose corpora, which allows them to perform as state of the art in many tasks such as text classification, textual entailment and question answering (qa) @xcite . yet, for many domain-specific scenarios, a framework like retrieval-augmented generation (rag) @xcite becomes convenient as it allows to extend the llm capabilities with additional explicit knowledge as context where to get the generated answer from. moreover, against frequent hallucinations, it useful to be able to verify that the claims that occur in a text generated by such a model are truthful @xcite . within the research on qa, a foundational task for any application of llms, several related problems such as attribution in question answering @xcite , evidentiality-guided generation @xcite , verifiability of gen-eration @xcite , and factuality in summarization @xcite aim for operationalizing such a verification. in the tasks we address, we treat the input (an sdg target in ei or an textual excerpt in td) as a question for which, after augmented with retrieved items, an llm must generate an answer with the correct outputs (identifiers of report passages or sdg targets, respectively) among the ones provided in the prompt. ## results and analysis throughout this section, tables 3 and 4 present the results for all the configurations in our experimentation. (the corresponding output files with the full rag results for all methods are made publicly available in our repository. foot_3 ) ## conclusions, limitations, and future work in this work, we study two dual problems on environmental analysis as a mean to approach towards the automatization of knowledge-intensive, timeconsuming tasks in the practice of assessing environmental impact in reports and its correspondence with the recent developments around sdgs. specifically, we propose and assess several methods within the rag framework powered by llms. our work approaches a paradigmatic scenario of environmental analysis, yet it is still limited in its capabilities to identify evidence and detect targets. on the one hand, the selected targets cover a meaningful part of the sdgs scope in regards to eia, yet there are more sdgs and targets that could be considered. on the other hand, the collection of reports where the eia passages come from suits well as information source for our experimentation, yet it is centered on particular regions of europe and so our study fails to capture phenomena about other environments and their corresponding sdg targets of relevance. furthermore, our data annotation is conducted with caution and good faith but it could present cases where the judgement could be different, especially as the concept of "addressing an sdg (target)" is already not exact in the literature and the eia practices described in the reports often take advantage of these uncertainties. in future work, we plan to further study the duality of these two tasks by approaching environmental analysis with a method were each task retrofits the other one. in this way, for example, a textual passage identified via ei for an sdg target can be the input of a subsequent td stage to possibly expand the space of targets of interest for that eia report, as well as exploiting relations between passages in the same report. another line of research is experimenting with the usage of a claim detector, this is, a dedicated model for identifying climate-aware claims in text, such as the one developed by @xcite . this component could complement the retrieval stage to improve the selection of passages that are finally fed into the llm during generation. a third possible area of work corresponds to automatically labeling larger volumes of test instances with an llm as assessor, which could extend the evaluation space, as well as allow for experimenting with fine-tuning a base pre-trained model with these instances. in a similar fashion, a fourth direction would investigate the automatic assessment, also via llm, of correctness for a predicted result. such an assessment would be validated by observing the inter-annotator agreement with manual assessments in a sample of the test collection.