Aligning Unstructured P aris Agreement Climate Plans with Sustainable Development Goals (2024)

Abstract: Aligning unstructured climate policy documents according to a particular classification taxonomy with little to no labeled examples is challenging and requires manual effort of climate policy researchers. In this work we examine whether large language models (LLMs) can act as an effective substitute or assist in the annotation process. Utilizing a large set of text spans from Paris Agreement Nationally Determined Contributions (NDCs) linked to United Nations Sustainable Development Goals (SDGs) and targets contained in the Climate Watch dataset from the World Resources Institute in combination with our own annotated data, we validate our approaches and establish a benchmark for model performance evaluation on this task. With our evaluation benchmarking we quantify the effectiveness of using zero-shot or few-shot prompted LLMs to align these documents.

## introduction the 2015 paris agreement established 165 country specific nationally determined contributions (ndcs) specifying global commitments to sustainability and resilience. revised ndcs were released in 2021-2022. the ndcs set ambitious climate action targets but are presented in unstructured texts making any analysis or tracking of goals over time difficult. the united nations sustainable development goals (sdgs) provide a structured framework of 17 goals and 169 sub-targets aimed at promoting global well-being and sustainability. the sdgs serve as a hierarchical taxonomy. linking ndc text spans to sdg goals and targets can enhance the understanding of global sustainability targets and offers a clear way to track progress. previous work by climate watch at the world resources institute manually linked ndc text spans to sdg goals and targets @xcite but such an effort * these authors contributed equally to this work. prompting llms provides a relatively unsophisticated yet powerful way to leverage the models' capabilities. furthermore, many of today's most advanced llms are easily accessible through apis and web interfaces, making them well suited for a wide range of climate policy researchers. however, there are concerns that llms can "hallucinate" and may struggle with understanding context, nuance, and long-term dependencies in text, leading to less coherent or relevant outputs in complex tasks. a formal evaluation of the utility of llms for the task at hand is currently lacking. our contributions are as follows: 1) we conduct an empirical study of the performance of llms and cross-encoder architectures on the task of aligning ndcs to sdgs. 2) we introduce a benchmark for comparing our models, annotators, and the existing climate watch dataset. 3) we analyze specific methods to further boost performance on this task. finally, we will release the full ndc reports with their predicted sdg alignment as an artifact for the community to use, fostering transparency and ensuring the aims of international agreements are better understood, monitored, and ultimately realized. ## datasets in this section we will introduce the world resources institute's climate watch dataset that we used for our experiments and additional benchmarks we constructed @xcite . the climate-watch dataset includes sentences from ndcs submitted before 2021, each of which are labeled with goals and targets. some sentences are labeled with a single goal and target, and others may also be labeled with multiple goals or targets. some statistics on the dataset, example sentences and their labels, as well as our data pre-processing can be found in appendix section a.1. ## experiments in this section, we introduce our experiments in which we use different prompting strategies with gpt models to classify sentences according to sdg. we will use chatgpt-3.5 and gpt-4-turbo as our main models to conduct prompt-based classification experiments. we will use json-mode api option to ensure the model outputs are properly structured for classification tasks. as our zeroshot classification baselines we will use minicdp, the cross-encoder model finetuned on the semistructured carbon disclosure project (cdp) questionnaire data from spokoyny et al. (2023) as well as its base model architecture minilm model. ## conclusion we have constructed benchmarks to compare the performance of models, annotators, using the climate-watch dataset on unstructured ndc documents. using this data we find that existing manual efforts provide low coverage, motivating the need for automated methods. finally, we found across various experiments that by prompting gpt models we could match the performance of our annotators on sdg-goal and sdg-target prediction. our findings highlight the potential of leveraging gptbased models to effectively annotate unstructured climate documents such as the ndcs. property mean sentence length (characters) 137.2 labelled sentences per document 66.4 goals per sentence 1.34 targets per sentence 1.49 additionally, these snippets are not directly linked to the exact locations in the ndc documents. we obtain a dataset of the full texts of the ndc documents as html files and using simple heuristics were able to match 94.8% of the annotations to their exact document spans. in appendix figure 3 we plot the distribution of where in the ndc documents the climate-watch annotations are found.