Climate Policy Transformer: Utilizing NLP to track the coherence of Climate Policy Documents in the Context of the P aris Agreement (2024)

Abstract: Climate policy implementation is pivotal inglobal efforts to mitigate and adapt to climatechange. In this context, this paper explores theuse of Natural Language Processing (NLP) as atool for policy advisors to efficiently track andassess climate policy and strategies, such asNationally Determined Contributions (NDCs).These documents are essential for monitoringcoherence with the Paris Agreement, yet theiranalysis traditionally demands significant la-bor and time. We demonstrate how to leverageNLP on existing climate policy databases totransform this process by structuring informa-tion extracted from these otherwise unstruc-tured policy documents and opening avenuesfor a more in-depth analysis of national and re-gional policies. Central to our approach is thecreation of a machine-learning (ML) dataset’CPo-CD’, based on data provided by the Inter-national Climate Initiative (IKI) and ClimateWatch (CW). The CPo-CD dataset is utilizedto fine-tune Transformer Models on classify-ing climate targets, actions, policies, and plans,along with their sector, mitigation-adaptation,and greenhouse gas (GHG) components. Wepublish our model and dataset on a HuggingFace repository.

## introduction the 2015 un climate change conference in paris produced a landmark agreement whereby all signatories agreed to hold "the increase in the global average temperature to well below 2 °c above preindustrial levels" (unfccc, 2016). the means for effecting this change are left to the countries, but each signatory is required to report progress every 5 years via nationally determined contributions (ndcs). signatories are also encouraged to periodically communicate long-term strategies (lts) to address climate change. the recent cop28 meeting in the uae was the first global stocktake (gst) making use of this reporting (unfccc, 2023). as the most frequent mandated reporting mechanism under the paris agreement, ndcs provide a consistent basis for tracking each country's progress and commitments. consequently, analysts utilize these reports to gauge global efforts towards climate goals. additionally, the agreement permits countries to revise their ndcs at any time @xcite . therefore, frequent review of these documents is important for holding signatories to account. the paris agreement and follow-up cops prescribed no standardized reporting framework. as a result, there is significant variation in the scope, format, and coverage of ndcs and ltss over jurisdiction and reporting periods @xcite . this variation is evident in numerous aspects of the reports, such as the articulation of mitigation contributions and the incorporation of adaptation strategies. additionally, the documents are often not intuitively structured, and in some cases voluminous. combined, these factors pose substantial challenges to aggregating and analyzing the data, thereby complicating the assessment of global and national efforts in addressing climate change. natural language processing (nlp) techniques based on deep learning have become increasingly viable for producing high-quality automated analyses in recent years, particularly with the advent of the transformer -and bert models built upon its architecture @xcite . the accessibility of pre-trained large language models via huggingface has further lowered barriers to entry, allowing easy fine-tuning on various nlp tasks, as well as model deployment. these tools add value to analytical workflows by providing analysts with the ability to extract knowledge from unstructured data to a much higher level than was previously possible. in this work, we seek to address the challenges with climate policy document analysis by applying sequence classification to the unstructured text in ndcs and ltss. our contribution consists of three main components. we first build and publish a training dataset 'cpo-cd' (climate policy classification dataset) derived from an agglomeration of two existing datasets: 1) the ndc transport tracker from the advancing transport climate strategies project of the international climate initiative (iki tracs) foot_0 ; and 2) the ndc sector data from climatewatch.org foot_1 . we then fine-tune 2 llms on the dataset to classify text according to binary, multi-class and multi-label domain categories aligning with the unfccc hierarchical taxonomy: targets, actions, policies and plans, mitigation / adaptation, sectors, target types and conditionality (see page @xcite for a breakdown of the taxonomy). finally, we publish cpo-cd and the fine-tuned models making them accessible via a web application foot_2 on huggingface -allowing analysts to upload and derive ad hoc insights from climate policy documents. ## related work the use of natural language processing (nlp) in document analysis has gained significant momentum in recent years, marking a transformative shift from cumbersome manual methods of knowledge discovery using unstructured text data. in an earlier paper, @xcite succinctly points out the benefit in the domain of policy analysis, where nlp techniques at the time had leveled the playing field, providing independent researchers and smaller teams of analysts the ability to perform "systematic analysis of large-scale text collections without massive funding support". encoder-based masked-language models trained on large text corpora have demonstrated high performance on downstream nlp tasks such as classification since @xcite introduced the bidirectional encoder representations from transformers (bert). subsequent variations have improved on the original architecture. with roberta, @xcite improved performance, resulting in higher performance across multiple nlp tasks. these base models are trained on a generalized task of next-word prediction and can be fine-tuned for a domain-specific context and downstream tasks such as the classification of targets, actions, policies, and plans. specific challenges to nlp tasks in the domain of policy analysis are inherent to its lexical properties -i.e. technical and domain-specific jargon. in domains with similar highly-specialized lexicons, applications have involved adapted approaches @xcite @xcite . concerning nlp application in the climate domain, the literature is surprisingly sparse. recent work by gonzalez et al. (2023) provides tangible evidence of this, finding that in over 76k acl anthology foot_3 nlp papers, "hardly [any papers] address other important goals such as poverty and climate", with only 50 climate-relevant nlp papers in the entire corpus since 1980 (cf. 2753 for health). meanwhile sietsma et al. (2024) noted 54 papers in the literature that either used or substantially discussed the use of nlp for climate adaptation specifically. however some recent efforts are quite prominent. concerning technical approaches, the field is quite active, with approaches using some combination of encoder/decoder architectures and pre-trained models primarily. peña et al. (2023) present a system for multi-class classification of policy documents using roberta coupled with an svm classifier. their results demonstrate that the combination with svm classifiers can achieve high accuracy (over 85%) over 30 classes, even in under-represented categories. other recent work in the climate domain involves domain-adaptive pre-training with roberta on climate-relevant corpora, before fine-tuning on downstream tasks including text classification to create climatebert @xcite . training on a dataset of climate-related literature (i.e. news reports, news, corporate esg disclosures, and scientific abstracts) resulted in climatebert outperforming a base-distilroberta model on cross-entropy loss and f1 @xcite . building on climate-bert, schimanski et al. (2023a) recently released climatebert-netzero which fine-tunes climate-bert to classify net zero and emissions reduction targets in corporate communications using a dataset of 3.5k expert-annotated text samples. classification using climatebert-netzero resulted in marginally better performance than larger bert base models. juhasz et al. (2024) showcases an approach for extracting mentions of net zero and other targets from national laws and policies. building on climatebert and manually annotated data they fine-tune a classification model. our work closely aligns with schimanski et al. (2023a) and juhasz et al. (2024) while leveraging existing policy databases to create a comprehensive training dataset and an array of classifiers corresponding to multiple unfccc mitigation contribution types. ## data the creation of the training dataset 'cpo-cd' was the most extensive task in this project and is the main contribution alongside the models. for this reason, we describe the creation in detail below. cpo-cd is comprised of labeled text passages extracted from policy documents (ndcs and lts) with accompanying labels. the data is sourced originally from 2 climate policy datasets: climate-watch ndc sector data foot_4 (cw) and iki tracs climate strategies for transport tracker foot_5 (iki). both datasets include text extracts from ndc/lts documents labeled by human annotators (domain experts), as well as the accompanying climate category labels in the form of metadata. however, the labeled text from both sources is not natively useful for text classification. the length of the labeled text differs from 2 up to 250 words. while some text passages are very focused and limited to short phrases with no peripheral context and often missing information relevant to determining all climate category labels; other contain more than one item of interest but are only annotated for one. additionally, the labeled text passages are often condensed, summarized versions of the original text and can appear multiple times in the document in varying contexts. therefore, we identify and retrieve the original source text from the policy documents. a sample of a short text observation is taken from the indian ndc "75 gw by 2022" which comes with additional meta-information: sectoral mitigation policy and energy sector. from the original ndc document, we extend the text so the metadata relevant context is included: "green generation for clean energy secure india: more than 5 times increase in renewable capacity from 35 gw (up to march 2015) to 175 gw by 2022. national solar mission scaled up five-fold from 20 gw to 100 gw by 2022. kochi airport is the world's first airport to fully run on solar power." this paragraph now includes a second sectoral policy information, the upscaling of the national solar mission. this missing metadata will be later added by grouping by paragraph (step 5). the whole process is described as follows: step 1: text processing: the required information is distributed across different files for both datasets. in the first step, we link the text passages with the metadata labels. for cw we utilize the sector file from the 'ndc content' dataset, taking the text passages and associated sector labels. we then merge with fields from the metadata file, which allows us to add additional labels (e.g. target, actions, plans, etc.) in subsequent steps. in some cases, the text has conjoined sequences between a separator character. in such cases, each unique sequence is broken out into its own sample. the iki data is structured much more simplyconsisting of tables for categories such as target, netzero, mitigation, and adaptation. in this case, we join all tables together and retain the table name as the label. we next combine both the cw and iki data. a basic cleaning process is applied to the dataset, involving the removal of duplicates and erroneous samples. during step 1, we also produce text length statistics for each country represented in the dataset. this object is used to calibrate the split strategy in step 2 for the source text. step 2: document processing: the text extractions from both cw and iki are narrowly focused and require expansion using the source text to make them usable for text classification. we collect the original ndc documents from the cw-associated wri repository repo 7 in html format. for further documents from the iki dataset, we source the original pdf versions of the documents from the unfccc website 8 using the document names provided in the iki dataset. after downloading, the iki pdf files are processed into raw text. both sets of source documents are then chunked into 60, 85, and 150-word sequences which respect sentence boundaries and include an overlap to ensure the labeled text passages from the datasets are fully covered. the inclusion of multiple sequence lengths allows for greater versatility in downstream nlp tasks (in this case, text classification). the arbi-trary choice of the sequence lengths reflects our informed estimate of the lower and higher limits of utility, based on knowledge of the dataset. step 3: secondary label processing and harmonization: various metadata accompanies the text passages for both cw and iki that can be used to apply further labels to the text. curating and harmonizing these metadata so that they can serve as useful labels is complex as both source datasets utilize slightly different methodologies. in the cw dataset, we take the broad "overview-category" to define adaptation and mitigation related text. we further use a subcategory "question-text" to define text relating to policies, targets, actions and plans (tapp), as well as conditional and unconditional commitments. a full mapping of cw questiontext subcategories to tapp is available on the cw website 9 . the structure of the iki dataset is less extensive and is processed to include labels using the associated tab from the original excel file. this includes 3 categories: target, adaptation-mitigation, and netzero. an additional 2 categories are defined from the "parameter" subcategories within the target spreadsheet: ghg and conditionality. the iki data presents a specific sectoral focus (i.e. transport) and differing nomenclature compared to climate watch. iki also contains no (mitigation) actions, nor the daughter categories of policies and plans, as found in cw. therefore these labels are not represented in samples sourced from iki. ## methodology to address the challenges of efficiently analyzing voluminous and complex climate policy documents, we adopt two distinct but complementary nlp methodologies: fine-tuning a generic llm embedding for classification tasks @xcite , and further fine-tuning a pre-trained domain-specific llm, climatebert @xcite . in case of a sparsity of positive examples, we fine-tune using setfit @xcite . set-fit represents an efficient few-shot learning framework based on sentence transformers @xcite which has proven to achieve high accuracy with a minimal number of samples. its process involves first fine-tuning a sentence transformer embedding model on a set of labeled examples through contrastive learning. following this, a classification head, in our case logistic regression model, was trained on these embeddings to classify new unseen data. our choice of the 'baai/bge-base-en-v1.5' @xcite -a recent 109m parameter model provided by the beijing academy of artificial intelligence -as the foundation for the generic llm was based on its superior performance in classification tasks and ranking on the hugging face leaderboard. for comparison purposes, we made use of cli-matebert, a climate domain-specific adaptation of the distilroberta, 82.4m params @xcite transformer model. climatebert was pre-trained on a large corpus of climate-related texts, imbuing it with a nuanced understanding of climate discourse. this makes it particularly suitable for classifying texts based on climate policy content. our methodology involves further fine-tuning climatebert on cpo-cd, leveraging its domainspecific pre-training to enhance classification performance. this approach is validated by its demonstrated superiority in net-zero classification tasks over larger models, including gpt-3.5-turbo, as reported by recent studies @xcite . given the prevalence of imbalanced classes, we chose the f1 score as the primary metric to assess model performance. the f1 score, a harmonic mean of precision and recall, provides a more comprehensive measure of a model's accuracy, especially in scenarios where class distribution is skewed. to address the inherent class imbalances within our dataset, we employed stratified sampling in the train-test split. this approach ensures that the class proportions are mirrored in the test set. additionally, we disclose the count of test samples (support) which account for 10% of the data. furthermore, to overcome class imbalance in a multi-label setting, we have used positive class weights in the loss function. the paragraphs extracted from the ndcs and lts climate policies often cover several topics, such as different climate actions or targets. acknowledging this, we train our models in a multilabel setup, that can recognize multiple topics in one paragraph. this method is more complex than the simpler multi-class classification where one one label per paragraph is attached. the additional complexity usually results in lower performance scores. however, multi-label is a better match for this use case, ensuring we accurately capture the wide range of climate policy discussions within a single paragraph. ## conclusion in conclusion, this paper explores the application of natural language processing (nlp) techniques to enhance the analysis and classification of climate policy documents, with a focus on nationally determined contributions (ndcs) and long-term strategies (lts). we show how existing policy databases can be used to create a machine-learningready dataset (cpo-cd) and fine-tune pre-trained transformer models for policy analysis. we have developed a methodology that significantly streamlines the process of structuring information from these critical documents. the use of our models has been shown to markedly reduce the time required for policy analysis, enhance the effectiveness of policy examination, and enable the inclusion of a broader array of documents in the analytical process. our approach facilitates the efficient assessment of climate targets, actions, policies, and plans (tapp), along with their associated mitigation/adaptation, greenhouse gas (ghg), and sector components. by achieving noteworthy accuracy in tapp, ghg, adaptation/mitigation as well as useful accuracy in sector classification, our research underscores the potential of nlp to offer meaningful insights into the alignment of international climate commitments with the paris agreement's objectives and support evidence-based policymaking. the release of our dataset 'cpo-cd' and model contributions marks a significant step forward towards advancing the capacity to monitor and analyze international climate commitments at scale, enhancing transparency, accountability, and informed decision-making in climate policy evaluation.