Constructing a Credible Estimation for Overreporting of Climate Adaptation Funds in the Creditor Reporting System (2023)

Abstract: Development funds are essential to finance climate change adaptation and are thus an important part of international climate policy. How ever, the absence of a common reporting practice makes it difficult to assess the amount and distribution of such funds. Research has questioned the credibility of reported figures, indicating that adaptation financing is in fact lower than published figures suggest. Projects claiming a greater relevance to climate change adaptation than they target are referred to as “overreported”. To estimate realistic rates of overreporting in large data sets over times, we propose an approach based on state-of-the-art text classification. To date, assessments of credibility have relied on small, manually evaluated samples. We use such a sample data set to train a classifier with an accuracy of 89.81%±0.83% (tenfold cross-validation) and extrapolate to larger data sets to identify overreporting. Additionally, we propose a method that incorporates evidence of smaller, higher-quality data to correct predicted rates using Bayes’ theorem. This enables a comparison of different annotation schemes to estimate the degree of overreporting in climate change adaptation. Our results support findings that indicate extensive overreporting of 32.03% with a credible interval of [19.81%; 48.34%].

## introduction the climate crisis is one of the greatest challenges of our time. climate change is accelerating toward a catastrophe that will almost certainly become a humanitarian crisis as well. according to a united nations committee, the next decade counts: the latest ipcc reports make it clear that limiting global warming to a relatively safe level still is possible, but it requires global cooperation and billions in financial support @xcite . failure of the global community to respond to the climate crisis will result in millions of people having to live with the consequences of extreme heat, food and water shortages, as well as the proliferation of pathogens, all of which add to the humanitarian crisis. therefore, ensuring that populations are able to secure their livelihoods necessitates focus on both adaptation to the impacts and mitigation of the effects of climate change. implementation of climate change adaptation measures is one of five targets set to reach the 13th sustainable development goal (sdg): "take urgent action to combat climate change and its impacts". there is international consensus on the need to respond to the global threat posed by climate change (paris accord, article 2). development funds are essential to finance climate change adaptation and are thus an important part of international climate policy. specifically, article 9.1 of the paris agreement states that "[d]eveloped country parties shall provide financial resources to assist developing country parties with respect to both mitigation and adaptation in continuation of their existing obligations under the convention." 1 . prior to that, the 2009 copenhagen accord had already declared a goal of mobilizing usd 100 billion by 2020. based on these agreements, the conference of the parties (cop) meets annually to assess implementation efforts and to ensure effective implementation of the conventions. one basis for this activity is the creditor reporting system (crs), which is administered by the oecd development assistance committee (dac). it monitors financial flows for adaptation and climate change mitigation activities that flow from oecd dac member countries to developing countries. as such, it is the central system for monitoring and evaluating the efforts of the international community to address climate change. to date, this dataset includes more than 1.5m aid activities. one of the challenges in ensuring valid reporting -or at least comparable figures -across reporting countries is that these agreements lack standardized indicators. these tasks and this area of activity of international politics can be classified as a field of research in international affairs, which is increasingly being researched using methods of the broader field of computational social science @xcite . to this end, in 2009 the oecd dac established the "rio markers" for climate change adaptation (cca) and mitigation (ccm). for each aid activity, donors self-report whether it contributes to cca, i.e. reducing "the vulnerability of human or natural systems to the current and expected impacts of climate change, including climate variability, by maintaining or increasing resilience, through increased ability to adapt to, or absorb, climate change stresses, shocks and variability and/or by helping reduce exposure to them" (oecd dac, 2022, 4). activities are eligible for a marker if "a) the climate change adaptation objective is explicitly indicated in the activity documentation; and b) the activity contains specific measures targeting the definition above." (oecd dac, 2022, 4). the rio marker r can take three values: 2, if cca is the principal objective; 1, if cca is a significant objective; and 0, if cca is neither a principal nor a significant objective. however, there is increasing evidence that the level of adaptation financing is in fact lower than public figures suggest @xcite . the authors refer to this phenomenon as overreporting, which primarily indicates a discrepancy between the qualitative descriptions of the financing purposes and the specified rio markers. one possible reason is that there is no common practice for reporting climate finance @xcite and reporting agencies thus follow different reporting rules. this makes it difficult to assess the total amount of cca or ccm finance, to compare commitments between donors, and to assess the geographical and sectoral distribution of funding @xcite .moreover, cca finance estimates vary among reporting agencies @xcite . hence, aggregate figures of adaptation finance are increasingly considered unreliable given that they comprise thousands of individual aid activity descriptions from the crs data. consequently, assessments of credibility have to date relied on analysis of small samples covering a limited period of time @xcite . returning to the fact that the cop is utilizing these reports, among other things, as source of information the unreliable nature of these reports jeopardizes the well-being of the people who are affected by climate change. regardless of whether this discrepancy is result of deliberate misreporting or due to the inconsistent nature of the reporting procedure, this system, with all its strengths and weaknesses, is part of global communication about the environment and environmental issues. this study applies state-of-the-art machine learning methods for natural language processing to estimate overreporting of cca finance for all aid activities as reported in the oecd dac crs since the introduction of rio markers. we model the information and knowledge contained in the reports using nlp methods to aid the requirement that the assessment of the reports is consistent, thorough and complete. this contributes to effective and targeted measures based on realistic assessments so that the necessary assistance can be provided to protect people's livelihoods and social structures. our main challenge in applying machine learning methodology is the quality and quantity of available annotated data. we have access to two data sets re-evaluated by experts and published in previous work: the first is small, but following a thorough re-evaluation process, we regard it as high-quality. one concern is that the current de facto standard of fine-tuning language models tends to be unstable with very small data sets and is hard to evaluate properly. the second set is much larger, but because it was re-evaluated with access to less information, we regard it as lower quality; nevertheless, its size makes it adequate for training. we propose to combine these two data sets, using the larger for training and extrapolation, and the smaller, higher-quality set to correct first estimates. the contribution of this paper is two-fold: 1. we propose and evaluate a machine learning model to detect overreporting in the crs data and discuss extrapolation. 2. we propose and attempt to use a bayesian framework for correction of extrapolated overreporting rates. @xcite . these studies are distinguished, first, by the rigor of their methodology to assess overreporting and, second, by the number of aid activities they analyze. some studies classify multiple aid activities but employ, rather simplistically, keyword searches only on short descriptions of aid activity @xcite @xcite . other studies examine only a few aid activities by scrutinizing extant project documentation against in-country expert assessments @xcite . @xcite strike a balance by manually assessing a large number of short project descriptions. we are among the first to apply state-of-the-art machine learning -which allows us to code all aid activities reported in the oecd dac crs database -to fully automate the process of detecting overreporting of cca finance. moreover, our method can be easily applied to future data releases of the oecd dac crs data as well as comparable text data. machine learning approaches to classify official development assistance are still rare. @xcite used machine learning to classify sdgs. more recently, toetzke et al. (2022) developed a machine-learning classifier to identify climate finance based on the title and descriptions of bilateral aid activities in the oecd dac crs dataset. cli-matefinancebert first classifies the relevance of aid activities to adaptation, mitigation, or the environment. subsequently, relevant activities are further differentiated into ten categories. in contrast, our classifier directly predicts rio markers for climate change adaptation. moreover, we address possible shortcomings due to the limited information contained in the oecd dac crs descriptions by integrating evidence from a high-quality re-evaluation. here, we rely on textual resources to automatically assign rio markers to crs reports. project reports typically contain both short and long descriptions of the project goals, ranging from one to a few sentences. currently, neural networks produce virtually every state-of-the-art result in text classification, either by training task-specific architectures, e.g. @xcite or adapting pre-trained language models to a given task @xcite @xcite @xcite . also, recent works have achieved both higher overall performance @xcite and greater sample efficiency, achieving better results with less data @xcite . the typical problems of previous classical machine learning approaches in text classification, like out-of-vocabulary or ambiguity, are directly handled by the language model, which is especially important in this case, because not all of the texts we deal with are free of orthographic and syntactical anomalies. we experiment with these models in various combinations to find the best fit for the task at hand. ## discussion our re-evaluation of aid activities reported as contributing to cca indicates a lack of quality in the self-reporting of donors. a substantial share of reported adaptation aid activities does not explicitly mention cca in project descriptions. this is problematic because valid indicators are required to assess whether the international community is meeting its climate policy obligations as described in, for example the paris accord. our finding indicates an overestimation of adaptation aid. even after downward adjustment of our estimates to account for insufficient information from short project descriptions, our best estimate suggests that about every third activity categorized by donors as adaptation aid is not adaptation related. however, we cannot say whether this is due to a lack of clear reporting standards @xcite , a lack of compliance with reporting standards, or even incentives to report more than is actually delivered @xcite . moreover, our estimates are subject to considerable uncertainty because an unambiguous classification of aid activities based on the rio marker methodology requires extensive knowledge of individual aid activities. although the estimates are somewhat uncertain, our results confirm earlier findings of a substantial discrepancy between the figures reported by donors and re-evaluations by independent re-searchers @xcite @xcite . this can be partly explained by the fact that our approach draws on earlier classifications as training data and is thus not completely independent. however, our study also goes well beyond existing research in temporal and geographical scope: we assess every adaptation aid activity reported by oecd dac donors since the rio marker on adaptation aid was introduced in 2010. if donors had changed their reporting practice, we would likely see this in our data, yet, overall, we find no indication that reporting practices have changed significantly since 2010. the share of overreporting remains at a high level between 2010 and 2019. although the classifier indicates a slight decrease of overreporting, the fluctuations are within the range of uncertainty of our adjusted estimates. nevertheless, the results are mainly driven by a significant decrease in overreporting by the united kingdom. however, we are careful to infer from our data substantial differences in reporting standards between countries; country-specific results should be examined more closely in future research. we have no reason to suspect that the classifier has particular problems with data reported by the uk based on the results of the cross-validation and given the fact that most project descriptions are in english. ## conclusion in this paper we propose an automated way of detecting overreporting of climate adaptation finance based on crs project descriptions. our approach is based on state-of-the-art text classification using finetuning neural language models. we consider the quality of the annotations of our training data when estimating overall overreporting rates, and propose a bayesian approach to estimate an extrapolation of high-quality annotations. our approach indicates significant overreporting throughout the study period. there are two key challenges with this approach: the quality and quantity of annotation scheme data, and the quality of the textual input. while, ultimately, the first challenge can be overcome by extending data with higher-quality annotations, the second proves trickier. unfortunately, the crs data does not have uniform quality built into its textual descriptions. we have discussed the influence of description length and language on the quality of our 106 classifier, but while future work may incorporate techniques from short-text classification research, in many cases the information the crs contains will likely not suffice to use techniques like augmentation or conceptualization. to improve on this, additional external data would be necessary, to which, at this point, we had no access. especially deciding if the cca aspects of a project comprises a "significant" or "principal" object should benefit from this. another way to improve would be to pay more attention to multi lingual classification research and either incorporate techniques for multi lingual text classification or utilize a high-quality pipeline for translation.