Assessing the Effectiveness of GPT -4o in Climate Change Evidence Synthesis and Systematic Assessments: Preliminary Insights (2024)

Abstract: In this research short, we examine the potential of using GPT-4o, a state-of-the-art large language model (LLM) to undertake evidence synthesis and systematic assessment tasks. Traditional workflows for such tasks involve large groups of domain experts who manually review and synthesize vast amounts of literature. The exponential growth of scientific literature and recent advances in LLMs provide an opportunity to complementing these traditional workflows with new age tools. We assess the efficacy of GPT-4o to do these tasks on a sample from the dataset created by the Global Adaptation Mapping Initiative (GAMI) where we check the accuracy of climate change adaptation related feature extraction from the scientific literature across three levels of expertise. Our results indicate that while GPT-4o can achieve high accuracy in low-expertise tasks like geographic location identification, their performance in intermediate and high-expertise tasks, such as stakeholder identification and assessment of depth of the adaptation response, is less reliable. The findings motivate the need for designing assessment workflows that utilize the strengths of models like GPT-4o while also providing refinements to improve their performance on these tasks.

## introduction climate change is one of the most pressing challenges that several regions across the world have to face in the coming decades @xcite . adapting to climate change is essential for ensuring long-term sustainability @xcite . for decision-makers to effectively respond to this challenge, they must carefully plan their strategies based on well-documented and assessed climate adaptation evidence. this involves reviewing a vast array of scientific documents and case studies that detail adaptation efforts in different regions. the intergovernmental panel on climate change (ipcc) has formalized the assessment of this evidence through the publication of time-sensitive reports. these reports play a crucial role in informing international treaties and country-specific legislative actions. while traditionally such assessments relied on domain experts working voluntarily in teams tasked with annotating the documents for specific aspects of climate change, this is changing of late with the incorporation of machine learning in the evidence gathering and synthesis process @xcite . this is because the exponential growth of scientific literature over time has made the process of managing and synthesizing the evidence increasingly challenging @xcite . automating the annotation of large volumes of scientific data can save valuable researcher time and reduce the assessment cycle, allowing decision-makers to receive quicker and more up-to-date information. recent advancements in neural network methods have shown to be very useful in processing documents and extracting useful information from documents in open domain data such as wikipedia articles @xcite . however, scientific documents present unique challenges due to their complex domain-specific terminologies and concepts. addressing these challenges requires training models with high-quality, human-labeled data at a sufficient scale. recent advances in large language models (llms) have shown promise in overcoming these challenges. their diverse training across various topics has made them effective at extracting information from scientific documents and supporting researchers in enhancing time efficiency. although they perform well in extracting information from scientific documents in certain domains, they still struggle to fully understand the evidence presented in these articles @xcite . given the importance of accurate information ex-traction for decision-making, it is crucial to have a reliable model that ensures factual accuracy during the extraction phase. in this preliminary work, we explore the potential of using gpt-4o, a state-ofthe-art llm by openai, without explicit domain specific training, as extractors of useful information from documents related to climate adaptation. our contributions are as follows ## related work in recent years, llms have gained significant attention across the world due to their ability to complete tasks on which no explicit training was provided. the variety of applications being explored by this technology has spawned a new interest in deploying them across different domains such as medical science, business, education etc. several studies have investigated the use of llms as text annotators, primarily in open domain settings @xcite . however, there is limited research on evaluating their performance in applications requiring domain expertise, specifically information extraction from scientific articles @xcite . studies have experimented with incorporating llms into data annotation pipelines, particularly for annotating texts that require domain expertise. these efforts have shown potential to reduce the time and overall cost of annotation @xcite . for instance, llms have been used to extract information about nanorod structure procedures from scientific texts @xcite and to extract information from clinical trials @xcite . in the context of climate change research, recent work such as the creation of expert confidence in climate statements (climatex) dataset @xcite to weigh assessment tasks in a few-shot learning setting has shown limited accuracy. however, prior to the task of assessment is the collection and streamlining of evidence from the peer reviewed literature where for example, llms can be useful for named entity recognition (ner) @xcite from the vast corpus to help organize the evidence more efficiently for synthesis. other similar trajectories in the use of llms in climate change research involve fact-checking of climate change claims @xcite or the utilisation of trained domain-specific climate models such as to synthesize interdisciplinary research on climate change @xcite or the use of llm agents to extract information from a database to improve climate change related information analysis @xcite . ## results to evaluate the information extraction capabilities of gpt-4o in the context of climate change data from scientific publications, we compared the annotation agreement between human-created labels and the information extracted by gpt-4o. furthermore, this specificity indicates the potential of gpt-4o to enhance the granularity of extracted data in low-expertise tasks. intermediate expertise tasks of identifying stakeholders involved in adaptation responses, gpt-4o effectively captured the primary stakeholders but also extracted extraneous information and occasionally misclassified categories. the performance metrics for this level included a micro f1 score of 0.54 (macro: 0.30), precision of 0.40 (macro: 0.27), and recall of 0.83 (macro: 0.33). manual checks of the disagreements highlighted that gpt-4o sometimes misidentified and extracted stakeholders mentioned in introductory sections or other parts of the text, which were not relevant to the specific adaptation measures being discussed in the document (high recall). given the task requires the model to use a taxonomy to classify the stakeholders, the model's performance on this task suggests that improvements can be made by integrating prompting methods that elicit reasoning and verification capabilities. high expertise tasks for high expertise feature extraction, our evaluation reveals that the model, in some cases (10.4% of the time), provides individual assessments for each adaptation response rather than an aggregate assessment of the impact of a set of responses. this behavior complicated the evaluation process. to address this, we isolated these instances and focused our evaluation on cases where the model provided a depth evaluation for the aggregate of set of adaptation responses. we treated the evaluation as a multi-class classification problem. in this context, gpt-4o achieved an accuracy of 22.7% and a micro-averaged f1 score of 0.22 (macro f1: 0.17). closer examination of the instances of disagreement revealed that, in all the cases with minimal agreement, gpt-4o exhibited a more optimistic view compared to human annotators, often overestimating the impact of adaptation responses. this discrepancy is likely due to the generalized nature of the model's training and instruction tuning on a wide range of tasks. these findings highlight significant challenges in using gpt-4o for tasks that require a deep understanding of complex and nuanced information. ## limitations in this study, we tried to cover a diverse set of information extraction tasks in the context of climate change adaptation research to understand the feasi-bility of using gpt-4o, but it is in not an exhaustive list. additionally, our findings are specific to the climate change adaptation literature and in food sector, limiting their generalizability to other domains or sub-fields of climate change research. further studies are necessary to assess the applicability of gpt-4o across a broader range of climate change topics. we did not explore complex prompting techniques, such as chain of verification (dhuliawala et al., 2023), which could potentially enhance the accuracy and reliability of gpt-4o outputs. incorporating such advanced techniques in future research might address some of the challenges we encountered, such as misclassification and the extraction of irrelevant information. our evaluation was conducted exclusively on gpt-4o, and we did not test other llm models, which may perform differently. future research should include a comparison of multiple llms to determine if our findings are consistent across different models and architectures. ## conclusion from our study we find that there are opportunities and challenges for the deployment of pre-trained llms in the climate evidence synthesis and assessments. we assessed the efficiency of gpt-4o's role as an annotator and find that tasks requiring beyond low levels of expertise are challenging for gpt-4o. future work should explore using methods to integrate knowledge for medium expertise level and learning from human feedback to improve the model performance on extraction on information that requires high levels of expertise. further, models with such capacity when trained on task specific data, could play a complementary role in the task of adaptation tracking by governments and global agencies and eventually help in timely securing funding for necessary adaptation responses. however, it is important to emphasize that these models cannot completely replace the expert-driven process, rather a human-in-the loop system would be extremely beneficial for ensuring the integrity and effectiveness of this process.