Using LLM s to Build a Database of Climate Extreme Impacts (2024)

Abstract: To better understand how extreme climate events impact society, we need to increase the availability of accurate and comprehensive information about these impacts. We propose a method for building large-scale databases of climate extreme impacts from online textual sources, using LLMs for information extraction in combination with more traditional NLP techniques to improve accuracy and consistency. We evaluate the method against a small benchmark database created by human experts and find that extraction accuracy varies for different types of information. We compare three different LLMs and find that, while the commercial GPT-4 model gives the best performance overall, the open-source models Mistral and Mixtral are competitive for some types of information.

## introduction increasingly frequent and intense extreme climate events pose significant threats globally at both individual and collective levels. however, we still do not have a robust understanding of how extreme climate events impact society, which in turn hinders impact forecasting, early warning, and disaster risk management ( @xcite . accurate impact information is crucial for identifying areas disproportionately affected @xcite , enabling targeted allocation of climate adaptation efforts. such data can also provide support for the evaluation of whether adaptation measures effectively reduce loss and damage from climate extremes @xcite . existing publicly accessible global climate impact databases suffer from incomplete, inconsistent and/or biased data @xcite . one of the most used natural hazards-related impact databases is em-dat @xcite . 2 while em-dat is an extremely valuable database, events are often assigned non-standardized spatial information: from city to country scales, or geophysical areas without clear formal boundaries. similarly, temporal specifications may be a date range in days, in months or only a year. the impacts from a single physical event may further be listed under multiple separate entries if affecting an extended area. moreover, events in both developed and developing countries are likely underreported @xcite . many climate extremes also lack impact information in one or multiple categories @xcite . some of these constraints are also shared by other multi-hazard, multiimpact databases, such as desinventar (unisdr, n.d.). single-hazards databases (e.g., @xcite and/or databases focusing on national spatial scales @xcite have better coverage and completeness, yet they typically cannot be easily updated or scaled to multiple hazards or regions. moreover, they adopt differing impact categories and event definitions, preventing any multi-hazard impact analyses. in this paper, we propose a method for constructing a database of climate extreme impacts from online textual sources, using natural language processing (nlp). this has the potential to address the above-mentioned database limitations, ensuring broad spatiotemporal coverage, standardisation of information and easy updating. our approach leverages the power of large language models (llms) and in-context learning to extract semi-structured information, which is normalized and refined in post-processing and stored in a relational database. a crucial step in the refinement process is geoparsing, which maps place names to geographical entities in order to enhance the usefulness of the database for researchers. another important feature of the database is that we store the actual text from which the information has been extracted, allowing users to trace sources and validate the information. an empirical evaluation based on a benchmark database created by human experts for 170 extreme climate events shows that extraction accuracy varies for different types of information. while the main event category (such as "flood" or "wildfire") and the number of people killed can usually be determined with high accuracy, geographic locations and total economic damage are harder to extract reliably. a comparison of three different llms shows that the commercial gpt-4 model gives superior performance overall, but the opensource models mistral and mixtral give competitive results for some information categories. ## database design the first step towards an information extraction system for climate extreme impacts is the design of a database schema, which defines what type of information should be extracted and how this information should be formally represented. an important consideration here is compatibility with existing de facto standards in the field, and we have therefore chosen to base our categories mainly on the existing em-dat database @xcite , while trying to overcome some of its limitations. figure 1 gives a schematic overview of the kind of schema used in our system. the fundamental entity is an event, which is a climate-related extreme such as a storm or a heatwave. each such event must be associated with information about its location, time and event category. this is a basic requirement, because information about impacts that cannot be located in space and time is of no use to scientists, but in order for an event to be included in the database, there must also be some information about its impacts. by impacts we understand the socio-economic impacts of climate extremes, that is, the negative repercussions of such events on society ( @xcite . as shown in figure 1 , we subdivide these into (a) direct impacts to persons, such as the number of fatalities and of persons being injured, displaced or homeless, and (b) material and economic damage, such as insured and total economic damage, and building damage. the specific impact types are chosen to ensure compatibility with existing impact databases, in particular, em-dat: location a location is specified across multiple fields encoding different levels of information and as standardized as possible. these fields are: these geometric shapes are pulled directly from openstreetmap and enable users to visualize impact locations on a world map. time the time of an event is specified by a start date and an end date, which are the same if the event took place within a single day. (we do not consider shorter time periods than one day.) the dates are specified in yyyy-mm-dd format, where the year is strictly required, while the month and day fields are nullable in case the information is not available. formally, this is represented by a tuple @xmath0, smon, sday, eyear , emon, eday⟩, where syear and eyear are 4-digit integers, while smon, sday, emon, and eday are 2-digit integers or null. ## information extraction our method for populating a database of climate extreme impacts based on information extraction from online textual sources uses a pipeline consisting of three main components, as illustrated in figure 2 . the first component performs document selection using web scraping with keyword filtering and an llm-based text classifier. the second component uses llm prompting to extract information about extreme climate events and their impacts, storing the result in a semi-structured format. the third component post-processes the semi-structured information by converting all information items to their correct data type, normalizing all text elements, performing various consistency checks, and mapping location names to geographical entities, before storing the result in a relational database. below we describe each of the three components in more depth. ## evaluation we evaluate our method using development and test data annotated by domain experts in our team. the experimental evaluation involves a comparison of three different models in the second step of @xcite . the same prompts are used for all models (cf. appendix b), except for an additional final sentence to ensure responses are strictly in json format for the mistral models, to overcome their tendency to produce additional comments. below, we first describe the data annotation and define the evaluation metrics before reporting and discussing our experimental results. ## related work the notion of using nlp for extracting impact information from textual data is rapidly gaining traction in the fields of climate and impact science. while no previous work has attempted to build a global multi-hazard database, such as the one that we are presenting here, there have been a number of implementations of nlp approaches in more targeted contexts. for instance, @xcite extract and classify impact statements in newspaper articles for the 2018/19 german drought. this line of work is continued by @xcite and @xcite , who use supervised classification models to extract information from newspaper articles on the different socio-economic impacts of droughts in germany. nlp approaches have also been applied to social media, for example by @xcite , who use a bert model to identify mentions of seven different types of drought impacts in twitter data originating in california, united states. other authors have used automated processing of textual data to provide a broader categorisation of climate extremes going beyond categorical impacts, notably @xcite , who map the course, consequences, and aftermaths of the 2021 european floods. finally, as a direct precursor of the information extraction approach presented in this paper, we mention li (2023), who focuses on wikipedia articles and urls to extract impacts of multiple classes of climate extremes, achieving 86% accuracy for time and 92% for location with gpt-3.5, surpassing the performance of a bert model. ## discussion and conclusion we have presented the first evaluation of an llmbased system for building a database of climate extreme impacts. the results show that this is a challenging task, especially for certain types of information, and that llms still need to be supported by more traditional nlp techniques to ensure correct data typing and consistency. our comparison of different llms indicates that open-source models match the performance of gpt-4 on specific information types (in particular the main event category), and it is likely that the results can be improved further through model-specific prompt engineering and better pre-and post-processing. even discounting inaccuracies introduced by the llms, the quality of the database depends on the correctness of the data presented in the wikipedia and artemis articles. the issue of potentially incorrect or incomplete impact data is shared with other current state-of-the-art global impact datasets (e.g. desiinventar and em-dat; @xcite . in this respect, it is crucial to underscore that there is often no ground truth for impacts of a specific event, as many impacts cannot be or are not directly measured, but rather are estimated. despite the inherent biases in using wikipedia and artemis as data sources, our approach presents several advances upon existing global impact datasets that are routinely used. existing datasets typically include manual and unsystematic compilation steps, and do not connect entries to specific sources, thus hindering validation. in contrast, our proposed database enables users to trace each entry back to a specific textual source. moreover, unlike most current impact databases we include ranges where no precise numbers are reported in our sources or where multiple estimates are quoted, thus facilitating uncertainty quantification. finally, the highly automated pipeline that we developed enables frequent updates of the database, for example, if new impact information or data sources become available. we nonetheless recognize that several additional steps may further facilitate the use of our database in research, notably connecting entries to observed environmental variables (e.g. water levels, wind speeds, temperatures). we thus conclude that, notwithstanding practical and technical challenges, llms are a promising tool to develop a new generation of databases of climate extreme impacts.