From Text to Maps: LLM -Driven Extraction and Geotagging of Epidemiological Data (2024)

Abstract: Epidemiological datasets are essential for public health analysis and decision-making, yet they remain scarce and often difficult to compile due to inconsistent data formats, language barriers, and evolving political boundaries. Traditional methods of creating such datasets involve extensive manual effort and are prone to errors in accurate location extraction. To address these challenges, we propose utilizing large language models (LLMs) to automate the extraction and geotagging of epidemiological data from textual documents. Our approach significantly reduces the manual effort required, limiting human intervention to validating a subset of records against text snippets and verifying the geotagging reasoning, as opposed to reviewing multiple entire documents manually to extract, clean, and geotag. Additionally, the LLMs identify information often overlooked by human annotators, further enhancing the dataset’s completeness. Our findings demonstrate that LLMs can be effectively used to semi-automate the extraction and geotagging of epidemiological data, offering several key advantages: (1) comprehensive information extraction with minimal risk of missing critical details; (2) minimal human intervention; (3) higher-resolution data with more precise geotagging; and (4) significantly reduced resource demands compared to traditional methods.

## introduction epidemiology, the study of disease prevalence, comes from the greek word "epidemios", meaning "among the people, of one's countrymen at home" @xcite . each country documents the diseases within its borders, but they do so in their own ways. analyzing epidemiological reports at a global scale thus becomes a challenging task due to the large number of heterogeneous reports. even so, researchers currently read through epidemiological reports to extract the valuable data reported within. epidemiological data refers to data collected on the occurrence of diseases and is used to understand the distributions, trends, and dynamics of disease through analyzing historical events and training models to understand drivers behind various disease outbreaks. such data is crucial for public health analysis, policy development, and decision-making, as it helps identify risk factors for disease and targets for preventive healthcare. however, compiling epidemiological datasets poses significant challenges due to varying factors. there are numerous legal, technical, political, and cultural barriers, many of which are beyond our control, for efficient epidemiological data sharing and utilization @xcite . furthermore, epidemiological data often exists in different formats, frequently embedded within textual reports. the dynamic nature of political boundaries further complicates data collection and makes geotagging these records challenging. additionally, the potential use of different languages by different countries in their reporting adds another layer of complexity. traditional ways of compiling such datasets have relied mostly on human effort, involving manual reading of source documents, data extraction, and subsequent postprocessing. this standard method suffers from multiple issues, including the potential for human error and difficulty in correctly geo-tagging such datasets. to address these challenges, we propose utilizing large language models (llms) to automate the extraction and geo-tagging of epidemiological data at scale. by leveraging the capabilities of llms in event extraction and geospatial reasoning, we propose using llms to extract epidemiological data from text and geotag this information accurately. the process is a two-step approach: (i) extraction of epidemiological data from text and (ii) geotagging this data using contextual information. first, we employ a hand-crafted prompt to instruct the llm to output the epidemiological data embedded within a given text in a structured format, such as json. this can be considered as structured information extraction from unstructured sources. second, the llm utilizes a global administrative regions database to geo-tag each extracted information. with access to a global administrative region database, we then prompt the llm with another prompt, incorporating the extracted data, and the context from which it was extracted. the goal is to identify the correct administrative regions from the ones in the database, accurately geotagging the extracted data. our findings reveal that llms are highly effective at extracting structured information from textual documents (@xmath0. while llms generate more data than present in the humancurated dataset (p @xmath1, some of this additional information may be incorrect. however, even when considering only the correctly generated entries, the llm-generated dataset is three times larger than the human-curated one while capturing all relevant entries present in the human dataset. this highlights the significant advantage of leveraging llms for enhanced data coverage. further, llms exhibit impressive geospatial reasoning capabilities, accurately geotagging data points through logical inference. overall, our results high-light the strong performance of llms in both extraction and geotagging tasks, indicating that these models could semi-automate these processes, with minimal human validation needed. ## related work modern llms have transformed the field of natural language processing and artificial intelligence by eliminating the need for task-specific models trained using vast amounts of human-annotated datasets. through pre-training techniques, llms can be pre-trained on large textual corpora, enabling them to encode various types of knowledge within their parameters and potentially even function as knowledge bases @xcite . llms encode world knowledge and exhibit common sense reasoning capabilities, enabling them to understand and generate human-like text across diverse contexts. demonstrating this capability, @xcite showed that sufficiently scaled llms like gpt-3 can handle diverse downstream tasks just by receiving a task description, with or without a few sample task examples, as context, a technique known as "prompting". recent advancements in prompting techniques foot_0 have further enhanced the ability of llms to handle complex tasks, including those requiring intricate reasoning. additionally, researchers @xcite @xcite have shown that llms possess encoded geospatial knowledge, making them geospatially aware and capable of reasoning with geospatial data during text generation. instruction tuning (it) is another emerging technique where llms are further trained on datasets containing instructions and desired output in a supervised manner. instruction tuning aligns the nextword prediction objective of llms with user objectives, enabling the creation of general-purpose chatbots like chatgpt foot_1 and gemini. 3 these instruction-tuned llms excel at following human instructions and have shown impressive performance in several downstream tasks, such as event extraction @xcite . information extraction (ie) using llms for event extraction has gained significant research attention, primarily due to the excellent instructionfollowing capabilities of instruction-tuned llms. recent advancements in this field have shown varying degrees of success. @xcite introduced chatie, a framework that transforms the zero-shot ie task into a multi-turn question-answering problem suited for llms. the authors evaluated their framework on three ie tasks: entity-relation triple extraction, named entity recognition, and event extraction. their results show that chatie achieves impressive performance, even surpassing some multishot models on several datasets. similarly, others have achieved success using llms for specific ie tasks, with various modifications to enhance performance @xcite @xcite . however, some researchers have found notable challenges in using llms for event extraction. for instance, @xcite found that chat-gpt's performance was only half that of a taskspecific model for long-tail and complex scenarios. our research presents a different approach compared to the existing studies. while most research focuses on extracting singular events described in a text, our approach aims to extract multiple events from a single text using llms' comprehensive understanding capabilities. additionally, rather than solely relying on absolute performance metrics, we also measure success in terms of reduction in human effort for creating epidemiological datasets. furthermore, our approach involves geotagging extracted data using llms, a novel concept that enhances the quality of the epidemiological dataset. ## methodology the extraction and geotagging of epidemiological data involve a two-step process, as outlined in figure 1 . in the first step, data is extracted from small sections of text, which are then processed and merged to form the final database. in the second step, each extracted record, along with its contextual information, is passed to an llm with access to a global administrative region database. the llm is tasked with selecting the correct entries from this database to represent the record and provide reasoning for its choices, facilitating human validation. ## experiments our experiments aim to evaluate the accuracy and viability of using llms to extract and geotag epi-demiological data. we focus on the global spread of rift valley fever (rvf) by extracting relevant information from a collection of documents using an llm to create a database of rvf outbreaks. these documents are sourced from a human-created rvf outbreak dataset. below, we first introduce the rvf dataset and then outline the experimental setup, post-processing steps, and evaluation metrics for our two experiments: extracting rvf data and geotagging the extracted data. ## results our experimental results indicate that while llms can extract significant amounts of information overlooked by human annotators, they also produce some inaccuracies. additionally, llms equipped with relevant contextual data show promise in effective geo-tagging. in the following sections, we first present our empirical findings, followed by insights from human evaluations. we conclude by discussing the implications of these results and their potential impact on the future of epidemiological data extraction and geo-tagging using llms. ## conclusion and future work we propose an automated approach for extracting and geotagging epidemiological data from textual documents using large language models (llms). current methods of collecting such data rely on human effort and are prone to errors. to overcome these challenges, our approach leverages llms to automate the extraction and geotagging processes. we tested our method by applying it to rvf outbreak data, where a human-curated dataset was available for comparison. our findings demonstrate the effectiveness of this approach. the llm was able to capture significantly more information than manual efforts, albeit with some inaccuracies. additionally, the llm exhibited notable geospatial reasoning abilities, accurately geotagging data points. these results suggest that llms can be effectively employed to semi-automate the extraction and geotagging of epidemiological data with some level of human verification. while our experiments focused on a single disease, the promising results encourage us to extend this work to multiple diseases with the aim of creating publicly available datasets for epidemiological studies. we also plan to extend our research to include multilingual llms, broadening the scope to non-english documents.