How LLM s React to Industrial Spatio-Temporal Data? Assessing Hallucination with a Novel Traffic Incident Benchmark Dataset (2025)

Abstract: Large language models (LLMs) hold revolutionary potential to digitize and enhance the Health & Public Services (H&PS) industry. Despite their advanced linguistic abilities, concerns about accuracy, stability, and traceability still persist, especially in high-stakes areas such as transportation systems. Moreover, the predominance of English in LLM development raises questions about how they perform in non-English contexts. This study originated from a real world industrial GenAI application, introduces a novel cross-lingual benchmark dataset comprising nearly 99,869 real traffic incident records from Vienna (2013-2023) to assess the robustness of state-of-the-art LLMs ( \geq 9) in the spatio vs temporal domain for traffic incident classification. We then explored three hypotheses — sentence indexing, date-to-text conversion, and German-to-English translation — and incorporated Retrieval Augmented Generation (RAG) to further examine the LLM hallucinations in both spatial and temporal domain. Our experiments reveal significant performance disparities in the spatio-temporal domain and demonstrate what types of hallucinations that RAG can mitigate and how it achieves this. We also provide open access to our H&PS traffic incident dataset, with the project demo and code available at Website https://sites.google.com/view/llmhallucination/home

## introduction large language models (llms) such as gpt-3.5/4 @xcite , and lamda @xcite have substantially enhanced public access to complex information, particularly in sectors such as healthcare and public services. these models are celebrated for their capability to demystify intricate information, assisting in tasks ranging from routine inquiries to aiding clinical decisionmaking @xcite . chatgpt, a derivative 1 * indicates co-first authorship and shared corresponding author. furthermore, substantial efforts have been directed towards developing llms, such as ur-bangpt @xcite , to make accurate predictions on synthetic data. given that llms are trained on extensive internet datasets, it is crucial to explore how these models perform with real industrial proprietary spatio-temporal data @xcite and to understand variations in performance across different spatio-temporal contexts. to address these challenges, our study originates from a real-world industrial genai application task, gathering lessons learned and introducing a novel, comprehensive multilingual benchmark from the industry for evaluating llms in sensitive sectors such as health and public services @xcite @xcite across spatio-temporal domains. our contributions include: ## related work previous studies have explored the capabilities of models like chatgpt @xcite , suggesting various methods to mitigate its limitations. for instance, bang et al. @xcite show that chatgpt excels in zero-shot learning across 9 of 13 nlp datasets. however, they also report a noticeable performance decline when handling non-english languages, particularly in non-latin scripts. manakul et al. @xcite introduced selfcheckgpt for hallucination response detection. however, it rely mostly on response consistency, may overlook cases where llms deliver consistent but inaccurate information, leading to potential false negatives responses. the xlingeval framework @xcite assesses llm behavior across several languages (english, hindi, chinese, and spanish), focusing on metrics like correctness, consistency, and verifiability. their findings indicate significant performance disparities across languages, with non-english responses generally showing an 18.12% decrease in quality @xcite . however, it only investigated the influence of multilingualism with state-of-the-art llms, without further exploring how to avoid hallucinations or in languages such as german. additionally, it did not examine the impact of input data on these llms or other factors beyond language type. for example, the format of the data, the effects of prompts under different temperature. moreover, urbangpt @xcite utilizes llms specifically for modeling urban environments, applying a gpt variant to zero-shot learning tasks in traffic management and public safety. the study emphasizes the critical role of high-quality, representative spatio-temporal data in training effective models @xcite . moreover, widely adopted rag techniques sometimes generate responses that are misleading, incomplete, or contextually off-target, particularly with non-english data @xcite . additionally, rag systems are latency-sensitive, and training local llms with rag is technically more complex and costlier than methods such as prompt fine-tuning or data augmentation @xcite . ## our h&ps traffic incidents dataset due to the challenges of penalty payments, regular reporting regulations, and the complexity of analyzing over 20 types of traffic incident records, current manual and subjective legacy systems are ripe for transformation by llms (large language models) @xcite . llms can significantly enhance the efficiency of the entire traffic incident tagging and reporting process. as shown in figure 1 , llms can automate the classification process, suggest tags based on dialogues between drivers and support teams, minimize subjective ambiguities, and respond swiftly to avoid costly penalties associated with reporting delays, which are particularly costly in transport systems. moreover, llms can conduct additional analyses and prioritization, such as identifying problematic traffic lines or stations and enhancing human awareness. "start": "2023-11-21 12:26:12", traffic delay start time "traffic_start": "2023-11-21 12:27:42", incident end time "end": "", effect lines "lines": "u3" the subsequent sections will detail our dataset creation process and genai workflow for analysis, including the structure of incident records. this is visually represented in figure 1 . we have queried incident records from the past ten years in the city of vienna via api under a creative commons non-commercial 4.0 international license. the cooperation ogd austria @xcite has developed a recommendation for publishing survey data due to the transparency obligation under the b-vg (austrian constitutional law) @xcite -particularly allowing for academic research. similar platforms can also be found such as nrw zuginfo (zuginfo, 2023) and f59 stoerungen (f59 stoerungen, 2023), which indicate the traffic status of germany nrw state and vienna in real-time. we then select 14 categories of different traffic incidents from the data pool (as shown in appendix each traffic record starts with an id number indicating its index order, followed by a title that specifies the affected traffic line (bus, tram or subway) along with its id and tag as shown in ## experimental settings robustness of llms on spatial vs temporal domain: we assess the robustness of major sota llms includes the gpt series @xcite , tinyllama model @xcite evaluated different sota llms output scores by examining response quality across ground truth for both spatial and temporal tasks. for temporal tasks, we analyzed responses across 10 categories, (with additional details provided in the appendix hypothesis to improve hallucination under industrial practices: we conducted a total of 165 samples per model (11 temperature settings from 0.0 to 1.0 * 10 temporal + 11 temperature settings * 5 spatial), comparing results across 9 different llm models (as shown in would rag really help and how? we also included rag-driven @xcite llm experiments using our dataset. these experiments were conducted with datastax (dat) and langflow(lan), where we vectorized dataset samples as context, used astra db(ast) as the vector database. we incorporated spatial and temporal queries as embeddings, adhering to the allowable tpm (tokens per minute) limit of 15,000 imposed by the api rate limits. we then also made ablation studies on comparing our dataset with existing benchmarks (see appendix for primary evaluation metric, we focus on the stability and accuracy (matching to ground-truth) of each model's responses. to test our hypotheses, we employed multiple linear regression (mlr) (yule, 1897), using p-value within 95% confidence interval (ci) as the confidence level @xcite . ## main results in this study, we first evaluate the top nine stateof-the-art (sota) llms with the cover of mostly well-known models. we conducted over 126 sets of experiments using our dataset, which covers data from 2013 to 2023. these experiments were designed to assess the llms' performance in spatial vs temporal domains. unbalanced hallucinations performance on spatio vs temporal domain. using our pro- ## conclusion in this work, we introduce a novel industrial spatiotemporal benchmark dataset (h&ps traffic incidents) from industry for enabling researchers to rigorously assess hallucinations in llms when handling real-world spatio-temporal challenges. it features diverse scenarios requiring both temporal and spatial reasoning. and we further conclude the following interesting findings: 1) major llms exhibit a significant number of unbalanced spatiotemporal hallucinations, and struggling more in the temporal domain. 2) three useful data preprocessing techniques offers practical guidance for optimizing data workflows in generative ai. 3) while rag improves contextual factual errors, it does not always enhance logical reasoning when handling more complex spatial problems or intricate temporal queries. limitation: despite being the first to release such large industrial dataset on accident information, our data still have limitations. to more effectively test the temporal and spatial awareness capabilities of llms, we need to manually annotate more spatial and temporal data and ground truths. expanding to other regions or cities would require additional approvals from governments or institutions, which could further enhance our dataset. future work: to address these limitations, we will continually collect accident information from various cities. additionally, we plan to exploring various other functionalities of llms beyond just hallucinations.