TRANSIENTTABLES : Evaluating LLM s’ Reasoning on Temporally Evolving Semi-structured Tables (2025)

Abstract: Humans continuously make new discoveries, and understanding temporal sequence of events leading to these breakthroughs is essential for advancing science and society. This ability to reason over time allows us to identify future steps and understand the effects of financial and political decisions on our lives. However, large language models (LLMs) are typically trained on static datasets, limiting their ability to perform effective temporal reasoning. To assess the temporal reasoning capabilities of LLMs, we present the TRANSIENTTABLES dataset, which comprises 3,971 questions derived from over 14,000 tables, spanning 1,238 entities across multiple time periods. We introduce a template-based question-generation pipeline that harnesses LLMs to refine both templates and questions. Additionally, we establish baseline results using state-of-the-art LLMs to create a benchmark. We also introduce novel modeling strategies centered around task decomposition, enhancing LLM performance.

## introduction in this day and age, information is constantly being updated depending on new facts and figures released in the public domain. information is inherently transient and often subject to periodic or non-periodic updates. for instance, the profits, losses, and revenues of publicly traded companies fluctuate regularly, political figures shift with each election, bestseller lists change frequently, public transportation schedules are revised, rankings of women's football teams evolve, quarterly gdp growth varies, and the number of moons 1 surrounding earth can change based on new discoveries or this example of the indian cricket team presents three tables sampled at different time points: 2017, 2020, and 2023. it clearly illustrates how certain values, such as captain, icc ranking, tests played , change over time. however, inconsistencies exist in the tables, including missing keys and incorrect values, such as the test status acquired field, as noted in @xcite . in this work, we are only focusing on transient (or temporally changing) information. semi-structured tables are everywhere in this modern world, from web pages (see figure 1 ) to nutrition labels on food products. semistructured tables such as wikipedia infoboxes (entity-centric) combine elements of both structured data (databases) and unstructured text. they offer structure while storing information in implicit forms, making them more flexible than traditional databases. by presenting information in a systematic, organized manner, tables allow for easy updates while maintaining organization and historical context, making them ideal for documenting the constant flux of dynamic information about entities such as public figures, corporations' revenues, and scientific concepts, among others @xcite . temporal reasoning is particularly challenging in natural language processing (nlp) tasks, due to the constant updation of information required as stated above. to understand transient information in the natural language, the model must understand explicit and implicit temporal relations and track an entity's changing attributes. recent works like temptabqa @xcite and tiq @xcite are both focused on temporal question-answering on tabular data. however, temptabqa only considers entity tables where each entity has a single table containing temporal information, such as an athlete winning different medals (gold, silver, or bronze) over multiple years. tiq explores temporal qa with implicit time constraints from various sources, including knowledge bases, text, and infoboxes (e.g., who was the captain of the indian test cricket team before rohit sharma?). neither of these studies investigates understanding entity-centric tables that contain information that evolves, such as multiple tables of a certain entity across time (as highlighted in figure 1 ). to the best of our knowledge, we are the first to study this problem. specifically, we ask: can current language models understand and reason about the temporally evolving information (both periodic and nonperiodic) in semi-structured tables ?. to address this problem, we define a questionanswering task and create an associated dataset, transienttables, where the answers require understanding and reasoning across at least two distinct tables sampled from different time periods. even when a question can be answered using a single table, the model must still identify the correct table from a set of input tables to provide an accurate answer. transienttables consists of a qa dataset and a thorough analysis of the performance of stateof-the-art llms on entity-centric tables where values of keys in the table change over time. the resulting dataset comprises 3971 questions generated from more than 14k tables associated with 1238 entities (averaging 11.42 dynamic temporal tables per entity timeline). our results indicate that sota llms struggle with reasoning on many straightforward questions, which humans can easily answer with sufficient context. this highlights that llms still have significant challenges in reasoning with transient information. additionally, our experiments reveal that simple prompting often proves ineffective; therefore, breaking tasks down into smaller, more manageable components is necessary to achieve improved outcomes @xcite @xcite . analysis of our experimental results indicates significant deficiencies in llms' capacity for rightful evidence extraction and reasoning. even when presented with the right evidence, these models demonstrate poor performance across various reasoning tasks compared to scenarios where the entire context is provided, which requires both rightful evidence extraction and reasoning processes. these findings suggest that llms rely on spurious correlations rather than robust logical inference when formulating responses. this work makes the following contributions foot_0 : a question-answering dataset on temporally evolving information in tabular data. in addition, human evaluations were carried out, revealing a significant performance gap that llms must address. ## transienttables dataset transienttables consists of infobox tables from various categories, including countries, cricket teams, economies, government agencies, and more. each category features multiple entities, such as the usa, india, and kenya, in the 'countries' category, with 7 to 12 infoboxes per entity that capture temporal changes to form a timeline. figure 1 illustrates an example from the 'cricket teams' category, showing a timeline of three infobox tables for the indian cricket team. the categories were selected based on the infobox size (with more than 10 keys) and the degree of temporal variation. we manually chose infobox templates that met these criteria, prioritizing those with more substantial changes in key-value pairs over time. we chose ten categories: cyclists, equestrian players, economic data of a country, government agencies, cricket teams, field hockey players, golfers, table tennis players, countries, and cricketers. ## modelling techniques in order to respond to a query posed on a set of transient tables, a human evaluator must reason through the following steps: these sequence of operations is highly interdependent, leading to compounded errors as we progress through each step. to test llm capabilities and find areas where llm needs improvement, we define a compressive set of modeling techniques using instruction sets (prompts) with different granularity of information (context in terms of a number of tables given as input), and intermediate task decomposition, i.e., temporal grounding, attribute selection and analytical reasoning. ## results and analysis our experiments answer the following questions: transienttables is challenging. tables 2 , 3 , and 4 demonstrate that reasoning with temporally evolving information poses significant challenges. the gpt-4o model achieves the highest f1-score and exact match scores of 63 and 58, respectively, when leveraging all tables as context and chain of thought (cot) prompting. humans achieve an f1-score of 93 and an exact match of 88, significantly outperforming the best models, with the topperforming model lagging by 30 f1 points. these results indicate that current state-of-the-art models struggle to effectively comprehend temporally evolving information. see appendix a.1 for the complete human evaluation procedure. is using larger context better? to answer this question, we compare model performance on full tables vs. single and oracle tables. ## conclusion and future work in conclusion, our study reveals key limitations in nlp systems' ability to reason about transient information in semi-structured data. we introduce a novel task of question answering on temporally evolving tables, along with a new transient-tables dataset containing 3,971 question-answer pairs from over 14k tables and 1,238 entities across various time periods. evaluating state-of-the-art models on this dataset highlights shortcomings in evidence extraction and reasoning, underscoring the need for improved temporal reasoning in nlp models and guiding future research. future direction. (a) diverse structures: we plan to expand dynamic temporal qa beyond traditional tables to include hybrid formats with text, images, and graphs, as well as hierarchical structures capturing nested temporal data. this will better reflect real-world scenarios and improve model applicability. (b) neuro-symbolic learning: we aim to develop more robust, interpretable models by integrating neural networks with symbolic reasoning, enhancing accuracy and explainability in handling complex temporal queries.