T emp T ab QA : Temporal Question Answering for Semi-Structured Tables (2023)

Abstract: Semi-structured data, such as Infobox tables, often include temporal information about entities, either implicitly or explicitly. Can current NLP systems reason about such information in semi-structured tables? To tackle this question, we introduce the task of temporal question answering on semi-structured tables. We present a dataset, TEMPTABQA, which comprises 11,454 question-answer pairs extracted from 1,208 Wikipedia Infobox tables spanning more than 90 distinct domains. Using this dataset, we evaluate several state-of-the-art models for temporal reasoning. We observe that even the top-performing LLMs lag behind human performance by more than 13.5 F1 points. Given these results, our dataset has the potential to serve as a challenging benchmark to improve the temporal reasoning capabilities of NLP models.

## introduction reasoning about temporal aspects of factual information presents a fundamental challenge for contemporary natural language processing (nlp) systems. factual information related to an entity often evolves over time, and understanding it requires understanding the scope of knowledge and temporal intervals. furthermore, this factual information is also scattered across semi-structured data in several different forms. these forms include both implicit and explicit representations (see figure 1 for an example). the wide prevalence of these characteristics creates major challenges for nlp models. it requires these models to effectively handle changes over time and extract valuable insights from time-dependent data. previous studies have primarily concentrated on question answering @xcite and inference @xcite : how many years elapsed between nedelcheva's 1 st and last silver medals? a1: 6 years q2: how many bronze medals has nedelcheva's won since 2013? a2: 3 q3: which country did nedelcheva's represent in the 2015 european games? a3: bulgaria q4: what was the ranking difference between nedelcheva's top doubles and singles performances? a4: 5 figure 1 : a semi-structured table of women badminton players (source: wikipedia) along with accompanying temporal questions and their respective answers form temptabqa. @xcite concerning numerical aspects of semi-structured tables. they typically examine tables without time-related information and focus on queries with predominantly non-temporal contexts. research on temporal aspects in entity-centric tables, such as wikipedia infoboxes, has been limited @xcite @xcite . @xcite introduced question answering in an entity-centric table context. however, the questions in the dataset are simple and non-temporal, and it's worth noting that the dataset is not open source. existing studies have only considered a few temporal aspects and addressed a small number of time-related factors. advances in modeling techniques, including table pre-training and targeted fine-tuning, have substantially improved reasoning on semi-structured tables @xcite . moreover, large language models (llms) have exhibited impressive performance across various domains, such as general, finance, and medical, demonstrating their mathematical, knowledge-based, and common-sense reasoning capabilities @xcite @xcite . however, the effectiveness of these models in handling temporal aspects remains understudied. consequently, this paper seeks to address the following research question: "can modern nlp models effectively reason about temporal information in semi-structured tables?" to effectively address the above question, we introduce a new task called temporal question answering on entity-centric semi-structured tables. figure 1 shows an example. we curate a comprehensive (covering diverse domains), specialized (temporally aligned), and human-verified dataset, temptabqa. it consists of 11,454 questionanswer pairs extracted from 1,208 wikipedia infobox tables across more than 90 domains. both the tables and questions in temptabqa are encompass numerous temporal terms. for example, the figure 1 table involve multiple dates, age, years, and the corresponding questions also incorporate temporal terms, such as first and last, years (since 2013), ranking, and more. these enhanced tables incorporate temporal information, ensuring that all queries have time-related components. this dataset is the first to explore question-answering and temporal reasoning in semi-structured data. we conduct analysis of temporal reasoning challenges in temptabqa, offering both qualitative and quantitative insights. most questions in temptabqa are abstractive and necessitate mathematical calculations over temporal concepts to arrive at correct answers. the dataset also encompasses additional test sets to evaluate reasoning in rare domains. our findings indicate that temporal reasoning in temptabqa poses greater challenges compared to non-temporal reasoning in previous tabular datasets. our assessment of contemporary nlp systems on the temptabqa benchmark exposes their subpar performance in comparison to humans. humans excel at temporal reasoning, delivering accurate answers and in-depth explanations. in contrast, our error analysis shows that models frequently make mistakes, particularly when faced with complex temporal reasoning questions. consequently, our dataset serves as a challenging testbed for investigating effective temporal reasoning within semi-structured information. our paper marks a significant milestone by pioneering the creation of complex temporal question answering datasets, specifically tailored to entity-centric tables. our primary objective was to introduce a novel challenge -addressing intricate temporal questions within this context. the temptabqa dataset not only demands sophisticated reasoning but also necessitates a firm grasp of temporal common sense, adept handling of arithmetic and numerical aspects. our work sheds light on the unique temporal specificity of this dataset, setting it apart from existing models. furthermore, we delves deep into this differentiation, offering a comprehensive array of statistics and analyses that illuminate the multitude of temporal reasoning challenges posed by the dataset. the findings from above enhance our understanding of temporal reasoning in tables and encourage further research. the temptabqa dataset can be accessed at https://zenodo.org/records/10022927 . for relevant analysis and modeling scripts refer at https:// temptabqa.github.io. ## motivation dynamic nature of information. tables serve as structured representations that organize and record diverse information types, making them highly useful for studying an entity's timeline. they offer a comprehensive record of events, enabling clear visualization of the evolution of various aspects over time. by capturing a chronological sequence of events, tables allow us to analyze the progression of positions, changes in marital status, and the acquisition of awards, serving as reliable sources for temporal reasoning. additionally, entity-centric tables, such as wikipedia infoboxes, significantly differ from both unstructured and fully structured data (sql tables and kgs). these tables possess a semi-structured nature and store data in intricate implicit forms, as discussed in the context of semi-structured tables @xcite . tables in the real world. complex temporal question answering applied to entity-centric semistructured tables, like wikipedia infoboxes, has broad applicability across various fields. in historical research and education, it helps scholars, historians, and students extract precise historical details, while in financial analysis, it empowers analysts with historical financial data for informed investment decisions. in medical research, it aids in accessing historical medical data and clinical trial timelines, and legal professionals use it to review historical legal records. journalists gain historical context, linguists analyze language dynamics, and businesses optimize supply chains through historical data. environmental researchers, policy analysts, travelers, software developers, and archivists all benefit from this versatile tool. this underscores the significance of discovering valuable information within tables across a broad spectrum of diverse fields. why temporal questions answering? temporal questions require reasoning based on timerelated information, falling into two main categories: explicit and implicit: (a.) explicit temporal questions directly involve time-specific details like dates, years, or hours, demanding precise knowledge. for example, in figure 1 , question such as 'when was the nedelcheva's was born?' is an explicit temporal question. (b.) implicit temporal questions rely on temporal terms and relationships that indicate the temporal order or context of events or entities. these questions may include terms like "rank," "before," "after," "predecessor," "successor," and similar expressions. in such cases, the temporal dimension isn't explicitly stated (e.g., mention of year not in the table) but must be inferred (or extracted) from the given context with understanding of the relationships between elements. for instance, in figure 1 'how many bronze medals did nedelcheva's won before 2013?' assumes an implicit understanding of the temporal sequence. in essence, addressing temporal questions involves comprehending and manipulating time-related information, whether explicit or implicit. this skill is vital in domains spanning historical research to natural language understanding, enabling effective reasoning about temporal aspects within data. why a new table qa dataset? current datasets such wikitablequestions (pasupat and liang, 2015), squall @xcite , finqa @xcite , tat-qa @xcite , hy-bridqa @xcite , fetaqa @xcite , sequentialqa(sqa) @xcite , and wikisql @xcite for questionanswering on table are limited in terms of both quantity and complexity of temporal questions. ## our temptabqa dataset we create a benchmark for answering temporal questions on entity-centric tables across domains. ## experimental evaluation we address the following research questions through our experiments: (a.) is the new dataset temptabqa challenging for existing models? (b.) does finetuning on temptabqa enhance model performance? (c.) does providing few-shot examples and chain of thought reasoning benefit them? (d.) is the performance on the tail domains worse than on the head domains? evaluation: we use the following metrics to evaluate the models: f1 score (f1), exact match (em), rouge 1 (r1) and 2 (r2), and meteor (met). for evaluation purposes, we treat december 2022 as the current month and year. to ensure models are aware of this, in all experiments, we add a new table row 'current date: december, 2022'. ## analysis breakdown of performance in our analysis, we examine the results (exact match) of our best model, gpt-4 few-shot with chain of thought, alongside human performance. question types. we categorize questions based on their types: starting with "what," "where," "when," "how," or "quantity" (also known as "how many"). the evaluation of the gpt-4 model's performance (exact match) compared to humans is presented in ## conclusion in conclusion, this study addresses the effectiveness of current nlp systems in reasoning about temporal information in semi-structured data, specifically infobox tables. we introduce the task of temporal question answering on semi-structured tables and present the temptabqa dataset, consisting of 11,454 question-answer pairs from 1,208 wikipedia infobox tables across varied domains. evaluating state-of-the-art models on this dataset reveals significant gaps compared to human performance, exceeding 13.5 f1 points. these findings emphasize the need for advancements in temporal reasoning capabilities of nlp models. the tempt-abqa dataset serves as a challenging benchmark to enhance temporal reasoning in nlp models. tailor large language models (llms) for tablespecific temporal logic, with more on this in appendix h. advanced prompts remain a potential area of exploration.