TANQ : An Open Domain Dataset of Table Answered Questions (2025)

Abstract: Language models, potentially augmented with tool usage such as retrieval, are becoming the go-to means of answering questions. Understanding and answering questions in real-world settings often requires retrieving information from different sources, processing and aggregating data to extract insights, and presenting complex findings in form of structured artifacts such as novel tables, charts, or infographics. In this paper, we introduce TANQ,1 the first open-domain question answering dataset where the answers require building tables from information across multiple sources. We release the full source attribution for every cell in the resulting table and benchmark state-of-the-art language models in open, oracle, and closed book setups. Our best-performing baseline, Gemini Flash, reaches an overall F1 score of 60.7, lagging behind human performance by 12.3 points. We analyze baselines’ performance across different dataset attributes such as different skills required for this task, including multi-hop reasoning, math operations, and unit conversions. We further discuss common failures in model-generated answers, suggesting that TANQ is a complex task with many challenges ahead.

## introduction understanding and solving problems in real-world scenarios often requires reasoning across multiple documents and data modalities. this includes (i) retrieving information from different sources, (ii) processing and aggregating data to extract insights, and (iii) presenting complex findings in a structured format, for example a table or infographics, to communicate them to readers. previous studies show that knowledge workers across domains, e.g., finance, science, and economics, spend around 20% of their time searching and gathering information from different files into one document to extract insights and consequently answer information-seeking questions @xcite . as a result, one of most challenging tasks for workers is to aggregate data and turn it into insights. tables as a structured representation of data are ubiquitous in real-world sources and are commonly used to communicate complex information. hence, they can be the perfect modality to answer complex questions. large language models (llms), often enhanced with external tools, have become a primary method for various application such as answering questions. state-of-the-art question-answering (qa) systems integrate llms in various ways, from decomposing complex queries to retrieving documents using external tools or generating context data from knowledge acquired during model training. however, their evaluation is mostly limited to simple datasets, e.g., tabfact @xcite or hotpotqa @xcite , whose questions can be answered by reasoning over a single table or text document and generating a short text sequence as answer. this limits the applicability of such systems to perform complex multi-step research explorations. moreover, it differs from real-world needs where relevant information can be spread across documents and represented in different forms (e.g., text or tables). more often than not, generating a short text as answer is not sufficient for complex information-seeking questions. in this paper, we investigate llms' capabilities in reasoning over multiple data sources and formats (i.e., text, tables, infoboxes) to answer entity-centric questions and generate table answers as structured artifacts. to address these challenges, we introduce tanq, an open-domain, ## related work various benchmarks for qa have been released in recent years. each one addresses different challenges related to the task. qa with text and/or table input . a number of datasets use text and one or multiple tables for qa. while both text and tables have been considered as input modalities, the output of the datasets is mostly limited to short textual answers. hybridqa @xcite , for example, is a multi-hop qa dataset that requires reasoning over one table and multiple wikipedia passages related to entities occurring in the table. hybridqa answers are short texts with location names being the most common answer types, followed by numbers, dates, and person names. moreover, many qa benchmarks for reasoning over text and tabular context concentrate on the finance domain. for example, the multihiertt (zhao et al., 2022) benchmark is created from financial reports. questions require reasoning over texts and multiple tables. the answers are short numerical values with a focus on numerical reasoning. other financial qa benchmarks are finqa @xcite and tatqa @xcite where the context is one table and minimum two paragraphs related to the table. tatqa answers are short texts consisting of either one or multiple text spans from context paragraphs/tables or are free-form answers. multimodalqa is a multi-hop, open-domain qa dataset that takes one table and related images and text paragraphs as input with answers similar to previously described datasets. open-domain benchmarks. most of the earlier described datasets have context provided in form of text and/or tables, whereas open-domain qa datasets first require extracting the relevant context, before answering the given question. the majority of open-domain qa datasets, such as wikiqa @xcite , triviaqa @xcite , and robustqa @xcite , are limited to textual context. nq tables @xcite extends table-qa to an open-domain setting where first top-k tables are retrieved from a given corpus. these tables are processed by a reader component for generating the correct short-text answer. built on hybridqa, the open table -and-text question answering (ott-qa) benchmark @xcite extends this setting by requiring to extract both tables and texts given multi-hop questions. qa with table answers . the work closest to ours with respect to input and output modalities is multitabqa @xcite . multitabqa seeds on the spider dataset, a text-to-sql dataset containing sql queries, database tables, and natural language translations of the queries. pal et al. (2023) use table names occurring in sql queries to extract the input tables and query spider databases for answer table generation. while the datas et also generates answers as tables, it has certain limitations: (i) the benchmark input is limited to tables; (ii) the input and output tables are highly structured database tables which differ from real-world scenarios where tables occur in documents and websites in various formats; (iii) the questions are limited to sql-based queries. 3 building the tanq benchmark tanq evaluates the capability to answer open domain, multi-hop questions by aggregating data and generating answer tables. example question in tanq and its corresponding table answer. supporting evidence from multiple pages in a wikipedia snapshot is provided for each data point inside the table. we highlight the rationale inside each snippet in yellow. llms are evaluated with or without access to the evidence. ## baselines & evaluation we evaluate tanq in (i) closed book, (ii) oracle, and (iii) open book setups. we give an overview of the different approaches in figure 3 . closed book. the closed book setup evaluates llms' capabilities in extracting relevant information acquired during training to answer tanq questions. for all experiments, we use the palm-2 unicorn model @xcite , @xcite , gemini pro and flash @xcite , as well as gemma 9b @xcite . we evaluate all baselines in a few-shot setting with three examples provided in the prompt (see top left prompt in figure 3 ). oracle. in the oracle setup, we provide models' source attribution for every cell of the answer table in form of oracle documents (i.e., text, tables, and infoboxes). hence, the prompt consists of the tanq question and multiple evidence sentences, infobox entries and/or tables (see bottom left prompt in figure 3 ). to provide further context, we include the wikipedia page title (for text evidence also sub-/section titles) where the evidence was extracted. moreover, we add randomly selected oracle documents from other questions with similar attributes. this makes the oracle setting more challenging and requires models to filter the correct evidence from all provided ones first. we exclude models with a context length of less than 4k from the oracle evaluation. the evidence samples require a longer context to be fully displayed. otherwise, performance may decrease due to the input being cut off. we design the agent model to decompose and solve the tanq task in multiple sub-tasks, consisting of (i) retrieving requested entities in an iterative manner (e.g., movies in figure 1 ), (ii) searching for entity-related information (e.g., release year), (iii) post-processing information (filtering, calculation, etc.), and finally (iv) aggregating the information in form of a table. for some sub-tasks (e.g., entity retrieval), a separate sub-agent augmented with the required tools (e.g., wikisearch) is spawned. ## conclusion this paper introduces tanq, the first open domain question answering dataset where the answers require building tables from information across multiple sources. to create tanq, we design and apply an automated dataset pipeline using large language models and wikipedia and wikidata as knowledge sources. we further release for each cell of the answer tables source attribution in form of text, tabulation, or infobox proofs. we evaluate our dataset on state-of-the-art models in three different setting: oracle documents provided, closed, and open book setting. our results and analysis suggest that tanq is a complex task with many challenges ahead. shunyu yao, jeffrey zhao, dian yu, nan du, izhak shafran, karthik r. narasimhan, and yuan cao. 2023. react: synergizing reasoning and acting in language models. in the eleventh international conference on learning representations, iclr 2023, kigali, rwanda, may 1-5, 2023. openreview.net. yilun zhao, yunxiang li, chenying li, and rui zhang. 2022. multihiertt: numerical reasoning over multi hierarchical tabular and textual data. in proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pages 6588-6600, dublin, ireland. association for computational linguistics. https://doi.org/10 .18653/v1/2022.acl-long.454 fengbin zhu, wenqiang lei, youcheng huang, chao wang, shuo zhang, jiancheng lv, fuli feng, and tat-seng chua. 2021. tat-qa: a question answering benchmark on a hybrid of tabular and textual content in finance. in proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pages 3277-3287, online. association for computational linguistics. https://doi.org/10.18653/v1 /2021.acl-long.254