H i T ab: A Hierarchical Table Dataset for Question Answering and Natural Language Generation (2022)

Abstract: Tables are often created with hierarchies, but existing works on table reasoning mainly focus on flat tables and neglect hierarchical tables. Hierarchical tables challenge numerical reasoning by complex hierarchical indexing, as well as implicit relationships of calculation and semantics. We present a new dataset, HiTab, to study question answering (QA) and natural language generation (NLG) over hierarchical tables. HiTab is a cross-domain dataset constructed from a wealth of statistical reports and Wikipedia pages, and has unique characteristics: (1) nearly all tables are hierarchical, and (2) QA pairs are not proposed by annotators from scratch, but are revised from real and meaningful sentences authored by analysts. (3) to reveal complex numerical reasoning in statistical reports, we provide fine-grained annotations of quantity and entity alignment. Experiments suggest that this HiTab presents a strong challenge for existing baselines and a valuable benchmark for future research. Targeting hierarchical structure, we devise a hierarchy-aware logical form for symbolic reasoning over tables, which shows high effectiveness. Targeting table reasoning, we leverage entity and quantity alignment to explore partially supervised training in QA and conditional generation in NLG, and largely reduce spurious predictions in QA and produce better descriptions in NLG.

## introduction in recent years, there are a flurry of works on reasoning over semi-structured tables, e.g., answering questions over tables @xcite and generating fluent and faithful text from tables @xcite . but they mainly focus on simple flat tables and neglect complex tables, e.g., hierarchical tables. a table is regarded as hierarchical if its header exhibits a multi-level structure @xcite @xcite . hierarchical tables are widely used, especially in data products, statistical reports, and research papers in government, finance, and science-related domains. hierarchical tables challenge qa and nlg due to: (1) hierarchical indexing. hierarchical headers, such as d2:g3 and a4:a25 in figure 1 , are informative and intuitive for readers, but make cell selection much more compositional than flat tables, requiring multi-level and bi-dimensional indexing. for example, to select the cell e5 ("66.6"), one needs to specify two top header cells, "master's" and "percent", and two left header cells, "all fulltime" and "self-support". (2) implicit calculation relationships among quantities. in hierarchical tables, it is common to insert aggregated rows and columns without explicit indications, e.g., total (columns b,d,f and rows 4,6,7,20) and proportion (columns c,e,g), which challenge precise numeri-cal inference. (3) implicit semantic relationships among entities. there are various cross-row, crosscolumn, and cross-level entity relationships, but lack explicit indications, e.g., "source" and "mechanism" in a2 describe a6:a19 and a20:a25 respectively, and d2 ("master's") and f2 ("doctoral") can be jointly described by a virtual entity, "degree". how to identify semantic relationships and link entities correctly is also a challenge. in this paper, we aim to build a dataset for hierarchical table qa and nlg. but without sufficient data analysts, it's hard to ensure questions and descriptions are meaningful and diverse @xcite . fortunately, large amounts of statistical reports are public from a variety of organizations (statcan; nsf; census; cdc; bls; imf), containing rich hierarchical tables and textual descriptions. take statistics canada (stat-can) for example, it consists of 6, 039 reports in 27 domains authored by over 1,000 professionals. importantly, since both tables and sentences are authored by domain experts, sentences are natural and reflective of real understandings of tables. to this end, we propose a new dataset, hitab, for qa and nlg on hierarchical tables. (1) all sentence descriptions of hierarchical tables are carefully extracted and revised by human annotators. hitab presents a strong challenge to state-of-theart baselines. for the qa task, mapo @xcite only achieves 29.2% accuracy due to the ineffectiveness of the logical form customized for flat tables. to leverage the hierarchy for table reasoning, we devise a hierarchy-aware logical form for table qa, which shows high effectiveness. we propose partially supervised training given annotations of linked mentions and formulas, which helps models to largely reduce spurious predictions and achieve 45.1% accuracy. for the nlg task, models also have difficulties in understanding deep hierarchies and generate complex analytical texts. we explore controlled generation @xcite , showing that conditioning on both aligned cells and calculation types helps models to generate meaningful texts. ## dataset construction and analysis we design an annotation process with six steps. to well-handle the annotation complexity, we recruit 18 students or graduates (13 females and 5 males) in computer science, finance, and english majors from top universities, and provide them with comprehensive online training, documents, and qas. the annotation totally costs 2,400 working hours. we will discuss the ethical considerations in section 8. ## related work table-to-text existing datasets are restricted in flat tables or specific subjects @xcite @xcite @xcite @xcite . the most related table-to-text dataset to hitab is totto (parikh et al., 2020), in which complex tables are also included. there are two main differences between hitab and totto: (1) in totto, hierarchical tables only account for a small proportion (5%), and there are no indication and usage of table hierarchies. (2) in addition to cell highlights, hitab conditions on operators that reflect symbolic operations on cells. table qa mainly focuses on db tables @xcite @xcite and flat web tables @xcite . recently, there are some datasets on domain-specific table qa @xcite and jointly qa over tables and texts @xcite , but hierarchical tables still have not been studied in depth. cfgnn @xcite and grasslm @xcite uses gragh neural networks to encode tables for qa, but all tables are database tables and relational web tables without hierarchies, respectively. @xcite include some hierarchical tables but only focuses on table search. ## discussion hitab also presents cross-domain and complicatedcalculation challenges. (1) to explore crossdomain generalizability, we randomly split train/dev/test by domains for three times and present the average results of our best methods in ## conclusion we present a new dataset, hitab, that simultaneously supports qa and nlg on hierarchical tables, where tables are collected from statistical reports and wikipedia in various domains. importantly, we provide fine-grained annotations on entity and quantity alignment. in experiments, we introduce strong baselines and conduct detailed analysis on qa and nlg tasks on hitab. results suggest that hitab can serve as a challenging and valuable benchmark for future research on complex tables.