IM - TQA : A C hinese Table Question Answering Dataset with Implicit and Multi-type Table Structures (2023)

Abstract: Various datasets have been proposed to promote the development of Table Question Answering (TQA) technique. However, the problem setting of existing TQA benchmarks suffers from two limitations. First, they directly provide models with explicit table structures where row headers and column headers of the table are explicitly annotated and treated as model input during inference. Second, they only consider tables of limited types and ignore other tables especially complex tables with flexible header locations. Such simplified problem setting cannot cover practical scenarios where models need to process tables without header annotations in the inference phase or tables of different types. To address above issues, we construct a new TQA dataset with implicit and multi-type table structures, named IM-TQA, which not only requires the model to understand tables without directly available header annotations but also to handle multi-type tables including previously neglected complex tables. We investigate the performance of recent methods on our dataset and find that existing methods struggle in processing implicit and multi-type table structures. Correspondingly, we propose an RGCN-RCI framework outperforming recent baselines. We will release our dataset to facilitate future research.

## introduction to automatically gain valuable information from numerous tables, table question answering (tqa) technique was developed and can answer natural language questions over tables @xcite @xcite . correspondingly, researchers proposed various tqa datasets aiming at different application scenarios (zhong et @xcite @xcite . as the model performance continues to improve, tqa technique has been widely used in intelligent data analysis tools, e.g., power bi foot_1 , and tableau 3 though previous datasets have promoted the development of tqa technique, the problem setting of existing benchmarks suffers from two limitations. first, existing benchmarks only evaluate the performance of model on tables with explicit table structures, which means locations and directions of headers are explicitly annotated and treated as model input during inference. for example, text2sql benchmarks offer annotated column headers @xcite and recent hierarchical table datasets also contain hierarchical header annotations @xcite , which are available at inference time. this setting artificially lowers the difficulty of the task. nevertheless, in practical scenarios, model may encounter plenty of tables without labeled headers. manually annotating headers for these tables is prohibitively expensive and timeconsuming. as a result, a benchmark is needed to evaluate the performance of tqa models on tables with implicit table structures. here, "implicit table structures" represents that header annotations of the table are not directly available during inference. second, existing benchmarks only consider limited table types and ignore complex tables with flexible header locations. previous datasets mainly focus on vertical or hierarchical tables whose header cells only locate on the top or left side of the table @xcite @xcite , as shown in figure 1 (a) and (c), but neglect the fact that model may require handling multi-type tables, especially complex tables whose header cells also appear at other positions in the table such as the bottom-right region, as illustrated in figure 1 (d ). complex tables are prevalent in professional equipment specifications and record sheets, which are beyond the ability of current tqa systems. above analyses show that the problem setting of previous benchmarks restricts the application of tqa models. in this paper, we define a new problem, table question answering over implicit and multi-type table structures. in this problem setting, annotations of table headers are not available during inference and model needs to comprehend implicit and multi-type table structures to answer questions. to facilitate the study on this problem, we build the first chinese table question answering dataset with implicit and multi-type table structures, named im-tqa, which consists of 1,200 tables and 5,000 questions (section 3). our dataset includes tables of four types from different domains, especially complex tables neglected by published studies. we annotate tables not only with various lookup questions and their answer cells, but also with functional roles of table cells. im-tqa presents a strong challenge to existing models. because of the inability to understand implicit and diverse table structures, the state-of-theart baseline rci (row and column intersection) model @xcite only achieves 49.6% overall accuracy and 23.8% accuracy on complex tables. to improve the performance of rci, we propose an rgcn-rci framework (section 4). specifically, this framework solves the new problem in two stages: (1) table is modeled as graph and an rgcn @xcite is used to comprehend table structure and predict table header cells. (2) based on predictions of rgcn, header information is merged into the text sequence representation of each row or column, which helps rci model to predict whether a row or column contains the answer or not. though our framework shows high effectiveness and improves the accuracy on all tables and complex tables by 3.8% (49.6% → 53.4%) and 8.2% (23.8% → 32.0%) respectively, it still lags far behind human performance, i.e., 95.1% and 94.1% on all and complex tables. we hope this work could raise attention to the implicit and multi-type table structures, facilitating the development of tqa models to address more complex and realistic table data. to summarize, we conclude our contributions as follows: ## problem statement in this section, we define the problem of table question answering over implicit and multi-type table structures. consisting of multiple cells, a table t can be defined as t . = {p, r, v}, where p, r and v represents the set of position information, functional roles and values of cells in table t respectively. table cells can be categorized into different types according to their functional roles, e.g., some cells are column headers and others are data cells. under the setting of traditional tqa problem, table structures are explicitly annotated @xcite or can be easily obtained by heuristic methods designed for specific table type @xcite . in this setting, functional roles r are directly provided for the model f (•), which outputs answer y given a natural language question q, i.e., @xmath0, t) = f (q, p, r, v). however, in the setting of real applications where the model may need to process implicit and multi-type table structures, functional roles r are not available in advance and the model requires distinguishing useful headers from data cells. thus, tqa over implicit and multi-type table structures can be formalized as: @xmath1, t ′ ) = f (q, p, v). this problem setting poses a great challenge to existing methods. unfortunately, none of previous benchmarks served as testbed for this problem. to fill the gap, we build im-tqa, which includes multi-type tables and requires understanding implicit table structures. compared with previous datasets, the improvements of im-tqa are demonstrated in figure 2 . ## dataset construction in this section, we first introduce considered table types, and then elaborate the dataset construction procedure. we also compare im-tqa with other related datasets. we recruit 10 professional annotators and provide them with sufficient instructions. the dataset construction totally costs 1,200 working hours. the ethical considerations will be discussed in the section 9. ## method we propose the rgcn-rci framework for im-tqa, shown in figure 5 . rgcn-rci consists of an rgcn-based cell type classification module and an rci-based table question answering module. ctc module uses an rgcn to aggregate neighbour cell information and predicts cell's functional role. based on the predictions of ctc module, tqa module adopts an improved rci model to predict whether a row or column contains the answer or not. final answer cells are selected from intersections of positive rows and positive columns. ## limitations our proposed dataset is in chinese and focuses on single tables. though we translate the dataset from chinese into english, we think it is better to directly construct a corresponding large-scale english tqa dataset in consideration of data quality. to build such a dataset with limited resource, one can fully utilize abundant tables in existing english tqa datasets. as an exploration of table question answering over implicit and multi-type structures, this work focuses on lookup questions and we leave annotations of aggregation and other numerical reasoning questions in the future work. the proposed rgcn-rci framework is not an end-to-end model. we look forward to seeing more end-to-end models which can simultaneously learn table structures and question answering with the help of our dataset. in addition, it takes more time to convert very large tables into graph structures which are used to recognize cell functional roles. 5082 our proposed benchmark is a free and open resource for the community to study table question answering over implicit and multi-type tables. we follow existing works aitqa @xcite and dusql @xcite to select data sources when building our dataset. tables are collected from the public organization china securities regulatory commission and baidu encyclopedia of baidu company. we got the reply and permission from these two organizations which allow us to share and redistribute data for noncommercial use. tables in these annual reports and web pages are open to public so there is no privacy risk. in the annotation process, we asked annotators to check if there exist any offensive content such as insulting or discriminatory speech. they did not find any such content in our benchmark. we also checked for identifiers and replaced identifying information with mono-directional hashes. we recruit 10 professional annotators with bachelor degree (6 males and 4 females) and pay them at a price of 6.5 dollars per hour (above the average local payment of similar jobs). the total time cost for annotation is 1,200 working hours. annotators were informed that these labeled data would be used as a table question answering dataset. the data collection protocol was approved by an ethics review board of an it company. main experiments in this paper can be run on a single nvidia geforce rtx 3090 gpu. our dataset follows the computational use of data agreement v1.0 foot_5 . over tabular and textual data. in findings of the association for computational linguistics: emnlp 2020, pages 1026-1036, online. association for computational linguistics. zhoujun cheng, haoyu dong, zhiruo wang, ran jia, jiaqi guo, yan gao, shi han, jian-guang lou, and dongmei zhang. 2022. hitab: a hierarchical table dataset for question answering and natural language generation. in proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1094-1110, dublin, ireland. association for computational linguistics. jacob devlin, ming-wei chang, kenton lee, and kristina toutanova. 2019. bert: pre-training of deep bidirectional transformers for language understanding. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171-4186, minneapolis, minnesota. association for computational linguistics. haoyu dong, shijie liu, zhouyu fu, shi han, and dongmei zhang. 2019. semantic structure extraction for spreadsheet tables with a multi-task learning architecture. in workshop on document intelligence at neurips 2019. majid ghasemi gol, jay pujara, and pedro szekely. 2019. tabular cell classification using pre-trained cell embeddings. in 2019 ieee international conference on data mining (icdm), pages 230-239. majid ghasemi-gol and pedro szekely. 2018. tabvec: table vectors for classification of web tables. michael glass, mustafa canim, alfio gliozzo, saneem chemmengath, vishwajeet kumar, rishav chakravarti, avi sil, feifei pan, samarth bharadwaj, and nicolas rodolfo fauceglia. 2021. capturing row and column semantics in transformer based question answering over tables.