O pen RT : An Open-source Framework for Reasoning Over Tabular Data (2023)

Abstract: There are a growing number of table pre-training methods proposed for reasoning over tabular data (e.g., question answering, fact checking, and faithful text generation). However, most existing methods are benchmarked solely on a limited number of datasets, varying in configuration, which leads to a lack of unified, standardized, fair, and comprehensive comparison between methods. This paper presents OpenRT, the first open-source framework for reasoning over tabular data, to reproduce existing table pre-training models for performance comparison and develop new models quickly. We implemented and compared six table pre-training models on four question answering, one fact checking, and one faithful text generation datasets. Moreover, to enable the community to easily construct new table reasoning datasets, we developed TaRAT, an annotation tool which supports multi-person collaborative annotations for various kinds of table reasoning tasks. The researchers are able to deploy the newly-constructed dataset to OpenRT and compare the performances of different baseline systems.

## introduction with the increasing amount of structured data available, there is a growing interest in developing nlp systems for reasoning over tabular data to perform tasks such as question answering @xcite @xcite , fact checking @xcite , and faithful text generation @xcite . table pre-training has emerged as a promising approach for developing large language models (llms) that can perform various kinds of downstream table reasoning tasks with high accuracy after fine-tuning @xcite @xcite @xcite . however, existing table pre-training methods have been benchmarked on different datasets with varying configurations ( in this paper, we present openrt, the first open-source framework for reasoning over tabular data, which has the following three characteristics: (1) modularization: we developed openrt with highly reusable modules and integrated them in a unified framework, which enables researchers to study different table reasoning models at a conceptual level; (2) standardization: openrt includes popular table reasoning datasets and models. the evaluation of different models is standardized under the same experimental configuration; (3) extensibility: openrt enables researchers to easily develop their own models or add new datasets by extending corresponding modules with their proposed ones. moreover, in order to facilitate the construction of new table reasoning datasets by other researchers, we developed tarat, the first table reasoning annotation tool that supports the collaborative construction of various dataset types (i.e., question answering, fact checking, text generation). user-created datasets can be easily integrated into openrt for performance evaluation. the main structure of the paper is organized as follows: section 2 describes each table reason-dataset # examples # tables input ## openrt framework as shown in figure 1 , openrt consists of four main modules: configuration, data, modeling, and evaluation. the users are able to fine-tune or test the existing table pre-training models on the included dataset. they are also allowed to add their own models or datasets into openrt by extending corresponding modules with their proposed ones. ## tarat annotation tools in order to facilitate the construction of new table reasoning datasets for other researchers, we developed tarat, the first open-source table reasoning annotation tool that supports the collaborative construction of various dataset types (i.e., question answering, fact checking, text generation). tarat was designed, developed, and tested with the four design principles shown in figure 3 . as depicted in figure 2 , a typical annotation process using tarat consists of the following five steps: ## related work reasoning over tabular data the tasks related to reasoning over tables involves question answering @xcite @xcite , fact checking @xcite , and faithful text generation @xcite @xcite based on the information contained in the tables. previous work mainly investigated how to develop a task-specific model that can work on one or two table reasoning datasets. more recently, inspired by the huge success of pre-trained language models @xcite , researchers have attempted to adopt the "pre-training, then finetuning" paradigm to develop models that can handle different kinds of table reasoning tasks with high performance @xcite @xcite @xcite . however, existing table pre-training methods have been evaluated on different datasets with varying configurations and developed as individual systems, resulting in difficulties in re-implementing them for performance comparison in future studies. the development of open-source libraries such as transformers @xcite ) alleviate these issues to some extent, but they only cover a narrow range of table pre-training models and datasets. openrt implements existing table pre-training models in a unified and highly modularized framework, and provides standardized and comprehensive evaluation benchmarks for performance comparison. ## conclusion this work presents openrt, the first open-source framework for reasoning over tabular data, to reproduce existing table pre-training models for a standardized and fair performance comparison. openrt also enables users to quickly deploy their own models and datasets. moreover, we developed tarat to facilitate the construction of new table reasoning datasets by other researchers. in the future, we will continue to add more table reasoning datasets and the latest released table pre-training models to openrt as part of regular updates. we welcome researchers and engineers to join us in developing, maintaining, and improving openrt and tarat, in order to push forward the development of research on table reasoning.