FREB - TQA : A Fine-Grained Robustness Evaluation Benchmark for Table Question Answering (2024)

Abstract: Table Question Answering (TQA) aims at composing an answer to a question based on tabular data. While prior research has shown that TQA models lack robustness, understanding the underlying cause and nature of this issue remains predominantly unclear, posing a significant obstacle to the development of robust TQA systems. In this paper, we formalize three major desiderata for a fine-grained evaluation of robustness of TQA systems. They should (i) answer questions regardless of alterations in table structure, (ii) base their responses on the content of relevant cells rather than on biases, and (iii) demonstrate robust numerical reasoning capabilities. To investigate these aspects, we create and publish a novel TQA evaluation benchmark in English. Our extensive experimental analysis reveals that none of the examined state-of-the-art TQA systems consistently excels in these three aspects. Our benchmark is a crucial instrument for monitoring the behavior of TQA systems and paves the way for the development of robust TQA systems. We release our benchmark publicly.

## introduction table question answering (tqa) deals with answering natural language questions related to information organized in a table. tqa systems serve as a fundamental component for interacting with relational databases @xcite through natural language and for processing information across diverse domains, e.g., science @xcite and finance @xcite . processing tabular knowledge poses notable challenges. while tables are structured, there are no standard table layouts for representing a particular type of data. table cells may contain various data types, including text and numbers. moreover, tables can have nested structures. thus, tqa systems should be able to robustly integrate textual 1 https://github.com/boschresearch/freb-tqa votes gender leslie 15 f olsson 4 f name floors ids tower 55 fos tower 32 name votes leslie 15 olsson 361 name votes leslie 15 olsson 4 name votes gender leslie f olsson f name ids tower fos tower floors 55 32 name q: how many floors does the fosh tower have? q: who received the least amount of votes? aggregation/comparison robustness in case of value changes attention to relevant cells retrieval robustness in case of table structure changes q: who received the least amount of votes? untitled diagram.iodraw-11 file:///c:/users/zhe7rng/downloads/untitled%20diagram.iodraw-11.ht 1 of 1 3/21/2024, 3:05 p figure 1: our benchmark addresses three aspects of robustness shown in the yellow boxes. answers are bold in tables. original tables (left) are what exist in a tqa dataset and changed tables (right) show tables after perturbations. we demonstrate three perturbations in this figure (top to bottom): table transposing, removing relevant cells and modifying values to change answers. as a result, these benchmarks fail to differentiate between various aspects of robustness, such as errors stemming from incorrect cell retrieval versus those arising from failures in value aggregation steps. pinpointing the exact failures of tqa systems, however, is a necessary diagnostic step towards enhancing their robustness. in this paper, we lay the groundwork for a more fine-grained systematic evaluation of tqa systems. to foster the development of robust tqa systems, we address three foot_0 important desiderata, as illustrated in figure 1 . with these aspects, we intend to separate several steps required when answering questions based on tabular data, i.e., identifying the relevant table cells and aggregating over or comparing them. in particular, we propose to evaluate tqa systems according to three aspects: (1) retrieval robustness in case of table structure changes: systems should retrieve the correct relevant cells regardless of table structure changes; to evaluate tqa systems with regard to these aspects, we create freb-tqa, a new benchmark based on four well-studied tqa datasets using seven novel automatic perturbations, one perturbation from previous work @xcite , and extensive manual annotations. in our extensive experiments, we study pipeline and end-to-end state-of-the-art tqa systems @xcite @xcite , as well as large language models (llms) @xcite using our benchmark. the fine-grained results uncover shortcomings of these systems that have not been previously explored. for example, we show that model performance diminishes substantially when column order changes, or when cells containing the answer are positioned at the bottom of the table. this finding indicates a strong positional bias in tqa systems, motivating more informative table encodings. our main contributions are: (1) we propose a novel benchmark of 8,590 selected and partially manually annotated questions and tables, resulting in a total of 75,205 instances using seven perturbation methods; (2) using several inter-annotator agreement studies, we demonstrate the solidness of our benchmark; and (3) using our benchmark, we experiment extensively with state-of-the-art tqa systems, discovering that for almost all robustness tests, system performances drop, demonstrating the difficulty of our benchmark for tqa systems. ## related work we provide an overview of previous research on tqa systems and their evaluation for robustness. tqa systems. end-to-end tqa systems @xcite ) compose an answer given a question and a serialized version of the table without intermediate steps or using additional tools. pipeline systems convert questions into a command language, e.g., @xcite , filtered tables @xcite @xcite , and then generate answers either via executing the commands @xcite or via using a trained neural network @xcite . our benchmark reveals the key strengths and weaknesses of these types of systems. ## experimental settings we use our benchmark to evaluate the robustness of state-of-the-art tqa systems with regard to our proposed aspects. ## experimental results in this section, we discuss the performance of tqa systems on our benchmark, and provide a detailed analysis of these models for each robustness aspect. ## discussion in our experiments, we use our new freb-tqa benchmark to analyze end-to-end, pipeline, and llm-based tqa systems. our experimental results show that all examined systems suffer from substantial issues when it comes to robustness. however, different system types show different patterns. end-to-end tqa systems, for instance, seem to be more robust against changes in the row/column arrangement. however, they are more likely to fail on numerical reasoning questions. this might be because the datasets these models are pre-trained on do not feature complex questions requiring numerical reasoning. llms are more affected by row or column perturbations but much more robust against transposing tables. their performance in performing aggregations or comparisons is highly dependent on the length of the serialized tables, i.e., they perform much better on tables that can be serialized to fewer tokens. finally, the pipeline models in our study are more robust against changes in relevant cells, including value changes. this is likely due to their symbolic execution component, which executes a predicted function or query on the given structured table. however, the prediction of the function or query itself still suffers from various perturbations of table data, including changes in table structure. yet, another benefit of pipeline tqa systems is that the intermediate representation, i.e., the predicted function or query, makes the model explainable to a certain extent. we hence argue that more research in the pipeline-based paradigm is a promising step towards more robust tqa. ## conclusion and outlook in this paper, we have proposed to evaluate three aspects of robustness of tqa systems: retrieval robustness in case of table structure changes, attention to relevant cells, and aggregation/comparison robustness in case of value changes. we have pre-sented a novel benchmark for a fine-grained analysis of those aspects. the main building blocks of our benchmark are targeted table perturbation methods and high-quality human annotations. finally, we have evaluated a range of architecturally varied state-of-the-art tqa systems, as well as offthe-shelf llms. our study has shown that while none of the systems was consistently robust, their weaknesses and strengths differ from each other. systems trained in an end-to-end fashion are able to deal with changes in row/column arrangement but llms perform better when it comes to numeric operations. the answers of pipeline-based models, which use symbolic methods for part of their computations, are more faithful towards the table they are supposed to use when composing their output. our new benchmark constitutes an important first step for evaluating tqa systems in a finegrained way, thereby directing research efforts towards building more robust tqa systems. as pipeline systems offer at least some explainability, we argue that research should concentrate on these types of systems. in particular, future work could explore the possibility of utilizing llms in a pipeline manner to build more robust tqa systems. further, aggregating or extracting information from long tables is an important next step given the shortcomings of current systems in this regard which our analysis revealed.