Uncovering Limitations of Large Language Models in Information Seeking from Tables (2024)

Abstract: Tables are recognized for their high information density and widespread usage, serving as essential sources of information. Seeking information from tables (TIS) is a crucial capability for Large Language Models (LLMs), serving as the foundation of knowledge-based Q&A systems. However, this field presently suffers from an absence of thorough and reliable evaluation. This paper introduces a more reliable benchmark for Table Information Seeking (TabIS). To avoid the unreliable evaluation caused by text similarity-based metrics, TabIS adopts a single-choice question format (with two options per question) instead of a text generation format. We establish an effective pipeline for generating options, ensuring their difficulty and quality. Experiments conducted on 12 LLMs reveal that while the performance of GPT-4-turbo is marginally satisfactory, both other proprietary and open-source models perform inadequately. Further analysis shows that LLMs exhibit a poor understanding of table structures, and struggle to balance between TIS performance and robustness against pseudo-relevant tables (common in retrieval-augmented systems). These findings uncover the limitations and potential challenges of LLMs in seeking information from tables. We release our data and code to facilitate further research in this field.

## introduction tables are widespread and rich sources of information across the web and in various documents. statistics show that the number of tables on internet web pages has reached several hundred million @xcite ; in the corporate environment, the number of tables in excel-like spreadsheet files has exceeded 115 million @xcite . precisely seeking relevant information from ## tabis benchmark we curated a benchmark tabis to investigate the table information seeking capabilities of llms. we use table-to-text generation (ttg) datasets as the original data source in our benchmark. the task of ttg is that, given a table and a set of selected cells (t, c), produce a one-sentence description of the cells, and the annotated description is called "reference" r. we transform ttg into a single-choice question with two options for objective and accurate evaluation. the format of a sample in tabis is (t, q, r, o) where q is a question, r, o are correct and wrong options. in tabis, t and r are the same as the annotation in the ttg task, o is a wrong description of the table that we generate, and q is a question about the table that can be answered by r. so, the task of tabis is that, given t and q, choose an option from {r, o} as the answer. tabis contains three subsets: basic table information seeking (b-tis), tis requiring structure understanding (su-tis), and tis from multiple tables (m-tis). in the following, we will first introduce how to generate options, and then introduce these subsets respectively. ## experiments on tabis based on the curated tabis benchmark, we evaluate the table information seeking capabilities of 12 representative llms. ## potential challenges in this section, we conduct in-depth analysis to investigate the llms' limitations and potential challenges behind the two complex subsets: su-tis (section 4.1) and m-tis (section 4.2). we further show error analysis on hard samples in section 4.3. ## improving table information seeking in this section, we explore how supervised finetuning enhances table information seeking using weakly-supervised datasets. we first utilize our proposed data generation pipeline foot_4 (section 2) to construct weaklysupervised b-tis and su-tis training datasets without manual checking. the statistics of the training dataset are shown in ## conclusion this paper introduces tabis, a new benchmark designed to evaluate the table information seeking (tis) abilities of large language models (llms). tabis is comprised of three typical tis scenarios and employs a single-choice question format to ensure reliable evaluation. extensive experiments on 12 representative llms have shown that tabis presents a significant challenge for current llms, with gpt-4-turbo showing only marginal satisfaction. further analysis points out two main issues: firstly, llms perform almost randomly on basic tasks involving comprehension of table structures; secondly, they face difficulties in improving performance and maintaining robustness against pseudorelevant tables, which could lead to sub-optimal results in real-world tis tasks. these observations underscore the current limitations and potential challenages in tis, calling for further exploration and advancement in this area.