Tables as Texts or Images: Evaluating the Table Reasoning Ability of LLM s and MLLM s (2024)

Abstract: Tables contrast with unstructured text data by its structure to organize the information.In this paper, we investigate the efficiency of various LLMs in interpreting tabular data through different prompting strategies and data formats. Our analysis extends across six benchmarks for table-related tasks such as question-answering and fact-checking. We pioneer in the assessment of LLMs’ performance on image-based table representation. Specifically, we compare five text-based and three image-based table representations, revealing the influence of representation and prompting on LLM performance. We hope our study provides researchers insights into optimizing LLMs’ application in table-related tasks.

## introduction recent years have witnessed an explosion of large language models (llms), with impressive performance on various natural language processing (nlp) tasks @xcite @xcite . research to date has examined the performance of llms for various aspects and abilities @xcite @xcite , but their effectiveness on structured data such as tables is less explored. unlike unstructured text, tables are systematically organized structures of a large amount of information. this characteristic makes tabular data serve as the foundations for numerous applications, including medical diagnostics, virtual personal assistants, customer relationship management @xcite @xcite , etc. the evaluation of llms on processing tabular data involves many challenges. first, there are many ways to represent the information in tables. if we represent the table in pure text, we may use * contributed equally to this work. appendix a lists the detailed contributions. in this paper, we systematically evaluate model performance on tabular data for both textual llms and multi-modal llms. specifically, we investigate several research questions, including the effectiveness of image-based representation of tabular data and how different text-based or imagebased prompt methods affect llms' performance on table-related tasks. in addition, we provide analysis and hypothesis of llms' behaviors. our findings include: to the best of our knowledge, we are the first to study how llms perform with image-based table representations. we believe this paper draws new insights into optimizing table-based information processing. figure 1 : concept diagram. in this paper, we study differences in table representations. for each example, we prompt llms with the question and the context information, as well as one of the table representations. ## related work table-related tasks. tasks involving structured data have attracted interest in various tasks from diverse communities @xcite @xcite , among which there is a huge focus on tabular data @xcite . researchers have investigated various ways to encode tabular data. @xcite ; @xcite ; @xcite linearize the table content. others employ model-specific techniques such as adapting the attention mechanism to better align transformer-based models with the tabular data @xcite or designing hierarchical encoding to capture the table structure @xcite , further tuning llms on tabular data @xcite , etc. in contrast, our work focuses on exploring various table representations and prompts llms directly. prompting llms. researchers have prompted llms to evaluate llms' performance on traditional nlp tasks @xcite ) as well as on various complex reasoning tasks @xcite @xcite . on the contrary, to the best of our knowledge, few works have prompted these llms on tasks involving tabular data. for closed-source llms, researchers adopt hard prompts to manually craft text prompts with discrete tokens @xcite @xcite . @xcite develop chainof-though prompting, @xcite develop expert prompting. in our work, we include the comparison between vanilla, chain-of-thought, and expert prompting for llms on table-related tasks. @xcite . "+v?" indicates whether the visual input is allowed for the llm. "company" indicates which company the llm is from. we conduct the majority of our experiments with llama-2. gpt-3.5, gpt-4 and gemini pro . we prompt the january 2024 versions of the gpt-3.5 and gpt-4. we include additional experiments for gpt-4o, llama-3, and gemma models in appendix g. ## conclusion we have explored various representation strategies, including both text-based and innovative imagebased approaches, to understand how to use llms effectively in tasks involving tabular data. we demonstrate the effectiveness of image-based representations and reveal the impact of prompting strategies on the performance of llms. we believe our insights contribute to the understanding of llms and how to optimize llms for tabular data processing.