Multimodal Table Understanding (2024)

Abstract: Although great progress has been made by previous table understanding methods including recent approaches based on large language models (LLMs), they rely heavily on the premise that given tables must be converted into a certain text sequence (such as Markdown or HTML) to serve as model input. However, it is difficult to access such high-quality textual table representations in some real-world scenarios, and table images are much more accessible. Therefore, how to directly understand tables using intuitive visual information is a crucial and urgent challenge for developing more practical applications. In this paper, we propose a new problem, multimodal table understanding, where the model needs to generate correct responses to various table-related requests based on the given table image. To facilitate both the model training and evaluation, we construct a large-scale dataset named MMTab, which covers a wide spectrum of table images, instructions and tasks. On this basis, we develop Table-LLaVA, a generalist tabular multimodal large language model (MLLM), which significantly outperforms recent open-source MLLM baselines on 23 benchmarks under held-in and held-out settings.

## introduction tables are widely used to store and present data across various fields, e.g., financial analysis, scientific research and government reports @xcite . to make the most of the abundant tabular data, the table understanding (tu) technique has been proposed to automatically understand tables and perform table-based tasks, such as question answering (pasupat and liang, table-llava 7b and existing mllms on various multimodal table understanding benchmarks. table-llava outperforms recent open-source mllms and is even competitive with the powerful gpt-4v on most tasks. ## 9105 for each task, we use the above-mentioned method to generate input requests and design scripts to automatically extract the final answer from the textual tables in collected datasets. finally, 8k training samples, 1k or 1.25k evaluation samples were constructed for each structure understanding task. except above strategies, we also combine singleturn samples of the same table to compose 37k multi-turn conversation samples. at last, we obtain 232k samples on 82k table images for instructiontuning (named mmtab-instruct), and 45k held-in and 4k held-out test samples on 23k table images for evaluation (named mmtab-eval). inspired by existing mllms which align textual descriptions with input images through image-text pre-training, we introduce the table recognition task as an important pre-training task for multimodal table understanding. in this task, mllms learn to generate a textual table representation such as an html sequence given the table image, which helps aligning structure and text information in the table image with the ground-truth. we additionally collect 20k table images from the totto @xcite training split and merge them with table images in the mmtab-instruct to construct sufficient pre-training data. based on these table images and their original textual tables, we write scripts to construct table representations of three formats (html, markdown and latex), and then build instruction-following samples in the same way of mmtab-instruct. the resulting pre-training data contains 150k table recognition samples (html: 96k, markdown: 27k, latex: 27k) on 97k table images, which is denoted as mmtab-pre. more details about mmtab are given in appendix a. ## table-llava after constructing the mmtab dataset, we endeavor to fully leverage this data to promote models' multimodal table understanding ability. inspired by the widely adopted training paradigm of previous mllms @xcite @xcite , we devise an enhanced two-stage training procedure and choose llava-1.5 @xcite as the backbone to develop a versatile tabular mllm named table -llava. the whole training process is illustrated in the figure 4 . ## ethical considerations the proposed mmtab dataset is constructed based on the academic datasets like wtq and tabfact, which are free and open datasets for research use with licenses like mit license foot_0 or cc-by-sa-4.0 license foot_1 . we write python scripts to render textual table sequences (like html) in these datasets to obtain table images, and build multimodal instruction-following data based on original samples. the resulting dataset mmtab is also a free and open resource for the community to study the multimodal table understanding problem. thus, the authors foresee no ethical concerns with the research in this paper.