Vision Language Models for Spreadsheet Understanding: Challenges and Opportunities (2024)

Abstract: This paper explores capabilities of Vision Language Models on spreadsheet comprehension. We propose three self-supervised challenges with corresponding evaluation metrics to comprehensively evaluate VLMs on Optical Character Recognition (OCR), spatial perception, and visual format recognition. Additionally, we utilize the spreadsheet table detection task to assess the overall performance of VLMs by integrating these challenges. To probe VLMs more finely, we propose three spreadsheet-to-image settings: column width adjustment, style change, and address augmentation. We propose variants of prompts to address the above tasks in different settings. Notably, to leverage the strengths of VLMs in understanding text rather than two-dimensional positioning, we propose to decode cell values on the four boundaries of the table in spreadsheet boundary detection. Our findings reveal that VLMs demonstrate promising OCR capabilities but produce unsatisfactory results due to cell omission and misalignment, and they notably exhibit insufficient spatial and format recognition skills, motivating future work to enhance VLMs’ spreadsheet data comprehension capabilities using our methods to generate extensive spreadsheet-image pairs in various settings.

## introduction spreadsheets are widely-used for data management and analysis @xcite . however, they are designed to be "human-friendly, not "machine-friendly" 1. cells are arranged on the grid and illustrated by various visual formats like borders, colors, and bold fonts. unlike machines, humans naturally leverage these visual cues to understand the layouts and structures of spreadsheets, such as the location of the table (e.g., "a2:n32") using borders, the headers (e.g., "a2:n3") using bold fonts, and aggregated rows and columns (e.g., rows 17, 19, and 20) using fill colors. while llms have shown promising performance in serializing spreadsheets as text sequences @xcite , representing spreadsheets in this manner loses critical visual signals. with the recent surge in vision language models (vlms) @xcite , we propose studying the capability of language models to leverage visual signals for spreadsheet understanding. fortunately, a spreadsheet can be straightforwardly processed using third-party tools like interop and converted into an image. this motivates us to construct spreadsheet-image pairwise data for self-supervised tasks. to this end, we propose three self-supervised tasks to comprehensively examine critical abilities of vlms separately: optical character recognition (ocr) of cells, two-dimensional spatial position perception, and visual format recognition. finally, we use spreadsheet table detection @xcite , a fundamental and enabling task in microsoft excel and google sheets, to jointly examine the effectiveness of vlms, as this task combines the challenges of all three self-supervised tasks. specifically, as shown in figure . 1, spreadsheet images present the following challenges: 1) the rows and columns are very compact, even overlapping, which makes the ocr task difficult. specifically, vlms sometimes struggle to split multiple cells and mistakenly treat them as a single cell. 2) the absence of explicit cell addresses and clear boundaries between rows and columns makes it difficult to perceive spatial locations. 3) spreadsheets often contain a variety of formats, making it hard to recognize all formats precisely at the pixel level. to address these issues, we propose three different spreadsheet-to-image settings to probe the vlms' performance: column width adjustment, style change, and address augmentation respectively, as shown in figure . 2. we explore variants of prompts to address the above tasks in different settings. notably, to leverage the strengths of vlms in understanding text rather than two-dimensional positioning, we propose to decode cell values on the four boundaries of the table rather than decoding regions like "a2:e5" directly in the task of spreadsheet boundary detection. by analyzing the experiment results, we draw the following conclusions: firstly, vlms possess strong ocr capabilities, yet they often encounter issues of cell omission and prediction misalignment when dealing with spreadsheet images. secondly, vlms lack robust spatial perception in images because they need to infer the number of rows and columns implicitly on a large two-dimensional cell grid rather than reading it directly. it is highly noteworthy that their performance on recognizing visual formats on a cell grid is far from satisfactory; they are far from human-level in comprehending spreadsheet formats. lastly, in the task of spreadsheet table detection, vlms do not perform as well as the existing cnn-based tablesense @xcite , which is well-trained using a human-labeled dataset, indicating that there is still a long way to go in understanding spreadsheet images for vlms. ## experiment setting we conducted experiments using gpt-4v (2024-02-15 preview) @xcite and gemini (1.5-pro lateset until 2024-05-16) @xcite . to ensure consistent experimental parameters, we set the generation temperature for both gpt-4v and gemini-pro to 0.7, top_p to 0.95, and max_output_token to 4096. due to gpt-4v's input image restrictions-specifically, that the image file size must be less than 4mb and the resolution must be between 50 × 50 and 10, 000 × 10, 000 pixels-we filtered 76 images from the test dataset of tablesense @xcite to meet these criteria. for each experiment, we evaluate the models using precision, recall, and f1, repeating each experiment three times and taking the average results. ## conclusion and future work in this paper, we develop a suite of probing tasks aimed at evaluating the critical capabilities of vlms in ocr, comprehension of formatting details, and recognition of spatial positioning within spreadsheet images. our findings demonstrate that while vlms possess strong ocr capabilities, they are prone to cell omission and prediction misalignment during ocr tasks on spreadsheet images. furthermore, their spatial perception is insufficient, as they struggle to accurately determine the row and column numbers of cells in a two-dimensional spreadsheet grid. surprisingly, vlms cannot comprehend visual formats well like humans. additionally, we introduce a spreadsheet table detection task designed to thoroughly assess the ability of vlms to interpret spreadsheet images effectively. however, the performance of this task falls short of that achieved by existing sota method, indicating that processing and comprehending spreadsheets remains a significant challenge. future research could focus on handling larger spreadsheet images and segmenting these spreadsheets without compromising the integrity of their format and spatial relationships. despite these challenges, the potential benefits of treating spread-sheets as images are substantial. in this paper, we have proposed methods that can massively generate spreadsheet-image pairs, and under our proposed settings, we can control various challenges. utilizing these methods to generate large amounts of data, we train open-source large models to enhance their understanding of structured data on grids, further advancing the comprehensive capabilities of large models towards agi. b case study b.1 a case of ocr task in order to deeply explore the impact of different reconstructed spreadsheet images on gpt-4v's ocr capabilities, we will analyze the case shown in figure. 12 in detail.