T able VLM : Multi-modal Pre-training for Table Structure Recognition (2023)

Abstract: Tables are widely used in research and business, which are suitable for human consumption, but not easily machine-processable, particularly when tables are present in images. One of the main challenges to extracting data from images of tables is accurately recognizing table structures, especially for complex tables with cross rows and columns. In this study, we propose a novel multi-modal pre-training model for table structure recognition, named TableVLM.With a two-stream multi-modal transformer-based encoder-decoder architecture, TableVLM learns to capture rich table structure-related features by multiple carefully-designed unsupervised objectives inspired by the notion of masked visual-language modeling. To pre-train this model, we also created a dataset, called ComplexTable, which consists of 1,000K samples to be released publicly. Experiment results show that the model built on pre-trained TableVLM can improve the performance up to 1.97% in tree-editing-distance-score on ComplexTable.

## introduction tables are quite useful for displaying data in an organized manner and they are widely used in research and business due to their readability and simplicity. recently, such semi-structured (tabular) data has attracted more attention because of its ubiquitous presence in almost all types of documents such as medical records, insurance files, and scientific articles @xcite . however, in many cases, we can only access images of tabular data. the format information will be lost if a table is turned into an image. it is very hard to recover the structure of tables from their images because tables differ significantly in structure, notation, and representation. once the table structure is accurately recognized, its texts can be easily extracted with the help of optical character recognition (ocr) toolkit and reorganized into a ta-(a) an example of table image with multi-column headers, multi-row headers and some missing dividing lines. tables vary greatly in structure and style, which seriously hinders the machine from accurately recognizing their structures. tabular data is typically organized in rows and columns, but possibly in a more complex structure. tables may contain multi-row and multi-column cells or their combinations @xcite . certain styles might be applied by intentionally removing some horizontal or vertical dividing lines, using non-standard spacing and different text formatting @xcite . the diversity and complexity in the table's structure and presentation pose a major challenge for recovering the structures of tables from their images. a couple of methods have been proposed to address this challenge by applying the recent deep neural architectures, including graph neural networks (gnns) @xcite and transformers @xcite , to image-based table structure recognition task @xcite @xcite . however, these methods still perform unsatisfactory, especially when encountering tables with more complex structures. for example, we show in figure 1 some mistakes made by pdflux and tabby @xcite , two representative table recognition toolkits. pdflux fails to recognize the multi-column header of "parental illness type (pit)", and tabby can not arrange all the headers as they were presented in the original image. such typical mistakes were also commonly observed when applying other table structure recognition models to similar tables. in this study, we explore the feasibility of pretraining a multi-modal model particularly designed for table structure recognition. in order to improve the recognition accuracy for tables with complex structures, two new pre-training tasks (or objectives) are introduced: prediction for column headers, and prediction for the relative position of texts, in addition to existing masked image modeling, text-image matching and text-image alignment tasks. observing that there are no datasets that include a large number of complex tables, we created a new dataset, named complextable, consisting of over 1, 000k tables and their images, ranging from tables in scientific articles to those in financial reports. based on the proposed training methods and the created dataset, we developed a pre-trained multi-modal model, named tablevlm (table visual language model). through extensive experimentation, we show that tablevlm pretrained on complextable dataset with the newlyintroduced training objectives and fine-tuned afterward achieved the highest accuracy in the table structure recognition across multiple datasets. our contributions of this study are summarized as follows: ## multi-modal pre-training scheme in the following, we first present the architecture of tablevlm. then, we depict our introduced embedding layer and proposed pre-training tasks. finally, our pre-training method is described. ## the complextable dataset the scarcity of comprehensive and intricate publicly accessible datasets stands out as a significant barrier that impedes the advancement of table structure recognition. previous studies have typically required manual annotation of such datasets, yet the limited number of tables available is insufficient for training a large-scale model capable of effectively handling complex table structures. for example, @xcite collected a dataset comprising only 2000 tables extracted from a diverse array of subject-specific e-books, encompassing over 120 sources. similarly, the icdar 2013 dataset @xcite encompasses a total of 67 englishlanguage pdf documents spanning 238 pages. the primary rationale behind this scarcity stems from the arduous, expensive and time-intensive process of manual annotation. in recent years, the introduction of tablebank @xcite has led to the emergence of numerous large-scale datasets for table structure recognition @xcite @xcite . however, a predominant focus of these datasets lies in scientific tables. for instance, tablex @xcite was meticulously constructed by preprocessing and postprocessing la-tex code derived from articles on arxiv. similarly, scitsr @xcite was also generated from latex source files. consequently, the table styles present in these datasets often exhibit similarities, rendering them challenging to apply to other domains such as finance. moreover, these datasets lack the richness and complexity necessary to accurately simulate real-world intricate table structures. in this study, we present our newly developed large-scale dataset for tabular structure recognition, named complextable. this dataset is synthetically generated using our auto html table creator, which generates table images along with corresponding structured html code. the com-plextable dataset comprises over 1, 000k tables, provided as annotated png images, with annotations representing the table structure in html format. similar to the approach adopted in syn-thtabnet @xcite , we classify tables as either "simple" or "complex." a table is considered "simple" if it lacks multi-column or multi-row cells; otherwise, it is classified as "complex." notably, compared to synthtabnet, complextable exhibits a significantly higher proportion of complex tables, and the variety of table styles within the dataset is more diverse. for a detailed comparison, please refer to in order to construct a dataset that encompasses greater complexity and stylistic diversity, we implemented the following procedures. firstly, we developed a wide array of style templates to encompass a broad spectrum of table appearances. these templates drew inspiration from various realworld sources, including scientific journals, financial statements, and general tables, among others. recognizing that spanning cells often pose challenges for accurate table structure identification by models, we deliberately increased the proportion of complex tables in our dataset. specifically, 75% of the tables in complextable contain merged cells. in certain instances, extreme table cells span five rows and five columns simultaneously. following the creation of table structures, we populate the table cells with purely random text. notably, to augment difficulty and complexity, some cell contents entail lengthy text that requires display across multiple lines. a style is randomly assigned to format the appearance of the synthesized table. finally, to generate complete tables, we employ a web browser engine, which renders the table image. ## conclusions in this study, we present tablevlm, a pre-trained multi-modal model particularly designed for recognizing the structures of complex tables from their images. a task-specific pre-training scheme with three new pre-training tasks has been proposed for training tablevlm, and the pre-training scheme