U ni T ab N et: Bridging Vision and Language Models for Enhanced Table Structure Recognition (2024)

Abstract: In the digital era, table structure recognition technology is a critical tool for processing and analyzing large volumes of tabular data. Previous methods primarily focus on visual aspects of table structure recovery but often fail to effectively comprehend the textual semantics within tables, particularly for descriptive textual cells. In this paper, we introduce UniTabNet, a novel framework for table structure parsing based on the image-to-text model. UniTabNet employs a “divide-and-conquer” strategy, utilizing an image-to-text model to decouple table cells and integrating both physical and logical decoders to reconstruct the complete table structure. We further enhance our framework with the Vision Guider, which directs the model’s focus towards pertinent areas, thereby boosting prediction accuracy. Additionally, we introduce the Language Guider to refine the model’s capability to understand textual semantics in table images. Evaluated on prominent table structure datasets such as PubTabNet, PubTables1M, WTW, and iFLYTAB, UniTabNet achieves a new state-of-the-art performance, demonstrating the efficacy of our approach. The code will also be made publicly available.

## introduction in this era of knowledge and information, documents serve as crucial repositories for various cognitive processes, including the creation of knowledge databases, optical character recognition (ocr), and document retrieval. among the various document elements, tabular structures are particularly notable. these structures distill complex information into a concise format, playing a pivotal role in fields such as finance, administration, research, and archival management @xcite . table structure recognition (tsr) focuses structure recognition method semv2 @xcite . (c) presents the ground truth label for table structure. the red dashed box highlights the discrepancy between the prediction and the ground truth label. on converting these tabular structures into machinereadable data, facilitating their interpretation and utilization. therefore, tsr as a precursor to contextual document understanding will be beneficial in a wide range of applications @xcite . table images efficiently convey information through visual clues, layout structures, and plain text. however, most previous methods @xcite @xcite in tsr primarily utilize visual or spatial features, neglecting the textual content within each table cell. the structures of some tables exhibit inherent ambiguities when assessed solely based on visual appearance, especially for wireless tables which contain cells with descriptive content, as illustrated in figure 1. to enhance accuracy in tsr, it is crucial to leverage the cross-modality characteristics of visually-rich table images by jointly modeling both visual and textual information @xcite . recent advancements in document understanding, exemplified by methods such as donut @xcite and pix2struct @xcite , have embraced an end-to-end image-to-text paradigm. these approaches leverage the transformer architecture @xcite during pre-training to decode ocr results, demonstrating superior perception of image content. by diminishing reliance on traditional ocr engines, they exhibit remarkable adaptability across diverse document understanding tasks, highlighting their robust ability to comprehend text embedded in images. despite these advancements, the application of this framework to tsr remains unexplored. while there are related works @xcite that employ this framework, they primarily focus on reconstructing table structures from a visual perspective, without adequately addressing the depth of textual understanding in images. in this work, we adopt the image-to-text framework and introduce a visually linguistic unified model for tsr, named unitabnet. this model is built on a "divide and conquer" design philosophy, initially using the image-to-text model to decouple table cells. according to the attributes of the table structure @xcite , the decoupled cells contain two types of attributes: logical and physical. the logical attributes cover the row and column span information of each cell, while the physical attributes include the bounding box coordinates of the cells. to parse these attributes independently, we design a logical decoder and a physical decoder. since table images differ significantly from regular document images, each step of the decoding output is grounded in a clear visual basis, specifically visual cues from rows, columns, and cells. therefore, we design a vision guider module, which directs the model to focus on relevant areas and make more precise predictions. furthermore, to enhance the unitabnet's understanding of text content in images, we develop a language guider. this module enables the model to perceive the corresponding text content at each decoding step, thereby understanding the textual semantics within the image. experimental results on multiple public tsr datasets, such as pubta-bles1m @xcite , pubtabnet @xcite , iflytab @xcite , and wtw @xcite , demonstrate that our approach achieves state-of-the-art performance, validating the effectiveness of our method. the main contributions of this paper are as follows: ## related work due to the rapid development of deep learning in documents, many deep learning-based tsr approaches have been presented. these methods can be roughly divided into three categories: bottomup methods, split-and-merge based methods and image-to-text based methods. one group of bottom-up methods @xcite @xcite treat words or cell contents as nodes in a graph and use graph neural networks to predict whether each sampled node pair belongs to the same cell, row, or column. these methods depend on the availability of bounding boxes for words or cell contents as additional inputs, which are challenging to obtain directly from table images. to eliminate this assumption, another group of methods @xcite has proposed directly detecting the bounding boxes of table cells. after cell detection, they design some rules to cluster cells into rows and columns. however, these methods regard the cells as bounding box, which is difficult to handle the cells in distorted tables. other methods @xcite detect cells through detecting the corner points of cells, making them more suitable for handling distorted cells. nevertheless, they suffer from tables containing a lot of empty cells and wireless tables. split-and-merge based methods initially split a table into basic grid pattern, followed by a merging process to reconstruct the table cells. previous methods @xcite utilize semantic segmentation @xcite for identifying rows, columns within tables in the "split" stage. however, segmenting table row/column separation lines in a pixel-wise manner is inaccurate due to the limited receptive field, and heuristic mask-to-line modules designed with strong assumptions in split stage make these methods work only on tables in digital documents. to enhance the accuracy of grid splitting in distorted tables, robusttabnet @xcite uses a spatial cnn-based separation line predictor to propagate contextual information across the entire table image in both horizontal and vertical directions. semv2 @xcite formulates the table separation line detection as the instance segmentation task. the table separation line can be accurately obtained by processing the table separation line mask in a row-wise/column-wise manner. tsrformer with sepretr @xcite formulates the table separation line prediction as a line regression problem and regresses separation line by detr @xcite , but it can't regress too long separation line well. tsrformer with dq-detr @xcite progressively regresses separation lines, which further enhances localization accuracy for distorted tables. image-to-text based methods conceptualize the structure of tables as sequential data (html or latex), utilizing an end-to-end image-to-text paradigm to decode table structures. the edd model @xcite ) employs an encoderdual-decoder architecture to generate both the logical structure and the cell content. during the decoding phase, edd utilizes two attention-based recurrent neural networks; one is tasked with decoding the structural code of the table, while the other decodes the content. building on this framework, tableformer @xcite employs a transformer-based decoder to enhance the capabilities of edd's decoder. additionally, it introduces a regression decoder that predicts bounding boxes rather than content, thus refining the focus on spatial elements. addressing the challenge of limited local visual cues, vast @xcite redefines bounding box prediction as a coordinate sequence generation task and incorporates a visual alignment loss to achieve more accurate bounding box outcomes. ## task definition as illustrated in figure 2 , given a table image @xmath0, our objective is to enable the model to predict the table structure sequence @xmath1, . . . , t }, where t is the length of the sequence and v is the the size of token vocabulary, to reconstruct the table's layout. previous methods @xcite @xcite have employed various formats for the output table structure sequence s, such as html and latex. in contrast, our approach simplifies the decoding process significantly by using only two types of tokens: <@xmath2. <@xmath3, @xmath4, facilitating a concise representation of the table structure. according to the attributes of the table structure @xcite , each table cell encompasses both logical attribute @xmath5, l col | l row , l @xmath6, . . . , 8}. the logical attribute l specifies the cell's span across rows and columns, while the physical attribute p defines the spatial positioning of the cell within the image. consequently, the output of our proposed model, unitabnet, includes the structure sequence s, along with logical attributes @xmath7, . . . , t and physical attributes @xmath8, . . . , t , providing a comprehensive description of the table's layout. additionally design a logical decoder and a physical decoder to predict the cell attributes l and p, respectively. considering the nature of table images, we incorporate a vision guider and a language guider at the output of the text decoder. the vision guider directs the model to focus on relevant areas during cell decoding, while the language guider aids in understanding the corresponding textual information within the cells. detailed descriptions of these modules will follow. vision encoder. the vision encoder converts the table image i into a set of embeddings @xmath9, . . . , n }, where n is feature map size and d is the dimension of the latent vectors of the encoder. as depicted in figure 3 , we adopt the swin transformer @xcite as our primary vision backbone, following the donut, to encode i into feature map f . additionally, we incorporate positional encoding @xcite into f to generate the final vision embeddings z. text decoder. similar to donut, we utilize the bart @xcite decoder to generate the table structure sequence s, conditioned on the z. since unitabnet is trained to predict the next tokens like llms (openai, 2023), it only requires maximizing the likelihood of loss at training time. physical decoder. given the output @xmath10, . . . , t } from the last layer of the text decoder, the physical decoder decodes these hidden states to obtain the polygon coordinates p i in the image. to facilitate this prediction, we introduce a set of 1,000 special tokens-<0>, <1>, ..., <999>-which are utilized for quantizing the coordinates of the polygons, forming a specialized vocabulary @xmath11. specifically, for each coordinate point p j in the polygon p i , the prediction process is as follows: the corresponding hidden state h i is transformed via a linear mapping to produce the h p j i , which serves as a query against the vocabulary loc. unlike previous method @xcite , which perform direct classification over the location vocabulary, we define the final position of p j as the expected location based on the distribution given by h the polygon regression loss is defined as follows: given the extreme imbalance in the distribution of rowspan and colspan across cells, we optimize our model using sigmoid focal loss @xcite . the span prediction loss for the logical decoder is defined as follows: the loss function for the vision guider is defined as: our methodology employs the following hyperparameters: the longest side of the image is resized to 1600 while maintaining the original aspect ratio. the downsampling factor of the visual backbone is set to 32. the dimension d of the feature is set to 1024. the decoders consist of a stack of 4 identical layers, and the number of multi-heads is set to 16. training. to train unitabnet, we design three training tasks as depicted in figure 4 . these tasks aim to enable the model to comprehensively perceive tabular images. specifically, the training process is divided into two phases. initially, during the pre-training phase, we use a synthetic dataset comprising 1.4 million chinese and english entries from synthdog @xcite , along with the training set from pubtables1m @xcite . after pre-training, the model is fine-tuned on specialized datasets dedicated to table structure recognition. we fine-tune unitabnet using the adam (kingma and ba, 2015) optimizer with the learning rate of 5 × 10 -5 .the learning rate is linearly warmed up over the first 10% steps then linearly decayed. the training is conducted on 8 telsa a40 48gb gpus. the model is trained for 100 epochs on the iflytab @xcite and wtw @xcite datasets, and for 10 epochs on the pubtables1m and pubtab-net @xcite datasets. in the overall loss of unitabnet, there are primarily two categories: regression losses (l poly , l lang ) and classification losses (l lm , l span , l vis ). given the significant scale differences among these losses, it is necessary to adjust their coefficients. inspired by @xcite , we optimize the model by maximising the gaussian likelihood with homoscedastic uncertainty. the σ is a learnable factor that adaptively adjusts the weight ratios among these losses. l k represents the five losses mentioned above. inference. during the inference phase, we feed @xmath12