P ix T 3: Pixel-based Table-To-Text Generation (2024)

Abstract: Table-to-text generation involves generating appropriate textual descriptions given structured tabular data. It has attracted increasing attention in recent years thanks to the popularity of neural network models and the availability of large-scale datasets. A common feature across existing methods is their treatment of the input as a string, i.e., by employing linearization techniques that do not always preserve information in the table, are verbose, and lack space efficiency. We propose to rethink data-to-text generation as a visual recognition task, removing the need for rendering the input in a string format. We present PixT3, a multimodal table-to-text model that overcomes the challenges of linearization and input size limitations encountered by existing models. PixT3 is trained with a new self-supervised learning objective to reinforce table structure awareness and is applicable to open-ended and controlled generation settings. Experiments on the ToTTo and Logic2Text benchmarks show that PixT3 is competitive and, in some settings, superior to generators that operate solely on text.

## introduction generating text from structured inputs such as tables, tuples, or graphs, is commonly referred to as data-to-text generation @xcite @xcite . this umbrella term includes several tasks ranging from generating sport summaries based on boxscore statistics @xcite , to producing fun facts from superlative wikipedia tables @xcite , and creating textual descriptions given biographical data @xcite . from a modeling perspective, data-to-text generation is challenging as it is not immediately obvious how to best describe the given input. for instance, the table in figure 1 can be verbalized in different ways, depending on the specific content we choose to focus on. in controlled data-to-text generation @xcite , models are expected to generate descriptions for pre-selected parts of the input (see the highlighted cells in figure 1 ). regardless of the generation setting, numerous approaches have emerged in recent years with different characteristics. a few exploit the structural information of the input @xcite @xcite , use neural templates @xcite , or resort to content planning @xcite . while others @xcite @xcite improve on fluency and generalization by leveraging large-scale pre-trained language models @xcite . a common feature across these methods is their treatment of tabular input as a string, following various linearization methods. as an example, figure 1 shows the representation of tabular data (top) as a sequence of (column, row, value) tuples (bottom). problematically, representing tabular information as a linear sequence results in a verbose representation that often exceeds the context window limit of popular transformer models @xcite . the challenge of processing such long sequences has fostered the development of even more controlled methods which refrain from encoding the table as a whole, concentrating exclusively on highlighted content (e.g., only the yellow cells in figure 1 ). unfortunately, models trained on abridged input have difficulty generalizing to new domains while being practically ineffective in scenarios where content selection is not provided. in this paper we propose to rethink data-to-text generation as a visual recognition task, allowing us to represent and preserve tabular information compactly. vision transformers (vits; @xcite have significantly advanced table title: shuttle america section title: fleet aircraft total orders passengers operated for notes f y+ y embraer e170 5 -6 16 48 70 united express transferred to republic airline 14 -9 12 69 delta connection delta shuttle 2 planes on wet lease from republic airline embraer e175 15 -12 12 52 76 total 35 -linearized table: <@xmath0. . . . . . target description: shuttle america operated the e-170 and the larger e-175 aircraft for delta air lines. -to-text generation taken from the totto dataset @xcite . in the controlled setting, a natural language description is generated only for highlighted (yellow) cells. the table is linearized by encoding each value as a (column, row, value) tuple. we only show the first row, for the sake of brevity. the field of visual language understanding @xcite demonstrating proficiency in various tasks, including language modeling @xcite , visual document understanding @xcite , and visual question answering @xcite . our work builds on pix2struct @xcite , a pretrained image-to-text model which can be fine-tuned for visually-situated language tasks. we recast datato-text generation as an image-to-text problem and present pixt3, a pixel-based table-to-text model, which is generally applicable to open-ended and controlled generation settings, overcoming the challenges of linearization and input size limitations encountered by existing models. our contributions can be summarized as follows: (a) we introduce the first pixel-based model for table-to-text generation and showcase its robustness across generation settings with varying table sizes; (b) we propose a new training curriculum and self-supervised learning objective to reinforce table structure awareness; (c) automatic and human evaluation results on the totto benchmark @xcite show that pixt3 excels in open-ended generation, leading to improved faithfulness and generation quality, while being competitive with existing methods in controlled scenarios; and (d) we present a new dataset based on logic2text @xcite , which allows us to evaluate generalization capabilities of current table-to-text models. ## related work the bulk of previous work treats tables as textual objects. several techniques have been developed to extract accurate information from them @xcite using templates @xcite , enforcing table structure awareness @xcite , applying contrastive learning @xcite or focusing on content planning @xcite . other techniques @xcite @xcite improve fluency and generalization by leveraging large-scale pretrained language models @xcite . tables are generally linearized, even when special-purpose techniques are developed for encoding table structure @xcite . dedicated table understanding techniques @xcite eschew linearization but have not been integrated with generation tasks. previous attempts to address table-to-text generation from a visual recognition perspective @xcite have relied on ocr methods which first extract text from the image and then feed it as a string to a generation model. aside from being noisy, these techniques typically embrace a text-centric point of view, treating the image as a limitation rather than an informative modality. our work builds on recent visual language understanding models @xcite @xcite which are based exclusively on pixels and have managed to outperform ocr methods in several natural language processing tasks @xcite @xcite . ## problem formulation the task of table-to-text generation aims to take a structured table t as input and output a natural language description @xmath1, . . . , y k ] where k is the length of the description. table t is typically reformatted as a sequence of textual records @xmath2,1 , t 1,2 , . . . , t i,j , . . . , t m,n ] where m and n respectively denote the number of rows and columns of t. we approach this task from a visual recognition perspective, and expect the input table to be an image x. the image is reshaped into a sequence of patches analogous to linguistic tokens. more formally, for an input image @xmath3, we create n image patches denoted as x @xmath4. (h, w ) is the resolution of the original image, c is the number of channels, (p, p ) is the resolution of each image patch, and @xmath5, which serves effectively as the input sequence length. our proposed model learns to autoregressively estimate the conditional probability of a text sequence from title: jane doe 2d absolute posi.onal embedding layer vit encoder transformer decoder pixt3 in 2015, jane odyssey as m title: jane d pixt3 2d abso in 2015, jane doe starred in the american odyssey as maya decker. title: jane doe 2d absolute posi.onal embedding layer vit encoder transformer decoder ## results pixt3 is the best performing model in loosely controlled and open-ended generation settings. we first discuss our results on the tightly controlled generation setting (tcontrol) where models are not given the full table, just the highlighted cells. we would not expect pixt3 to excel at this setting, which is better suited to text-to-text models (highlighted cells make for non-descriptive images, see appendix b, figure 6 ). pixt3 is indeed unable to outperform cont, lattice, and related t5 variants, falling 3.5 bleu points behind on the development set and 3.7 on the test set. however, lcontrol, the loosely controlled generation setting, better showcases the advantages of pixt3, which in this case demonstrates almost a two times improvement over cont and t5 models. performance degrades drastically for all systems in the open-ended setting (opene) which is challenging; models are expected to perform content selection in addition to text generation, and could produce table descriptions which are valid but different from the reference. automatic metrics based on n-gram overlap are particularly punitive in this case. nevertheless, pixt3 is superior to cont, lattice, and t5 across evaluation metrics. pixt3 generalizes to out-of-domain tables which require reasoning skills. we next evaluate whether pixt3 generalizes to unseen tables, outside totto's distribution. the three generation settings. compared to totto, logic2text is a more challenging dataset as most descriptions rely on reasoning over the entire table. this results in poor model performance in the tcontrol setting which does not include the table as input. nonetheless, we observe that pixt3 excels at the lcontrol setting, even though it has to process and reason over the entire table. the opene setting is challenging for all models as they are asked to identify interesting cells to talk about in out-of-domain tables. pixt3 still maintains an edge over t5 and llava, performing on par with cont and lattice. we observe that llava cannot match the performance of pixt3 and t5-based models. this underscores the importance of task-specific fine-tuning over parameter size. we present output examples in appendix e. ## conclusion avenues for future research are many and varied. there are several downstream tasks which stand to benefit from a pixel-based view of textual information, including multilingual table-to-text generation, and semantic parsing. we would also like to investigate additional objectives and inductive biases that can better capture the structure of tables and inter-cell dependencies.