Table-To-Text generation and pre-training with T ab T 5 (2022)

Abstract: Encoder-only transformer models have been successfully applied to different table understanding tasks, as in TAPAS. A major limitation of these architectures is that they are constrained to classification-like tasks such as cell selection or entailment detection. We present TabT5, an encoder-decoder model that generates natural language text based on tables and textual inputs. TabT5 overcomes the encoder-only limitation by incorporating a decoder component and leverages the input structure with table specific embeddings and pre-training. TabT5 achieves new state-of-the-art results on several domains, including spreadsheet formula prediction with a 15% increase in sequence accuracy, QA with a 2.5% increase in sequence accuracy and data-to-text generation with a 2.5% increase in BLEU.

## introduction large language models (llms) such as bert @xcite or t5 @xcite have shown impressive abilities to encode and generate fluent and coherent natural language text @xcite @xcite . however their representation and generational capabilities are limited when it comes to structured or semi-structured domains like tables. this is mainly due to two reasons: (i) llms are only pre-trained on large amount of unstructured data (e.g., documents, news, etc.); (ii) their underlying model architecture lacks a way to fully leverage this structure information. yet, structured and semi-structured data is ubiquitous on the web (e.g. web tables, database tables, pdf tables, spreadsheets store rich numerical information and provide concise summaries of data), and widely studied in the academia (chen * corresponding authors † main advisor @xcite @xcite and the industry (e.g. formula prediction in excel foot_0 and google sheets foot_1 , or extracting data from tables in text-to-speech assistants). recently, several solutions propose to alleviate aforementioned issue by introducing pre-training or intermediate training strategies for tables. for instance, @xcite propose to use a masked language model (mlm) as pre-training objective to improve the contextual representation of bert @xcite over table inputs. to train their model, they introduce additional input embeddings that help the model understand the table structure. these pre-training models are designed and evaluated on datasets where the answers contain only table cells or aggregations of multiple cells, and not full sentences. in this paper, we tackle a set of distinct, complex tasks such as question answering and formula prediction that require full generation capabilities. in particular, our contributions are as follows: ## problem definition the objective of our model is to learn a conditional sequence generator p (y|x) where x is endowed with extra two-dimensional structure. to encode said structure, each instance of x is as a variable length sequence of tuples (u i , t i , c i , r i ) n @xmath0. in each component, table caption, table header, table cell, etc.), and c i and r i represent the two dimensional column and row coordinates for this component. this approach is general to represent the information layout in web documents and in particular tables where each table cell and each piece of metadata can map into a single component. table language models. several works use a common serialization where table contents are linearized row by row @xcite @xcite @xcite . another design choice is to use structural positional encoding, in addition to 1-d encoding, to represent two dimensional information such as the row and column positions of tokens @xcite @xcite ). an alternative is the the use of a structure-aware attention, in contrast to a standard self-attention mechanism, to better leverage the table structure @xcite . all of these models are encoder-only. concurrent with our work @xcite propose a similar method to adapt to t5 to tabular data, however their pretraining approach relies on existing annotated datasets and focuses solely on qa applications. table pre-training. most pretraining methods follow the masked language modeling (mlm) scheme, where some percentage of input tokens are randomly masked and successively predicted in an encoder only setup @xcite @xcite . some approaches @xcite @xcite apply the masking on a cell-level, where the full contents of a given cell is masked and then predicted. our work differs in training the encoder and decoder jointly by using a de-noising scheme similar to the one used in t5 @xcite . table qa . given an input table, the task consists in producing an answer to a natural language question. we focus on wikisql @xcite , and learn an encoder-decoder model with row/column embeddings in the weakly supervised setting without logical forms. herzig et al. (2020) formula prediction. the task is to predict formula conditioned on headers and other contextual information, without an explicit natural language question. @xcite propose to use a bert-based architecture to compute an input header and a cell data vector that are fed to a two-step lstm decoder. the decoder proposes a formula sketch and refines it with cell ranges. @xcite propose a similar approach where the representation of the target cell output by a table encoder @xcite is an input to a two-step lstm-based decoder. our approach is simpler as a single model is used to solve the task end to end. data-to-text. the task consists in generating a natural language description given structured data input. @xcite employ an encoderdecoder model where the encoder and decoder are both initialized with bert @xcite . kale and rastogi (2020) use a t5 model. in both, tables are linearized with row/column separator tokens. our work differs as we use row/column embeddings, and we employ two pretraining schemes. ## tabt5 model tabt5 uses the t5 pre-trained language model as a baseline architecture.we linearize the table into a sequence of words, split words into word pieces (tokens) and concatenate the question and table tokens to create the input sequence. we include in the model row and column embeddings to encode table structure @xcite . we add them on top of the token embeddings as model inputs and optimize them during training (figure 1 ). the target sequence is a free-form answer. this can be an answer to a question for question-answering tasks, a table summary when no question is specified or a formula for the formula prediction tasks. ## pre-training as a starting point, we use publicly available t5 checkpoints released by @xcite . next, we pre-train tabt5 on wikipedia tables. we use the pre-training dataset proposed by @xcite which contains 6.2m tables (3.3m of class infobox foot_2 and 2.9m of class wikitable). we also extract related passages that caption the table. we define two pre-training strategies described below. ## experiments in this section, we discuss the experiments we performed to show the effectiveness of our method. ## conclusions and discussion we introduced tabt5 a new t5-based encoderdecoder model that improves the encoding ability of tables for pre-trained sequence-to-sequence language model. tabt5 achieves new sota results on spreadsheet formula prediction, question answering with text or complex mathematical formulas, and data-to-text generation. this work opens up different paths for future work. we plan to explore different datasets for pre-training. raffel et al. show that it is beneficial for the unstructured data to train on datasets bigger than wikipedia. thus, we plan to use larger and task specific datasets for pre-training (e.g. scrape tables from web, sheets). finally, we want to extend this work to multiple languages, especially low resource ones.