Transformers for Tabular Data Representation: A Survey of Models and Applications (2023)

Abstract: In the last few years, the natural language processing community has witnessed advances in neural representations of free texts with transformer-based language models (LMs). Given the importance of knowledge available in tabular data, recent research efforts extend LMs by developing neural representations for structured data. In this article, we present a survey that analyzes these efforts. We first abstract the different systems according to a traditional machine learning pipeline in terms of training data, input representation, model training, and supported downstream tasks. For each aspect, we characterize and compare the proposed solutions. Finally, we discuss future work directions.

## introduction many researchers are studying how to represent tabular data with neural models for traditional and new natural language processing (nlp) and data management tasks. these models enable effective data-driven systems that go beyond the limits of traditional declarative specifications built around first order logic and sql. examples of tasks include answering queries expressed in natural language @xcite @xcite , performing fact-checking @xcite @xcite , doing semantic parsing @xcite , retrieving relevant tables @xcite @xcite , understanding tables @xcite , and predicting table content @xcite . indeed, tabular data contain an extensive amount of knowledge, necessary in a multitude of tasks, such as business @xcite and medical operations @xcite , hence, the importance of developing table representations. given the success of transformers in developing pre-trained language models (lms) @xcite , we focus our analysis on the extension of this architecture for producing representations of tabular data. such architectures, based on the attention mechanism, have proven to be successful as well on visual @xcite , audio @xcite , and time series data @xcite . indeed, pre-trained lms are very versatile, as demonstrated by the large number of tasks that practitioners solve by using these models with fine-tuning, such as improving arabic opinion and emotion mining @xcite @xcite @xcite . recent work shows that a similar pre-training strategy leads to successful results when language models are developed for tabular data. 1 as depicted in figure 1 , this survey covers both (1) the transformer-based encoder for pre-training neural representations of tabular data and (2) the target models that use the resulting lm to address downstream tasks. for (1), the training data consist of a large corpus of tables. once the representation for this corpus has been learned, it can be used in (2) for a target task on a given (unseen at pre-training) table, such as the population table, along with its context, namely, relevant text information such as table header and captions. in most cases, the lm obtained from pre-training with large datasets in (1) is used in (2) by fine-tuning the lm with a labeled downstream dataset, for example, a table with cells annotated as the answer for the question in the context. extensions on the typical transformer architecture are applied to account for the tabular structure, which is different and richer in some aspects than traditional free text. figure 1 : the overall framework for developing and consuming neural representations for tabular data. wikipedia tables or wdc web table corpus are typically used in the pre-training step (1). in (2), a table along with its context (header, the additional question and the highlighted answer) are used for a question answering downstream task. both processes combine the serialized table data with natural language text in the context. while all proposed solutions make contributions to the neural representation of tabular data, there is still no systematic study to compare those representations given the different assumptions and target tasks. in this work, we aim to bring clarity in this space and to provide suitable axes for a classification that highlights the main trends and enables future work to clearly position new results. our contributions are threefold: we conclude the survey with a discussion of the limitations of existing works and future research directions (section 8). related surveys. some recent surveys cover the use of deep learning on tabular data @xcite @xcite . these surveys are different from ours in multiple aspects. first, they investigate how deep learning models (including non-transformer-based) compare against classical ml models (mainly gradient boosted trees) by providing empirical results with lack of details over the changes to the model, while our work taxonomizes transformer-based models in depth. second, other works focus on standard classification and regression tasks where the term ''tabular data'' is used for labeled training data, representing data points with columns consisting of features. every row is an input to the model together with a label (e.g., predicting house prices) @xcite . in contrast, we do not consider tabular data necessarily as a training dataset with features and labels, but rather as an input containing information needed for prediction by a target model, such as question answering on tables. for this reason, while they focus on data generation and classification, we relate design choices and contributions to a large set of downstream tasks. a recent survey @xcite covers approaches for table pre-training and usage of lms in downstream tasks with an overlap in terms of surveyed works. in contrast with their work, we provide a more detailed study of the extensions to the transformer architecture and analyze in detail the relevant datasets' properties and pre-processing steps. ## terminology and overview we focus on tabular data that come in the form of a table, as depicted in the two examples in an entity table has the same properties, but with a horizontal structure with only one entity whose properties are organized as rows. spreadsheets, or matrix tables, are the most general kind with information that can be organized both horizontally and vertically, possibly with hierarchies in the header and formatting metadata, such as in financial tables. tables can have rich metadata, such as attribute types (e.g., date), domain constraints, functional dependencies across columns and integrity constraints such as primary keys. in the table sample in figure 1 , column country is a primary key. most systems focus on a single table, with or without metadata. however, a few systems, such as gtr @xcite , grappa @xcite , and dtr @xcite , consume databases, which are collections of relational tables, possibly under referential constraints. we identify as input the table(s) and its context. the context is a text associated to the table. depending on the dataset and task at hand, it varies from table metadata, the text surrounding the tables or their captions up to questions, expressed in natural language, that can be answered with the tabular data @xcite . the main advantage of the transformer architecture is its ability to generate a lm, a large neural network, with self-supervised pre-training. this pre-trained lm is then usually followed by supervised fine-tuning to adapt it to the target task with a small amount of training data. while transformers have proven to be effective in modeling textual content, tables have a rich structure that comes with its own relationships, such as those across values in the rows and attributes. new solutions are therefore needed to jointly model the characteristics of the table, its text content, and the text in the table context. as shown in figure 1 , we distinguish two main phases to spell out these contributions. first, we focus on the development of tabular lms by using transformer-based deep neural networks (1). given a table and its context, the goal is to learn a pre-trained representation of the structured data (cell values, rows, attributes) in a continuous vector space. we then discuss the use of those representations in the downstream tasks (2). figure 1 shows the reference pipeline and the aspects that we propose to model existing systems. table based fact-checking (tfc), question answering (qa), semantic parsing (sp), table retrieval (tr), table metadata prediction (tmp), and table content population (tcp). ## training datasets we present both the datasets used for pre-training and for fine-tuning in the downstream tasks. pre-training tables are not annotated, in some cases scraped from the web, while data used for fine-tuning have task-dependent annotation labels. the datasets consist of tables and their context, such as table metadata, surrounding texts, claims or questions. to construct large pre-training datasets and in an attempt to reduce bias, multiple sources can be used, independently of the target task at hand. for instance, unlike tapas @xcite , which only uses wikipedia tables for qa, and tabularnet @xcite , which only uses spreadsheets for tmp, tabert @xcite uses wikipedia tables and wdc for sp; grappa uses wikipedia tables, spider, and wikisql for sp; mmr @xcite uses nq, ott-qa, and wikisql for tr; mate @xcite uses wikipedia tables and hybridqa for qa; and tuta @xcite uses wikipedia tables, wdc, and spreadsheets for tmp. in general, it is recommended to utilize different data sources for pre-training to ensure covering different kinds and content, and thus, improve the scope of representations. for instance, wikipedia tables has a large number of relational tables @xcite , while wdc and spreadsheets include also entity tables and spreadsheets with complex structure. ## input processing as for the original text setting, transformers for tabular data require as input a sequence of tokens. however, in addition to the typical tokenization executed before feeding the text to the neural network @xcite , tabular data requires some steps to be processed correctly. some requirements come from the nature of the transformer, such as the limitation on the input data size (section 4.1). other requirements are due to the nature of the tabular data, with structural information expressed in two dimensions that need to be converted into a one dimensional space (section 4.2). finally, given a table, its data and its context must be jointly fed to the transformer (section 4.3). ## adaptation of transformers to account for structured tables in the input, several pre-trained transformer-based lm and systems have been developed. vanilla lms are customized to make the model more ''data structure-aware'', thus rendering a modified transformer-based encoder to be utilized on other tasks. these encoders, depicted in part (1) of figure 1 , capture both structure and semantic information. to exploit such resources and build applications, as in part (2) of figure 1 , several systems build on top of the encoder, usually with more modules and fine-tuning. in a different approach, other systems use the encoder as part of a bigger architecture in a more task-oriented fashion rather than encoder-oriented. we first briefly revise the vanilla transformer architecture, then discuss customizations to lms. ## using the language model the initial neural representations of tabular data are the result of the pre-training. systems use the output lm in different ways. it can be fine-tuned or used ''as-is''-for example, by using its representations as features in traditional ml algorithms (section 7.1). however, as pre-trained lms can act as encoders of the input, they are also used as a module in bigger systems (section 7.2). ## future directions tabular lms effectively address some of the challenges that arise with classical ml models @xcite , such as transfer learning and self-supervision. however, several challenges remain unaddressed. interpretability. only a few systems expose a justification of their model output, for example, tapas, cltr, and mate, thus model usage remains a black box. one direction is to use the attention mechanism to derive interpretations @xcite . looking at self-attention weights of particular layers and layer-wise propagation with respect to the input tokens, we can capture the influence of each cell value/tuple on the output through backpropagation @xcite . for instance, in tfc, providing explanation is crucial when the decision is derived from aggregating several cells, such as sum or average operation. in this case, a basic explanation would be to show all the cells that led to the final true/false decision. ## conclusion we conducted a survey on the efforts in developing transformer-based representations for tabular data. we introduced a high level framework to categorize those efforts and characterized each step in terms of solutions to model structured data, with special attention to the extensions to the transformer architecture. as future work, we envision a generic system to perform an experimental study based on our framework. the first part of the system would develop tabular data representations with alternative design choices, while the second part would evaluate them in downstream tasks. this work would help identifying the impact of alternative techniques on the performance in the final applications.