Revisiting Multimodal Transformers for Tabular Data with Text Fields (2024)

Abstract: Tabular data with text fields can be leveraged in applications such as financial risk assessment or medical diagnosis prediction. When employing multimodal approaches to make predictions based on these modalities, it is crucial to make the most appropriate modeling choices in terms of numerical feature encoding or fusion strategy. In this paper, we focus on multimodal classification tasks based on tabular datasets with text fields. We build on multimodal Transformers to propose the Tabular-Text Transformer (TTT), a tabular/text dual-stream Transformer network. This architecture includes a distance-to-quantile embedding scheme for numerical features and an overall attention module which concurrently considers self-attention and cross-modal attention. Further, we leverage the two well-informed modality streams to estimate whether a prediction is uncertain or not. To explain uncertainty in terms of feature values, we use a sampling-based approximation of Shapley values in a bimodal context, with two options for the value function. To show the efficacy and relevance of this approach, we compare it to six baselines and measure its ability to quantify and explain uncertainty against various methods. Our code is available at https://github.com/thomas-bonnier/TabularTextTransformer.

## introduction tabular datasets with text fields can be leveraged in various critical applications such as finance or healthcare. in financial risk assessment, numerical, categorical, and text data can be used by a classification model in order to assess companies' creditworthiness @xcite . in the medical field, clinical data and caregiver notes could be employed for diagnosis prediction. transformers with an attention mechanism have become popular by achieving state-of-the-art performance in natural language processing @xcite . in particular, multimodal transformers have been used with various modalities such as audio, language, and vision @xcite . motivation. employing transformers with multimodal tabular/text datasets requires adequate modeling choices. first, relevant encoding schemes should be used for numerical features. the traditional mapping methods used to construct embeddings for these features, e.g. linear functions @xcite , are more difficult to interpret in high-dimensional settings. numerical features are ordered and can have various distributions including extreme values. this should be expressed through the choice of a relevant embedding scheme. regarding the attention mechanism, it may be more relevant to consider the whole surroundings of an item from the input sequence rather than exclusively considering the features from the same modality or from the other modality. in other words, the attention weights may be more informative by taking into account the whole context rather than considering partial contexts (e.g. cross-modal attention only). lastly, when a multimodal model makes predictions based on unlabeled data, it is essential for subject matter experts to understand whether the predictions are trustworthy. if the uncertainty is significant for a given prediction, the source of uncertainty should be explained in terms of feature values. for instance, a data scientist could identify regions of the feature space where the model is uncertain because it is under-specified by the data. or a doctor might want to know why a diagnosis prediction is uncertain and might then override the prediction thanks to expert knowledge. perimeter and contribution. the focus is on multimodal classification tasks based on tabular datasets with text fields in english. these datasets contain numerical and categorical features (together referred to as the tabular modality here) and fields with free-form text (i.e. the text modal-ity). numerical features have continuous scalar values while categorical features have discrete values. the latter could be a finite number of unquantifiable categories (e.g. country), boolean values (e.g. true/false flag), or ordinal values (e.g. rating from 1 to 5). our mains contributions are fourfold: ## related work embeddings for tabular features. the quality of feature embeddings is key for the performance of models such as transformers. the objective is to find a map from a scalar value or a category name to an embedding vector of higher dimension. to mimic word embeddings, a categorical feature can be embedded through a look-up table . numerical features can naturally be encoded by using linear functions @xcite or piece-wise linear encoding @xcite . ## tabular-text transformer we consider a dataset @xmath0, y i )} n @xmath1, where each input @xmath2, x cat , x num ) = (x text , x tab ) ∈ x contains text fields, categorical, and numerical features. the true class is @xmath3, 2, . . . , c} for c-class classification tasks. ## experiments we empirically test the relevance of our approach on eight classification datasets. in the appendix, we provide further details on the experimental settings and results (e.g. details on datasets, data preprocessing, hyper-parameters, baselines). ## conclusion we presented the tabular-text transformer, a multimodal approach which shows promising results on classification tasks. we demonstrated how uncertainty can be quantified and explained. future work will focus on pretraining strategies for ttt.