A Package for Learning on Tabular and Text Data with Transformers (2021)

Abstract: Recent progress in natural language processing has led to Transformer architectures becoming the predominant model used for natural language tasks. However, in many real- world datasets, additional modalities are included which the Transformer does not directly leverage. We present Multimodal- Toolkit, an open-source Python package to incorporate text and tabular (categorical and numerical) data with Transformers for downstream applications. Our toolkit integrates well with Hugging Face’s existing API such as tokenization and the model hub which allows easy download of different pre-trained models.

## introduction in recent years, transformers @xcite have become popular for model pre-training @xcite @xcite and have yielded state-of-the-art results on many natural language processing (nlp) tasks. in addition, well-documented transformer libraries such as hugging face transformers @xcite , and allennlp @xcite have democratized nlp, making it easier to productionize and experiment on transformers. however, there are not a lot of comprehensive tools for transformers to work with tabular data. often in real-world datasets, there are tabular data as well as unstructured text data which can provide meaningful signals for the task at hand. for instance, in the small example in figure 1 , each row is a data point. columns title and review text contain text features, columns division name, class name, and department name contain categorical features, and the age column is a numerical feature. to the best of our knowledge, no tool exists that makes it simple for transformers to handle this extra modality. therefore, 1 github: https://git.io/jo5a6 2 https://huggingface.co/docs given the advances of transformers for natural language tasks and the maturity of existing transformer libraries, we introduce multimodal-toolkit, a lightweight python package built on top of hugging face transformers. our package extends existing transformers in the hugging face's transformers library to seamlessly handle structured tabular data while keeping the existing tokenization (including subword segmentation), experimental pipeline, and pre-trained model hub functionalities of hugging face transformers. we show the effectiveness of our toolkit on three real-world datasets. ## related work there have been several proposed transformer models that aim to handle text features and additional features of another modality. for pre-trained transformers on images and text, models such as vilbert @xcite and vlbert @xcite are mainly the same as the original bert model but treat the extra image modality as additional tokens to the input. these models require pre-training on multimodal image and text data. on the other hand, while treating image features as additional input tokens, mmbt @xcite proposes to use pre-trained bert directly and fine-tune on image and text data. this is similar to multimodal-toolkit in which no pre-training on text and tabular data is needed. likewise, transformers have been adapted to align, audio, visual, and text modalities in which there is a natural ground truth alignment. mult @xcite is similar to vilbert in which co-attention is used between pairs of modalities but also includes temporal convolutions so that input tokens are aware of their temporal neighbors. meanwhile, @xcite injects cross modality attention at certain transformer layers via a gating mechanism. finally, knowledge graph embeddings have also been effectively combined with input text tokens in transformers. @xcite combines knowledge graph embeddings on authors with book titles and other metadata features via simple concatenation for book genre classification. on the other hand, for more general language tasks, ernie @xcite first matches the tokens in the input text with entities in the knowledge graph. with this matching, the model fuses these embeddings to produce entity-aware text embeddings and text-aware entity embeddings. however, these models do not capture categorical and numerical data explicitly. hugging face does include lxmert @xcite to handle language and vision modality but this can not be easily adapted for categorical and numerical data. nevertheless, existing multimodal transformer models do give good insights into how to combine categorical and numerical features. vil-bert and vlbert for example include image modality as input tokens which lead to one of our simple baseline of categorical and numerical features as additional token inputs to the model. likewise, the gating mechanism @xcite , attention, and different weighting schemes have all been shown to be useful in combining different modalities. ## design the goal of multimodal-toolkit is to allow users to quickly adapt state-of-the-art transformer models for situations involving text and tabular data which occur often in real-world datasets. moreover, we want to bring the benefits of transformers to more use cases while making it simple for users there is a data processing module that outputs processed text, numerical, and categorical features that are then fed as input to our transformer with tabular module consisting of a hugging face transformer and our combining module. this design enables us to easily include more transformer models, leverage strengths of specific models, use a feature-rich training pipeline, and integrate the thousands of community trained models on hugging face's model hub. we support a variety of transformers (e.g. bert, al-bert, roberta, xlnet) for both classification and regression tasks. all together, this becomes a reusable transformer with tabular component. we also provide a data preprocessing module for categorical and numerical features. an overview of the system is shown in figure 2 . currently, the library supports pytorch transformers implementations. ## experiments in this section, we study the effectiveness of leveraging tabular features on data with text and tabular data. we evaluate multimodal-toolkit on three real-world datasets from kaggle. ## conclusion this paper presents multimodal-toolkit, an opensource python library powered by hugging face transformers to learn on data that contains both text and tabular data. we show the effectiveness of incorporating tabular data and treating it as a separate modality with the already powerful transformers. the modular design and shared api with hugging face allow users quick access to hugging face's community uploaded transformer models. for future work, we aim to include support for more transformers and integrate the combining module at earlier layers in the transformer. we hope the toolkit brings more research attention to this data scenario and we welcome open-source contributions to the project.