Few-Shot Tabular Data Enrichment Using Fine-Tuned Transformer Architectures (2022)

Abstract: The enrichment of tabular datasets using external sources has gained significant attention in recent years. Existing solutions, however, either ignore external unstructured data completely or devise dataset-specific solutions. In this study we proposed Few-Shot Transformer based Enrichment (FeSTE), a generic and robust framework for the enrichment of tabular datasets using unstructured data. By training over multiple datasets, our approach is able to develop generic models that can be applied to additional datasets with minimal training (i.e., few-shot). Our approach is based on an adaptation of BERT, for which we present a novel fine-tuning approach that reformulates the tuples of the datasets as sentences. Our evaluation, conducted on 17 datasets, shows that FeSTE is able to generate high quality features and significantly outperform existing fine-tuning solutions.

## introduction tabular data is the most diverse format of data representation, spanning domains from nutrition to banking. it does, however, suffer from a lack of contextual information that could make its analysis more effective. data scientists seek to overcome this limitation by using feature engineering (fe), which involves applying transformations on existing features to create additional representations of the data. when the available data is not sufficiently diverse (or when additional improvement is sought), one may attempt to use external information sources to enrich the data. we refer to this process as external enrichment of datasets (eed). the use of external sources for feature engineering is both computationally-heavy and time consuming. the process first involves matching entities in the data to those in the external source, a process known as entity linking @xcite . once entities in the external source have been matched, candidate features need to be generated, evaluated, and finally integrated into the tabular dataset. while multiple studies in recent years @xcite @xcite @xcite have sought to automate the eed process, a large majority focuses solely on structured external sources, e.g., dbpedia tables, and do not attempt to use the large amounts of available unstructured data (i.e., free text). in this study, we present few-shot transformer based enrichment (feste) a generic and robust framework for the enrichment of tabular datasets using unstructured data. our approach utilizes transformer-based, pre-trained language models (lm) @xcite to identify and prioritize promising candidate features in the external data source. feste then applies a novel process of analyzing the relationships between the unstructured features and the dataset's target class values, and automatically generating new tabular features. to overcome the difficulty imposed by datasets of limited size, we train feste on multiple datasets in order to create a generic model that can later be applied to additional datasets. additionally, we propose a novel fine-tuning (ft) process that enables pre-trained lm to quickly adapt to new datasets (i.e., perform few-shot learning). the result of this process is a more robust model that is also more effective on small datasets. while previous studies-tapas @xcite , tabert @xcite , turl @xcite , and tpn @xcite -have attempted to use transformers for analyzing tabular data, feste focuses on analyzing the connection between external texts and the dataset's entities. we are therefore able to leverage the transformer architecture to generate additional features and finetune the generation process in a novel way. we evaluate feste on 17 tabular datasets with diverse characteristics (number of samples, feature composition, etc.). for our evaluation, we use bert as the transformer architecture and wikipedia as the external source, with its page abstracts as our unstructured texts. our results show that feste outperforms existing bert finetuning strategies and that feste is highly effective, achieving an average improvement of 9.2% when combined with the datasets' original features. finally, we show feste performs well even when it is applied on its own (without any original features), achieving an average auc of 0.664. to summarize, our contributions in this study are as follows: ## problem definition for our task of feature generation from the free text of external sources, we assume a target tabular dataset d t with a classification tasks. additionally, we assume a set of pre-analyzed tabular datasets with different classification tasks (i.e. different number of classes) @xmath0...d n }. for each dataset, let there be target class values tc i and original features f i . of f i , let there be at least one feature representing entities e @xmath1,1 ...e i,m }. for the purpose of generating new features, we assume an external data source ex which consists of entities e ex and text related to these entities. we denote this set of texts as t . for the purpose of linking e i and e ex , we assume an entity linking function γ. we generate a set of new features f new t from t using a language model lm . ## the proposed method overview. our proposed approach is presented in figure 1 and algorithm 1. feste consists of three main phases: a) entity linking; b) fine-tuning, and; c) features generation. in the entity linking phase, feste automatically matches entity names in the tabular dataset to their corresponding entries in the external data source. in the fine-tuning phase, we fine-tune a pre-trained lm for the task of feature generation. this phase consists of two stages: a preliminary stage where we fine-tune the model "offline" on multiple datasets, and an "online" stage where we fine-tune the model on the training samples of the analyzed dataset. finally, in the features generation phase, we add the newly-generated features to the original features of the tabular dataset. ## discussion cases where the original features outperformed the augmented features set. the results in section 5.3 clearly show that feste significantly outperforms the baselines in a large majority of the evaluated datasets. in this section, however, we focus on datasets where our approach did not perform well compared to the original set of features. as shown in finally, we compare the performance achieved using only feste's generated features ( ## conclusions we present feste, a framework for generating new features for tabular datasets from unstructured sources. our approach uses a novel two-step finetuning process that enables it to effectively apply transformer based lm for the extraction of useful features even when the target dataset is limited in size. our ft approach significantly outperforms the existing sota.