Edit model card

Keras Implementation of Structured data learning with TabTransformer

This repo contains the trained model of Structured data learning with TabTransformer. The full credit goes to: Khalid Salama

Spaces Link:

Model summary:

  • The trained model uses self-attention based Transformers structure following by multiple feed forward layers in order to serve supervised and semi-supervised learning.
  • The model's inputs can contain both numerical and categorical features.
  • All the categorical features will be encoded into embedding vector with the same number of embedding dimensions, before adding (point-wise) with each other and feeding into a stack of Transformer blocks.
  • The contextual embeddings of the categorical features after the final Transformer layer, are concatenated with the input numerical features, and fed into a final MLP block.
  • A SoftMax function is applied at the end of the model.

Intended uses & limitations:

  • This model can be used for both supervised and semi-supervised tasks on tabular data.

Training and evaluation data:

  • This model was trained using the United States Census Income Dataset provided by the UC Irvine Machine Learning Repository. The task of the dataset is to predict whether a person is likely to be making over USD 50,000 a year (binary classification).
  • The dataset consists of 14 input features: 5 numerical features and 9 categorical features.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • optimizer: 'AdamW'
  • learning_rate: 0.001
  • weight decay: 1e-04
  • loss: 'sparse_categorical_crossentropy'
  • beta_1: 0.9
  • beta_2: 0.999
  • epsilon: 1e-07
  • epochs: 50
  • batch_size: 16
  • training_precision: float32

Training Metrics

Model history needed

Model Plot

View Model Plot

Model Image

Downloads last month
Hosted inference API
This model can be loaded on the Inference API on-demand.

Space using keras-io/tab_transformer