keras-io/text-classification-with-transformer

Model description

Implement a Transformer block as a Keras layer and use it for text classification.

For details on the implementation, please see the original link on keras

Full credits to: Apoorv Nandan

Training and evaluation data

The model is trained and evaluated on the IMDB dataset for sentiment analysis.

Details on the dataset can be found on keras

Note that the keras dataset is already tokenized, so the model doesn't have an associated tokenizer. Since the hosted text classification pipeline only accepts text as input, it is currently broken.

For a live demo of the model using my own tokenizer, please use the space

Training hyperparameters

The following hyperparameters were used during training:

name	learning_rate	decay	beta_1	beta_2	epsilon	amsgrad	training_precision
Adam	0.0010000000474974513	0.0	0.8999999761581421	0.9990000128746033	1e-07	False	float32

Training Metrics

Epochs	Train Loss	Train Accuracy	Validation Loss	Validation Accuracy
1	0.385	0.819	0.298	0.868
2	0.198	0.925	0.333	0.863

Model Plot

View Model Plot