deberta-v3-base-kaggle-mlm

This model is a fine-tuned version of microsoft/deberta-v3-base on the None dataset. It achieves the following results on the evaluation set:

• Loss: 1.5600

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 25
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
3.1466	1.0	6848	2.9173
2.6316	2.0	13696	2.4139
2.3281	3.0	20544	2.2020
2.2122	4.0	27392	2.0776
2.0794	5.0	34240	1.9780
2.0299	6.0	41088	1.8861
1.9629	7.0	47936	1.8213
1.9001	8.0	54784	1.7946
1.8508	9.0	61632	1.7551
1.8157	10.0	68480	1.7485
1.7815	11.0	75328	1.7100
1.7423	12.0	82176	1.6970
1.7318	13.0	89024	1.6813
1.7173	14.0	95872	1.6493
1.6902	15.0	102720	1.6243
1.7002	16.0	109568	1.6313
1.6714	17.0	116416	1.6181
1.6605	18.0	123264	1.6026
1.6331	19.0	130112	1.5825
1.6143	20.0	136960	1.5903
1.6136	21.0	143808	1.5812
1.6151	22.0	150656	1.5708
1.6122	23.0	157504	1.5806
1.6025	24.0	164352	1.5492
1.614	25.0	171200	1.5555

Framework versions

• Transformers 4.41.2
• Pytorch 2.3.1+cu121
• Datasets 2.19.2
• Tokenizers 0.19.1