hbertv1-massive-logit_KD-small

This model is a fine-tuned version of gokuls/model_v1_complete_training_wt_init_48_small_freeze_new on the massive dataset. It achieves the following results on the evaluation set:

• Loss: 0.4139
• Accuracy: 0.8736

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 64
• eval_batch_size: 64
• seed: 33
• distributed_type: multi-GPU
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 50

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
2.2301	1.0	180	0.8611	0.7565
0.8039	2.0	360	0.5989	0.8151
0.5542	3.0	540	0.5036	0.8396
0.4134	4.0	720	0.4535	0.8569
0.3187	5.0	900	0.4432	0.8569
0.251	6.0	1080	0.4280	0.8637
0.2201	7.0	1260	0.4311	0.8598
0.1879	8.0	1440	0.4443	0.8608
0.168	9.0	1620	0.4136	0.8677
0.153	10.0	1800	0.4286	0.8598
0.137	11.0	1980	0.4148	0.8701
0.1276	12.0	2160	0.4158	0.8711
0.1196	13.0	2340	0.3975	0.8721
0.1137	14.0	2520	0.4221	0.8662
0.1066	15.0	2700	0.4085	0.8677
0.1024	16.0	2880	0.4048	0.8687
0.0995	17.0	3060	0.4139	0.8736
0.0949	18.0	3240	0.3953	0.8706
0.0908	19.0	3420	0.3984	0.8716
0.0882	20.0	3600	0.4006	0.8701
0.0864	21.0	3780	0.3943	0.8731
0.0837	22.0	3960	0.3912	0.8692

Framework versions

• Transformers 4.35.2
• Pytorch 1.14.0a0+410ce96
• Datasets 2.15.0
• Tokenizers 0.15.0