enlm-roberta

This model is a fine-tuned version of manirai91/enlm-roberta on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.4193

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 6e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• gradient_accumulation_steps: 128
• total_train_batch_size: 8192
• total_eval_batch_size: 64
• optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-06
• lr_scheduler_type: polynomial
• num_epochs: 10

Training results

Training Loss	Epoch	Step	Validation Loss
1.5984	0.13	160	1.4905
1.6149	0.27	320	1.4969
1.6285	0.61	480	1.5087
1.6473	0.96	640	1.5173
1.679	1.3	800	1.5441
1.683	1.64	960	1.5576
1.6942	1.99	1120	1.5681
1.6921	2.33	1280	1.5659
1.6986	2.67	1440	1.5684
1.8496	3.02	1600	1.5586
1.6807	3.36	1760	1.5503
1.6744	3.7	1920	1.5466
1.6838	4.05	2080	1.5471
1.6725	4.39	2240	1.5371
1.6663	4.73	2400	1.5433
1.6644	5.08	2560	1.5321
1.66	5.42	2720	1.5325
1.6535	5.76	2880	1.5272
1.651	6.11	3040	1.5253
1.6432	6.45	3200	1.5207
1.6452	6.79	3360	1.5239
1.6398	7.14	3520	1.5168
1.6308	7.48	3680	1.5088
1.6332	7.82	3840	1.5065
1.6261	8.17	4000	1.5024
1.6194	8.51	4160	1.5085
1.6178	8.85	4320	1.4961
1.6137	9.2	4480	1.4949
1.613	9.54	4640	1.4953
1.6048	9.88	4800	1.4905
1.6058	10.23	4960	1.4893
1.6036	10.57	5120	1.4826
1.5976	10.91	5280	1.4846
1.593	11.26	5440	1.4792
1.5914	11.6	5600	1.4734
1.5863	11.94	5760	1.4731
1.5828	12.29	5920	1.4702
1.5831	12.63	6080	1.4649
1.5796	12.97	6240	1.4611
1.5717	13.32	6400	1.4580
1.5737	13.66	6560	1.4576
1.7137	14.0	6720	1.4571
1.5651	14.35	6880	1.4543
1.561	14.69	7040	1.4469
1.5578	15.25	7200	1.4469
1.5531	15.6	7360	1.4430
1.5548	15.94	7520	1.4408
1.5523	16.28	7680	1.4390
1.5467	16.63	7840	1.4357
1.5467	16.97	8000	1.4328
1.5406	17.23	8160	1.4290
1.5379	17.58	8320	1.4321
1.5349	17.92	8480	1.4277
1.5343	18.26	8640	1.4238
1.5302	18.61	8800	1.4206
1.5293	18.95	8960	1.4198
1.5278	19.29	9120	1.4207
1.523	19.64	9280	1.4193

Framework versions

• Transformers 4.20.1
• Pytorch 1.11.0
• Datasets 2.3.2
• Tokenizers 0.12.1