amazon_domain_pretrained_model

This model is a fine-tuned version of roberta-base on the None dataset. It achieves the following results on the evaluation set:

• Loss: 1.4165

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0005
• train_batch_size: 32
• eval_batch_size: 64
• seed: 42
• gradient_accumulation_steps: 64
• total_train_batch_size: 2048
• optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-06
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.06
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss
No log	0.0	10	1.6922
No log	0.01	20	1.6359
No log	0.01	30	1.6075
No log	0.02	40	1.5957
No log	0.02	50	1.5806
No log	0.03	60	1.5801
No log	0.03	70	1.5776
No log	0.04	80	1.5719
No log	0.04	90	1.5710
No log	0.05	100	1.5837
No log	0.05	110	1.5795
No log	0.06	120	1.5760
No log	0.06	130	1.5867
No log	0.07	140	1.5892
No log	0.07	150	1.5853
No log	0.08	160	1.5803
No log	0.08	170	1.5909
No log	0.09	180	1.5810
No log	0.09	190	1.5763
No log	0.1	200	1.5805
No log	0.1	210	1.5725
No log	0.1	220	1.5938
No log	0.11	230	1.5735
No log	0.11	240	1.5735
No log	0.12	250	1.5692
No log	0.12	260	1.5634
No log	0.13	270	1.5649
No log	0.13	280	1.5660
No log	0.14	290	1.5695
No log	0.14	300	1.5689
No log	0.15	310	1.5648
No log	0.15	320	1.5565
No log	0.16	330	1.5622
No log	0.16	340	1.5580
No log	0.17	350	1.5516
No log	0.17	360	1.5537
No log	0.18	370	1.5537
No log	0.18	380	1.5506
No log	0.19	390	1.5543
No log	0.19	400	1.5447
No log	0.19	410	1.5471
No log	0.2	420	1.5459
No log	0.2	430	1.5480
No log	0.21	440	1.5313
No log	0.21	450	1.5353
No log	0.22	460	1.5423
No log	0.22	470	1.5408
No log	0.23	480	1.5399
No log	0.23	490	1.5330
1.6837	0.24	500	1.5389
1.6837	0.24	510	1.5360
1.6837	0.25	520	1.5244
1.6837	0.25	530	1.5230
1.6837	0.26	540	1.5348
1.6837	0.26	550	1.5223
1.6837	0.27	560	1.5165
1.6837	0.27	570	1.5288
1.6837	0.28	580	1.5195
1.6837	0.28	590	1.5208
1.6837	0.29	600	1.5217
1.6837	0.29	610	1.5249
1.6837	0.29	620	1.5138
1.6837	0.3	630	1.5154
1.6837	0.3	640	1.5194
1.6837	0.31	650	1.5206
1.6837	0.31	660	1.5178
1.6837	0.32	670	1.5195
1.6837	0.32	680	1.5134
1.6837	0.33	690	1.5118
1.6837	0.33	700	1.5135
1.6837	0.34	710	1.5178
1.6837	0.34	720	1.5106
1.6837	0.35	730	1.5077
1.6837	0.35	740	1.5065
1.6837	0.36	750	1.5076
1.6837	0.36	760	1.5003
1.6837	0.37	770	1.5049
1.6837	0.37	780	1.5094
1.6837	0.38	790	1.5022
1.6837	0.38	800	1.4966
1.6837	0.38	810	1.5028
1.6837	0.39	820	1.4946
1.6837	0.39	830	1.5052
1.6837	0.4	840	1.4997
1.6837	0.4	850	1.5073
1.6837	0.41	860	1.4915
1.6837	0.41	870	1.5003
1.6837	0.42	880	1.4918
1.6837	0.42	890	1.4994
1.6837	0.43	900	1.4975
1.6837	0.43	910	1.4993
1.6837	0.44	920	1.4931
1.6837	0.44	930	1.4962
1.6837	0.45	940	1.4947
1.6837	0.45	950	1.4911
1.6837	0.46	960	1.4918
1.6837	0.46	970	1.4893
1.6837	0.47	980	1.4866
1.6837	0.47	990	1.4845
1.6086	0.48	1000	1.4829
1.6086	0.48	1010	1.4847
1.6086	0.48	1020	1.4801
1.6086	0.49	1030	1.4831
1.6086	0.49	1040	1.4860
1.6086	0.5	1050	1.4832
1.6086	0.5	1060	1.4814
1.6086	0.51	1070	1.4825
1.6086	0.51	1080	1.4780
1.6086	0.52	1090	1.4742
1.6086	0.52	1100	1.4821
1.6086	0.53	1110	1.4770
1.6086	0.53	1120	1.4730
1.6086	0.54	1130	1.4739
1.6086	0.54	1140	1.4718
1.6086	0.55	1150	1.4706
1.6086	0.55	1160	1.4729
1.6086	0.56	1170	1.4712
1.6086	0.56	1180	1.4699
1.6086	0.57	1190	1.4659
1.6086	0.57	1200	1.4685
1.6086	0.58	1210	1.4721
1.6086	0.58	1220	1.4716
1.6086	0.58	1230	1.4604
1.6086	0.59	1240	1.4619
1.6086	0.59	1250	1.4716
1.6086	0.6	1260	1.4643
1.6086	0.6	1270	1.4640
1.6086	0.61	1280	1.4616
1.6086	0.61	1290	1.4638
1.6086	0.62	1300	1.4605
1.6086	0.62	1310	1.4634
1.6086	0.63	1320	1.4580
1.6086	0.63	1330	1.4591
1.6086	0.64	1340	1.4597
1.6086	0.64	1350	1.4585
1.6086	0.65	1360	1.4555
1.6086	0.65	1370	1.4509
1.6086	0.66	1380	1.4591
1.6086	0.66	1390	1.4525
1.6086	0.67	1400	1.4479
1.6086	0.67	1410	1.4511
1.6086	0.67	1420	1.4545
1.6086	0.68	1430	1.4542
1.6086	0.68	1440	1.4469
1.6086	0.69	1450	1.4525
1.6086	0.69	1460	1.4452
1.6086	0.7	1470	1.4509
1.6086	0.7	1480	1.4524
1.6086	0.71	1490	1.4470
1.5617	0.71	1500	1.4479
1.5617	0.72	1510	1.4444
1.5617	0.72	1520	1.4485
1.5617	0.73	1530	1.4433
1.5617	0.73	1540	1.4380
1.5617	0.74	1550	1.4387
1.5617	0.74	1560	1.4383
1.5617	0.75	1570	1.4438
1.5617	0.75	1580	1.4338
1.5617	0.76	1590	1.4446
1.5617	0.76	1600	1.4376
1.5617	0.77	1610	1.4407
1.5617	0.77	1620	1.4384
1.5617	0.77	1630	1.4354
1.5617	0.78	1640	1.4344
1.5617	0.78	1650	1.4335
1.5617	0.79	1660	1.4364
1.5617	0.79	1670	1.4342
1.5617	0.8	1680	1.4319
1.5617	0.8	1690	1.4359
1.5617	0.81	1700	1.4389
1.5617	0.81	1710	1.4352
1.5617	0.82	1720	1.4324
1.5617	0.82	1730	1.4330
1.5617	0.83	1740	1.4281
1.5617	0.83	1750	1.4298
1.5617	0.84	1760	1.4243
1.5617	0.84	1770	1.4277
1.5617	0.85	1780	1.4253
1.5617	0.85	1790	1.4300
1.5617	0.86	1800	1.4272
1.5617	0.86	1810	1.4284
1.5617	0.86	1820	1.4293
1.5617	0.87	1830	1.4242
1.5617	0.87	1840	1.4267
1.5617	0.88	1850	1.4240
1.5617	0.88	1860	1.4193
1.5617	0.89	1870	1.4273
1.5617	0.89	1880	1.4174
1.5617	0.9	1890	1.4199
1.5617	0.9	1900	1.4239
1.5617	0.91	1910	1.4240
1.5617	0.91	1920	1.4201
1.5617	0.92	1930	1.4161
1.5617	0.92	1940	1.4222
1.5617	0.93	1950	1.4102
1.5617	0.93	1960	1.4177
1.5617	0.94	1970	1.4157
1.5617	0.94	1980	1.4100
1.5617	0.95	1990	1.4194
1.5215	0.95	2000	1.4232
1.5215	0.96	2010	1.4116
1.5215	0.96	2020	1.4243
1.5215	0.96	2030	1.4151
1.5215	0.97	2040	1.4202
1.5215	0.97	2050	1.4129
1.5215	0.98	2060	1.4138
1.5215	0.98	2070	1.4097
1.5215	0.99	2080	1.4143
1.5215	0.99	2090	1.4084
1.5215	1.0	2100	1.4132

Framework versions

• Transformers 4.30.2
• Pytorch 1.12.0+cu113
• Datasets 2.13.2
• Tokenizers 0.13.3