scenario-KD-SCR-MSV-D2_data-AmazonScience_massive_all_1_1_beta-jason

This model is a fine-tuned version of FacebookAI/xlm-roberta-base on the massive dataset. It achieves the following results on the evaluation set:

• Loss: 1.8147
• Accuracy: 0.8277
• F1: 0.8039

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 32
• eval_batch_size: 32
• seed: 4444
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 30

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	F1
6.0249	0.27	5000	5.8081	0.3147	0.1329
4.3373	0.53	10000	4.3107	0.5396	0.3628
3.4678	0.8	15000	3.4302	0.6411	0.5106
2.7173	1.07	20000	3.0007	0.6913	0.5899
2.4577	1.34	25000	2.7295	0.7203	0.6347
2.2593	1.6	30000	2.5136	0.7454	0.6683
2.1902	1.87	35000	2.3803	0.7600	0.7085
1.6969	2.14	40000	2.2853	0.7723	0.7110
1.6768	2.41	45000	2.2101	0.7813	0.7304
1.6905	2.67	50000	2.1416	0.7879	0.7439
1.6272	2.94	55000	2.0825	0.7935	0.7525
1.3056	3.21	60000	2.1342	0.7916	0.7485
1.3281	3.47	65000	2.0752	0.7961	0.7578
1.3173	3.74	70000	2.0450	0.7997	0.7627
1.25	4.01	75000	2.0475	0.8050	0.7647
1.1003	4.28	80000	2.0769	0.8008	0.7647
1.085	4.54	85000	2.0143	0.8055	0.7729
1.1138	4.81	90000	2.0097	0.8062	0.7741
0.8998	5.08	95000	2.0293	0.8042	0.7687
0.941	5.34	100000	2.0463	0.8035	0.7730
0.9522	5.61	105000	2.0034	0.8080	0.7740
0.9612	5.88	110000	1.9783	0.8129	0.7818
0.8171	6.15	115000	2.0325	0.8105	0.7782
0.8311	6.41	120000	2.0537	0.8083	0.7822
0.8211	6.68	125000	2.0400	0.8117	0.7812
0.8373	6.95	130000	2.0087	0.8089	0.7817
0.7282	7.22	135000	2.0047	0.8129	0.7851
0.7229	7.48	140000	2.0254	0.8092	0.7847
0.7496	7.75	145000	1.9874	0.8159	0.7890
0.712	8.02	150000	2.0405	0.8122	0.7851
0.671	8.28	155000	2.0360	0.8128	0.7891
0.6774	8.55	160000	2.0099	0.8136	0.7899
0.7075	8.82	165000	1.9979	0.8131	0.7852
0.6209	9.09	170000	1.9955	0.8152	0.7908
0.6128	9.35	175000	1.9814	0.8161	0.7927
0.6485	9.62	180000	1.9914	0.8159	0.7925
0.6385	9.89	185000	1.9978	0.8169	0.7939
0.5789	10.15	190000	2.0151	0.8157	0.7938
0.5933	10.42	195000	1.9624	0.8191	0.7952
0.602	10.69	200000	1.9850	0.8185	0.7931
0.5901	10.96	205000	1.9757	0.8175	0.7913
0.5686	11.22	210000	1.9776	0.8168	0.7928
0.5642	11.49	215000	2.0007	0.8155	0.7918
0.5734	11.76	220000	1.9678	0.8183	0.7939
0.5281	12.03	225000	1.9652	0.8172	0.7948
0.5051	12.29	230000	1.9726	0.8188	0.7933
0.5247	12.56	235000	1.9615	0.8188	0.7937
0.5239	12.83	240000	1.9451	0.8188	0.7964
0.4843	13.09	245000	1.9342	0.8203	0.7939
0.5033	13.36	250000	1.9629	0.8182	0.7964
0.5131	13.63	255000	1.9466	0.8181	0.7959
0.5116	13.9	260000	1.9256	0.8206	0.7966
0.4832	14.16	265000	1.9252	0.8206	0.7993
0.4746	14.43	270000	1.9285	0.8197	0.7956
0.483	14.7	275000	1.9409	0.8188	0.7933
0.4955	14.96	280000	1.9275	0.8211	0.7963
0.4609	15.23	285000	1.9160	0.8213	0.7963
0.477	15.5	290000	1.9189	0.8218	0.7997
0.4685	15.77	295000	1.9135	0.8216	0.7970
0.4449	16.03	300000	1.8993	0.8232	0.7964
0.444	16.3	305000	1.8900	0.8231	0.7979
0.4584	16.57	310000	1.9016	0.8220	0.7991
0.4401	16.84	315000	1.8982	0.8213	0.7951
0.4252	17.1	320000	1.8930	0.8228	0.7997
0.438	17.37	325000	1.8836	0.8222	0.7984
0.4391	17.64	330000	1.8935	0.8220	0.7991
0.4471	17.9	335000	1.8970	0.8212	0.7988
0.4159	18.17	340000	1.8877	0.8235	0.8001
0.4227	18.44	345000	1.8950	0.8226	0.7995
0.4178	18.71	350000	1.8888	0.8233	0.7992
0.4214	18.97	355000	1.8758	0.8222	0.7974
0.4113	19.24	360000	1.8703	0.8235	0.7975
0.4066	19.51	365000	1.8790	0.8243	0.7994
0.413	19.77	370000	1.8561	0.8248	0.8013
0.4099	20.04	375000	1.8576	0.8240	0.7993
0.4092	20.31	380000	1.8591	0.8252	0.8030
0.3952	20.58	385000	1.8563	0.8258	0.8023
0.4017	20.84	390000	1.8603	0.8253	0.8009
0.3964	21.11	395000	1.8472	0.8251	0.8028
0.3891	21.38	400000	1.8540	0.8251	0.8014
0.3881	21.65	405000	1.8558	0.8247	0.8010
0.3953	21.91	410000	1.8594	0.8244	0.8005
0.3872	22.18	415000	1.8576	0.8249	0.7995
0.3754	22.45	420000	1.8486	0.8255	0.8008
0.3798	22.71	425000	1.8511	0.8252	0.7998
0.3744	22.98	430000	1.8491	0.8242	0.7987
0.3691	23.25	435000	1.8343	0.8247	0.7995
0.3718	23.52	440000	1.8496	0.8247	0.7997
0.3761	23.78	445000	1.8433	0.8251	0.8016
0.3634	24.05	450000	1.8397	0.8247	0.8000
0.3704	24.32	455000	1.8334	0.8254	0.8007
0.3651	24.58	460000	1.8367	0.8255	0.8022
0.3649	24.85	465000	1.8321	0.8258	0.8022
0.3573	25.12	470000	1.8358	0.8255	0.8020
0.355	25.39	475000	1.8301	0.8259	0.8023
0.3595	25.65	480000	1.8257	0.8263	0.8042
0.3625	25.92	485000	1.8244	0.8265	0.8025
0.3508	26.19	490000	1.8303	0.8254	0.8021
0.3578	26.46	495000	1.8261	0.8264	0.8035
0.3506	26.72	500000	1.8200	0.8267	0.8029
0.3514	26.99	505000	1.8247	0.8266	0.8032
0.3496	27.26	510000	1.8255	0.8269	0.8046
0.3435	27.52	515000	1.8173	0.8262	0.8019
0.3502	27.79	520000	1.8137	0.8269	0.8035
0.3463	28.06	525000	1.8167	0.8265	0.8024
0.344	28.33	530000	1.8144	0.8275	0.8023
0.3469	28.59	535000	1.8133	0.8268	0.8021
0.3458	28.86	540000	1.8153	0.8254	0.8013
0.3387	29.13	545000	1.8111	0.8269	0.8019
0.3346	29.39	550000	1.8097	0.8271	0.8036
0.3393	29.66	555000	1.8179	0.8268	0.8033
0.3398	29.93	560000	1.8147	0.8277	0.8039

Framework versions

• Transformers 4.33.3
• Pytorch 2.1.1+cu121
• Datasets 2.14.5
• Tokenizers 0.13.3