scenario-KD-SCR-MSV-D2_data-AmazonScience_massive_all_1_1_alpha-jason

This model is a fine-tuned version of FacebookAI/xlm-roberta-base on the massive dataset. It achieves the following results on the evaluation set:

• Loss: 1.8097
• Accuracy: 0.8259
• F1: 0.8016

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 32
• eval_batch_size: 32
• seed: 1123
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 30

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	F1
5.8591	0.27	5000	5.6952	0.3264	0.1501
4.1699	0.53	10000	4.1191	0.5584	0.4058
3.363	0.8	15000	3.3295	0.6517	0.5362
2.5996	1.07	20000	2.8831	0.7073	0.6101
2.4092	1.34	25000	2.6156	0.7336	0.6540
2.2034	1.6	30000	2.4512	0.7535	0.6792
2.0892	1.87	35000	2.3290	0.7658	0.6941
1.6869	2.14	40000	2.2489	0.7767	0.7205
1.6442	2.41	45000	2.2293	0.7780	0.7261
1.633	2.67	50000	2.1483	0.7854	0.7392
1.6348	2.94	55000	2.0885	0.7906	0.7468
1.2977	3.21	60000	2.1314	0.7903	0.7490
1.3298	3.47	65000	2.0696	0.7975	0.7590
1.3	3.74	70000	2.0638	0.7961	0.7611
1.2101	4.01	75000	2.0296	0.8022	0.7627
1.0841	4.28	80000	2.0720	0.8008	0.7656
1.0928	4.54	85000	2.0490	0.8031	0.7684
1.0846	4.81	90000	1.9852	0.8068	0.7751
0.9008	5.08	95000	2.0298	0.8076	0.7749
0.9178	5.34	100000	2.0931	0.8025	0.7735
0.9507	5.61	105000	2.0079	0.8066	0.7790
0.9577	5.88	110000	1.9660	0.8103	0.7780
0.7877	6.15	115000	2.0676	0.8072	0.7772
0.7916	6.41	120000	2.0080	0.8089	0.7832
0.8493	6.68	125000	2.0347	0.8078	0.7780
0.8544	6.95	130000	2.0131	0.8093	0.7806
0.7207	7.22	135000	2.0612	0.8089	0.7827
0.7387	7.48	140000	2.0334	0.8100	0.7829
0.7341	7.75	145000	2.0446	0.8096	0.7826
0.6886	8.02	150000	2.0384	0.8114	0.7853
0.6826	8.28	155000	2.0159	0.8103	0.7850
0.6944	8.55	160000	1.9987	0.8136	0.7879
0.6858	8.82	165000	2.0162	0.8124	0.7905
0.6204	9.09	170000	2.0336	0.8128	0.7875
0.6063	9.35	175000	2.0218	0.8125	0.7879
0.6253	9.62	180000	2.0256	0.8130	0.7874
0.6354	9.89	185000	1.9910	0.8149	0.7889
0.5804	10.15	190000	2.0027	0.8139	0.7898
0.5932	10.42	195000	1.9711	0.8157	0.7919
0.5965	10.69	200000	1.9713	0.8158	0.7930
0.6028	10.96	205000	2.0039	0.8135	0.7884
0.5417	11.22	210000	1.9622	0.8164	0.7926
0.5556	11.49	215000	1.9953	0.8157	0.7937
0.5552	11.76	220000	1.9741	0.8166	0.7928
0.5146	12.03	225000	1.9948	0.8146	0.7892
0.5328	12.29	230000	1.9546	0.8175	0.7969
0.5224	12.56	235000	1.9565	0.8171	0.7927
0.5491	12.83	240000	1.9538	0.8178	0.7932
0.5001	13.09	245000	1.9559	0.8184	0.7944
0.4904	13.36	250000	1.9734	0.8165	0.7947
0.5091	13.63	255000	1.9647	0.8177	0.7936
0.5157	13.9	260000	1.9391	0.8194	0.7953
0.4824	14.16	265000	1.9494	0.8189	0.7967
0.4757	14.43	270000	1.9423	0.8174	0.7920
0.4859	14.7	275000	1.9255	0.8193	0.7949
0.4878	14.96	280000	1.9229	0.8197	0.7957
0.4629	15.23	285000	1.9201	0.8191	0.7950
0.4634	15.5	290000	1.9189	0.8209	0.7990
0.4593	15.77	295000	1.9161	0.8200	0.7991
0.4484	16.03	300000	1.8980	0.8210	0.7952
0.4473	16.3	305000	1.9098	0.8204	0.7983
0.4531	16.57	310000	1.8917	0.8210	0.7964
0.4493	16.84	315000	1.8937	0.8205	0.7979
0.4288	17.1	320000	1.8914	0.8200	0.7989
0.4291	17.37	325000	1.8920	0.8216	0.7988
0.4215	17.64	330000	1.8951	0.8224	0.7987
0.4351	17.9	335000	1.8831	0.8220	0.7964
0.4164	18.17	340000	1.8704	0.8223	0.7971
0.4205	18.44	345000	1.8835	0.8227	0.7985
0.4239	18.71	350000	1.8768	0.8227	0.7985
0.4269	18.97	355000	1.8723	0.8226	0.7988
0.4051	19.24	360000	1.8555	0.8235	0.8016
0.4122	19.51	365000	1.8716	0.8234	0.7997
0.3921	19.77	370000	1.8650	0.8231	0.7978
0.3973	20.04	375000	1.8550	0.8236	0.7983
0.4	20.31	380000	1.8512	0.8225	0.7966
0.4027	20.58	385000	1.8653	0.8236	0.7982
0.3932	20.84	390000	1.8594	0.8243	0.7974
0.392	21.11	395000	1.8373	0.8247	0.8006
0.3887	21.38	400000	1.8420	0.8252	0.8012
0.3887	21.65	405000	1.8425	0.8241	0.7984
0.38	21.91	410000	1.8413	0.8244	0.8017
0.3793	22.18	415000	1.8325	0.8240	0.7978
0.3806	22.45	420000	1.8338	0.8249	0.7990
0.3726	22.71	425000	1.8488	0.8231	0.7990
0.3771	22.98	430000	1.8441	0.8243	0.7998
0.3728	23.25	435000	1.8380	0.8238	0.8005
0.3677	23.52	440000	1.8289	0.8246	0.7999
0.368	23.78	445000	1.8334	0.8256	0.8012
0.3659	24.05	450000	1.8188	0.8261	0.8010
0.3706	24.32	455000	1.8239	0.8250	0.7992
0.3649	24.58	460000	1.8236	0.8258	0.8013
0.3537	24.85	465000	1.8327	0.8254	0.7991
0.3548	25.12	470000	1.8175	0.8258	0.8020
0.3483	25.39	475000	1.8225	0.8255	0.8008
0.3516	25.65	480000	1.8200	0.8254	0.8004
0.3588	25.92	485000	1.8265	0.8256	0.8001
0.3492	26.19	490000	1.8052	0.8270	0.8015
0.3497	26.46	495000	1.8165	0.8268	0.8022
0.3467	26.72	500000	1.8172	0.8265	0.8026
0.3463	26.99	505000	1.8084	0.8266	0.8023
0.3448	27.26	510000	1.8105	0.8267	0.8021
0.3414	27.52	515000	1.8109	0.8267	0.8014
0.3439	27.79	520000	1.8146	0.8254	0.8005
0.3374	28.06	525000	1.8081	0.8264	0.8033
0.3412	28.33	530000	1.8125	0.8264	0.8022
0.3396	28.59	535000	1.8141	0.8264	0.8022
0.3451	28.86	540000	1.8072	0.8258	0.8005
0.337	29.13	545000	1.8056	0.8265	0.8028
0.3335	29.39	550000	1.8083	0.8263	0.8010
0.3402	29.66	555000	1.8107	0.8260	0.8013
0.3409	29.93	560000	1.8097	0.8259	0.8016

Framework versions

• Transformers 4.33.3
• Pytorch 2.1.1+cu121
• Datasets 2.14.5
• Tokenizers 0.13.3