scenario-KD-SCR-MSV-D2_data-AmazonScience_massive_all_1_1_gamma-jason

This model is a fine-tuned version of FacebookAI/xlm-roberta-base on the massive dataset. It achieves the following results on the evaluation set:

• Loss: 1.8080
• Accuracy: 0.8266
• F1: 0.8031

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 32
• eval_batch_size: 32
• seed: 123621
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 30

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	F1
6.283	0.27	5000	6.2020	0.2676	0.1018
4.4008	0.53	10000	4.4005	0.5229	0.3509
3.3814	0.8	15000	3.4464	0.6349	0.5009
2.6796	1.07	20000	2.9367	0.6972	0.6000
2.4097	1.34	25000	2.6464	0.7279	0.6536
2.2303	1.6	30000	2.4685	0.7522	0.6898
2.0826	1.87	35000	2.3359	0.7657	0.7120
1.6889	2.14	40000	2.3083	0.7710	0.7094
1.6854	2.41	45000	2.2360	0.7767	0.7203
1.621	2.67	50000	2.1137	0.7867	0.7391
1.6051	2.94	55000	2.0718	0.7929	0.7467
1.3071	3.21	60000	2.1140	0.7929	0.7508
1.3165	3.47	65000	2.0525	0.7962	0.7574
1.3278	3.74	70000	2.0554	0.7993	0.7620
1.203	4.01	75000	2.0619	0.7989	0.7606
1.1003	4.28	80000	2.0386	0.8011	0.7667
1.0856	4.54	85000	2.0191	0.8024	0.7723
1.0941	4.81	90000	2.0063	0.8019	0.7722
0.9191	5.08	95000	2.0552	0.8051	0.7680
0.9443	5.34	100000	2.0511	0.8023	0.7745
0.9377	5.61	105000	2.0379	0.8059	0.7772
0.9659	5.88	110000	2.0115	0.8058	0.7760
0.7971	6.15	115000	2.0532	0.8083	0.7798
0.8237	6.41	120000	2.0635	0.8070	0.7798
0.8419	6.68	125000	2.0257	0.8079	0.7756
0.8498	6.95	130000	2.0144	0.8117	0.7858
0.7382	7.22	135000	2.0307	0.8101	0.7828
0.7385	7.48	140000	2.0336	0.8117	0.7879
0.7622	7.75	145000	1.9982	0.8126	0.7849
0.6757	8.02	150000	2.0168	0.8147	0.7929
0.667	8.28	155000	2.0176	0.8130	0.7882
0.6735	8.55	160000	2.0328	0.8121	0.7888
0.6927	8.82	165000	1.9887	0.8127	0.7877
0.6113	9.09	170000	2.0148	0.8145	0.7885
0.6098	9.35	175000	2.0184	0.8139	0.7898
0.6308	9.62	180000	1.9917	0.8120	0.7870
0.6361	9.89	185000	1.9818	0.8134	0.7877
0.5727	10.15	190000	2.0203	0.8126	0.7888
0.59	10.42	195000	1.9819	0.8143	0.7930
0.5947	10.69	200000	2.0151	0.8143	0.7906
0.602	10.96	205000	1.9809	0.8165	0.7923
0.5482	11.22	210000	1.9816	0.8160	0.7935
0.5669	11.49	215000	1.9793	0.8160	0.7904
0.5724	11.76	220000	1.9677	0.8153	0.7905
0.5295	12.03	225000	1.9569	0.8171	0.7924
0.5277	12.29	230000	1.9549	0.8178	0.7959
0.5324	12.56	235000	1.9477	0.8175	0.7929
0.5374	12.83	240000	1.9587	0.8176	0.7960
0.4818	13.09	245000	1.9764	0.8168	0.7935
0.5064	13.36	250000	1.9439	0.8180	0.7945
0.507	13.63	255000	1.9332	0.8160	0.7941
0.5081	13.9	260000	1.9293	0.8180	0.7990
0.4791	14.16	265000	1.9500	0.8183	0.7953
0.4949	14.43	270000	1.9520	0.8181	0.7952
0.4746	14.7	275000	1.9375	0.8197	0.7966
0.4918	14.96	280000	1.9161	0.8210	0.7949
0.4758	15.23	285000	1.9281	0.8184	0.7939
0.4605	15.5	290000	1.9164	0.8194	0.7934
0.4637	15.77	295000	1.9372	0.8192	0.7986
0.4387	16.03	300000	1.9123	0.8213	0.8005
0.4405	16.3	305000	1.9115	0.8191	0.7966
0.4455	16.57	310000	1.8867	0.8212	0.7981
0.4562	16.84	315000	1.9136	0.8199	0.7967
0.4316	17.1	320000	1.8907	0.8218	0.7986
0.4281	17.37	325000	1.8942	0.8222	0.7990
0.4296	17.64	330000	1.9041	0.8215	0.7998
0.4327	17.9	335000	1.8844	0.8239	0.7999
0.4157	18.17	340000	1.8902	0.8219	0.8001
0.4184	18.44	345000	1.8874	0.8227	0.7991
0.4224	18.71	350000	1.8701	0.8224	0.7991
0.4264	18.97	355000	1.8816	0.8217	0.7974
0.4044	19.24	360000	1.8879	0.8212	0.7974
0.4119	19.51	365000	1.8577	0.8229	0.7991
0.4046	19.77	370000	1.8675	0.8235	0.8003
0.4011	20.04	375000	1.8604	0.8231	0.7997
0.4036	20.31	380000	1.8500	0.8240	0.8000
0.3887	20.58	385000	1.8624	0.8231	0.7999
0.4057	20.84	390000	1.8588	0.8222	0.7972
0.3883	21.11	395000	1.8524	0.8233	0.7990
0.3881	21.38	400000	1.8481	0.8245	0.8024
0.3956	21.65	405000	1.8503	0.8245	0.8005
0.3828	21.91	410000	1.8538	0.8240	0.7999
0.3776	22.18	415000	1.8495	0.8241	0.7999
0.3896	22.45	420000	1.8513	0.8226	0.7991
0.3759	22.71	425000	1.8518	0.8251	0.8007
0.3769	22.98	430000	1.8388	0.8242	0.8019
0.3675	23.25	435000	1.8307	0.8245	0.8002
0.3704	23.52	440000	1.8402	0.8227	0.7992
0.3698	23.78	445000	1.8409	0.8238	0.7991
0.3672	24.05	450000	1.8180	0.8248	0.7979
0.3709	24.32	455000	1.8300	0.8235	0.8003
0.361	24.58	460000	1.8265	0.8252	0.8012
0.3649	24.85	465000	1.8288	0.8250	0.8012
0.3534	25.12	470000	1.8216	0.8253	0.8025
0.3535	25.39	475000	1.8240	0.8261	0.8017
0.3578	25.65	480000	1.8216	0.8259	0.8011
0.3569	25.92	485000	1.8257	0.8253	0.8025
0.3515	26.19	490000	1.8191	0.8263	0.8026
0.3551	26.46	495000	1.8209	0.8264	0.8036
0.3577	26.72	500000	1.8199	0.8254	0.8011
0.3548	26.99	505000	1.8190	0.8252	0.8006
0.3498	27.26	510000	1.8072	0.8257	0.8023
0.3419	27.52	515000	1.8131	0.8259	0.8019
0.3452	27.79	520000	1.8140	0.8253	0.8023
0.3364	28.06	525000	1.8145	0.8254	0.8017
0.346	28.33	530000	1.8087	0.8256	0.8019
0.3391	28.59	535000	1.8142	0.8259	0.8025
0.3535	28.86	540000	1.8044	0.8270	0.8029
0.333	29.13	545000	1.8150	0.8264	0.8026
0.3397	29.39	550000	1.8099	0.8266	0.8032
0.3429	29.66	555000	1.8090	0.8259	0.8017
0.3422	29.93	560000	1.8080	0.8266	0.8031

Framework versions

• Transformers 4.33.3
• Pytorch 2.1.1+cu121
• Datasets 2.14.5
• Tokenizers 0.13.3