scenario-NON-KD-PR-COPY-D2_data-AmazonScience_massive_all_1_1_gamma-jason

This model is a fine-tuned version of xlm-roberta-base on the massive dataset. It achieves the following results on the evaluation set:

• Loss: 1.5166
• Accuracy: 0.8307
• F1: 0.8049

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 32
• eval_batch_size: 32
• seed: 333
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 30

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	F1
1.6464	0.27	5000	1.5147	0.5833	0.4304
1.1994	0.53	10000	1.1439	0.6859	0.5701
0.9913	0.8	15000	0.9761	0.7368	0.6485
0.808	1.07	20000	0.9064	0.7579	0.6741
0.7284	1.34	25000	0.8415	0.7782	0.7145
0.6627	1.6	30000	0.8130	0.7846	0.7253
0.6359	1.87	35000	0.7713	0.7961	0.7412
0.5309	2.14	40000	0.7650	0.8029	0.7544
0.5095	2.41	45000	0.7703	0.8019	0.7581
0.492	2.67	50000	0.7485	0.8119	0.7671
0.4905	2.94	55000	0.7258	0.8133	0.7767
0.3813	3.21	60000	0.7600	0.8149	0.7802
0.385	3.47	65000	0.7538	0.8167	0.7777
0.3869	3.74	70000	0.7434	0.8173	0.7765
0.3519	4.01	75000	0.7548	0.8219	0.7836
0.3087	4.28	80000	0.7899	0.8169	0.7806
0.3088	4.54	85000	0.8028	0.8190	0.7866
0.3263	4.81	90000	0.7830	0.8191	0.7839
0.2382	5.08	95000	0.8014	0.8250	0.7912
0.2457	5.34	100000	0.8182	0.8208	0.7844
0.2481	5.61	105000	0.8435	0.8191	0.7805
0.2586	5.88	110000	0.8087	0.8233	0.7924
0.2071	6.15	115000	0.8568	0.8252	0.7922
0.2014	6.41	120000	0.8656	0.8243	0.7923
0.2034	6.68	125000	0.8602	0.8245	0.7896
0.2249	6.95	130000	0.8541	0.8225	0.7926
0.1743	7.22	135000	0.8963	0.8238	0.7931
0.1786	7.48	140000	0.8958	0.8255	0.7953
0.1847	7.75	145000	0.9003	0.8256	0.7970
0.1558	8.02	150000	0.9111	0.8260	0.7979
0.1491	8.28	155000	0.9490	0.8240	0.7950
0.1569	8.55	160000	0.9553	0.8256	0.7970
0.1571	8.82	165000	0.9445	0.8243	0.7972
0.1213	9.09	170000	0.9658	0.8280	0.8032
0.1265	9.35	175000	1.0384	0.8215	0.7964
0.134	9.62	180000	0.9988	0.8241	0.7990
0.1276	9.89	185000	0.9948	0.8293	0.8028
0.1049	10.15	190000	1.0334	0.8249	0.7995
0.1105	10.42	195000	1.0462	0.8267	0.7984
0.1052	10.69	200000	1.0433	0.8276	0.8069
0.113	10.96	205000	1.0673	0.8248	0.7963
0.0933	11.22	210000	1.0933	0.8244	0.7976
0.096	11.49	215000	1.1244	0.8229	0.7963
0.0941	11.76	220000	1.1033	0.8255	0.7984
0.083	12.03	225000	1.1119	0.8276	0.8009
0.0846	12.29	230000	1.1474	0.8283	0.8028
0.0848	12.56	235000	1.1601	0.8245	0.7981
0.0881	12.83	240000	1.1509	0.8281	0.8013
0.0687	13.09	245000	1.1838	0.8265	0.7973
0.0736	13.36	250000	1.2040	0.8256	0.8011
0.0722	13.63	255000	1.2075	0.8271	0.8044
0.0752	13.9	260000	1.1894	0.8257	0.7999
0.0555	14.16	265000	1.2432	0.8261	0.7987
0.0677	14.43	270000	1.2326	0.8284	0.8005
0.0678	14.7	275000	1.2249	0.8281	0.8017
0.0652	14.96	280000	1.2391	0.8258	0.7990
0.0531	15.23	285000	1.2643	0.8266	0.8031
0.0561	15.5	290000	1.2688	0.8275	0.8003
0.064	15.77	295000	1.2652	0.8286	0.8016
0.0433	16.03	300000	1.2915	0.8286	0.8044
0.0482	16.3	305000	1.3119	0.8258	0.7986
0.05	16.57	310000	1.3283	0.8269	0.8017
0.0574	16.84	315000	1.3050	0.8283	0.8006
0.0481	17.1	320000	1.3210	0.8264	0.8000
0.0456	17.37	325000	1.3479	0.8283	0.8009
0.0438	17.64	330000	1.3568	0.8277	0.8011
0.0487	17.9	335000	1.3517	0.8276	0.8017
0.0413	18.17	340000	1.3726	0.8265	0.7985
0.0381	18.44	345000	1.3834	0.8274	0.8007
0.0447	18.71	350000	1.3754	0.8276	0.8014
0.0403	18.97	355000	1.3614	0.8282	0.8025
0.0359	19.24	360000	1.3996	0.8267	0.8011
0.0392	19.51	365000	1.4055	0.8263	0.7990
0.037	19.77	370000	1.4205	0.8271	0.8009
0.0275	20.04	375000	1.4213	0.8260	0.8005
0.0357	20.31	380000	1.4220	0.8273	0.7992
0.0279	20.58	385000	1.4286	0.8273	0.8027
0.034	20.84	390000	1.4348	0.8273	0.7993
0.0256	21.11	395000	1.4429	0.8291	0.8027
0.0274	21.38	400000	1.4480	0.8276	0.8006
0.0287	21.65	405000	1.4488	0.8263	0.8001
0.0272	21.91	410000	1.4457	0.8284	0.8021
0.0255	22.18	415000	1.4497	0.8285	0.8030
0.0272	22.45	420000	1.4753	0.8282	0.8026
0.0219	22.71	425000	1.4563	0.8295	0.8020
0.028	22.98	430000	1.4635	0.8301	0.8029
0.0204	23.25	435000	1.4824	0.8289	0.8029
0.0254	23.52	440000	1.4726	0.8291	0.8036
0.0286	23.78	445000	1.4786	0.8295	0.8033
0.0177	24.05	450000	1.4603	0.8301	0.8041
0.0204	24.32	455000	1.4873	0.8294	0.8031
0.0245	24.58	460000	1.4858	0.8298	0.8031
0.0237	24.85	465000	1.4910	0.8292	0.8013
0.0194	25.12	470000	1.4910	0.8289	0.8016
0.0204	25.39	475000	1.4945	0.8310	0.8034
0.0212	25.65	480000	1.4980	0.8294	0.8030
0.0256	25.92	485000	1.4874	0.8299	0.8036
0.015	26.19	490000	1.4961	0.8298	0.8036
0.0194	26.46	495000	1.4987	0.8299	0.8047
0.0197	26.72	500000	1.5119	0.8299	0.8029
0.0203	26.99	505000	1.5063	0.8308	0.8042
0.0148	27.26	510000	1.5112	0.8292	0.8019
0.0134	27.52	515000	1.5136	0.8301	0.8044
0.0199	27.79	520000	1.5042	0.8297	0.8036
0.0146	28.06	525000	1.5049	0.8305	0.8040
0.0159	28.33	530000	1.5063	0.8294	0.8034
0.0141	28.59	535000	1.5145	0.8310	0.8054
0.0134	28.86	540000	1.5125	0.8307	0.8049
0.0129	29.13	545000	1.5142	0.8303	0.8045
0.0152	29.39	550000	1.5152	0.8303	0.8045
0.0125	29.66	555000	1.5169	0.8305	0.8050
0.0138	29.93	560000	1.5166	0.8307	0.8049

Framework versions

• Transformers 4.33.3
• Pytorch 2.1.1+cu121
• Datasets 2.14.5
• Tokenizers 0.13.3