scenario-NON-KD-SCR-D2_data-AmazonScience_massive_all_1_1_b

This model is a fine-tuned version of xlm-roberta-base on the massive dataset. It achieves the following results on the evaluation set:

• Loss: 1.9349
• Accuracy: 0.8048
• F1: 0.7709

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 32
• eval_batch_size: 32
• seed: 12334
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 30

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	F1
1.5221	0.27	5000	1.4996	0.6091	0.4809
1.0768	0.53	10000	1.1287	0.7012	0.6353
0.9058	0.8	15000	0.9466	0.7494	0.6859
0.6321	1.07	20000	0.8635	0.7758	0.7172
0.6206	1.34	25000	0.8650	0.7799	0.7389
0.6011	1.6	30000	0.8515	0.7851	0.7471
0.582	1.87	35000	0.7911	0.7927	0.7500
0.381	2.14	40000	0.8422	0.7969	0.7582
0.3811	2.41	45000	0.7978	0.8049	0.7634
0.3962	2.67	50000	0.8069	0.8056	0.7683
0.3928	2.94	55000	0.7850	0.8088	0.7734
0.2613	3.21	60000	0.8674	0.8020	0.7636
0.2626	3.47	65000	0.8891	0.8082	0.7787
0.2705	3.74	70000	0.8417	0.8095	0.7810
0.2428	4.01	75000	0.9267	0.8102	0.7786
0.1796	4.28	80000	0.9683	0.8073	0.7729
0.1978	4.54	85000	0.9646	0.8071	0.7786
0.2013	4.81	90000	1.0033	0.8058	0.7773
0.1243	5.08	95000	1.0949	0.8050	0.7805
0.1398	5.34	100000	1.1007	0.8070	0.7750
0.1517	5.61	105000	1.0743	0.8073	0.7728
0.1547	5.88	110000	1.0960	0.8063	0.7801
0.0951	6.15	115000	1.2261	0.8019	0.7701
0.1117	6.41	120000	1.2443	0.8052	0.7785
0.1307	6.68	125000	1.1847	0.8057	0.7760
0.122	6.95	130000	1.2076	0.8072	0.7782
0.0901	7.22	135000	1.2994	0.8046	0.7769
0.0916	7.48	140000	1.3590	0.8039	0.7736
0.1014	7.75	145000	1.2950	0.8033	0.7770
0.0752	8.02	150000	1.3555	0.8038	0.7750
0.0759	8.28	155000	1.3642	0.8042	0.7743
0.0806	8.55	160000	1.3728	0.8035	0.7770
0.0826	8.82	165000	1.3975	0.8053	0.7782
0.0498	9.09	170000	1.4578	0.8048	0.7752
0.0681	9.35	175000	1.4855	0.8043	0.7742
0.0725	9.62	180000	1.4749	0.8036	0.7752
0.0729	9.89	185000	1.4769	0.8027	0.7730
0.0458	10.15	190000	1.5603	0.7987	0.7690
0.0643	10.42	195000	1.5069	0.8041	0.7739
0.0659	10.69	200000	1.5066	0.8040	0.7737
0.0698	10.96	205000	1.5022	0.8048	0.7771
0.0506	11.22	210000	1.5427	0.8040	0.7737
0.057	11.49	215000	1.6248	0.7998	0.7667
0.0563	11.76	220000	1.5648	0.8006	0.7715
0.0401	12.03	225000	1.6269	0.8018	0.7733
0.0428	12.29	230000	1.6632	0.8017	0.7731
0.0542	12.56	235000	1.6165	0.8057	0.7726
0.0563	12.83	240000	1.6232	0.8025	0.7695
0.0327	13.09	245000	1.6111	0.8042	0.7698
0.0327	13.36	250000	1.7038	0.8031	0.7695
0.0414	13.63	255000	1.6716	0.8032	0.7717
0.0489	13.9	260000	1.6327	0.8040	0.7728
0.0349	14.16	265000	1.7026	0.8000	0.7617
0.0338	14.43	270000	1.7035	0.8029	0.7705
0.0438	14.7	275000	1.6773	0.8012	0.7688
0.0354	14.96	280000	1.7168	0.8035	0.7737
0.0293	15.23	285000	1.7188	0.7986	0.7669
0.0312	15.5	290000	1.7136	0.8010	0.7691
0.0371	15.77	295000	1.7322	0.8008	0.7703
0.0232	16.03	300000	1.7465	0.8018	0.7720
0.0264	16.3	305000	1.7673	0.8031	0.7722
0.0321	16.57	310000	1.7565	0.8029	0.7710
0.0361	16.84	315000	1.7732	0.8005	0.7694
0.0193	17.1	320000	1.8060	0.7995	0.7625
0.0312	17.37	325000	1.8417	0.7961	0.7666
0.0278	17.64	330000	1.7811	0.8002	0.7677
0.0304	17.9	335000	1.7688	0.8031	0.7741
0.0212	18.17	340000	1.8303	0.8022	0.7753
0.0295	18.44	345000	1.8007	0.8000	0.7668
0.0243	18.71	350000	1.8216	0.8052	0.7731
0.0306	18.97	355000	1.7698	0.8015	0.7691
0.0214	19.24	360000	1.8350	0.7996	0.7623
0.0191	19.51	365000	1.8295	0.8011	0.7638
0.0191	19.77	370000	1.8074	0.8053	0.7731
0.0172	20.04	375000	1.8719	0.8009	0.7715
0.0178	20.31	380000	1.8405	0.8031	0.7717
0.023	20.58	385000	1.8839	0.8004	0.7643
0.0185	20.84	390000	1.9008	0.7975	0.7645
0.0167	21.11	395000	1.8486	0.8029	0.7712
0.0157	21.38	400000	1.8701	0.7990	0.7677
0.017	21.65	405000	1.8187	0.8010	0.7705
0.0214	21.91	410000	1.7915	0.8012	0.7688
0.0125	22.18	415000	1.8853	0.8031	0.7707
0.0147	22.45	420000	1.8881	0.8002	0.7673
0.0194	22.71	425000	1.8614	0.8016	0.7708
0.0136	22.98	430000	1.8593	0.8019	0.7693
0.0161	23.25	435000	1.8706	0.8002	0.7673
0.0155	23.52	440000	1.8959	0.8028	0.7709
0.0165	23.78	445000	1.9264	0.8008	0.7649
0.0106	24.05	450000	1.9267	0.8020	0.7678
0.0119	24.32	455000	1.9022	0.8017	0.7691
0.0081	24.58	460000	1.9048	0.8016	0.7704
0.0107	24.85	465000	1.9242	0.8033	0.7698
0.012	25.12	470000	1.9326	0.8022	0.7685
0.0085	25.39	475000	1.9369	0.8034	0.7701
0.0097	25.65	480000	1.9317	0.8027	0.7716
0.0106	25.92	485000	1.8838	0.8019	0.7700
0.0118	26.19	490000	1.9157	0.8000	0.7654
0.0108	26.46	495000	1.9159	0.8041	0.7713
0.0077	26.72	500000	1.9163	0.8027	0.7714
0.0088	26.99	505000	1.8850	0.8022	0.7678
0.009	27.26	510000	1.9129	0.8032	0.7691
0.0081	27.52	515000	1.9209	0.8025	0.7695
0.0069	27.79	520000	1.9289	0.8033	0.7701
0.0086	28.06	525000	1.9292	0.8029	0.7698
0.0052	28.33	530000	1.9519	0.8039	0.7701
0.0081	28.59	535000	1.9357	0.8028	0.7687
0.0086	28.86	540000	1.9125	0.8048	0.7708
0.0081	29.13	545000	1.9113	0.8048	0.7695
0.0064	29.39	550000	1.9320	0.8047	0.7702
0.0081	29.66	555000	1.9335	0.8047	0.7704
0.006	29.93	560000	1.9349	0.8048	0.7709

Framework versions

• Transformers 4.33.3
• Pytorch 2.1.1+cu121
• Datasets 2.14.5
• Tokenizers 0.13.3