scenario-NON-KD-PR-COPY-D2_data-AmazonScience_massive_all_1_1_betta-jason

This model is a fine-tuned version of xlm-roberta-base on the massive dataset. It achieves the following results on the evaluation set:

• Loss: 1.4867
• Accuracy: 0.8342
• F1: 0.8095

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 32
• eval_batch_size: 32
• seed: 222
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 30

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	F1
1.6555	0.27	5000	1.5175	0.5778	0.4204
1.1914	0.53	10000	1.1206	0.6936	0.5745
0.9781	0.8	15000	0.9691	0.7383	0.6438
0.7905	1.07	20000	0.8906	0.7631	0.6846
0.7201	1.34	25000	0.8162	0.7809	0.7151
0.6624	1.6	30000	0.7933	0.7914	0.7310
0.6399	1.87	35000	0.7562	0.8000	0.7518
0.5163	2.14	40000	0.7660	0.8057	0.7589
0.503	2.41	45000	0.7659	0.8060	0.7609
0.4855	2.67	50000	0.7446	0.8110	0.7707
0.4744	2.94	55000	0.7199	0.8160	0.7793
0.3807	3.21	60000	0.7582	0.8165	0.7787
0.3902	3.47	65000	0.7526	0.8177	0.7773
0.3757	3.74	70000	0.7647	0.8158	0.7771
0.3496	4.01	75000	0.7529	0.8234	0.7867
0.2917	4.28	80000	0.7961	0.8181	0.7830
0.3042	4.54	85000	0.7907	0.8190	0.7837
0.3098	4.81	90000	0.7699	0.8237	0.7878
0.2381	5.08	95000	0.8027	0.8224	0.7842
0.244	5.34	100000	0.8074	0.8242	0.7898
0.2509	5.61	105000	0.8052	0.8260	0.7924
0.2709	5.88	110000	0.8002	0.8258	0.7925
0.2001	6.15	115000	0.8449	0.8242	0.7909
0.2175	6.41	120000	0.8669	0.8236	0.7892
0.214	6.68	125000	0.8765	0.8252	0.7987
0.222	6.95	130000	0.8332	0.8288	0.8003
0.1713	7.22	135000	0.9019	0.8262	0.7951
0.1751	7.48	140000	0.8958	0.8255	0.7957
0.1794	7.75	145000	0.9065	0.8260	0.7982
0.1544	8.02	150000	0.9200	0.8263	0.8020
0.1445	8.28	155000	0.9510	0.8240	0.7975
0.155	8.55	160000	0.9418	0.8294	0.7999
0.157	8.82	165000	0.9463	0.8295	0.8055
0.1182	9.09	170000	0.9762	0.8273	0.8020
0.1303	9.35	175000	0.9829	0.8271	0.8032
0.131	9.62	180000	1.0006	0.8292	0.8019
0.127	9.89	185000	0.9976	0.8251	0.8009
0.092	10.15	190000	1.0280	0.8278	0.8002
0.1131	10.42	195000	1.0338	0.8271	0.8018
0.1135	10.69	200000	1.0388	0.8277	0.8009
0.115	10.96	205000	1.0341	0.8278	0.8010
0.0871	11.22	210000	1.0720	0.8282	0.8022
0.0992	11.49	215000	1.0691	0.8292	0.8040
0.1007	11.76	220000	1.0821	0.8279	0.8017
0.0776	12.03	225000	1.1169	0.8260	0.7960
0.0833	12.29	230000	1.1196	0.8283	0.8031
0.0843	12.56	235000	1.1386	0.8286	0.8035
0.0884	12.83	240000	1.1368	0.8281	0.8006
0.0637	13.09	245000	1.1611	0.8265	0.8032
0.0721	13.36	250000	1.1857	0.8248	0.7988
0.0731	13.63	255000	1.1788	0.8294	0.8028
0.0754	13.9	260000	1.1879	0.8266	0.8027
0.0615	14.16	265000	1.2059	0.8308	0.8049
0.0741	14.43	270000	1.2121	0.8280	0.8019
0.0677	14.7	275000	1.2192	0.8296	0.8033
0.0736	14.96	280000	1.2419	0.8266	0.7993
0.0561	15.23	285000	1.2439	0.8288	0.8006
0.0554	15.5	290000	1.2603	0.8282	0.7990
0.0634	15.77	295000	1.2692	0.8279	0.8009
0.0445	16.03	300000	1.2826	0.8284	0.8010
0.052	16.3	305000	1.2949	0.8287	0.8048
0.0568	16.57	310000	1.3029	0.8284	0.8031
0.0484	16.84	315000	1.2977	0.8298	0.8043
0.0446	17.1	320000	1.3212	0.8280	0.8031
0.0462	17.37	325000	1.3350	0.8277	0.8013
0.047	17.64	330000	1.3301	0.8297	0.8042
0.048	17.9	335000	1.3293	0.8297	0.8041
0.0421	18.17	340000	1.3249	0.8286	0.8023
0.0405	18.44	345000	1.3471	0.8277	0.8025
0.0458	18.71	350000	1.3654	0.8302	0.8037
0.0463	18.97	355000	1.3435	0.8311	0.8043
0.0392	19.24	360000	1.3816	0.8294	0.8051
0.0379	19.51	365000	1.3748	0.8315	0.8077
0.0358	19.77	370000	1.3599	0.8322	0.8064
0.0276	20.04	375000	1.3637	0.8318	0.8080
0.0359	20.31	380000	1.3649	0.8322	0.8068
0.0322	20.58	385000	1.3857	0.8305	0.8040
0.0404	20.84	390000	1.3926	0.8302	0.8048
0.0338	21.11	395000	1.3937	0.8311	0.8048
0.0307	21.38	400000	1.4248	0.8294	0.8043
0.0301	21.65	405000	1.4184	0.8296	0.8050
0.0289	21.91	410000	1.4154	0.8307	0.8053
0.0266	22.18	415000	1.4249	0.8304	0.8057
0.0282	22.45	420000	1.4311	0.8319	0.8085
0.0306	22.71	425000	1.4417	0.8306	0.8055
0.0272	22.98	430000	1.4490	0.8302	0.8036
0.0264	23.25	435000	1.4372	0.8321	0.8061
0.0232	23.52	440000	1.4548	0.8304	0.8057
0.0264	23.78	445000	1.4496	0.8317	0.8065
0.0194	24.05	450000	1.4454	0.8320	0.8073
0.0228	24.32	455000	1.4532	0.8327	0.8077
0.0229	24.58	460000	1.4574	0.8325	0.8068
0.0203	24.85	465000	1.4709	0.8310	0.8066
0.0183	25.12	470000	1.4707	0.8327	0.8090
0.0157	25.39	475000	1.4689	0.8334	0.8089
0.0175	25.65	480000	1.4704	0.8324	0.8076
0.0211	25.92	485000	1.4806	0.8319	0.8065
0.0158	26.19	490000	1.4881	0.8326	0.8067
0.0209	26.46	495000	1.4771	0.8335	0.8084
0.0193	26.72	500000	1.4882	0.8325	0.8076
0.0184	26.99	505000	1.4740	0.8333	0.8088
0.0145	27.26	510000	1.4818	0.8339	0.8095
0.0141	27.52	515000	1.4909	0.8327	0.8075
0.0157	27.79	520000	1.4787	0.8331	0.8086
0.0168	28.06	525000	1.4842	0.8336	0.8078
0.0179	28.33	530000	1.4847	0.8338	0.8084
0.014	28.59	535000	1.4846	0.8339	0.8089
0.0145	28.86	540000	1.4856	0.8333	0.8087
0.0135	29.13	545000	1.4864	0.8336	0.8085
0.0127	29.39	550000	1.4852	0.8340	0.8088
0.0159	29.66	555000	1.4874	0.8342	0.8093
0.0144	29.93	560000	1.4867	0.8342	0.8095

Framework versions

• Transformers 4.33.3
• Pytorch 2.1.1+cu121
• Datasets 2.14.5
• Tokenizers 0.13.3