scenario-KD-SCR-MSV-D2_data-AmazonScience_massive_all_1_1_delta-jason

This model is a fine-tuned version of FacebookAI/xlm-roberta-base on the massive dataset. It achieves the following results on the evaluation set:

• Loss: 1.7914
• Accuracy: 0.8274
• F1: 0.8029

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 32
• eval_batch_size: 32
• seed: 1254
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 30

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	F1
5.7233	0.27	5000	5.5374	0.3569	0.1678
3.9329	0.53	10000	3.9116	0.5835	0.4241
3.1699	0.8	15000	3.1794	0.6720	0.5600
2.5004	1.07	20000	2.7930	0.7147	0.6299
2.289	1.34	25000	2.5799	0.7385	0.6658
2.1906	1.6	30000	2.4334	0.7534	0.6852
2.0248	1.87	35000	2.3295	0.7659	0.6957
1.7159	2.14	40000	2.2350	0.7731	0.7166
1.669	2.41	45000	2.2103	0.7811	0.7321
1.6191	2.67	50000	2.1297	0.7874	0.7426
1.587	2.94	55000	2.0855	0.7890	0.7448
1.3031	3.21	60000	2.1223	0.7920	0.7492
1.3	3.47	65000	2.0614	0.7952	0.7563
1.337	3.74	70000	2.0343	0.8002	0.7618
1.2406	4.01	75000	2.0565	0.7999	0.7616
1.1146	4.28	80000	2.0439	0.8006	0.7686
1.0767	4.54	85000	2.0461	0.8001	0.7663
1.1244	4.81	90000	2.0096	0.8032	0.7699
0.91	5.08	95000	2.0492	0.8064	0.7710
0.9446	5.34	100000	2.0278	0.8059	0.7742
0.9581	5.61	105000	2.0434	0.8034	0.7750
0.9525	5.88	110000	2.0068	0.8073	0.7794
0.809	6.15	115000	2.0286	0.8072	0.7773
0.8597	6.41	120000	2.0080	0.8104	0.7819
0.8219	6.68	125000	2.0049	0.8093	0.7816
0.8507	6.95	130000	2.0095	0.8094	0.7866
0.7371	7.22	135000	2.0533	0.8105	0.7863
0.754	7.48	140000	1.9987	0.8110	0.7827
0.7829	7.75	145000	2.0120	0.8079	0.7825
0.6798	8.02	150000	1.9928	0.8144	0.7883
0.6619	8.28	155000	1.9941	0.8127	0.7861
0.6869	8.55	160000	2.0368	0.8119	0.7889
0.695	8.82	165000	2.0073	0.8133	0.7869
0.6395	9.09	170000	2.0095	0.8110	0.7879
0.6274	9.35	175000	1.9915	0.8156	0.7924
0.6186	9.62	180000	2.0114	0.8158	0.7912
0.643	9.89	185000	1.9917	0.8143	0.7936
0.5898	10.15	190000	2.0036	0.8156	0.7905
0.5948	10.42	195000	1.9868	0.8168	0.7904
0.6093	10.69	200000	1.9822	0.8153	0.7892
0.5942	10.96	205000	1.9939	0.8144	0.7912
0.5497	11.22	210000	1.9786	0.8169	0.7961
0.5516	11.49	215000	1.9650	0.8168	0.7913
0.5591	11.76	220000	1.9793	0.8175	0.7927
0.5103	12.03	225000	1.9715	0.8183	0.7942
0.5165	12.29	230000	1.9620	0.8172	0.7936
0.5248	12.56	235000	1.9760	0.8179	0.7950
0.5289	12.83	240000	1.9459	0.8190	0.7952
0.4995	13.09	245000	1.9564	0.8185	0.7959
0.4906	13.36	250000	1.9484	0.8186	0.7940
0.5011	13.63	255000	1.9320	0.8188	0.7923
0.4996	13.9	260000	1.9477	0.8164	0.7929
0.4844	14.16	265000	1.9110	0.8207	0.7942
0.4814	14.43	270000	1.9303	0.8190	0.7927
0.4953	14.7	275000	1.9211	0.8208	0.7951
0.4897	14.96	280000	1.9206	0.8209	0.7940
0.4473	15.23	285000	1.9059	0.8214	0.7959
0.4615	15.5	290000	1.9021	0.8229	0.7985
0.4687	15.77	295000	1.9177	0.8204	0.7960
0.4425	16.03	300000	1.9065	0.8225	0.7994
0.451	16.3	305000	1.8924	0.8219	0.7972
0.458	16.57	310000	1.9036	0.8210	0.7953
0.4514	16.84	315000	1.8810	0.8224	0.7960
0.4263	17.1	320000	1.8826	0.8241	0.8003
0.4355	17.37	325000	1.8685	0.8236	0.7991
0.4234	17.64	330000	1.8634	0.8249	0.7994
0.4346	17.9	335000	1.8640	0.8239	0.8001
0.4077	18.17	340000	1.8656	0.8245	0.8006
0.4156	18.44	345000	1.8666	0.8229	0.7990
0.4185	18.71	350000	1.8495	0.8235	0.8005
0.4211	18.97	355000	1.8784	0.8233	0.7982
0.3981	19.24	360000	1.8562	0.8235	0.7993
0.4139	19.51	365000	1.8417	0.8243	0.7986
0.4052	19.77	370000	1.8533	0.8249	0.7998
0.3915	20.04	375000	1.8413	0.8255	0.8020
0.4015	20.31	380000	1.8540	0.8232	0.7991
0.3923	20.58	385000	1.8592	0.8245	0.7995
0.3984	20.84	390000	1.8613	0.8257	0.8026
0.3886	21.11	395000	1.8350	0.8248	0.7985
0.3888	21.38	400000	1.8343	0.8238	0.7984
0.3878	21.65	405000	1.8207	0.8263	0.8013
0.3901	21.91	410000	1.8394	0.8266	0.8034
0.3765	22.18	415000	1.8250	0.8257	0.8017
0.3793	22.45	420000	1.8159	0.8262	0.7997
0.3825	22.71	425000	1.8220	0.8244	0.8009
0.383	22.98	430000	1.8325	0.8265	0.8013
0.3737	23.25	435000	1.8248	0.8259	0.8024
0.3741	23.52	440000	1.8139	0.8258	0.8014
0.3676	23.78	445000	1.8299	0.8264	0.8007
0.3611	24.05	450000	1.8136	0.8261	0.8018
0.3642	24.32	455000	1.8196	0.8263	0.8017
0.3654	24.58	460000	1.8241	0.8249	0.8012
0.3706	24.85	465000	1.8103	0.8255	0.8012
0.3585	25.12	470000	1.8137	0.8263	0.8029
0.3664	25.39	475000	1.8094	0.8260	0.8024
0.3544	25.65	480000	1.8000	0.8279	0.8041
0.3491	25.92	485000	1.8039	0.8264	0.8028
0.3523	26.19	490000	1.7989	0.8279	0.8037
0.3483	26.46	495000	1.8045	0.8276	0.8027
0.3482	26.72	500000	1.8058	0.8264	0.8022
0.3601	26.99	505000	1.8035	0.8269	0.8017
0.3461	27.26	510000	1.7959	0.8273	0.8041
0.3448	27.52	515000	1.8078	0.8271	0.8030
0.3454	27.79	520000	1.7968	0.8273	0.8035
0.3377	28.06	525000	1.7924	0.8270	0.8019
0.3497	28.33	530000	1.7950	0.8277	0.8041
0.3461	28.59	535000	1.7954	0.8282	0.8049
0.3448	28.86	540000	1.7968	0.8270	0.8031
0.3413	29.13	545000	1.7914	0.8279	0.8038
0.3367	29.39	550000	1.7976	0.8274	0.8027
0.3432	29.66	555000	1.7976	0.8271	0.8037
0.3429	29.93	560000	1.7914	0.8274	0.8029

Framework versions

• Transformers 4.33.3
• Pytorch 2.1.1+cu121
• Datasets 2.14.5
• Tokenizers 0.13.3