scenario-NON-KD-SCR-D2_data-AmazonScience_massive_all_1_1_betta-jason

This model is a fine-tuned version of xlm-roberta-base on the massive dataset. It achieves the following results on the evaluation set:

• Loss: 1.9142
• Accuracy: 0.8046
• F1: 0.7728

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 32
• eval_batch_size: 32
• seed: 222
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 30

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	F1
1.5859	0.27	5000	1.5238	0.5916	0.4568
1.0824	0.53	10000	1.0656	0.7161	0.6369
0.9043	0.8	15000	0.9531	0.7483	0.6851
0.6415	1.07	20000	0.8855	0.7695	0.7228
0.621	1.34	25000	0.8840	0.7753	0.7240
0.5937	1.6	30000	0.7962	0.7901	0.7516
0.5686	1.87	35000	0.7952	0.7967	0.7561
0.3671	2.14	40000	0.8418	0.7987	0.7574
0.3902	2.41	45000	0.8381	0.7957	0.7598
0.396	2.67	50000	0.8075	0.8048	0.7694
0.393	2.94	55000	0.7902	0.8080	0.7732
0.2474	3.21	60000	0.9032	0.8080	0.7735
0.2644	3.47	65000	0.9130	0.8025	0.7691
0.2718	3.74	70000	0.8801	0.8054	0.7697
0.2297	4.01	75000	0.9409	0.8096	0.7761
0.17	4.28	80000	1.0125	0.8032	0.7736
0.1986	4.54	85000	0.9995	0.8068	0.7782
0.2025	4.81	90000	0.9642	0.8104	0.7824
0.1189	5.08	95000	1.1317	0.8028	0.7769
0.1335	5.34	100000	1.1220	0.8055	0.7786
0.152	5.61	105000	1.0980	0.8086	0.7790
0.1583	5.88	110000	1.0602	0.8060	0.7772
0.0974	6.15	115000	1.2524	0.8045	0.7746
0.1184	6.41	120000	1.2211	0.8090	0.7829
0.1281	6.68	125000	1.1444	0.8077	0.7812
0.135	6.95	130000	1.1688	0.8062	0.7769
0.0922	7.22	135000	1.3128	0.8067	0.7724
0.0953	7.48	140000	1.3277	0.8065	0.7819
0.1066	7.75	145000	1.3270	0.8050	0.7756
0.0774	8.02	150000	1.3254	0.8050	0.7788
0.0764	8.28	155000	1.4299	0.8011	0.7738
0.0785	8.55	160000	1.4007	0.8034	0.7711
0.084	8.82	165000	1.3885	0.8045	0.7766
0.0563	9.09	170000	1.4746	0.8035	0.7752
0.0645	9.35	175000	1.4897	0.8045	0.7692
0.0759	9.62	180000	1.4882	0.8059	0.7769
0.0773	9.89	185000	1.4405	0.8045	0.7815
0.0546	10.15	190000	1.5088	0.8031	0.7734
0.0659	10.42	195000	1.5111	0.8002	0.7692
0.0626	10.69	200000	1.5119	0.8051	0.7776
0.0673	10.96	205000	1.5103	0.8043	0.7712
0.05	11.22	210000	1.5920	0.8018	0.7690
0.0523	11.49	215000	1.6002	0.8009	0.7671
0.0542	11.76	220000	1.5411	0.8041	0.7716
0.0383	12.03	225000	1.6058	0.8027	0.7704
0.0407	12.29	230000	1.6273	0.8061	0.7775
0.0455	12.56	235000	1.6502	0.7989	0.7748
0.0558	12.83	240000	1.5998	0.8011	0.7711
0.0294	13.09	245000	1.6627	0.7994	0.7732
0.0425	13.36	250000	1.7116	0.8003	0.7742
0.0422	13.63	255000	1.6802	0.8032	0.7779
0.0468	13.9	260000	1.6578	0.8011	0.7714
0.033	14.16	265000	1.7403	0.8017	0.7721
0.0349	14.43	270000	1.6947	0.8021	0.7709
0.0394	14.7	275000	1.7328	0.8007	0.7711
0.0423	14.96	280000	1.6948	0.8006	0.7721
0.0329	15.23	285000	1.7625	0.7996	0.7701
0.0297	15.5	290000	1.7449	0.8007	0.7717
0.0398	15.77	295000	1.7366	0.7981	0.7654
0.024	16.03	300000	1.7560	0.7997	0.7703
0.0276	16.3	305000	1.7644	0.8004	0.7675
0.0361	16.57	310000	1.7596	0.8028	0.7729
0.0268	16.84	315000	1.7816	0.8038	0.7736
0.0197	17.1	320000	1.7892	0.8022	0.7733
0.028	17.37	325000	1.8219	0.8027	0.7753
0.0287	17.64	330000	1.8050	0.8014	0.7765
0.0272	17.9	335000	1.8047	0.8000	0.7725
0.0213	18.17	340000	1.8086	0.8011	0.7710
0.0254	18.44	345000	1.8148	0.8005	0.7715
0.0243	18.71	350000	1.8234	0.7995	0.7675
0.0246	18.97	355000	1.7890	0.7980	0.7664
0.0214	19.24	360000	1.8467	0.7983	0.7677
0.0233	19.51	365000	1.8218	0.8013	0.7690
0.0216	19.77	370000	1.8382	0.8019	0.7737
0.0153	20.04	375000	1.8232	0.8023	0.7723
0.0193	20.31	380000	1.8526	0.8012	0.7724
0.0173	20.58	385000	1.8398	0.8034	0.7732
0.0205	20.84	390000	1.8113	0.8014	0.7690
0.0168	21.11	395000	1.8381	0.7991	0.7680
0.0175	21.38	400000	1.8405	0.8019	0.7717
0.0186	21.65	405000	1.9043	0.8002	0.7700
0.0184	21.91	410000	1.8670	0.8018	0.7704
0.0114	22.18	415000	1.8526	0.8026	0.7710
0.0135	22.45	420000	1.8631	0.8020	0.7692
0.019	22.71	425000	1.8668	0.8006	0.7709
0.0143	22.98	430000	1.8666	0.8028	0.7729
0.0114	23.25	435000	1.8901	0.8014	0.7720
0.016	23.52	440000	1.8869	0.7994	0.7678
0.0132	23.78	445000	1.8972	0.8025	0.7710
0.0129	24.05	450000	1.9345	0.7991	0.7667
0.0088	24.32	455000	1.9026	0.7997	0.7694
0.0136	24.58	460000	1.8971	0.8011	0.7681
0.0126	24.85	465000	1.9017	0.8017	0.7721
0.0104	25.12	470000	1.9358	0.8028	0.7724
0.0067	25.39	475000	1.9320	0.8019	0.7677
0.0084	25.65	480000	1.9150	0.8032	0.7729
0.0116	25.92	485000	1.9124	0.8020	0.7710
0.0076	26.19	490000	1.9507	0.8030	0.7712
0.012	26.46	495000	1.9480	0.8009	0.7707
0.0112	26.72	500000	1.8995	0.8039	0.7726
0.0092	26.99	505000	1.8921	0.8021	0.7711
0.007	27.26	510000	1.9380	0.8019	0.7703
0.0087	27.52	515000	1.9166	0.8014	0.7709
0.0074	27.79	520000	1.9080	0.8035	0.7743
0.0076	28.06	525000	1.9084	0.8029	0.7710
0.0086	28.33	530000	1.9215	0.8042	0.7729
0.0074	28.59	535000	1.9156	0.8032	0.7723
0.0077	28.86	540000	1.9076	0.8033	0.7717
0.0066	29.13	545000	1.9227	0.8038	0.7733
0.0071	29.39	550000	1.9142	0.8042	0.7729
0.008	29.66	555000	1.9119	0.8042	0.7725
0.0066	29.93	560000	1.9142	0.8046	0.7728

Framework versions

• Transformers 4.33.3
• Pytorch 2.1.1+cu121
• Datasets 2.14.5
• Tokenizers 0.13.3