kugler/distilbert-base-german-cased-defakts-fake-binary

destilbert-base-german-cased

This Model is finetuned for sequence classification (binary fake-news classification task) on the german DeFaktS-Dataset. It achieves the following results on the evaluation set:

• Loss: 0.3515
• Accuracy: 0.8526
• F1: 0.8476
• Precision: 0.8474
• Recall: 0.8478

Model description

This Model is finetuned for sequence classification

Dataset

Trained on the DeFactS dataset https://github.com/caisa-lab/DeFaktS-Dataset-Disinformaton-Detection, feature catposfake/catneutral to detect fake news

Intended uses & limitations

Fake news classification

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 6

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy@de	F1@de	Precision@de	Recall@de	Loss@de
0.6376	0.0888	50	0.5454	0.7081	0.6726	0.7130	0.6704	0.5453
0.4988	0.1776	100	0.4341	0.8021	0.7930	0.7971	0.7902	0.4341
0.4158	0.2664	150	0.4317	0.8026	0.7901	0.8030	0.7842	0.4317
0.4127	0.3552	200	0.3989	0.8166	0.8108	0.8100	0.8117	0.3989
0.4109	0.4440	250	0.3788	0.8271	0.8191	0.8237	0.8159	0.3787
0.3458	0.5329	300	0.3932	0.8116	0.8089	0.8076	0.8173	0.3932
0.3775	0.6217	350	0.3649	0.8336	0.8237	0.8355	0.8176	0.3649
0.3702	0.7105	400	0.3579	0.8341	0.8286	0.8282	0.8291	0.3579
0.3549	0.7993	450	0.3480	0.8421	0.8365	0.8368	0.8363	0.3480
0.3446	0.8881	500	0.3840	0.8386	0.8252	0.8534	0.8152	0.3840
0.3302	0.9769	550	0.3591	0.8366	0.8322	0.8303	0.8350	0.3591
0.2635	1.0657	600	0.3731	0.8401	0.8327	0.8374	0.8295	0.3731
0.2598	1.1545	650	0.3728	0.8436	0.8380	0.8384	0.8375	0.3728
0.2658	1.2433	700	0.3773	0.8431	0.8301	0.8585	0.8199	0.3773
0.2345	1.3321	750	0.3655	0.8516	0.8417	0.8588	0.8337	0.3656
0.2206	1.4210	800	0.3612	0.8601	0.8536	0.8587	0.8500	0.3612
0.2715	1.5098	850	0.3449	0.8481	0.8427	0.8430	0.8425	0.3450
0.2469	1.5986	900	0.3476	0.8516	0.8449	0.8492	0.8418	0.3476
0.2594	1.6874	950	0.3516	0.8441	0.8396	0.8380	0.8417	0.3516
0.2268	1.7762	1000	0.3515	0.8526	0.8476	0.8474	0.8478	0.3515
0.2431	1.8650	1050	0.3636	0.8531	0.8448	0.8554	0.8388	0.3636
0.2417	1.9538	1100	0.3842	0.8451	0.8333	0.8566	0.8241	0.3842
0.1908	2.0426	1150	0.4307	0.8576	0.8539	0.8517	0.8573	0.4306
0.1213	2.1314	1200	0.4600	0.8531	0.8462	0.8515	0.8425	0.4599
0.1422	2.2202	1250	0.4175	0.8551	0.8497	0.8507	0.8488	0.4174
0.1288	2.3091	1300	0.4744	0.8551	0.8482	0.8537	0.8444	0.4743
0.1333	2.3979	1350	0.4399	0.8566	0.8510	0.8526	0.8497	0.4399
0.1234	2.4867	1400	0.4777	0.8531	0.8440	0.8580	0.8369	0.4776
0.1409	2.5755	1450	0.4435	0.8561	0.8489	0.8557	0.8445	0.4434
0.1028	2.6643	1500	0.4783	0.8521	0.8459	0.8488	0.8436	0.4782
0.1178	2.7531	1550	0.4530	0.8606	0.8534	0.8612	0.8485	0.4529
0.1343	2.8419	1600	0.4320	0.8546	0.8490	0.8505	0.8476	0.4320
0.1408	2.9307	1650	0.4398	0.8541	0.8461	0.8556	0.8406	0.4397
0.1197	3.0195	1700	0.5015	0.8581	0.8494	0.8632	0.8422	0.5015
0.0603	3.1083	1750	0.6439	0.8536	0.8449	0.8573	0.8383	0.6438
0.0594	3.1972	1800	0.6179	0.8541	0.8476	0.8517	0.8445	0.6177
0.0501	3.2860	1850	0.6482	0.8596	0.8511	0.8642	0.8443	0.6481
0.0582	3.3748	1900	0.6923	0.8591	0.8506	0.8637	0.8437	0.6921
0.0513	3.4636	1950	0.7074	0.8511	0.8455	0.8466	0.8444	0.7072
0.1219	3.5524	2000	0.6395	0.8546	0.8483	0.8519	0.8455	0.6394
0.0643	3.6412	2050	0.6353	0.8596	0.8540	0.8560	0.8524	0.6351
0.0558	3.7300	2100	0.6717	0.8561	0.8513	0.8508	0.8519	0.6715
0.0687	3.8188	2150	0.6898	0.8501	0.8424	0.8499	0.8377	0.6897
0.0667	3.9076	2200	0.6625	0.8551	0.8482	0.8537	0.8444	0.6625
0.0695	3.9964	2250	0.6391	0.8561	0.8495	0.8543	0.8460	0.6390
0.026	4.0853	2300	0.7211	0.8566	0.8498	0.8553	0.8461	0.7210
0.0279	4.1741	2350	0.7191	0.8551	0.8501	0.8500	0.8503	0.7189
0.0315	4.2629	2400	0.7622	0.8551	0.8480	0.8543	0.8439	0.7621
0.0281	4.3517	2450	0.7876	0.8561	0.8484	0.8573	0.8430	0.7874
0.0403	4.4405	2500	0.7719	0.8606	0.8538	0.8600	0.8496	0.7717
0.0238	4.5293	2550	0.7860	0.8571	0.8503	0.8559	0.8465	0.7858
0.0349	4.6181	2600	0.7675	0.8536	0.8458	0.8543	0.8407	0.7674
0.052	4.7069	2650	0.7553	0.8536	0.8483	0.8489	0.8477	0.7552
0.0361	4.7957	2700	0.7740	0.8566	0.8494	0.8565	0.8448	0.7738
0.0462	4.8845	2750	0.7590	0.8556	0.8490	0.8537	0.8456	0.7588
0.0302	4.9734	2800	0.7862	0.8551	0.8482	0.8539	0.8442	0.7860
0.017	5.0622	2850	0.7971	0.8571	0.8508	0.8547	0.8478	0.7969
0.0144	5.1510	2900	0.8138	0.8576	0.8516	0.8545	0.8494	0.8136
0.0095	5.2398	2950	0.8287	0.8596	0.8532	0.8578	0.8499	0.8285
0.0212	5.3286	3000	0.8530	0.8571	0.8503	0.8560	0.8463	0.8529
0.0176	5.4174	3050	0.8345	0.8571	0.8515	0.8533	0.8499	0.8343
0.022	5.5062	3100	0.8400	0.8581	0.8521	0.8552	0.8496	0.8398
0.0261	5.5950	3150	0.8283	0.8576	0.8519	0.8539	0.8503	0.8280
0.0191	5.6838	3200	0.8346	0.8566	0.8506	0.8534	0.8483	0.8344
0.0131	5.7726	3250	0.8382	0.8576	0.8516	0.8546	0.8492	0.8380
0.0084	5.8615	3300	0.8402	0.8586	0.8528	0.8553	0.8508	0.8400
0.0136	5.9503	3350	0.8425	0.8581	0.8520	0.8554	0.8494	0.8423

Framework versions

• Transformers 4.45.2
• Pytorch 2.3.1+cu121
• Tokenizers 0.20.3