distilroberta-proppy

This model is a fine-tuned version of distilroberta-base on the proppy corpus. It achieves the following results on the evaluation set:

• Loss: 0.1838
• Acc: 0.9269

Training and evaluation data

The training data is the proppy corpus. See Proppy: Organizing the News Based on Their Propagandistic Content for details.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 32
• eval_batch_size: 32
• seed: 12345
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 16
• num_epochs: 20
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Acc
0.3179	1.0	732	0.2032	0.9146
0.2933	2.0	1464	0.2026	0.9206
0.2938	3.0	2196	0.1849	0.9252
0.3429	4.0	2928	0.1983	0.9221
0.2608	5.0	3660	0.2310	0.9106
0.2562	6.0	4392	0.1826	0.9270
0.2785	7.0	5124	0.1954	0.9228
0.307	8.0	5856	0.2056	0.9200
0.28	9.0	6588	0.1843	0.9259
0.2794	10.0	7320	0.1782	0.9299
0.2868	11.0	8052	0.1907	0.9242
0.2789	12.0	8784	0.2031	0.9216
0.2827	13.0	9516	0.1976	0.9229
0.2795	14.0	10248	0.1866	0.9255
0.2895	15.0	10980	0.1838	0.9269

Framework versions

• Transformers 4.11.2
• Pytorch 1.7.1
• Datasets 1.11.0
• Tokenizers 0.10.3