Edit model card

distilroberta-proppy

This model is a fine-tuned version of distilroberta-base on the proppy corpus. It achieves the following results on the evaluation set:

  • Loss: 0.1838
  • Acc: 0.9269

Training and evaluation data

The training data is the proppy corpus. See Proppy: Organizing the News Based on Their Propagandistic Content for details.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0001
  • train_batch_size: 32
  • eval_batch_size: 32
  • seed: 12345
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_steps: 16
  • num_epochs: 20
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss Acc
0.3179 1.0 732 0.2032 0.9146
0.2933 2.0 1464 0.2026 0.9206
0.2938 3.0 2196 0.1849 0.9252
0.3429 4.0 2928 0.1983 0.9221
0.2608 5.0 3660 0.2310 0.9106
0.2562 6.0 4392 0.1826 0.9270
0.2785 7.0 5124 0.1954 0.9228
0.307 8.0 5856 0.2056 0.9200
0.28 9.0 6588 0.1843 0.9259
0.2794 10.0 7320 0.1782 0.9299
0.2868 11.0 8052 0.1907 0.9242
0.2789 12.0 8784 0.2031 0.9216
0.2827 13.0 9516 0.1976 0.9229
0.2795 14.0 10248 0.1866 0.9255
0.2895 15.0 10980 0.1838 0.9269

Framework versions

  • Transformers 4.11.2
  • Pytorch 1.7.1
  • Datasets 1.11.0
  • Tokenizers 0.10.3
Downloads last month
672
Hosted inference API
Text Classification
Examples
Examples
This model can be loaded on the Inference API on-demand.