Edit model card

distilroberta-base-wikitextepoch_50

This model is a fine-tuned version of distilroberta-base on the None dataset. It achieves the following results on the evaluation set:

  • Loss: 1.6360

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 2e-05
  • train_batch_size: 8
  • eval_batch_size: 8
  • seed: 42
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • num_epochs: 50

Training results

Training Loss Epoch Step Validation Loss
1.9729 1.0 2145 1.7725
1.9158 2.0 4290 1.7521
1.8479 3.0 6435 1.7376
1.8081 4.0 8580 1.7272
1.7966 5.0 10725 1.7018
1.7284 6.0 12870 1.7010
1.7198 7.0 15015 1.6868
1.6985 8.0 17160 1.6879
1.6712 9.0 19305 1.6930
1.6489 10.0 21450 1.6594
1.6643 11.0 23595 1.6856
1.6215 12.0 25740 1.6816
1.6125 13.0 27885 1.6714
1.5936 14.0 30030 1.6760
1.5745 15.0 32175 1.6660
1.572 16.0 34320 1.6690
1.5614 17.0 36465 1.6807
1.558 18.0 38610 1.6711
1.5305 19.0 40755 1.6446
1.5021 20.0 42900 1.6573
1.4923 21.0 45045 1.6648
1.5086 22.0 47190 1.6757
1.4895 23.0 49335 1.6525
1.4918 24.0 51480 1.6577
1.4642 25.0 53625 1.6633
1.4604 26.0 55770 1.6462
1.4644 27.0 57915 1.6509
1.4633 28.0 60060 1.6417
1.4188 29.0 62205 1.6519
1.4066 30.0 64350 1.6363
1.409 31.0 66495 1.6419
1.4029 32.0 68640 1.6510
1.4013 33.0 70785 1.6522
1.3939 34.0 72930 1.6498
1.3648 35.0 75075 1.6423
1.3682 36.0 77220 1.6504
1.3603 37.0 79365 1.6511
1.3621 38.0 81510 1.6533
1.3783 39.0 83655 1.6426
1.3707 40.0 85800 1.6542
1.3628 41.0 87945 1.6671
1.3359 42.0 90090 1.6394
1.3433 43.0 92235 1.6409
1.3525 44.0 94380 1.6366
1.3312 45.0 96525 1.6408
1.3389 46.0 98670 1.6225
1.3323 47.0 100815 1.6309
1.3294 48.0 102960 1.6151
1.3356 49.0 105105 1.6374
1.3285 50.0 107250 1.6360

Framework versions

  • Transformers 4.21.0
  • Pytorch 1.5.0
  • Datasets 2.4.0
  • Tokenizers 0.12.1
Downloads last month
1
Inference API
Examples
Mask token: <mask>
This model can be loaded on Inference API (serverless).