Edit model card

t5-small-mlm-pubmed-15

This model is a fine-tuned version of t5-small on an unknown dataset. It achieves the following results on the evaluation set:

  • Loss: 0.5389
  • Rouge2 Precision: 0.7165
  • Rouge2 Recall: 0.5375
  • Rouge2 Fmeasure: 0.5981

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 2e-05
  • train_batch_size: 16
  • eval_batch_size: 16
  • seed: 42
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • num_epochs: 40
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss Rouge2 Precision Rouge2 Recall Rouge2 Fmeasure
1.1024 0.75 500 0.7890 0.6854 0.4813 0.5502
0.8788 1.51 1000 0.7176 0.6906 0.4989 0.5638
0.8086 2.26 1500 0.6830 0.6872 0.5052 0.5663
0.7818 3.02 2000 0.6650 0.6912 0.5104 0.5711
0.7466 3.77 2500 0.6458 0.6965 0.5167 0.5774
0.731 4.52 3000 0.6355 0.6955 0.5161 0.5763
0.7126 5.28 3500 0.6249 0.6924 0.517 0.576
0.6998 6.03 4000 0.6166 0.6995 0.5207 0.5809
0.6855 6.79 4500 0.6076 0.6981 0.5215 0.5813
0.676 7.54 5000 0.6015 0.7003 0.5242 0.5836
0.6688 8.3 5500 0.5962 0.7004 0.5235 0.583
0.6569 9.05 6000 0.5900 0.6997 0.5234 0.5827
0.6503 9.8 6500 0.5880 0.703 0.5257 0.5856
0.6455 10.56 7000 0.5818 0.7008 0.5259 0.5849
0.635 11.31 7500 0.5796 0.7017 0.5271 0.5861
0.6323 12.07 8000 0.5769 0.7053 0.5276 0.5877
0.6241 12.82 8500 0.5730 0.7011 0.5243 0.5838
0.6224 13.57 9000 0.5696 0.7046 0.5286 0.5879
0.6139 14.33 9500 0.5685 0.7047 0.5295 0.5886
0.6118 15.08 10000 0.5653 0.704 0.5297 0.5886
0.6089 15.84 10500 0.5633 0.703 0.5272 0.5865
0.598 16.59 11000 0.5613 0.7059 0.5293 0.5889
0.6003 17.35 11500 0.5602 0.7085 0.532 0.5918
0.5981 18.1 12000 0.5587 0.7106 0.5339 0.5938
0.5919 18.85 12500 0.5556 0.708 0.5319 0.5914
0.5897 19.61 13000 0.5556 0.7106 0.5327 0.5931
0.5899 20.36 13500 0.5526 0.7114 0.534 0.5939
0.5804 21.12 14000 0.5521 0.7105 0.5328 0.5928
0.5764 21.87 14500 0.5520 0.715 0.537 0.5976
0.5793 22.62 15000 0.5506 0.713 0.5346 0.5951
0.5796 23.38 15500 0.5492 0.7124 0.5352 0.5952
0.5672 24.13 16000 0.5482 0.7124 0.5346 0.5948
0.5737 24.89 16500 0.5470 0.7134 0.5352 0.5956
0.5685 25.64 17000 0.5463 0.7117 0.5346 0.5946
0.5658 26.4 17500 0.5457 0.7145 0.5359 0.5965
0.5657 27.15 18000 0.5447 0.7145 0.5367 0.597
0.5645 27.9 18500 0.5441 0.7141 0.5362 0.5964
0.565 28.66 19000 0.5436 0.7151 0.5367 0.5972
0.5579 29.41 19500 0.5426 0.7162 0.5378 0.5982
0.563 30.17 20000 0.5424 0.7155 0.5373 0.5977
0.556 30.92 20500 0.5418 0.7148 0.536 0.5966
0.5576 31.67 21000 0.5411 0.7141 0.5356 0.5961
0.5546 32.43 21500 0.5409 0.7149 0.5364 0.5967
0.556 33.18 22000 0.5405 0.7143 0.5356 0.596
0.5536 33.94 22500 0.5401 0.7165 0.5377 0.5982
0.5527 34.69 23000 0.5397 0.7188 0.5389 0.5999
0.5531 35.44 23500 0.5395 0.7172 0.538 0.5989
0.5508 36.2 24000 0.5392 0.7166 0.538 0.5985
0.5495 36.95 24500 0.5391 0.7176 0.5387 0.5993
0.5539 37.71 25000 0.5391 0.7169 0.5372 0.598
0.5452 38.46 25500 0.5390 0.7179 0.5384 0.5991
0.5513 39.22 26000 0.5390 0.717 0.5377 0.5984
0.5506 39.97 26500 0.5389 0.7165 0.5375 0.5981

Framework versions

  • Transformers 4.12.5
  • Pytorch 1.10.0+cu111
  • Datasets 1.15.1
  • Tokenizers 0.10.3
Downloads last month
6