Hugging Face's logo

gokuls
/
mobilebert_add_pre-training-complete

Fill-Mask Transformers PyTorch TensorBoard
mobilebert generated_from_trainer Eval Results Inference Endpoints
Model card Files Metrics Community
1
mobilebert_add_pre-training-complete / README.md
gokuls's picture
gokuls
update model card README.md
0fd8839
preview code
|
raw history blame contribute delete
12.8 kB
metadata
license: apache-2.0
tags:
  - generated_from_trainer
datasets:
  - wikitext
metrics:
  - accuracy
model-index:
  - name: mobilebert_add_pre-training-complete
    results:
      - task:
          name: Masked Language Modeling
          type: fill-mask
        dataset:
          name: wikitext wikitext-103-raw-v1
          type: wikitext
          config: wikitext-103-raw-v1
          split: validation
          args: wikitext-103-raw-v1
        metrics:
          - name: Accuracy
            type: accuracy
            value: 0.46066022044935584

mobilebert_add_pre-training-complete

This model is a fine-tuned version of google/mobilebert-uncased on the wikitext wikitext-103-raw-v1 dataset. It achieves the following results on the evaluation set:

  • Loss: 2.9849
  • Accuracy: 0.4607

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 5e-05
  • train_batch_size: 64
  • eval_batch_size: 64
  • seed: 10
  • distributed_type: multi-GPU
  • num_devices: 2
  • total_train_batch_size: 128
  • total_eval_batch_size: 128
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_steps: 100
  • training_steps: 300000

Training results

Training Loss Epoch Step Validation Loss Accuracy
4.8119 1.0 1787 4.3700 0.3199
4.2649 2.0 3574 4.0930 0.3445
4.0457 3.0 5361 3.9375 0.3545
3.9099 4.0 7148 3.8534 0.3644
3.8193 5.0 8935 3.7993 0.3669
3.7517 6.0 10722 3.7414 0.3730
3.6983 7.0 12509 3.6737 0.3818
3.6565 8.0 14296 3.6657 0.3794
3.619 9.0 16083 3.6129 0.3869
3.5899 10.0 17870 3.5804 0.3910
3.5597 11.0 19657 3.5432 0.3964
3.5329 12.0 21444 3.5397 0.3958
3.5088 13.0 23231 3.4896 0.4011
3.4904 14.0 25018 3.4731 0.4000
3.4703 15.0 26805 3.4971 0.3994
3.4533 16.0 28592 3.4609 0.4049
3.4369 17.0 30379 3.4411 0.4067
3.423 18.0 32166 3.4219 0.4066
3.4084 19.0 33953 3.4477 0.4014
3.3949 20.0 35740 3.4013 0.4087
3.3811 21.0 37527 3.3642 0.4130
3.3688 22.0 39314 3.4173 0.4031
3.3598 23.0 41101 3.4018 0.4101
3.3484 24.0 42888 3.3499 0.4143
3.3363 25.0 44675 3.3675 0.4119
3.3274 26.0 46462 3.3562 0.4154
3.3161 27.0 48249 3.3487 0.4159
3.3073 28.0 50036 3.3293 0.4159
3.2991 29.0 51823 3.3317 0.4160
3.2899 30.0 53610 3.3058 0.4183
3.2814 31.0 55397 3.2795 0.4235
3.2734 32.0 57184 3.3185 0.4143
3.266 33.0 58971 3.2682 0.4268
3.2578 34.0 60758 3.3145 0.4181
3.2506 35.0 62545 3.2726 0.4230
3.2423 36.0 64332 3.2735 0.4218
3.2359 37.0 66119 3.2845 0.4175
3.2293 38.0 67906 3.3067 0.4193
3.2207 39.0 69693 3.2586 0.4257
3.2138 40.0 71480 3.2543 0.4250
3.2077 41.0 73267 3.2395 0.4226
3.202 42.0 75054 3.2224 0.4270
3.1964 43.0 76841 3.2562 0.4234
3.1925 44.0 78628 3.2544 0.4251
3.1865 45.0 80415 3.2043 0.4353
3.1812 46.0 82202 3.2280 0.4286
3.1744 47.0 83989 3.2174 0.4276
3.1699 48.0 85776 3.1972 0.4317
3.1652 49.0 87563 3.2016 0.4302
3.1609 50.0 89350 3.2018 0.4338
3.1548 51.0 91137 3.1950 0.4327
3.1508 52.0 92924 3.2128 0.4279
3.1478 53.0 94711 3.2027 0.4303
3.1423 54.0 96498 3.1959 0.4312
3.1383 55.0 98285 3.1911 0.4340
3.1336 56.0 100072 3.1914 0.4320
3.129 57.0 101859 3.1855 0.4312
3.1233 58.0 103646 3.1570 0.4337
3.1198 59.0 105433 3.2042 0.4307
3.1153 60.0 107220 3.1370 0.4390
3.1122 61.0 109007 3.1612 0.4412
3.1093 62.0 110794 3.1642 0.4348
3.1048 63.0 112581 3.1807 0.4326
3.1013 64.0 114368 3.1449 0.4359
3.0977 65.0 116155 3.1408 0.4380
3.0926 66.0 117942 3.1723 0.4365
3.0901 67.0 119729 3.1473 0.4380
3.0882 68.0 121516 3.1401 0.4378
3.0839 69.0 123303 3.1281 0.4374
3.0794 70.0 125090 3.1356 0.4367
3.0766 71.0 126877 3.1019 0.4397
3.074 72.0 128664 3.1626 0.4355
3.0702 73.0 130451 3.1287 0.4387
3.0676 74.0 132238 3.1366 0.4379
3.0648 75.0 134025 3.1782 0.4346
3.0624 76.0 135812 3.1229 0.4427
3.0575 77.0 137599 3.1139 0.4430
3.0549 78.0 139386 3.0948 0.4431
3.052 79.0 141173 3.1030 0.4452
3.0527 80.0 142960 3.0929 0.4448
3.0466 81.0 144747 3.0888 0.4428
3.0439 82.0 146534 3.1035 0.4414
3.0409 83.0 148321 3.1112 0.4411
3.041 84.0 150108 3.1296 0.4399
3.0379 85.0 151895 3.1224 0.4428
3.0332 86.0 153682 3.1101 0.4398
3.0315 87.0 155469 3.1045 0.4423
3.0302 88.0 157256 3.0913 0.4446
3.0265 89.0 159043 3.0745 0.4447
3.0243 90.0 160830 3.0942 0.4443
3.0222 91.0 162617 3.0821 0.4432
3.021 92.0 164404 3.0616 0.4473
3.0183 93.0 166191 3.1021 0.4450
3.0155 94.0 167978 3.1163 0.4422
3.0132 95.0 169765 3.0645 0.4493
3.0118 96.0 171552 3.0922 0.4420
3.0105 97.0 173339 3.1187 0.4423
3.0063 98.0 175126 3.1061 0.4462
3.0035 99.0 176913 3.1098 0.4424
3.0025 100.0 178700 3.0856 0.4454
3.0001 101.0 180487 3.0584 0.4504
2.9979 102.0 182274 3.0897 0.4435
2.9963 103.0 184061 3.0712 0.4437
2.9944 104.0 185848 3.0853 0.4458
2.9931 105.0 187635 3.0809 0.4475
2.992 106.0 189422 3.0910 0.4426
2.9886 107.0 191209 3.0693 0.4490
2.986 108.0 192996 3.0906 0.4445
2.9834 109.0 194783 3.0320 0.4538
2.9829 110.0 196570 3.0760 0.4456
2.9814 111.0 198357 3.0423 0.4504
2.9795 112.0 200144 3.0411 0.4529
2.979 113.0 201931 3.0784 0.4463
2.9781 114.0 203718 3.0526 0.4537
2.9751 115.0 205505 3.0479 0.4512
2.9749 116.0 207292 3.0545 0.4493
2.9735 117.0 209079 3.0529 0.4485
2.9705 118.0 210866 3.0080 0.4581
2.9698 119.0 212653 3.0271 0.4537
2.9674 120.0 214440 3.0477 0.4482
2.9666 121.0 216227 3.0328 0.4558
2.9664 122.0 218014 3.0689 0.4463
2.9639 123.0 219801 3.0749 0.4459
2.9633 124.0 221588 3.0505 0.4489
2.9618 125.0 223375 3.0256 0.4535
2.9589 126.0 225162 3.0522 0.4496
2.9584 127.0 226949 3.0451 0.4530
2.9589 128.0 228736 3.0654 0.4502
2.9581 129.0 230523 2.9989 0.4580
2.9554 130.0 232310 3.0347 0.4508
2.9565 131.0 234097 3.0586 0.4498
2.9548 132.0 235884 3.0170 0.4536
2.9515 133.0 237671 3.0470 0.4492
2.9499 134.0 239458 3.0339 0.4515
2.9514 135.0 241245 3.0474 0.4473
2.9486 136.0 243032 3.0427 0.4493
2.9483 137.0 244819 3.0336 0.4534
2.9491 138.0 246606 3.0274 0.4516
2.9465 139.0 248393 3.0354 0.4539
2.9447 140.0 250180 3.0139 0.4526
2.9449 141.0 251967 3.0163 0.4548
2.9439 142.0 253754 3.0308 0.4534
2.9435 143.0 255541 3.0242 0.4579
2.943 144.0 257328 3.0437 0.4513
2.943 145.0 259115 3.0227 0.4544
2.9403 146.0 260902 3.0464 0.4478
2.9407 147.0 262689 3.0718 0.4465
2.9397 148.0 264476 3.0519 0.4487
2.9392 149.0 266263 3.0163 0.4558
2.9377 150.0 268050 3.0159 0.4518
2.9386 151.0 269837 3.0010 0.4545
2.9391 152.0 271624 3.0346 0.4530
2.9364 153.0 273411 3.0039 0.4541
2.9359 154.0 275198 3.0417 0.4519
2.9359 155.0 276985 3.0161 0.4544
2.936 156.0 278772 3.0169 0.4534
2.9329 157.0 280559 3.0594 0.4478
2.9336 158.0 282346 3.0265 0.4555
2.9341 159.0 284133 3.0276 0.4542
2.933 160.0 285920 3.0324 0.4524
2.9325 161.0 287707 3.0249 0.4489
2.932 162.0 289494 3.0444 0.4519
2.9334 163.0 291281 3.0420 0.4494
2.9318 164.0 293068 2.9972 0.4541
2.9316 165.0 294855 2.9973 0.4526
2.9318 166.0 296642 3.0389 0.4529
2.9301 167.0 298429 3.0131 0.4557
2.9291 167.88 300000 3.0067 0.4548

Framework versions

  • Transformers 4.26.0
  • Pytorch 1.14.0a0+410ce96
  • Datasets 2.8.0
  • Tokenizers 0.13.2