--- license: apache-2.0 tags: - generated_from_trainer datasets: - wikitext metrics: - accuracy model-index: - name: distilbert_add_pre-training-complete results: - task: name: Masked Language Modeling type: fill-mask dataset: name: wikitext wikitext-103-raw-v1 type: wikitext config: wikitext-103-raw-v1 split: validation args: wikitext-103-raw-v1 metrics: - name: Accuracy type: accuracy value: 0.23073914743840437 --- # distilbert_add_pre-training-complete This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the wikitext wikitext-103-raw-v1 dataset. It achieves the following results on the evaluation set: - Loss: 5.0239 - Accuracy: 0.2307 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 5e-05 - train_batch_size: 64 - eval_batch_size: 64 - seed: 10 - distributed_type: multi-GPU - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: linear - lr_scheduler_warmup_steps: 100 - training_steps: 300000 - mixed_precision_training: Native AMP ### Training results | Training Loss | Epoch | Step | Validation Loss | Accuracy | |:-------------:|:-----:|:------:|:---------------:|:--------:| | 6.295 | 1.0 | 3573 | 6.0701 | 0.1522 | | 6.0482 | 2.0 | 7146 | 5.9533 | 0.1565 | | 5.9799 | 3.0 | 10719 | 5.9008 | 0.1584 | | 5.9378 | 4.0 | 14292 | 5.8997 | 0.1545 | | 5.9057 | 5.0 | 17865 | 5.8905 | 0.1536 | | 5.8811 | 6.0 | 21438 | 5.8646 | 0.1550 | | 5.8617 | 7.0 | 25011 | 5.8322 | 0.1534 | | 5.844 | 8.0 | 28584 | 5.8563 | 0.1523 | | 5.8297 | 9.0 | 32157 | 5.8352 | 0.1548 | | 5.8175 | 10.0 | 35730 | 5.8136 | 0.1558 | | 5.8056 | 11.0 | 39303 | 5.8147 | 0.1526 | | 5.7921 | 12.0 | 42876 | 5.8020 | 0.1548 | | 5.7777 | 13.0 | 46449 | 5.7891 | 0.1545 | | 5.7596 | 14.0 | 50022 | 5.7370 | 0.1587 | | 5.7414 | 15.0 | 53595 | 5.7396 | 0.1604 | | 5.7243 | 16.0 | 57168 | 5.7490 | 0.1564 | | 5.6997 | 17.0 | 60741 | 5.7135 | 0.1561 | | 5.6698 | 18.0 | 64314 | 5.6858 | 0.1620 | | 5.6398 | 19.0 | 67887 | 5.6735 | 0.1644 | | 5.6135 | 20.0 | 71460 | 5.6174 | 0.1681 | | 5.5899 | 21.0 | 75033 | 5.6191 | 0.1684 | | 5.5699 | 22.0 | 78606 | 5.5977 | 0.1669 | | 5.5487 | 23.0 | 82179 | 5.6139 | 0.1669 | | 5.529 | 24.0 | 85752 | 5.5272 | 0.1741 | | 5.512 | 25.0 | 89325 | 5.5271 | 0.1727 | | 5.4939 | 26.0 | 92898 | 5.5190 | 0.1721 | | 5.4765 | 27.0 | 96471 | 5.4824 | 0.1770 | | 5.4604 | 28.0 | 100044 | 5.5159 | 0.1747 | | 5.4422 | 29.0 | 103617 | 5.4577 | 0.1807 | | 5.4243 | 30.0 | 107190 | 5.4546 | 0.1772 | | 5.408 | 31.0 | 110763 | 5.4297 | 0.1837 | | 5.3915 | 32.0 | 114336 | 5.4089 | 0.1866 | | 5.3766 | 33.0 | 117909 | 5.3996 | 0.1848 | | 5.3594 | 34.0 | 121482 | 5.3974 | 0.1841 | | 5.3451 | 35.0 | 125055 | 5.3718 | 0.1908 | | 5.3294 | 36.0 | 128628 | 5.3706 | 0.1878 | | 5.3155 | 37.0 | 132201 | 5.3677 | 0.1903 | | 5.2996 | 38.0 | 135774 | 5.2970 | 0.1994 | | 5.287 | 39.0 | 139347 | 5.3127 | 0.1977 | | 5.2735 | 40.0 | 142920 | 5.3145 | 0.1955 | | 5.26 | 41.0 | 146493 | 5.2985 | 0.2017 | | 5.2487 | 42.0 | 150066 | 5.2661 | 0.2025 | | 5.2362 | 43.0 | 153639 | 5.2712 | 0.2031 | | 5.2248 | 44.0 | 157212 | 5.2452 | 0.2049 | | 5.2115 | 45.0 | 160785 | 5.2325 | 0.2054 | | 5.1998 | 46.0 | 164358 | 5.2233 | 0.2075 | | 5.188 | 47.0 | 167931 | 5.1994 | 0.2118 | | 5.1779 | 48.0 | 171504 | 5.2436 | 0.2069 | | 5.1664 | 49.0 | 175077 | 5.2203 | 0.2129 | | 5.1546 | 50.0 | 178650 | 5.1820 | 0.2134 | | 5.1431 | 51.0 | 182223 | 5.2029 | 0.2122 | | 5.133 | 52.0 | 185796 | 5.1458 | 0.2140 | | 5.1226 | 53.0 | 189369 | 5.1757 | 0.2163 | | 5.1138 | 54.0 | 192942 | 5.1380 | 0.2193 | | 5.1046 | 55.0 | 196515 | 5.1498 | 0.2178 | | 5.0984 | 56.0 | 200088 | 5.1094 | 0.2194 | | 5.0907 | 57.0 | 203661 | 5.1354 | 0.2202 | | 5.0812 | 58.0 | 207234 | 5.0662 | 0.2256 | | 5.0748 | 59.0 | 210807 | 5.1163 | 0.2181 | | 5.067 | 60.0 | 214380 | 5.1193 | 0.2199 | | 5.0609 | 61.0 | 217953 | 5.0919 | 0.2224 | | 5.0536 | 62.0 | 221526 | 5.0899 | 0.2239 | | 5.0491 | 63.0 | 225099 | 5.1125 | 0.2224 | | 5.0433 | 64.0 | 228672 | 5.0892 | 0.2226 | | 5.0373 | 65.0 | 232245 | 5.0644 | 0.2260 | | 5.032 | 66.0 | 235818 | 5.0623 | 0.2253 | | 5.0283 | 67.0 | 239391 | 5.1004 | 0.2213 | | 5.0223 | 68.0 | 242964 | 5.0573 | 0.2279 | | 5.0184 | 69.0 | 246537 | 5.0488 | 0.2271 | | 5.014 | 70.0 | 250110 | 5.0482 | 0.2280 | | 5.0102 | 71.0 | 253683 | 5.0600 | 0.2269 | | 5.0079 | 72.0 | 257256 | 5.0271 | 0.2279 | | 5.0029 | 73.0 | 260829 | 5.0629 | 0.2267 | | 4.9994 | 74.0 | 264402 | 5.0304 | 0.2297 | | 4.9978 | 75.0 | 267975 | 5.0485 | 0.2269 | | 4.9945 | 76.0 | 271548 | 5.0380 | 0.2306 | | 4.9917 | 77.0 | 275121 | 5.0590 | 0.2265 | | 4.9913 | 78.0 | 278694 | 5.0585 | 0.2262 | | 4.987 | 79.0 | 282267 | 5.0339 | 0.2278 | | 4.9862 | 80.0 | 285840 | 5.0214 | 0.2305 | | 4.9841 | 81.0 | 289413 | 5.0393 | 0.2271 | | 4.983 | 82.0 | 292986 | 5.0200 | 0.2298 | | 4.9816 | 83.0 | 296559 | 5.0289 | 0.2300 | | 4.9801 | 83.96 | 300000 | 4.9972 | 0.2332 | ### Framework versions - Transformers 4.26.0 - Pytorch 1.14.0a0+410ce96 - Datasets 2.9.0 - Tokenizers 0.13.2