|
--- |
|
license: apache-2.0 |
|
tags: |
|
- generated_from_trainer |
|
datasets: |
|
- wikitext |
|
metrics: |
|
- accuracy |
|
model-index: |
|
- name: distilbert_add_pre-training-complete |
|
results: |
|
- task: |
|
name: Masked Language Modeling |
|
type: fill-mask |
|
dataset: |
|
name: wikitext |
|
type: wikitext |
|
config: wikitext-103-raw-v1 |
|
split: validation |
|
args: wikitext-103-raw-v1 |
|
metrics: |
|
- name: Accuracy |
|
type: accuracy |
|
value: 0.23321614400225005 |
|
--- |
|
|
|
<!-- This model card has been generated automatically according to the information the Trainer had access to. You |
|
should probably proofread and complete it, then remove this comment. --> |
|
|
|
# distilbert_add_pre-training-complete |
|
|
|
This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the wikitext dataset. |
|
It achieves the following results on the evaluation set: |
|
- Loss: 4.9972 |
|
- Accuracy: 0.2332 |
|
|
|
## Model description |
|
|
|
More information needed |
|
|
|
## Intended uses & limitations |
|
|
|
More information needed |
|
|
|
## Training and evaluation data |
|
|
|
More information needed |
|
|
|
## Training procedure |
|
|
|
### Training hyperparameters |
|
|
|
The following hyperparameters were used during training: |
|
- learning_rate: 5e-05 |
|
- train_batch_size: 64 |
|
- eval_batch_size: 64 |
|
- seed: 10 |
|
- distributed_type: multi-GPU |
|
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 |
|
- lr_scheduler_type: linear |
|
- lr_scheduler_warmup_steps: 100 |
|
- training_steps: 300000 |
|
- mixed_precision_training: Native AMP |
|
|
|
### Training results |
|
|
|
| Training Loss | Epoch | Step | Validation Loss | Accuracy | |
|
|:-------------:|:-----:|:------:|:---------------:|:--------:| |
|
| 6.295 | 1.0 | 3573 | 6.0701 | 0.1522 | |
|
| 6.0482 | 2.0 | 7146 | 5.9533 | 0.1565 | |
|
| 5.9799 | 3.0 | 10719 | 5.9008 | 0.1584 | |
|
| 5.9378 | 4.0 | 14292 | 5.8997 | 0.1545 | |
|
| 5.9057 | 5.0 | 17865 | 5.8905 | 0.1536 | |
|
| 5.8811 | 6.0 | 21438 | 5.8646 | 0.1550 | |
|
| 5.8617 | 7.0 | 25011 | 5.8322 | 0.1534 | |
|
| 5.844 | 8.0 | 28584 | 5.8563 | 0.1523 | |
|
| 5.8297 | 9.0 | 32157 | 5.8352 | 0.1548 | |
|
| 5.8175 | 10.0 | 35730 | 5.8136 | 0.1558 | |
|
| 5.8056 | 11.0 | 39303 | 5.8147 | 0.1526 | |
|
| 5.7921 | 12.0 | 42876 | 5.8020 | 0.1548 | |
|
| 5.7777 | 13.0 | 46449 | 5.7891 | 0.1545 | |
|
| 5.7596 | 14.0 | 50022 | 5.7370 | 0.1587 | |
|
| 5.7414 | 15.0 | 53595 | 5.7396 | 0.1604 | |
|
| 5.7243 | 16.0 | 57168 | 5.7490 | 0.1564 | |
|
| 5.6997 | 17.0 | 60741 | 5.7135 | 0.1561 | |
|
| 5.6698 | 18.0 | 64314 | 5.6858 | 0.1620 | |
|
| 5.6398 | 19.0 | 67887 | 5.6735 | 0.1644 | |
|
| 5.6135 | 20.0 | 71460 | 5.6174 | 0.1681 | |
|
| 5.5899 | 21.0 | 75033 | 5.6191 | 0.1684 | |
|
| 5.5699 | 22.0 | 78606 | 5.5977 | 0.1669 | |
|
| 5.5487 | 23.0 | 82179 | 5.6139 | 0.1669 | |
|
| 5.529 | 24.0 | 85752 | 5.5272 | 0.1741 | |
|
| 5.512 | 25.0 | 89325 | 5.5271 | 0.1727 | |
|
| 5.4939 | 26.0 | 92898 | 5.5190 | 0.1721 | |
|
| 5.4765 | 27.0 | 96471 | 5.4824 | 0.1770 | |
|
| 5.4604 | 28.0 | 100044 | 5.5159 | 0.1747 | |
|
| 5.4422 | 29.0 | 103617 | 5.4577 | 0.1807 | |
|
| 5.4243 | 30.0 | 107190 | 5.4546 | 0.1772 | |
|
| 5.408 | 31.0 | 110763 | 5.4297 | 0.1837 | |
|
| 5.3915 | 32.0 | 114336 | 5.4089 | 0.1866 | |
|
| 5.3766 | 33.0 | 117909 | 5.3996 | 0.1848 | |
|
| 5.3594 | 34.0 | 121482 | 5.3974 | 0.1841 | |
|
| 5.3451 | 35.0 | 125055 | 5.3718 | 0.1908 | |
|
| 5.3294 | 36.0 | 128628 | 5.3706 | 0.1878 | |
|
| 5.3155 | 37.0 | 132201 | 5.3677 | 0.1903 | |
|
| 5.2996 | 38.0 | 135774 | 5.2970 | 0.1994 | |
|
| 5.287 | 39.0 | 139347 | 5.3127 | 0.1977 | |
|
| 5.2735 | 40.0 | 142920 | 5.3145 | 0.1955 | |
|
| 5.26 | 41.0 | 146493 | 5.2985 | 0.2017 | |
|
| 5.2487 | 42.0 | 150066 | 5.2661 | 0.2025 | |
|
| 5.2362 | 43.0 | 153639 | 5.2712 | 0.2031 | |
|
| 5.2248 | 44.0 | 157212 | 5.2452 | 0.2049 | |
|
| 5.2115 | 45.0 | 160785 | 5.2325 | 0.2054 | |
|
| 5.1998 | 46.0 | 164358 | 5.2233 | 0.2075 | |
|
| 5.188 | 47.0 | 167931 | 5.1994 | 0.2118 | |
|
| 5.1779 | 48.0 | 171504 | 5.2436 | 0.2069 | |
|
| 5.1664 | 49.0 | 175077 | 5.2203 | 0.2129 | |
|
| 5.1546 | 50.0 | 178650 | 5.1820 | 0.2134 | |
|
| 5.1431 | 51.0 | 182223 | 5.2029 | 0.2122 | |
|
| 5.133 | 52.0 | 185796 | 5.1458 | 0.2140 | |
|
| 5.1226 | 53.0 | 189369 | 5.1757 | 0.2163 | |
|
| 5.1138 | 54.0 | 192942 | 5.1380 | 0.2193 | |
|
| 5.1046 | 55.0 | 196515 | 5.1498 | 0.2178 | |
|
| 5.0984 | 56.0 | 200088 | 5.1094 | 0.2194 | |
|
| 5.0907 | 57.0 | 203661 | 5.1354 | 0.2202 | |
|
| 5.0812 | 58.0 | 207234 | 5.0662 | 0.2256 | |
|
| 5.0748 | 59.0 | 210807 | 5.1163 | 0.2181 | |
|
| 5.067 | 60.0 | 214380 | 5.1193 | 0.2199 | |
|
| 5.0609 | 61.0 | 217953 | 5.0919 | 0.2224 | |
|
| 5.0536 | 62.0 | 221526 | 5.0899 | 0.2239 | |
|
| 5.0491 | 63.0 | 225099 | 5.1125 | 0.2224 | |
|
| 5.0433 | 64.0 | 228672 | 5.0892 | 0.2226 | |
|
| 5.0373 | 65.0 | 232245 | 5.0644 | 0.2260 | |
|
| 5.032 | 66.0 | 235818 | 5.0623 | 0.2253 | |
|
| 5.0283 | 67.0 | 239391 | 5.1004 | 0.2213 | |
|
| 5.0223 | 68.0 | 242964 | 5.0573 | 0.2279 | |
|
| 5.0184 | 69.0 | 246537 | 5.0488 | 0.2271 | |
|
| 5.014 | 70.0 | 250110 | 5.0482 | 0.2280 | |
|
| 5.0102 | 71.0 | 253683 | 5.0600 | 0.2269 | |
|
| 5.0079 | 72.0 | 257256 | 5.0271 | 0.2279 | |
|
| 5.0029 | 73.0 | 260829 | 5.0629 | 0.2267 | |
|
| 4.9994 | 74.0 | 264402 | 5.0304 | 0.2297 | |
|
| 4.9978 | 75.0 | 267975 | 5.0485 | 0.2269 | |
|
| 4.9945 | 76.0 | 271548 | 5.0380 | 0.2306 | |
|
| 4.9917 | 77.0 | 275121 | 5.0590 | 0.2265 | |
|
| 4.9913 | 78.0 | 278694 | 5.0585 | 0.2262 | |
|
| 4.987 | 79.0 | 282267 | 5.0339 | 0.2278 | |
|
| 4.9862 | 80.0 | 285840 | 5.0214 | 0.2305 | |
|
| 4.9841 | 81.0 | 289413 | 5.0393 | 0.2271 | |
|
| 4.983 | 82.0 | 292986 | 5.0200 | 0.2298 | |
|
| 4.9816 | 83.0 | 296559 | 5.0289 | 0.2300 | |
|
| 4.9801 | 83.96 | 300000 | 4.9972 | 0.2332 | |
|
|
|
|
|
### Framework versions |
|
|
|
- Transformers 4.26.0 |
|
- Pytorch 1.14.0a0+410ce96 |
|
- Datasets 2.9.0 |
|
- Tokenizers 0.13.2 |
|
|
