Hugging Face's logo

gokuls
/
distilbert_add_pre-training-complete

Fill-Mask
Transformers
PyTorch
TensorBoard
distilbert
generated_from_trainer
Eval Results
Inference Endpoints
Model card Files Metrics Community
gokuls commited on
Commit
066c7ff
•
1 Parent(s): da0d8c4

update model card README.md

Browse files
Files changed (1) hide show
  1. README.md +160 -0
README.md ADDED
@@ -0,0 +1,160 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ tags:
4
+ - generated_from_trainer
5
+ datasets:
6
+ - wikitext
7
+ metrics:
8
+ - accuracy
9
+ model-index:
10
+ - name: distilbert_add_pre-training-complete
11
+ results:
12
+ - task:
13
+ name: Masked Language Modeling
14
+ type: fill-mask
15
+ dataset:
16
+ name: wikitext
17
+ type: wikitext
18
+ config: wikitext-103-raw-v1
19
+ split: validation
20
+ args: wikitext-103-raw-v1
21
+ metrics:
22
+ - name: Accuracy
23
+ type: accuracy
24
+ value: 0.23321614400225005
25
+ ---
26
+
27
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
28
+ should probably proofread and complete it, then remove this comment. -->
29
+
30
+ # distilbert_add_pre-training-complete
31
+
32
+ This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the wikitext dataset.
33
+ It achieves the following results on the evaluation set:
34
+ - Loss: 4.9972
35
+ - Accuracy: 0.2332
36
+
37
+ ## Model description
38
+
39
+ More information needed
40
+
41
+ ## Intended uses & limitations
42
+
43
+ More information needed
44
+
45
+ ## Training and evaluation data
46
+
47
+ More information needed
48
+
49
+ ## Training procedure
50
+
51
+ ### Training hyperparameters
52
+
53
+ The following hyperparameters were used during training:
54
+ - learning_rate: 5e-05
55
+ - train_batch_size: 64
56
+ - eval_batch_size: 64
57
+ - seed: 10
58
+ - distributed_type: multi-GPU
59
+ - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
60
+ - lr_scheduler_type: linear
61
+ - lr_scheduler_warmup_steps: 100
62
+ - training_steps: 300000
63
+ - mixed_precision_training: Native AMP
64
+
65
+ ### Training results
66
+
67
+ | Training Loss | Epoch | Step | Validation Loss | Accuracy |
68
+ |:-------------:|:-----:|:------:|:---------------:|:--------:|
69
+ | 6.295 | 1.0 | 3573 | 6.0701 | 0.1522 |
70
+ | 6.0482 | 2.0 | 7146 | 5.9533 | 0.1565 |
71
+ | 5.9799 | 3.0 | 10719 | 5.9008 | 0.1584 |
72
+ | 5.9378 | 4.0 | 14292 | 5.8997 | 0.1545 |
73
+ | 5.9057 | 5.0 | 17865 | 5.8905 | 0.1536 |
74
+ | 5.8811 | 6.0 | 21438 | 5.8646 | 0.1550 |
75
+ | 5.8617 | 7.0 | 25011 | 5.8322 | 0.1534 |
76
+ | 5.844 | 8.0 | 28584 | 5.8563 | 0.1523 |
77
+ | 5.8297 | 9.0 | 32157 | 5.8352 | 0.1548 |
78
+ | 5.8175 | 10.0 | 35730 | 5.8136 | 0.1558 |
79
+ | 5.8056 | 11.0 | 39303 | 5.8147 | 0.1526 |
80
+ | 5.7921 | 12.0 | 42876 | 5.8020 | 0.1548 |
81
+ | 5.7777 | 13.0 | 46449 | 5.7891 | 0.1545 |
82
+ | 5.7596 | 14.0 | 50022 | 5.7370 | 0.1587 |
83
+ | 5.7414 | 15.0 | 53595 | 5.7396 | 0.1604 |
84
+ | 5.7243 | 16.0 | 57168 | 5.7490 | 0.1564 |
85
+ | 5.6997 | 17.0 | 60741 | 5.7135 | 0.1561 |
86
+ | 5.6698 | 18.0 | 64314 | 5.6858 | 0.1620 |
87
+ | 5.6398 | 19.0 | 67887 | 5.6735 | 0.1644 |
88
+ | 5.6135 | 20.0 | 71460 | 5.6174 | 0.1681 |
89
+ | 5.5899 | 21.0 | 75033 | 5.6191 | 0.1684 |
90
+ | 5.5699 | 22.0 | 78606 | 5.5977 | 0.1669 |
91
+ | 5.5487 | 23.0 | 82179 | 5.6139 | 0.1669 |
92
+ | 5.529 | 24.0 | 85752 | 5.5272 | 0.1741 |
93
+ | 5.512 | 25.0 | 89325 | 5.5271 | 0.1727 |
94
+ | 5.4939 | 26.0 | 92898 | 5.5190 | 0.1721 |
95
+ | 5.4765 | 27.0 | 96471 | 5.4824 | 0.1770 |
96
+ | 5.4604 | 28.0 | 100044 | 5.5159 | 0.1747 |
97
+ | 5.4422 | 29.0 | 103617 | 5.4577 | 0.1807 |
98
+ | 5.4243 | 30.0 | 107190 | 5.4546 | 0.1772 |
99
+ | 5.408 | 31.0 | 110763 | 5.4297 | 0.1837 |
100
+ | 5.3915 | 32.0 | 114336 | 5.4089 | 0.1866 |
101
+ | 5.3766 | 33.0 | 117909 | 5.3996 | 0.1848 |
102
+ | 5.3594 | 34.0 | 121482 | 5.3974 | 0.1841 |
103
+ | 5.3451 | 35.0 | 125055 | 5.3718 | 0.1908 |
104
+ | 5.3294 | 36.0 | 128628 | 5.3706 | 0.1878 |
105
+ | 5.3155 | 37.0 | 132201 | 5.3677 | 0.1903 |
106
+ | 5.2996 | 38.0 | 135774 | 5.2970 | 0.1994 |
107
+ | 5.287 | 39.0 | 139347 | 5.3127 | 0.1977 |
108
+ | 5.2735 | 40.0 | 142920 | 5.3145 | 0.1955 |
109
+ | 5.26 | 41.0 | 146493 | 5.2985 | 0.2017 |
110
+ | 5.2487 | 42.0 | 150066 | 5.2661 | 0.2025 |
111
+ | 5.2362 | 43.0 | 153639 | 5.2712 | 0.2031 |
112
+ | 5.2248 | 44.0 | 157212 | 5.2452 | 0.2049 |
113
+ | 5.2115 | 45.0 | 160785 | 5.2325 | 0.2054 |
114
+ | 5.1998 | 46.0 | 164358 | 5.2233 | 0.2075 |
115
+ | 5.188 | 47.0 | 167931 | 5.1994 | 0.2118 |
116
+ | 5.1779 | 48.0 | 171504 | 5.2436 | 0.2069 |
117
+ | 5.1664 | 49.0 | 175077 | 5.2203 | 0.2129 |
118
+ | 5.1546 | 50.0 | 178650 | 5.1820 | 0.2134 |
119
+ | 5.1431 | 51.0 | 182223 | 5.2029 | 0.2122 |
120
+ | 5.133 | 52.0 | 185796 | 5.1458 | 0.2140 |
121
+ | 5.1226 | 53.0 | 189369 | 5.1757 | 0.2163 |
122
+ | 5.1138 | 54.0 | 192942 | 5.1380 | 0.2193 |
123
+ | 5.1046 | 55.0 | 196515 | 5.1498 | 0.2178 |
124
+ | 5.0984 | 56.0 | 200088 | 5.1094 | 0.2194 |
125
+ | 5.0907 | 57.0 | 203661 | 5.1354 | 0.2202 |
126
+ | 5.0812 | 58.0 | 207234 | 5.0662 | 0.2256 |
127
+ | 5.0748 | 59.0 | 210807 | 5.1163 | 0.2181 |
128
+ | 5.067 | 60.0 | 214380 | 5.1193 | 0.2199 |
129
+ | 5.0609 | 61.0 | 217953 | 5.0919 | 0.2224 |
130
+ | 5.0536 | 62.0 | 221526 | 5.0899 | 0.2239 |
131
+ | 5.0491 | 63.0 | 225099 | 5.1125 | 0.2224 |
132
+ | 5.0433 | 64.0 | 228672 | 5.0892 | 0.2226 |
133
+ | 5.0373 | 65.0 | 232245 | 5.0644 | 0.2260 |
134
+ | 5.032 | 66.0 | 235818 | 5.0623 | 0.2253 |
135
+ | 5.0283 | 67.0 | 239391 | 5.1004 | 0.2213 |
136
+ | 5.0223 | 68.0 | 242964 | 5.0573 | 0.2279 |
137
+ | 5.0184 | 69.0 | 246537 | 5.0488 | 0.2271 |
138
+ | 5.014 | 70.0 | 250110 | 5.0482 | 0.2280 |
139
+ | 5.0102 | 71.0 | 253683 | 5.0600 | 0.2269 |
140
+ | 5.0079 | 72.0 | 257256 | 5.0271 | 0.2279 |
141
+ | 5.0029 | 73.0 | 260829 | 5.0629 | 0.2267 |
142
+ | 4.9994 | 74.0 | 264402 | 5.0304 | 0.2297 |
143
+ | 4.9978 | 75.0 | 267975 | 5.0485 | 0.2269 |
144
+ | 4.9945 | 76.0 | 271548 | 5.0380 | 0.2306 |
145
+ | 4.9917 | 77.0 | 275121 | 5.0590 | 0.2265 |
146
+ | 4.9913 | 78.0 | 278694 | 5.0585 | 0.2262 |
147
+ | 4.987 | 79.0 | 282267 | 5.0339 | 0.2278 |
148
+ | 4.9862 | 80.0 | 285840 | 5.0214 | 0.2305 |
149
+ | 4.9841 | 81.0 | 289413 | 5.0393 | 0.2271 |
150
+ | 4.983 | 82.0 | 292986 | 5.0200 | 0.2298 |
151
+ | 4.9816 | 83.0 | 296559 | 5.0289 | 0.2300 |
152
+ | 4.9801 | 83.96 | 300000 | 4.9972 | 0.2332 |
153
+
154
+
155
+ ### Framework versions
156
+
157
+ - Transformers 4.26.0
158
+ - Pytorch 1.14.0a0+410ce96
159
+ - Datasets 2.9.0
160
+ - Tokenizers 0.13.2