Hugging Face's logo

gokuls
/
mobilebert_sa_pre-training-complete

Fill-Mask
Transformers
PyTorch
TensorBoard
mobilebert
Generated from Trainer
Eval Results
Inference Endpoints
Model card Files Metrics Community
gokuls commited on
Commit
494dc3d
•
1 Parent(s): 6675fdb

update model card README.md

Browse files
Files changed (1) hide show
  1. README.md +247 -0
README.md ADDED
@@ -0,0 +1,247 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ tags:
4
+ - generated_from_trainer
5
+ datasets:
6
+ - wikitext
7
+ metrics:
8
+ - accuracy
9
+ model-index:
10
+ - name: mobilebert_sa_pre-training-complete
11
+ results:
12
+ - task:
13
+ name: Masked Language Modeling
14
+ type: fill-mask
15
+ dataset:
16
+ name: wikitext
17
+ type: wikitext
18
+ config: wikitext-103-raw-v1
19
+ split: train
20
+ args: wikitext-103-raw-v1
21
+ metrics:
22
+ - name: Accuracy
23
+ type: accuracy
24
+ value: 0.6354230747721045
25
+ ---
26
+
27
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
28
+ should probably proofread and complete it, then remove this comment. -->
29
+
30
+ # mobilebert_sa_pre-training-complete
31
+
32
+ This model is a fine-tuned version of [google/mobilebert-uncased](https://huggingface.co/google/mobilebert-uncased) on the wikitext dataset.
33
+ It achieves the following results on the evaluation set:
34
+ - Loss: nan
35
+ - Accuracy: 0.6354
36
+
37
+ ## Model description
38
+
39
+ More information needed
40
+
41
+ ## Intended uses & limitations
42
+
43
+ More information needed
44
+
45
+ ## Training and evaluation data
46
+
47
+ More information needed
48
+
49
+ ## Training procedure
50
+
51
+ ### Training hyperparameters
52
+
53
+ The following hyperparameters were used during training:
54
+ - learning_rate: 5e-05
55
+ - train_batch_size: 64
56
+ - eval_batch_size: 64
57
+ - seed: 10
58
+ - distributed_type: multi-GPU
59
+ - num_devices: 2
60
+ - total_train_batch_size: 128
61
+ - total_eval_batch_size: 128
62
+ - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
63
+ - lr_scheduler_type: linear
64
+ - lr_scheduler_warmup_steps: 100
65
+ - training_steps: 300000
66
+ - mixed_precision_training: Native AMP
67
+
68
+ ### Training results
69
+
70
+ | Training Loss | Epoch | Step | Validation Loss | Accuracy |
71
+ |:-------------:|:------:|:------:|:---------------:|:--------:|
72
+ | 0.0 | 1.0 | 1787 | nan | 0.6390 |
73
+ | 0.0 | 2.0 | 3574 | nan | 0.6426 |
74
+ | 0.0 | 3.0 | 5361 | nan | 0.6415 |
75
+ | 0.0 | 4.0 | 7148 | nan | 0.6340 |
76
+ | 0.0 | 5.0 | 8935 | nan | 0.6360 |
77
+ | 0.0 | 6.0 | 10722 | nan | 0.6341 |
78
+ | 0.0 | 7.0 | 12509 | nan | 0.6378 |
79
+ | 0.0 | 8.0 | 14296 | nan | 0.6335 |
80
+ | 0.0 | 9.0 | 16083 | nan | 0.6363 |
81
+ | 0.0 | 10.0 | 17870 | nan | 0.6383 |
82
+ | 0.0 | 11.0 | 19657 | nan | 0.6379 |
83
+ | 0.0 | 12.0 | 21444 | nan | 0.6346 |
84
+ | 0.0006 | 13.0 | 23231 | nan | 0.6409 |
85
+ | 0.0 | 14.0 | 25018 | nan | 0.6406 |
86
+ | 0.0 | 15.0 | 26805 | nan | 0.6323 |
87
+ | 0.0 | 16.0 | 28592 | nan | 0.6402 |
88
+ | 0.0 | 17.0 | 30379 | nan | 0.6400 |
89
+ | 0.0 | 18.0 | 32166 | nan | 0.6328 |
90
+ | 0.0 | 19.0 | 33953 | nan | 0.6352 |
91
+ | 0.0 | 20.0 | 35740 | nan | 0.6380 |
92
+ | 0.0 | 21.0 | 37527 | nan | 0.6463 |
93
+ | 0.0 | 22.0 | 39314 | nan | 0.6313 |
94
+ | 0.0 | 23.0 | 41101 | nan | 0.6386 |
95
+ | 0.0 | 24.0 | 42888 | nan | 0.6413 |
96
+ | 0.0 | 25.0 | 44675 | nan | 0.6323 |
97
+ | 0.0008 | 26.0 | 46462 | nan | 0.6359 |
98
+ | 0.0 | 27.0 | 48249 | nan | 0.6397 |
99
+ | 0.0 | 28.0 | 50036 | nan | 0.6377 |
100
+ | 0.0 | 29.0 | 51823 | nan | 0.6383 |
101
+ | 0.0 | 30.0 | 53610 | nan | 0.6374 |
102
+ | 0.0 | 31.0 | 55397 | nan | 0.6476 |
103
+ | 0.0 | 32.0 | 57184 | nan | 0.6305 |
104
+ | 0.0011 | 33.0 | 58971 | nan | 0.6451 |
105
+ | 0.0 | 34.0 | 60758 | nan | 0.6372 |
106
+ | 0.0 | 35.0 | 62545 | nan | 0.6368 |
107
+ | 0.0006 | 36.0 | 64332 | nan | 0.6385 |
108
+ | 0.0 | 37.0 | 66119 | nan | 0.6349 |
109
+ | 0.0 | 38.0 | 67906 | nan | 0.6334 |
110
+ | 0.0 | 39.0 | 69693 | nan | 0.6391 |
111
+ | 0.0 | 40.0 | 71480 | nan | 0.6345 |
112
+ | 0.0 | 41.0 | 73267 | nan | 0.6423 |
113
+ | 0.0 | 42.0 | 75054 | nan | 0.6375 |
114
+ | 0.0 | 43.0 | 76841 | nan | 0.6292 |
115
+ | 0.0 | 44.0 | 78628 | nan | 0.6337 |
116
+ | 0.0 | 45.0 | 80415 | nan | 0.6451 |
117
+ | 0.0 | 46.0 | 82202 | nan | 0.6376 |
118
+ | 0.0 | 47.0 | 83989 | nan | 0.6355 |
119
+ | 0.0 | 48.0 | 85776 | nan | 0.6411 |
120
+ | 0.0 | 49.0 | 87563 | nan | 0.6358 |
121
+ | 0.0 | 50.0 | 89350 | nan | 0.6428 |
122
+ | 0.0 | 51.0 | 91137 | nan | 0.6421 |
123
+ | 0.004 | 52.0 | 92924 | nan | 0.6352 |
124
+ | 0.0 | 53.0 | 94711 | nan | 0.6411 |
125
+ | 0.0 | 54.0 | 96498 | nan | 0.6377 |
126
+ | 0.0 | 55.0 | 98285 | nan | 0.6375 |
127
+ | 0.0 | 56.0 | 100072 | nan | 0.6368 |
128
+ | 0.0 | 57.0 | 101859 | nan | 0.6365 |
129
+ | 0.0 | 58.0 | 103646 | nan | 0.6413 |
130
+ | 0.0 | 59.0 | 105433 | nan | 0.6347 |
131
+ | 0.0 | 60.0 | 107220 | nan | 0.6407 |
132
+ | 0.0 | 61.0 | 109007 | nan | 0.6395 |
133
+ | 0.0 | 62.0 | 110794 | nan | 0.6373 |
134
+ | 0.0 | 63.0 | 112581 | nan | 0.6356 |
135
+ | 0.0 | 64.0 | 114368 | nan | 0.6367 |
136
+ | 0.0 | 65.0 | 116155 | nan | 0.6441 |
137
+ | 0.0017 | 66.0 | 117942 | nan | 0.6380 |
138
+ | 0.0 | 67.0 | 119729 | nan | 0.6348 |
139
+ | 0.0 | 68.0 | 121516 | nan | 0.6356 |
140
+ | 0.0 | 69.0 | 123303 | nan | 0.6391 |
141
+ | 0.0006 | 70.0 | 125090 | nan | 0.6362 |
142
+ | 0.0 | 71.0 | 126877 | nan | 0.6388 |
143
+ | 0.0 | 72.0 | 128664 | nan | 0.6354 |
144
+ | 0.0 | 73.0 | 130451 | nan | 0.6362 |
145
+ | 0.0013 | 74.0 | 132238 | nan | 0.6347 |
146
+ | 0.0 | 75.0 | 134025 | nan | 0.6327 |
147
+ | 0.0 | 76.0 | 135812 | nan | 0.6382 |
148
+ | 0.0 | 77.0 | 137599 | nan | 0.6411 |
149
+ | 0.0 | 78.0 | 139386 | nan | 0.6404 |
150
+ | 0.0 | 79.0 | 141173 | nan | 0.6392 |
151
+ | 0.0 | 80.0 | 142960 | nan | 0.6404 |
152
+ | 0.0 | 81.0 | 144747 | nan | 0.6421 |
153
+ | 0.0 | 82.0 | 146534 | nan | 0.6364 |
154
+ | 0.0 | 83.0 | 148321 | nan | 0.6364 |
155
+ | 0.0 | 84.0 | 150108 | nan | 0.6370 |
156
+ | 0.0 | 85.0 | 151895 | nan | 0.6357 |
157
+ | 0.0 | 86.0 | 153682 | nan | 0.6353 |
158
+ | 0.0 | 87.0 | 155469 | nan | 0.6393 |
159
+ | 0.0 | 88.0 | 157256 | nan | 0.6397 |
160
+ | 0.0006 | 89.0 | 159043 | nan | 0.6396 |
161
+ | 0.0013 | 90.0 | 160830 | nan | 0.6378 |
162
+ | 0.0 | 91.0 | 162617 | nan | 0.6386 |
163
+ | 0.0 | 92.0 | 164404 | nan | 0.6415 |
164
+ | 0.0 | 93.0 | 166191 | nan | 0.6342 |
165
+ | 0.0 | 94.0 | 167978 | nan | 0.6356 |
166
+ | 0.0 | 95.0 | 169765 | nan | 0.6410 |
167
+ | 0.0 | 96.0 | 171552 | nan | 0.6366 |
168
+ | 0.0 | 97.0 | 173339 | nan | 0.6329 |
169
+ | 0.0013 | 98.0 | 175126 | nan | 0.6352 |
170
+ | 0.0 | 99.0 | 176913 | nan | 0.6340 |
171
+ | 0.0 | 100.0 | 178700 | nan | 0.6358 |
172
+ | 0.0 | 101.0 | 180487 | nan | 0.6367 |
173
+ | 0.0006 | 102.0 | 182274 | nan | 0.6368 |
174
+ | 0.0 | 103.0 | 184061 | nan | 0.6353 |
175
+ | 0.0 | 104.0 | 185848 | nan | 0.6370 |
176
+ | 0.0 | 105.0 | 187635 | nan | 0.6333 |
177
+ | 0.0 | 106.0 | 189422 | nan | 0.6316 |
178
+ | 0.0006 | 107.0 | 191209 | nan | 0.6394 |
179
+ | 0.0 | 108.0 | 192996 | nan | 0.6323 |
180
+ | 0.0 | 109.0 | 194783 | nan | 0.6406 |
181
+ | 0.0012 | 110.0 | 196570 | nan | 0.6331 |
182
+ | 0.0 | 111.0 | 198357 | nan | 0.6398 |
183
+ | 0.0 | 112.0 | 200144 | nan | 0.6402 |
184
+ | 0.0 | 113.0 | 201931 | nan | 0.6345 |
185
+ | 0.0 | 114.0 | 203718 | nan | 0.6416 |
186
+ | 0.0 | 115.0 | 205505 | nan | 0.6352 |
187
+ | 0.0 | 116.0 | 207292 | nan | 0.6357 |
188
+ | 0.0032 | 117.0 | 209079 | nan | 0.6358 |
189
+ | 0.0013 | 118.0 | 210866 | nan | 0.6406 |
190
+ | 0.0 | 119.0 | 212653 | nan | 0.6354 |
191
+ | 0.0 | 120.0 | 214440 | nan | 0.6345 |
192
+ | 0.0 | 121.0 | 216227 | nan | 0.6433 |
193
+ | 0.0 | 122.0 | 218014 | nan | 0.6326 |
194
+ | 0.0 | 123.0 | 219801 | nan | 0.6358 |
195
+ | 0.0 | 124.0 | 221588 | nan | 0.6409 |
196
+ | 0.0 | 125.0 | 223375 | nan | 0.6405 |
197
+ | 0.0 | 126.0 | 225162 | nan | 0.6376 |
198
+ | 0.0 | 127.0 | 226949 | nan | 0.6396 |
199
+ | 0.0 | 128.0 | 228736 | nan | 0.6356 |
200
+ | 0.0 | 129.0 | 230523 | nan | 0.6432 |
201
+ | 0.0 | 130.0 | 232310 | nan | 0.6385 |
202
+ | 0.0 | 131.0 | 234097 | nan | 0.6337 |
203
+ | 0.0 | 132.0 | 235884 | nan | 0.6390 |
204
+ | 0.0 | 133.0 | 237671 | nan | 0.6362 |
205
+ | 0.0 | 134.0 | 239458 | nan | 0.6332 |
206
+ | 0.0 | 135.0 | 241245 | nan | 0.6367 |
207
+ | 0.0016 | 136.0 | 243032 | nan | 0.6334 |
208
+ | 0.0 | 137.0 | 244819 | nan | 0.6412 |
209
+ | 0.0 | 138.0 | 246606 | nan | 0.6367 |
210
+ | 0.0 | 139.0 | 248393 | nan | 0.6378 |
211
+ | 0.0 | 140.0 | 250180 | nan | 0.6390 |
212
+ | 0.0 | 141.0 | 251967 | nan | 0.6376 |
213
+ | 0.0 | 142.0 | 253754 | nan | 0.6363 |
214
+ | 0.0033 | 143.0 | 255541 | nan | 0.6425 |
215
+ | 0.0 | 144.0 | 257328 | nan | 0.6360 |
216
+ | 0.0 | 145.0 | 259115 | nan | 0.6377 |
217
+ | 0.0 | 146.0 | 260902 | nan | 0.6302 |
218
+ | 0.0 | 147.0 | 262689 | nan | 0.6320 |
219
+ | 0.0 | 148.0 | 264476 | nan | 0.6358 |
220
+ | 0.0 | 149.0 | 266263 | nan | 0.6381 |
221
+ | 0.0 | 150.0 | 268050 | nan | 0.6414 |
222
+ | 0.0 | 151.0 | 269837 | nan | 0.6401 |
223
+ | 0.0012 | 152.0 | 271624 | nan | 0.6415 |
224
+ | 0.0 | 153.0 | 273411 | nan | 0.6425 |
225
+ | 0.0 | 154.0 | 275198 | nan | 0.6367 |
226
+ | 0.0 | 155.0 | 276985 | nan | 0.6356 |
227
+ | 0.0 | 156.0 | 278772 | nan | 0.6411 |
228
+ | 0.0 | 157.0 | 280559 | nan | 0.6343 |
229
+ | 0.0007 | 158.0 | 282346 | nan | 0.6369 |
230
+ | 0.0 | 159.0 | 284133 | nan | 0.6361 |
231
+ | 0.0013 | 160.0 | 285920 | nan | 0.6396 |
232
+ | 0.0008 | 161.0 | 287707 | nan | 0.6381 |
233
+ | 0.0 | 162.0 | 289494 | nan | 0.6352 |
234
+ | 0.0 | 163.0 | 291281 | nan | 0.6370 |
235
+ | 0.0 | 164.0 | 293068 | nan | 0.6399 |
236
+ | 0.0031 | 165.0 | 294855 | nan | 0.6401 |
237
+ | 0.0 | 166.0 | 296642 | nan | 0.6358 |
238
+ | 0.0 | 167.0 | 298429 | nan | 0.6390 |
239
+ | 0.0 | 167.88 | 300000 | nan | 0.6354 |
240
+
241
+
242
+ ### Framework versions
243
+
244
+ - Transformers 4.25.1
245
+ - Pytorch 1.14.0a0+410ce96
246
+ - Datasets 2.8.0
247
+ - Tokenizers 0.13.2