Hugging Face's logo

lapp0
/
distily_bench_gpt2_simple_objectives2

TensorBoard
Safetensors
Distily
gpt2
Generated from Trainer
8-bit precision
bitsandbytes
Model card Files Metrics Community
lapp0 commited on
Commit
7c05548
•
1 Parent(s): 6a3eb73

End of training

Browse files
Files changed (2) hide show
  1. README.md +35 -35
  2. logs/distillation_objective=MultiObjective(logits_weight_1__logits_loss_fn_(fn_jsd_loss())__activations_weight_0.2__activations_loss_fn_(fn_soft_mse_loss())__attentions_weight_0__attentions_loss_fn_(fn_sof/events.out.tfevents.1723539895.93d6cbb3ad53 +3 -0
README.md CHANGED
@@ -16,13 +16,13 @@ This student model is distilled from the teacher model [gpt2](https://huggingfac
16
  The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
17
 
18
  It achieves the following results on the evaluation set:
19
- - eval_enwikippl: 321.9943
20
- - eval_frwikippl: 1758.7812
21
- - eval_zhwikippl: 13365.0029
22
- - eval_loss: 1.4163
23
- - eval_runtime: 38.2164
24
- - eval_samples_per_second: 52.334
25
- - eval_steps_per_second: 6.542
26
 
27
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
28
  should probably proofread and complete it, then remove this comment.
@@ -45,7 +45,7 @@ More information needed
45
  ### Training hyperparameters
46
 
47
  The following hyperparameters were used during training:
48
- - distillation_objective: MultiObjective(logits_weight=1, logits_loss_fn=(fn:cakld_loss()), activations_weight=0.2, activations_loss_fn=(fn:soft_mse_loss()), attentions_weight=0, attentions_loss_fn=(fn:soft_mse_loss()))
49
  - train_embeddings: True
50
  - learning_rate: 4e-05
51
  - train_batch_size: 8
@@ -56,38 +56,38 @@ The following hyperparameters were used during training:
56
  - num_epochs: 1.0
57
 
58
  ### Resource Usage
59
- Peak GPU Memory: 10.3900 GB
60
 
61
  ### Eval-Phase Metrics
62
  | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | zhwikippl |
63
  | --- | --- | --- | --- | --- | --- | --- | --- | --- |
64
  | **teacher eval** | | 30.2086 | 57.2728 | | | | | 18.1784 |
65
- | 0 | 0 | 54069.2930 | 57285.3438 | 6.3621 | 38.3646 | 52.131 | 6.516 | 54227.1016 |
66
- | 1000 | 0.0404 | 1067.6234 | 5608.6260 | 2.0789 | 38.0715 | 52.533 | 6.567 | 96472.8984 |
67
- | 2000 | 0.0808 | 757.3948 | 3725.6042 | 1.9128 | 38.2045 | 52.35 | 6.544 | 17909.9004 |
68
- | 3000 | 0.1212 | 658.3904 | 3416.6064 | 1.8048 | 38.242 | 52.299 | 6.537 | 7773.6802 |
69
- | 4000 | 0.1616 | 560.8882 | 3119.1289 | 1.7206 | 38.3142 | 52.2 | 6.525 | 2095.8083 |
70
- | 5000 | 0.2020 | 497.5148 | 2912.9402 | 1.6453 | 38.3234 | 52.187 | 6.523 | 1419.0811 |
71
- | 6000 | 0.2424 | 431.3407 | 2770.3191 | 1.5753 | 38.2865 | 52.238 | 6.53 | 1246.3398 |
72
- | 7000 | 0.2828 | 380.4417 | 2245.9678 | 1.5120 | 38.2419 | 52.299 | 6.537 | 2103.0974 |
73
- | 8000 | 0.3232 | 348.0757 | 1937.6904 | 1.4605 | 38.1726 | 52.394 | 6.549 | 1825.5139 |
74
- | 9000 | 0.3636 | 321.9943 | 1758.7812 | 1.4163 | 38.2164 | 52.334 | 6.542 | 13365.0029 |
75
- | 10000 | 0.4040 | 302.1302 | 1743.9637 | 1.3804 | 38.2909 | 52.232 | 6.529 | 1458.2706 |
76
- | 11000 | 0.4444 | 283.6677 | 1578.4840 | 1.3479 | 38.0626 | 52.545 | 6.568 | 2750.6077 |
77
- | 12000 | 0.4848 | 270.2290 | 1480.8057 | 1.3235 | 38.2617 | 52.272 | 6.534 | 1864.9377 |
78
- | 13000 | 0.5253 | 253.0666 | 1502.0465 | 1.2928 | 38.0543 | 52.556 | 6.57 | 5540.5454 |
79
- | 14000 | 0.5657 | 241.2089 | 1442.4714 | 1.2666 | 38.1426 | 52.435 | 6.554 | 1532.5410 |
80
- | 15000 | 0.6061 | 225.8018 | 1385.6504 | 1.2418 | 38.089 | 52.509 | 6.564 | 984.5054 |
81
- | 16000 | 0.6465 | 219.4576 | 1220.6696 | 1.2177 | 38.2186 | 52.33 | 6.541 | 876.7557 |
82
- | 17000 | 0.6869 | 208.0084 | 1206.6365 | 1.1978 | 38.223 | 52.324 | 6.541 | 1094.0415 |
83
- | 18000 | 0.7273 | 204.6121 | 1193.9426 | 1.1802 | 38.2118 | 52.34 | 6.542 | 1641.8646 |
84
- | 19000 | 0.7677 | 197.2790 | 1129.5767 | 1.1681 | 38.2359 | 52.307 | 6.538 | 693.8603 |
85
- | 20000 | 0.8081 | 191.9000 | 1025.8606 | 1.1505 | 38.008 | 52.621 | 6.578 | 992.1599 |
86
- | 21000 | 0.8485 | 190.8004 | 1014.3529 | 1.1373 | 38.0057 | 52.624 | 6.578 | 1086.7607 |
87
- | 22000 | 0.8889 | 185.3244 | 978.5265 | 1.1308 | 38.0856 | 52.513 | 6.564 | 728.6151 |
88
- | 23000 | 0.9293 | 185.4108 | 969.4617 | 1.1227 | 38.0654 | 52.541 | 6.568 | 623.9739 |
89
- | 24000 | 0.9697 | 178.5300 | 955.3481 | 1.1107 | 38.1566 | 52.416 | 6.552 | 568.7457 |
90
- | 24750 | 1.0 | 179.6843 | 993.8232 | 1.1067 | 38.1733 | 52.393 | 6.549 | 685.6628 |
91
 
92
  ### Framework versions
93
  - Distily 0.2.0
 
16
  The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
17
 
18
  It achieves the following results on the evaluation set:
19
+ - eval_enwikippl: 512.2950
20
+ - eval_frwikippl: 3101.1487
21
+ - eval_zhwikippl: 191798.5312
22
+ - eval_loss: 0.1841
23
+ - eval_runtime: 38.3167
24
+ - eval_samples_per_second: 52.197
25
+ - eval_steps_per_second: 6.525
26
 
27
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
28
  should probably proofread and complete it, then remove this comment.
 
45
  ### Training hyperparameters
46
 
47
  The following hyperparameters were used during training:
48
+ - distillation_objective: MultiObjective(logits_weight=1, logits_loss_fn=(fn:jsd_loss()), activations_weight=0.2, activations_loss_fn=(fn:soft_mse_loss()), attentions_weight=0, attentions_loss_fn=(fn:soft_mse_loss()))
49
  - train_embeddings: True
50
  - learning_rate: 4e-05
51
  - train_batch_size: 8
 
56
  - num_epochs: 1.0
57
 
58
  ### Resource Usage
59
+ Peak GPU Memory: 10.3934 GB
60
 
61
  ### Eval-Phase Metrics
62
  | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | zhwikippl |
63
  | --- | --- | --- | --- | --- | --- | --- | --- | --- |
64
  | **teacher eval** | | 30.2086 | 57.2728 | | | | | 18.1784 |
65
+ | 0 | 0 | 55129.1953 | 56939.0469 | 0.5624 | 38.2552 | 52.28 | 6.535 | 54824.1562 |
66
+ | 1000 | 0.0404 | 1574.1282 | 9231.6934 | 0.2450 | 38.197 | 52.36 | 6.545 | 323126.3438 |
67
+ | 2000 | 0.0808 | 1109.2875 | 6477.1401 | 0.2300 | 38.4705 | 51.988 | 6.498 | 361869.5 |
68
+ | 3000 | 0.1212 | 959.5045 | 5362.6807 | 0.2196 | 38.2081 | 52.345 | 6.543 | 331961.1875 |
69
+ | 4000 | 0.1616 | 849.7711 | 4466.3486 | 0.2122 | 38.1788 | 52.385 | 6.548 | 229626.5781 |
70
+ | 5000 | 0.2020 | 715.5438 | 3672.9194 | 0.2054 | 38.1399 | 52.439 | 6.555 | 179842.4219 |
71
+ | 6000 | 0.2424 | 674.3274 | 3799.3496 | 0.1993 | 38.1179 | 52.469 | 6.559 | 222800.1875 |
72
+ | 7000 | 0.2828 | 571.2620 | 3382.5713 | 0.1930 | 38.2139 | 52.337 | 6.542 | 174077.0156 |
73
+ | 8000 | 0.3232 | 530.7191 | 2989.9294 | 0.1883 | 38.251 | 52.286 | 6.536 | 218381.6875 |
74
+ | 9000 | 0.3636 | 512.2950 | 3101.1487 | 0.1841 | 38.3167 | 52.197 | 6.525 | 191798.5312 |
75
+ | 10000 | 0.4040 | 465.7365 | 2662.6936 | 0.1801 | 38.3474 | 52.155 | 6.519 | 149415.4531 |
76
+ | 11000 | 0.4444 | 435.4128 | 2513.4690 | 0.1768 | 38.3141 | 52.2 | 6.525 | 239843.7656 |
77
+ | 12000 | 0.4848 | 418.3436 | 2475.8303 | 0.1744 | 38.4612 | 52.001 | 6.5 | 213309.0469 |
78
+ | 13000 | 0.5253 | 386.7266 | 2253.5813 | 0.1722 | 38.3131 | 52.201 | 6.525 | 170715.9219 |
79
+ | 14000 | 0.5657 | 387.9898 | 2286.2295 | 0.1699 | 38.4198 | 52.057 | 6.507 | 168226.6875 |
80
+ | 15000 | 0.6061 | 381.0330 | 2336.0906 | 0.1681 | 38.3633 | 52.133 | 6.517 | 192619.9219 |
81
+ | 16000 | 0.6465 | 358.8618 | 2008.6333 | 0.1662 | 38.5711 | 51.852 | 6.482 | 109902.5547 |
82
+ | 17000 | 0.6869 | 354.6786 | 1894.4617 | 0.1651 | 38.3338 | 52.173 | 6.522 | 185501.1719 |
83
+ | 18000 | 0.7273 | 351.6073 | 1982.4641 | 0.1639 | 38.3968 | 52.088 | 6.511 | 157025.25 |
84
+ | 19000 | 0.7677 | 349.6740 | 2298.5125 | 0.1630 | 38.4227 | 52.053 | 6.507 | 302094.75 |
85
+ | 20000 | 0.8081 | 331.0454 | 1852.9810 | 0.1615 | 38.3923 | 52.094 | 6.512 | 188850.5469 |
86
+ | 21000 | 0.8485 | 325.8680 | 1841.2605 | 0.1604 | 38.5044 | 51.942 | 6.493 | 98031.1953 |
87
+ | 22000 | 0.8889 | 325.0340 | 2070.4631 | 0.1595 | 38.4226 | 52.053 | 6.507 | 161017.3438 |
88
+ | 23000 | 0.9293 | 312.5102 | 1947.8259 | 0.1585 | 38.4551 | 52.009 | 6.501 | 129694.3594 |
89
+ | 24000 | 0.9697 | 313.1418 | 1909.7499 | 0.1579 | 38.2352 | 52.308 | 6.538 | 171997.4531 |
90
+ | 24750 | 1.0 | 319.7268 | 2126.0857 | 0.1576 | 38.5377 | 51.897 | 6.487 | 784044.3125 |
91
 
92
  ### Framework versions
93
  - Distily 0.2.0
logs/distillation_objective=MultiObjective(logits_weight_1__logits_loss_fn_(fn_jsd_loss())__activations_weight_0.2__activations_loss_fn_(fn_soft_mse_loss())__attentions_weight_0__attentions_loss_fn_(fn_sof/events.out.tfevents.1723539895.93d6cbb3ad53 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1520eb0c916cfb63fa0e7c81f0c64676bae33b85d75a6c2d39b8fa43d10087ff
3
+ size 253