Hugging Face's logo

lapp0
/
distily_bench_gpt2_simple_objectives2

TensorBoard
Safetensors
Distily
gpt2
Generated from Trainer
8-bit precision
bitsandbytes
Model card Files Metrics Community
lapp0 commited on
Commit
902d4d0
•
1 Parent(s): 43a1107

End of training

Browse files
Files changed (2) hide show
  1. README.md +34 -34
  2. logs/distillation_objective=MultiObjective(logits_weight_1__logits_loss_fn_(fn_cross_entropy())__activations_weight_0.2__activations_loss_fn_(fn_soft_mse_loss())__attentions_weight_0__attentions_loss_fn_(f/events.out.tfevents.1723547906.93d6cbb3ad53 +3 -0
README.md CHANGED
@@ -16,13 +16,13 @@ This student model is distilled from the teacher model [gpt2](https://huggingfac
16
  The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
17
 
18
  It achieves the following results on the evaluation set:
19
- - eval_enwikippl: 512.2950
20
- - eval_frwikippl: 3101.1487
21
- - eval_zhwikippl: 191798.5312
22
- - eval_loss: 0.1841
23
- - eval_runtime: 38.3167
24
- - eval_samples_per_second: 52.197
25
- - eval_steps_per_second: 6.525
26
 
27
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
28
  should probably proofread and complete it, then remove this comment.
@@ -45,7 +45,7 @@ More information needed
45
  ### Training hyperparameters
46
 
47
  The following hyperparameters were used during training:
48
- - distillation_objective: MultiObjective(logits_weight=1, logits_loss_fn=(fn:jsd_loss()), activations_weight=0.2, activations_loss_fn=(fn:soft_mse_loss()), attentions_weight=0, attentions_loss_fn=(fn:soft_mse_loss()))
49
  - train_embeddings: True
50
  - learning_rate: 4e-05
51
  - train_batch_size: 8
@@ -62,32 +62,32 @@ Peak GPU Memory: 10.3934 GB
62
  | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | zhwikippl |
63
  | --- | --- | --- | --- | --- | --- | --- | --- | --- |
64
  | **teacher eval** | | 30.2086 | 57.2728 | | | | | 18.1784 |
65
- | 0 | 0 | 55129.1953 | 56939.0469 | 0.5624 | 38.2552 | 52.28 | 6.535 | 54824.1562 |
66
- | 1000 | 0.0404 | 1574.1282 | 9231.6934 | 0.2450 | 38.197 | 52.36 | 6.545 | 323126.3438 |
67
- | 2000 | 0.0808 | 1109.2875 | 6477.1401 | 0.2300 | 38.4705 | 51.988 | 6.498 | 361869.5 |
68
- | 3000 | 0.1212 | 959.5045 | 5362.6807 | 0.2196 | 38.2081 | 52.345 | 6.543 | 331961.1875 |
69
- | 4000 | 0.1616 | 849.7711 | 4466.3486 | 0.2122 | 38.1788 | 52.385 | 6.548 | 229626.5781 |
70
- | 5000 | 0.2020 | 715.5438 | 3672.9194 | 0.2054 | 38.1399 | 52.439 | 6.555 | 179842.4219 |
71
- | 6000 | 0.2424 | 674.3274 | 3799.3496 | 0.1993 | 38.1179 | 52.469 | 6.559 | 222800.1875 |
72
- | 7000 | 0.2828 | 571.2620 | 3382.5713 | 0.1930 | 38.2139 | 52.337 | 6.542 | 174077.0156 |
73
- | 8000 | 0.3232 | 530.7191 | 2989.9294 | 0.1883 | 38.251 | 52.286 | 6.536 | 218381.6875 |
74
- | 9000 | 0.3636 | 512.2950 | 3101.1487 | 0.1841 | 38.3167 | 52.197 | 6.525 | 191798.5312 |
75
- | 10000 | 0.4040 | 465.7365 | 2662.6936 | 0.1801 | 38.3474 | 52.155 | 6.519 | 149415.4531 |
76
- | 11000 | 0.4444 | 435.4128 | 2513.4690 | 0.1768 | 38.3141 | 52.2 | 6.525 | 239843.7656 |
77
- | 12000 | 0.4848 | 418.3436 | 2475.8303 | 0.1744 | 38.4612 | 52.001 | 6.5 | 213309.0469 |
78
- | 13000 | 0.5253 | 386.7266 | 2253.5813 | 0.1722 | 38.3131 | 52.201 | 6.525 | 170715.9219 |
79
- | 14000 | 0.5657 | 387.9898 | 2286.2295 | 0.1699 | 38.4198 | 52.057 | 6.507 | 168226.6875 |
80
- | 15000 | 0.6061 | 381.0330 | 2336.0906 | 0.1681 | 38.3633 | 52.133 | 6.517 | 192619.9219 |
81
- | 16000 | 0.6465 | 358.8618 | 2008.6333 | 0.1662 | 38.5711 | 51.852 | 6.482 | 109902.5547 |
82
- | 17000 | 0.6869 | 354.6786 | 1894.4617 | 0.1651 | 38.3338 | 52.173 | 6.522 | 185501.1719 |
83
- | 18000 | 0.7273 | 351.6073 | 1982.4641 | 0.1639 | 38.3968 | 52.088 | 6.511 | 157025.25 |
84
- | 19000 | 0.7677 | 349.6740 | 2298.5125 | 0.1630 | 38.4227 | 52.053 | 6.507 | 302094.75 |
85
- | 20000 | 0.8081 | 331.0454 | 1852.9810 | 0.1615 | 38.3923 | 52.094 | 6.512 | 188850.5469 |
86
- | 21000 | 0.8485 | 325.8680 | 1841.2605 | 0.1604 | 38.5044 | 51.942 | 6.493 | 98031.1953 |
87
- | 22000 | 0.8889 | 325.0340 | 2070.4631 | 0.1595 | 38.4226 | 52.053 | 6.507 | 161017.3438 |
88
- | 23000 | 0.9293 | 312.5102 | 1947.8259 | 0.1585 | 38.4551 | 52.009 | 6.501 | 129694.3594 |
89
- | 24000 | 0.9697 | 313.1418 | 1909.7499 | 0.1579 | 38.2352 | 52.308 | 6.538 | 171997.4531 |
90
- | 24750 | 1.0 | 319.7268 | 2126.0857 | 0.1576 | 38.5377 | 51.897 | 6.487 | 784044.3125 |
91
 
92
  ### Framework versions
93
  - Distily 0.2.0
 
16
  The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
17
 
18
  It achieves the following results on the evaluation set:
19
+ - eval_enwikippl: 50740220.0
20
+ - eval_frwikippl: 39742000.0
21
+ - eval_zhwikippl: 41711904.0
22
+ - eval_loss: -1133756.375
23
+ - eval_runtime: 32.9889
24
+ - eval_samples_per_second: 60.627
25
+ - eval_steps_per_second: 7.578
26
 
27
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
28
  should probably proofread and complete it, then remove this comment.
 
45
  ### Training hyperparameters
46
 
47
  The following hyperparameters were used during training:
48
+ - distillation_objective: MultiObjective(logits_weight=1, logits_loss_fn=(fn:cross_entropy()), activations_weight=0.2, activations_loss_fn=(fn:soft_mse_loss()), attentions_weight=0, attentions_loss_fn=(fn:soft_mse_loss()))
49
  - train_embeddings: True
50
  - learning_rate: 4e-05
51
  - train_batch_size: 8
 
62
  | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | zhwikippl |
63
  | --- | --- | --- | --- | --- | --- | --- | --- | --- |
64
  | **teacher eval** | | 30.2086 | 57.2728 | | | | | 18.1784 |
65
+ | 0 | 0 | 54069.2930 | 57285.3438 | -552648.6875 | 32.884 | 60.82 | 7.602 | 54227.1016 |
66
+ | 1000 | 0.0404 | 58406536.0 | 55278164.0 | -1128988.625 | 32.9702 | 60.661 | 7.583 | 60044060.0 |
67
+ | 2000 | 0.0808 | 47996096.0 | 32493628.0 | -1115226.125 | 32.92 | 60.753 | 7.594 | 30615724.0 |
68
+ | 3000 | 0.1212 | 53309036.0 | 52586468.0 | -1142620.125 | 32.8831 | 60.822 | 7.603 | 63406108.0 |
69
+ | 4000 | 0.1616 | 48853388.0 | 39451680.0 | -1129824.25 | 32.9886 | 60.627 | 7.578 | 40786548.0 |
70
+ | 5000 | 0.2020 | 48792764.0 | 39496180.0 | -1129889.75 | 32.94 | 60.716 | 7.59 | 41224608.0 |
71
+ | 6000 | 0.2424 | 49218992.0 | 39786900.0 | -1130381.25 | 32.8582 | 60.868 | 7.608 | 41379088.0 |
72
+ | 7000 | 0.2828 | 48641396.0 | 39540800.0 | -1130201.125 | 33.0051 | 60.597 | 7.575 | 41202596.0 |
73
+ | 8000 | 0.3232 | 48641396.0 | 39742000.0 | -1130168.375 | 33.0186 | 60.572 | 7.572 | 41312844.0 |
74
+ | 9000 | 0.3636 | 50740220.0 | 39742000.0 | -1133756.375 | 32.9889 | 60.627 | 7.578 | 41711904.0 |
75
+ | 10000 | 0.4040 | 50945516.0 | 40215528.0 | -1134297.125 | 33.0575 | 60.501 | 7.563 | 42227584.0 |
76
+ | 11000 | 0.4444 | 51968464.0 | 40443060.0 | -1135476.75 | 33.1355 | 60.358 | 7.545 | 42589960.0 |
77
+ | 12000 | 0.4848 | 50410312.0 | 40465824.0 | -1131806.75 | 32.913 | 60.766 | 7.596 | 42453708.0 |
78
+ | 13000 | 0.5253 | 49020652.0 | 39674820.0 | -1130709.0 | 33.1318 | 60.365 | 7.546 | 41778864.0 |
79
+ | 14000 | 0.5657 | 47536192.0 | 38998080.0 | -1128382.5 | 33.163 | 60.308 | 7.539 | 40504356.0 |
80
+ | 15000 | 0.6061 | 47936448.0 | 38822560.0 | -1127694.375 | 33.0155 | 60.578 | 7.572 | 40461116.0 |
81
+ | 16000 | 0.6465 | 49035896.0 | 42119312.0 | -1133248.5 | 33.0001 | 60.606 | 7.576 | 44687476.0 |
82
+ | 17000 | 0.6869 | 49081556.0 | 43840280.0 | -1134542.875 | 33.0658 | 60.485 | 7.561 | 47391796.0 |
83
+ | 18000 | 0.7273 | 48823112.0 | 45311016.0 | -1135951.875 | 32.8757 | 60.835 | 7.604 | 48054540.0 |
84
+ | 19000 | 0.7677 | 51358604.0 | 46384208.0 | -1138917.375 | 33.0459 | 60.522 | 7.565 | 52342104.0 |
85
+ | 20000 | 0.8081 | 52308568.0 | 47335696.0 | -1139867.625 | 33.1167 | 60.393 | 7.549 | 52819580.0 |
86
+ | 21000 | 0.8485 | 49633508.0 | 44375164.0 | -1133396.0 | 33.0584 | 60.499 | 7.562 | 45774616.0 |
87
+ | 22000 | 0.8889 | 48853388.0 | 44827992.0 | -1131577.375 | 32.9397 | 60.717 | 7.59 | 48441136.0 |
88
+ | 23000 | 0.9293 | 49726084.0 | 45221656.0 | -1133002.75 | 33.0244 | 60.561 | 7.57 | 49513708.0 |
89
+ | 24000 | 0.9697 | 53292568.0 | 48497804.0 | -1141473.25 | 33.1059 | 60.412 | 7.552 | 53730116.0 |
90
+ | 24750 | 1.0 | 59081604.0 | 55590812.0 | -1151893.5 | 33.0542 | 60.507 | 7.563 | 59374192.0 |
91
 
92
  ### Framework versions
93
  - Distily 0.2.0
logs/distillation_objective=MultiObjective(logits_weight_1__logits_loss_fn_(fn_cross_entropy())__activations_weight_0.2__activations_loss_fn_(fn_soft_mse_loss())__attentions_weight_0__attentions_loss_fn_(f/events.out.tfevents.1723547906.93d6cbb3ad53 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:487a72c710a5a13d4fefd637dba1139b09acda4af1bd4f2721e81da5f6ecb40f
3
+ size 253