End of training
Browse files
README.md
CHANGED
@@ -18,14 +18,14 @@ This student model is distilled from the teacher model [gpt2](https://huggingfac
|
|
18 |
The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
|
19 |
|
20 |
It achieves the following results on the evaluation set:
|
21 |
-
- eval_enwikippl:
|
22 |
-
- eval_frwikippl:
|
23 |
-
- eval_zhwikippl:
|
24 |
-
- eval_tinystoriesppl:
|
25 |
-
- eval_loss: 1.
|
26 |
-
- eval_runtime: 29.
|
27 |
-
- eval_samples_per_second: 83.
|
28 |
-
- eval_steps_per_second: 10.
|
29 |
|
30 |
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
|
31 |
should probably proofread and complete it, then remove this comment.
|
@@ -55,80 +55,80 @@ The following hyperparameters were used during training:
|
|
55 |
- eval_batch_size: 8
|
56 |
- seed: 42
|
57 |
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
|
58 |
-
- lr_scheduler_type:
|
59 |
- lr_scheduler_warmup_ratio: 0.5
|
60 |
- num_epochs: 1.0
|
61 |
|
62 |
### Resource Usage
|
63 |
-
Peak GPU Memory: 7.
|
64 |
|
65 |
### Eval-Phase Metrics
|
66 |
| step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | tinystoriesppl | zhwikippl |
|
67 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
68 |
| **teacher eval** | | 43.25 | 61.25 | | | | | 11.6875 | 19.125 |
|
69 |
-
| 0 | 0 |
|
70 |
-
| 1000 | 0.0162 |
|
71 |
-
| 2000 | 0.0323 |
|
72 |
-
| 3000 | 0.0485 |
|
73 |
-
| 4000 | 0.0646 |
|
74 |
-
| 5000 | 0.0808 |
|
75 |
-
| 6000 | 0.0970 |
|
76 |
-
| 7000 | 0.1131 |
|
77 |
-
| 8000 | 0.1293 |
|
78 |
-
| 9000 | 0.1455 |
|
79 |
-
| 10000 | 0.1616 |
|
80 |
-
| 11000 | 0.1778 |
|
81 |
-
| 12000 | 0.1939 |
|
82 |
-
| 13000 | 0.2101 |
|
83 |
-
| 14000 | 0.2263 |
|
84 |
-
| 15000 | 0.2424 |
|
85 |
-
| 16000 | 0.2586 |
|
86 |
-
| 17000 | 0.2747 |
|
87 |
-
| 18000 | 0.2909 |
|
88 |
-
| 19000 | 0.3071 | 87.0 |
|
89 |
-
| 20000 | 0.3232 |
|
90 |
-
| 21000 | 0.3394 |
|
91 |
-
| 22000 | 0.3556 | 70.0 |
|
92 |
-
| 23000 | 0.3717 |
|
93 |
-
| 24000 | 0.3879 |
|
94 |
-
| 25000 | 0.4040 | 65.5 |
|
95 |
-
| 26000 | 0.4202 |
|
96 |
-
| 27000 | 0.4364 |
|
97 |
-
| 28000 | 0.4525 |
|
98 |
-
| 29000 | 0.4687 | 64.
|
99 |
-
| 30000 | 0.4848 |
|
100 |
-
| 31000 | 0.5010 |
|
101 |
-
| 32000 | 0.5172 |
|
102 |
-
| 33000 | 0.5333 |
|
103 |
-
| 34000 | 0.5495 |
|
104 |
-
| 35000 | 0.5657 |
|
105 |
-
| 36000 | 0.5818 |
|
106 |
-
| 37000 | 0.5980 | 61.
|
107 |
-
| 38000 | 0.6141 |
|
108 |
-
| 39000 | 0.6303 |
|
109 |
-
| 40000 | 0.6465 | 59.
|
110 |
-
| 41000 | 0.6626 | 58.
|
111 |
-
| 42000 | 0.6788 |
|
112 |
-
| 43000 | 0.6949 | 57.75 |
|
113 |
-
| 44000 | 0.7111 |
|
114 |
-
| 45000 | 0.7273 |
|
115 |
-
| 46000 | 0.7434 |
|
116 |
-
| 47000 | 0.7596 |
|
117 |
-
| 48000 | 0.7758 | 52.0 |
|
118 |
-
| 49000 | 0.7919 |
|
119 |
-
| 50000 | 0.8081 | 51.
|
120 |
-
| 51000 | 0.8242 |
|
121 |
-
| 52000 | 0.8404 |
|
122 |
-
| 53000 | 0.8566 | 50.
|
123 |
-
| 54000 | 0.8727 | 50.5 |
|
124 |
-
| 55000 | 0.8889 | 50.0 |
|
125 |
-
| 56000 | 0.9051 | 50.
|
126 |
-
| 57000 | 0.9212 |
|
127 |
-
| 58000 | 0.9374 |
|
128 |
-
| 59000 | 0.9535 |
|
129 |
-
| 60000 | 0.9697 |
|
130 |
-
| 61000 | 0.9859 |
|
131 |
-
| 61875 | 1.0 |
|
132 |
|
133 |
### Framework versions
|
134 |
- Distily 0.2.0
|
|
|
18 |
The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
|
19 |
|
20 |
It achieves the following results on the evaluation set:
|
21 |
+
- eval_enwikippl: 184.0
|
22 |
+
- eval_frwikippl: 744.0
|
23 |
+
- eval_zhwikippl: 180.0
|
24 |
+
- eval_tinystoriesppl: 148.0
|
25 |
+
- eval_loss: 1.1860
|
26 |
+
- eval_runtime: 29.84
|
27 |
+
- eval_samples_per_second: 83.78
|
28 |
+
- eval_steps_per_second: 10.489
|
29 |
|
30 |
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
|
31 |
should probably proofread and complete it, then remove this comment.
|
|
|
55 |
- eval_batch_size: 8
|
56 |
- seed: 42
|
57 |
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
|
58 |
+
- lr_scheduler_type: cosine
|
59 |
- lr_scheduler_warmup_ratio: 0.5
|
60 |
- num_epochs: 1.0
|
61 |
|
62 |
### Resource Usage
|
63 |
+
Peak GPU Memory: 7.5008 GB
|
64 |
|
65 |
### Eval-Phase Metrics
|
66 |
| step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | tinystoriesppl | zhwikippl |
|
67 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
68 |
| **teacher eval** | | 43.25 | 61.25 | | | | | 11.6875 | 19.125 |
|
69 |
+
| 0 | 0 | 841813590016.0 | 42880953483264.0 | 19.1388 | 29.7619 | 84.0 | 10.517 | 2533359616.0 | 18691697672192.0 |
|
70 |
+
| 1000 | 0.0162 | 8256.0 | 104448.0 | 3.7570 | 29.792 | 83.915 | 10.506 | 4608.0 | 250880.0 |
|
71 |
+
| 2000 | 0.0323 | 1488.0 | 8576.0 | 2.5011 | 29.7856 | 83.933 | 10.508 | 828.0 | 40448.0 |
|
72 |
+
| 3000 | 0.0485 | 668.0 | 4384.0 | 2.0176 | 29.8589 | 83.727 | 10.483 | 442.0 | 1648.0 |
|
73 |
+
| 4000 | 0.0646 | 444.0 | 2304.0 | 1.7541 | 29.853 | 83.744 | 10.485 | 308.0 | 672.0 |
|
74 |
+
| 5000 | 0.0808 | 328.0 | 1288.0 | 1.5507 | 29.8272 | 83.816 | 10.494 | 258.0 | 242.0 |
|
75 |
+
| 6000 | 0.0970 | 266.0 | 1168.0 | 1.3948 | 29.9048 | 83.599 | 10.467 | 217.0 | 253.0 |
|
76 |
+
| 7000 | 0.1131 | 229.0 | 1048.0 | 1.3140 | 29.8053 | 83.878 | 10.501 | 181.0 | 189.0 |
|
77 |
+
| 8000 | 0.1293 | 202.0 | 760.0 | 1.2384 | 29.8461 | 83.763 | 10.487 | 166.0 | 187.0 |
|
78 |
+
| 9000 | 0.1455 | 184.0 | 744.0 | 1.1860 | 29.84 | 83.78 | 10.489 | 148.0 | 180.0 |
|
79 |
+
| 10000 | 0.1616 | 161.0 | 564.0 | 1.0820 | 29.8521 | 83.746 | 10.485 | 132.0 | 170.0 |
|
80 |
+
| 11000 | 0.1778 | 139.0 | 478.0 | 0.9691 | 29.7904 | 83.92 | 10.507 | 112.5 | 139.0 |
|
81 |
+
| 12000 | 0.1939 | 122.5 | 446.0 | 0.8903 | 29.8277 | 83.815 | 10.494 | 91.0 | 153.0 |
|
82 |
+
| 13000 | 0.2101 | 130.0 | 450.0 | 0.8290 | 29.8764 | 83.678 | 10.476 | 113.5 | 148.0 |
|
83 |
+
| 14000 | 0.2263 | 111.5 | 410.0 | 0.7867 | 29.8386 | 83.784 | 10.49 | 85.5 | 116.5 |
|
84 |
+
| 15000 | 0.2424 | 103.0 | 394.0 | 0.7550 | 29.7824 | 83.942 | 10.51 | 81.5 | 126.5 |
|
85 |
+
| 16000 | 0.2586 | 95.5 | 368.0 | 0.7130 | 29.8395 | 83.781 | 10.489 | 74.0 | 137.0 |
|
86 |
+
| 17000 | 0.2747 | 91.0 | 370.0 | 0.6869 | 29.8002 | 83.892 | 10.503 | 72.0 | 110.0 |
|
87 |
+
| 18000 | 0.2909 | 89.5 | 356.0 | 0.6569 | 29.8522 | 83.746 | 10.485 | 65.0 | 124.5 |
|
88 |
+
| 19000 | 0.3071 | 87.0 | 354.0 | 0.6839 | 29.8823 | 83.661 | 10.474 | 68.5 | 137.0 |
|
89 |
+
| 20000 | 0.3232 | 79.5 | 290.0 | 0.6065 | 29.8977 | 83.618 | 10.469 | 65.0 | 113.5 |
|
90 |
+
| 21000 | 0.3394 | 75.0 | 251.0 | 0.5674 | 29.8207 | 83.834 | 10.496 | 59.75 | 112.5 |
|
91 |
+
| 22000 | 0.3556 | 70.0 | 250.0 | 0.5363 | 29.8336 | 83.798 | 10.492 | 56.25 | 81.0 |
|
92 |
+
| 23000 | 0.3717 | 69.0 | 220.0 | 0.5125 | 29.8003 | 83.892 | 10.503 | 53.75 | 86.5 |
|
93 |
+
| 24000 | 0.3879 | 65.5 | 226.0 | 0.5047 | 29.8312 | 83.805 | 10.492 | 52.25 | 91.0 |
|
94 |
+
| 25000 | 0.4040 | 65.5 | 211.0 | 0.4917 | 29.8281 | 83.813 | 10.493 | 55.0 | 141.0 |
|
95 |
+
| 26000 | 0.4202 | 63.25 | 204.0 | 0.4817 | 29.8227 | 83.829 | 10.495 | 50.75 | 86.5 |
|
96 |
+
| 27000 | 0.4364 | 64.5 | 213.0 | 0.4738 | 29.9242 | 83.544 | 10.46 | 51.25 | 94.5 |
|
97 |
+
| 28000 | 0.4525 | 62.75 | 192.0 | 0.4619 | 29.9106 | 83.583 | 10.465 | 48.75 | 113.5 |
|
98 |
+
| 29000 | 0.4687 | 64.5 | 204.0 | 0.4840 | 29.8026 | 83.885 | 10.502 | 52.5 | 81.5 |
|
99 |
+
| 30000 | 0.4848 | 65.0 | 217.0 | 0.4796 | 29.8897 | 83.641 | 10.472 | 49.25 | 140.0 |
|
100 |
+
| 31000 | 0.5010 | 63.5 | 206.0 | 0.4689 | 29.8072 | 83.872 | 10.501 | 48.25 | 141.0 |
|
101 |
+
| 32000 | 0.5172 | 63.25 | 217.0 | 0.4726 | 29.8682 | 83.701 | 10.479 | 46.25 | 112.5 |
|
102 |
+
| 33000 | 0.5333 | 66.5 | 231.0 | 0.4654 | 29.7912 | 83.917 | 10.506 | 51.25 | 87.5 |
|
103 |
+
| 34000 | 0.5495 | 62.75 | 200.0 | 0.4547 | 29.8255 | 83.821 | 10.494 | 49.75 | 89.5 |
|
104 |
+
| 35000 | 0.5657 | 63.75 | 196.0 | 0.4552 | 29.8185 | 83.841 | 10.497 | 49.25 | 83.5 |
|
105 |
+
| 36000 | 0.5818 | 63.75 | 215.0 | 0.4588 | 29.8868 | 83.649 | 10.473 | 46.0 | 113.5 |
|
106 |
+
| 37000 | 0.5980 | 61.5 | 193.0 | 0.4382 | 29.825 | 83.822 | 10.495 | 46.25 | 130.0 |
|
107 |
+
| 38000 | 0.6141 | 61.5 | 193.0 | 0.4237 | 29.8213 | 83.833 | 10.496 | 45.75 | 75.5 |
|
108 |
+
| 39000 | 0.6303 | 61.5 | 187.0 | 0.4218 | 29.8194 | 83.838 | 10.497 | 44.0 | 82.5 |
|
109 |
+
| 40000 | 0.6465 | 59.75 | 178.0 | 0.4127 | 29.8348 | 83.795 | 10.491 | 42.75 | 100.5 |
|
110 |
+
| 41000 | 0.6626 | 58.0 | 184.0 | 0.4133 | 29.778 | 83.955 | 10.511 | 42.25 | 119.0 |
|
111 |
+
| 42000 | 0.6788 | 56.75 | 184.0 | 0.4072 | 29.8696 | 83.697 | 10.479 | 40.75 | 109.0 |
|
112 |
+
| 43000 | 0.6949 | 57.75 | 184.0 | 0.3986 | 29.8393 | 83.782 | 10.49 | 41.75 | 87.0 |
|
113 |
+
| 44000 | 0.7111 | 58.0 | 180.0 | 0.4014 | 29.8433 | 83.771 | 10.488 | 40.5 | 101.0 |
|
114 |
+
| 45000 | 0.7273 | 55.75 | 158.0 | 0.3611 | 29.8497 | 83.753 | 10.486 | 38.25 | 67.0 |
|
115 |
+
| 46000 | 0.7434 | 55.0 | 148.0 | 0.3377 | 29.8619 | 83.719 | 10.482 | 36.0 | 63.75 |
|
116 |
+
| 47000 | 0.7596 | 52.25 | 143.0 | 0.3271 | 29.8199 | 83.837 | 10.496 | 35.0 | 50.75 |
|
117 |
+
| 48000 | 0.7758 | 52.0 | 141.0 | 0.3185 | 29.8125 | 83.857 | 10.499 | 34.0 | 49.25 |
|
118 |
+
| 49000 | 0.7919 | 52.5 | 142.0 | 0.3146 | 29.9037 | 83.602 | 10.467 | 33.5 | 43.5 |
|
119 |
+
| 50000 | 0.8081 | 51.25 | 134.0 | 0.3096 | 29.8931 | 83.631 | 10.471 | 33.25 | 46.25 |
|
120 |
+
| 51000 | 0.8242 | 51.25 | 133.0 | 0.3025 | 30.0212 | 83.274 | 10.426 | 32.5 | 40.0 |
|
121 |
+
| 52000 | 0.8404 | 51.5 | 132.0 | 0.2984 | 29.8459 | 83.764 | 10.487 | 32.5 | 39.5 |
|
122 |
+
| 53000 | 0.8566 | 50.5 | 131.0 | 0.2951 | 29.8292 | 83.81 | 10.493 | 32.5 | 36.0 |
|
123 |
+
| 54000 | 0.8727 | 50.5 | 132.0 | 0.2934 | 29.9146 | 83.571 | 10.463 | 32.25 | 37.75 |
|
124 |
+
| 55000 | 0.8889 | 50.0 | 131.0 | 0.2918 | 29.8217 | 83.831 | 10.496 | 32.5 | 35.75 |
|
125 |
+
| 56000 | 0.9051 | 50.0 | 130.0 | 0.2911 | 29.8366 | 83.79 | 10.49 | 32.25 | 35.5 |
|
126 |
+
| 57000 | 0.9212 | 50.0 | 130.0 | 0.2903 | 29.8261 | 83.819 | 10.494 | 32.25 | 35.5 |
|
127 |
+
| 58000 | 0.9374 | 50.0 | 130.0 | 0.2901 | 29.8639 | 83.713 | 10.481 | 32.25 | 35.5 |
|
128 |
+
| 59000 | 0.9535 | 50.0 | 130.0 | 0.2900 | 29.8256 | 83.821 | 10.494 | 32.25 | 35.25 |
|
129 |
+
| 60000 | 0.9697 | 50.0 | 130.0 | 0.2899 | 29.8767 | 83.677 | 10.476 | 32.25 | 35.25 |
|
130 |
+
| 61000 | 0.9859 | 50.0 | 130.0 | 0.2900 | 29.8188 | 83.84 | 10.497 | 32.25 | 35.25 |
|
131 |
+
| 61875 | 1.0 | 50.0 | 130.0 | 0.2899 | 29.869 | 83.699 | 10.479 | 32.25 | 35.25 |
|
132 |
|
133 |
### Framework versions
|
134 |
- Distily 0.2.0
|
logs/lr_scheduler_type=cosine, warmup_ratio=0.5/events.out.tfevents.1724181244.5f530b1cf724
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:8b2a5514ca6b1c6850da55776e8f5fb3de0a74575b0b252791613944102570b2
|
3 |
+
size 312
|