End of training

README.md

@@ -0,0 +1,96 @@
+---
+base_model: gpt2
+library_name: Distily
+license: mit
+tags:
+- generated_from_trainer
+model-index:
+- name: distily_bench_gpt2_activation_loss
+  results: []
+---
+
+# distily_bench_gpt2_activation_loss
+
+This student model is distilled from the teacher model [gpt2](https://huggingface.co/gpt2) using the dataset (unspecified).
+
+The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
+
+It achieves the following results on the evaluation set:
+- eval_enwikippl: 216.1260
+- eval_frwikippl: 1157.4716
+- eval_zhwikippl: 725.3146
+- eval_loss: 1.2660
+- eval_runtime: 34.0138
+- eval_samples_per_second: 58.8
+- eval_steps_per_second: 7.35
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment.
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+-->
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- distillation_objective: MultiObjective(logits_weight=1, logits_loss_fn=(fn:kl_divergence_loss()), activations_weight=0, activations_loss_fn=(fn:soft_mse_loss()), attentions_weight=0, attentions_loss_fn=(fn:soft_mse_loss()))
+- train_embeddings: True
+- learning_rate: 4e-05
+- train_batch_size: 8
+- eval_batch_size: 8
+- seed: 42
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: constant
+- num_epochs: 1.0
+
+### Resource Usage
+Peak GPU Memory: 7.9371 GB
+
+### Eval-Phase Metrics
+| step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | zhwikippl |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| **teacher eval** |  | 30.2086 | 57.2728 |  |  |  |  | 18.1784 |
+| 0 | 0 | 58863.3555 | 58419.2617 | 5.9616 | 34.0389 | 58.756 | 7.345 | 60583.1562 |
+| 1000 | 0.0404 | 726.5175 | 4290.4102 | 1.9634 | 33.8546 | 59.076 | 7.385 | 10492.5918 |
+| 2000 | 0.0808 | 522.3779 | 3038.8101 | 1.7799 | 34.2286 | 58.431 | 7.304 | 1967.7870 |
+| 3000 | 0.1212 | 436.0217 | 2697.4614 | 1.6705 | 33.9025 | 58.993 | 7.374 | 1091.1235 |
+| 4000 | 0.1616 | 370.4137 | 2260.9023 | 1.5809 | 33.9602 | 58.892 | 7.362 | 907.7313 |
+| 5000 | 0.2020 | 327.6950 | 1962.1614 | 1.5021 | 33.7774 | 59.211 | 7.401 | 865.3570 |
+| 6000 | 0.2424 | 285.5019 | 1729.7585 | 1.4289 | 33.8547 | 59.076 | 7.385 | 883.8088 |
+| 7000 | 0.2828 | 258.4085 | 1491.2831 | 1.3662 | 33.7766 | 59.213 | 7.402 | 626.0604 |
+| 8000 | 0.3232 | 233.4870 | 1228.4403 | 1.3132 | 33.8614 | 59.064 | 7.383 | 850.6914 |
+| 9000 | 0.3636 | 216.1260 | 1157.4716 | 1.2660 | 34.0138 | 58.8 | 7.35 | 725.3146 |
+| 10000 | 0.4040 | 202.6199 | 1178.7196 | 1.2261 | 33.8348 | 59.111 | 7.389 | 783.9336 |
+| 11000 | 0.4444 | 186.9287 | 1086.7654 | 1.1812 | 33.885 | 59.023 | 7.378 | 767.5665 |
+| 12000 | 0.4848 | 168.7043 | 1002.8327 | 1.1325 | 33.9839 | 58.851 | 7.356 | 807.3055 |
+| 13000 | 0.5253 | 157.9653 | 885.8019 | 1.0944 | 33.8811 | 59.03 | 7.379 | 597.1538 |
+| 14000 | 0.5657 | 150.5168 | 838.8755 | 1.0732 | 33.9477 | 58.914 | 7.364 | 586.4849 |
+| 15000 | 0.6061 | 145.0886 | 758.6350 | 1.0454 | 33.9544 | 58.902 | 7.363 | 1364.0833 |
+| 16000 | 0.6465 | 140.9902 | 774.3610 | 1.0288 | 33.8344 | 59.111 | 7.389 | 882.1583 |
+| 17000 | 0.6869 | 135.8739 | 742.5475 | 1.0145 | 33.7421 | 59.273 | 7.409 | 747.0387 |
+| 18000 | 0.7273 | 132.0769 | 685.4412 | 0.9989 | 33.917 | 58.967 | 7.371 | 1488.5706 |
+| 19000 | 0.7677 | 131.6060 | 731.5826 | 0.9887 | 33.9182 | 58.965 | 7.371 | 618.0857 |
+| 20000 | 0.8081 | 128.5655 | 678.8043 | 0.9780 | 33.9033 | 58.991 | 7.374 | 587.1904 |
+| 21000 | 0.8485 | 126.3780 | 655.1987 | 0.9683 | 33.9405 | 58.927 | 7.366 | 467.9376 |
+| 22000 | 0.8889 | 125.1474 | 660.4861 | 0.9594 | 33.8307 | 59.118 | 7.39 | 436.3720 |
+| 23000 | 0.9293 | 123.5350 | 660.9055 | 0.9523 | 33.9643 | 58.885 | 7.361 | 394.6270 |
+| 24000 | 0.9697 | 123.2763 | 653.1232 | 0.9485 | 33.8028 | 59.167 | 7.396 | 397.2708 |
+| 24750 | 1.0 | 122.3797 | 623.2513 | 0.9423 | 33.9114 | 58.977 | 7.372 | 412.1859 |
+
+### Framework versions
+- Distily 0.2.0
+- Transformers 4.44.0
+- Pytorch 2.3.0
+- Datasets 2.20.0

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 50256,
+  "eos_token_id": 50256,
+  "transformers_version": "4.44.0"
+}

logs/distillation_objective=MultiObjective(logits_weight_1__logits_loss_fn_(fn_kl_divergence_loss())__activations_weight_0__activations_loss_fn_(fn_soft_mse_loss())__attentions_weight_0__attentions_loss_fn/events.out.tfevents.1723582039.93d6cbb3ad53

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1f32346227dc28a76b5048d7c966ee047106262bea4b507d683351dc0306324e
+size 253