End of training

README.md

@@ -0,0 +1,113 @@
+---
+base_model: gpt2
+library_name: Distily
+license: mit
+tags:
+- generated_from_trainer
+model-index:
+- name: distily_bench_obj_cross_v2.12_gpt2
+  results: []
+---
+
+# distily_bench_obj_cross_v2.12_gpt2
+
+This student model is distilled from the teacher model [gpt2](https://huggingface.co/gpt2) using the dataset (unspecified).
+
+The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
+
+It achieves the following results on the evaluation set:
+- eval_enwikippl: 503.7859
+- eval_frwikippl: 1047.1145
+- eval_zhwikippl: 264.5559
+- eval_tinystoriesppl: 666.5348
+- eval_loss: 1.4958
+- eval_runtime: 12.5585
+- eval_samples_per_second: 47.776
+- eval_steps_per_second: 11.944
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment.
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+-->
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- distillation_objective: DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl, layer_mapper=None, projector=None), hs_loss_component=LossComponent(label=hs, weight=0, loss_fn=None, layer_mapper=None, projector=None), attn_loss_component=LossComponent(label=attn, weight=0, loss_fn=None, layer_mapper=None, projector=None))
+- train_embeddings: True
+- learning_rate: 0.0001
+- train_batch_size: 1
+- eval_batch_size: 4
+- seed: 42
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: linear
+- lr_scheduler_warmup_ratio: 0.5
+- num_epochs: 1.0
+
+### Resource Usage
+Peak GPU Memory: 3.9293 GB
+
+### Eval-Phase Metrics
+| step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | tinystoriesppl | zhwikippl |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| **teacher eval** |  | 270.2348 | 76.8142 |  |  |  |  | 671.1238 | 22.8030 |
+| 0 | 0 | 183229.1875 | 5048000577536.0 | 20.5358 | 12.4801 | 48.077 | 12.019 | 80.5552 | 3082468982784.0 |
+| 1500 | 0.0253 | 9113.8789 | 30548.5391 | 3.2712 | 12.5479 | 47.817 | 11.954 | 14231.5039 | 456229.9375 |
+| 3000 | 0.0505 | 693.9990 | 4267.4482 | 2.2570 | 12.5474 | 47.819 | 11.955 | 594.9199 | 19603.9199 |
+| 4500 | 0.0758 | 476.8070 | 2154.4661 | 1.8703 | 12.4952 | 48.019 | 12.005 | 449.2081 | 843.1834 |
+| 6000 | 0.1010 | 532.1569 | 1726.7161 | 1.6299 | 12.5593 | 47.773 | 11.943 | 626.2061 | 337.6483 |
+| 7500 | 0.1263 | 735.5850 | 1569.8848 | 1.5580 | 12.6411 | 47.464 | 11.866 | 1143.5127 | 252.8755 |
+| 9000 | 0.1515 | 503.7859 | 1047.1145 | 1.4958 | 12.5585 | 47.776 | 11.944 | 666.5348 | 264.5559 |
+| 10500 | 0.1768 | 533.2093 | 1002.5167 | 1.3808 | 12.6194 | 47.546 | 11.886 | 774.5290 | 288.0 |
+| 12000 | 0.2020 | 637.2250 | 876.1205 | 1.2946 | 12.7141 | 47.192 | 11.798 | 1129.3741 | 217.3894 |
+| 13500 | 0.2273 | 576.6252 | 764.2595 | 1.2314 | 12.6111 | 47.577 | 11.894 | 975.4443 | 331.7217 |
+| 15000 | 0.2525 | 681.7961 | 769.0110 | 1.2003 | 12.5217 | 47.917 | 11.979 | 1309.4331 | 293.8405 |
+| 16500 | 0.2778 | 571.7438 | 690.9651 | 1.1554 | 12.5347 | 47.867 | 11.967 | 1037.2352 | 426.1898 |
+| 18000 | 0.3030 | 528.8489 | 711.8596 | 1.1426 | 12.5384 | 47.853 | 11.963 | 924.9495 | 181.5751 |
+| 19500 | 0.3283 | 499.4630 | 819.7951 | 1.1576 | 12.5733 | 47.72 | 11.93 | 840.5123 | 360.4440 |
+| 21000 | 0.3535 | 529.5048 | 626.7708 | 1.0882 | 12.5636 | 47.757 | 11.939 | 958.8156 | 239.4642 |
+| 22500 | 0.3788 | 512.8919 | 640.6055 | 1.0511 | 12.5386 | 47.852 | 11.963 | 954.6632 | 248.5280 |
+| 24000 | 0.4040 | 529.3919 | 663.4023 | 1.0387 | 12.5736 | 47.719 | 11.93 | 1015.1377 | 198.7630 |
+| 25500 | 0.4293 | 604.9487 | 660.0699 | 0.9904 | 12.5494 | 47.811 | 11.953 | 1328.6781 | 307.4746 |
+| 27000 | 0.4545 | 451.6754 | 803.4192 | 0.9591 | 12.57 | 47.733 | 11.933 | 865.2579 | 270.4616 |
+| 28500 | 0.4798 | 522.6483 | 711.9848 | 0.9809 | 12.4962 | 48.014 | 12.004 | 1094.2220 | 301.7252 |
+| 30000 | 0.5051 | 454.2809 | 689.1668 | 0.9558 | 12.5709 | 47.729 | 11.932 | 907.8295 | 317.3508 |
+| 31500 | 0.5303 | 471.7919 | 614.4454 | 0.9386 | 12.6563 | 47.407 | 11.852 | 951.1581 | 230.7619 |
+| 33000 | 0.5556 | 478.2406 | 545.3618 | 0.8857 | 12.5628 | 47.76 | 11.94 | 1024.1143 | 178.0135 |
+| 34500 | 0.5808 | 492.7567 | 565.6649 | 0.8603 | 12.4885 | 48.044 | 12.011 | 1135.7874 | 291.7314 |
+| 36000 | 0.6061 | 481.1389 | 532.6078 | 0.8346 | 12.5622 | 47.762 | 11.941 | 1090.4297 | 222.2125 |
+| 37500 | 0.6313 | 453.8061 | 465.9002 | 0.8123 | 12.5471 | 47.82 | 11.955 | 992.4849 | 198.2467 |
+| 39000 | 0.6566 | 441.5903 | 493.3379 | 0.7979 | 12.6155 | 47.56 | 11.89 | 983.0132 | 219.6486 |
+| 40500 | 0.6818 | 424.5010 | 446.6855 | 0.7815 | 12.6326 | 47.496 | 11.874 | 924.7585 | 218.5816 |
+| 42000 | 0.7071 | 398.7913 | 369.2028 | 0.7056 | 12.5938 | 47.642 | 11.911 | 915.3652 | 203.1051 |
+| 43500 | 0.7323 | 384.6968 | 325.4155 | 0.6372 | 12.5794 | 47.697 | 11.924 | 909.6699 | 228.6780 |
+| 45000 | 0.7576 | 374.8567 | 310.6573 | 0.6152 | 12.6565 | 47.406 | 11.852 | 890.8421 | 147.1766 |
+| 46500 | 0.7828 | 369.8378 | 282.4115 | 0.5953 | 12.5746 | 47.715 | 11.929 | 884.1658 | 125.3724 |
+| 48000 | 0.8081 | 367.4035 | 286.3572 | 0.5886 | 12.5775 | 47.704 | 11.926 | 874.0283 | 130.5612 |
+| 49500 | 0.8333 | 364.9781 | 283.6874 | 0.5853 | 12.5977 | 47.628 | 11.907 | 872.0796 | 119.1516 |
+| 51000 | 0.8586 | 364.5755 | 272.5054 | 0.5703 | 12.6417 | 47.462 | 11.865 | 881.6116 | 116.9159 |
+| 52500 | 0.8838 | 359.8615 | 263.3172 | 0.5553 | 12.6306 | 47.504 | 11.876 | 871.5392 | 104.2855 |
+| 54000 | 0.9091 | 359.1236 | 256.3636 | 0.5503 | 12.6682 | 47.363 | 11.841 | 875.1489 | 92.7346 |
+| 55500 | 0.9343 | 354.3635 | 253.1253 | 0.5462 | 12.6106 | 47.579 | 11.895 | 860.3723 | 95.9754 |
+| 57000 | 0.9596 | 354.4595 | 253.5534 | 0.5444 | 12.5855 | 47.674 | 11.918 | 860.3010 | 92.5677 |
+| 58500 | 0.9848 | 354.4595 | 251.5613 | 0.5425 | 12.5359 | 47.863 | 11.966 | 863.1859 | 92.5616 |
+| 59400 | 1.0 | 354.5420 | 251.2780 | 0.5425 | 12.6074 | 47.591 | 11.898 | 863.4004 | 92.3149 |
+
+### Framework versions
+- Distily 0.2.0
+- Transformers 4.44.0
+- Pytorch 2.3.0
+- Datasets 2.21.0

config.json

@@ -0,0 +1,54 @@
+{
+  "_name_or_path": "gpt2",
+  "activation_function": "gelu_new",
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.1,
+  "bos_token_id": 50256,
+  "embd_pdrop": 0.1,
+  "eos_token_id": 50256,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_ctx": 1024,
+  "n_embd": 768,
+  "n_head": 12,
+  "n_inner": null,
+  "n_layer": 12,
+  "n_positions": 1024,
+  "quantization_config": {
+    "_load_in_4bit": false,
+    "_load_in_8bit": true,
+    "bnb_4bit_compute_dtype": "float32",
+    "bnb_4bit_quant_storage": "uint8",
+    "bnb_4bit_quant_type": "fp4",
+    "bnb_4bit_use_double_quant": false,
+    "llm_int8_enable_fp32_cpu_offload": false,
+    "llm_int8_has_fp16_weight": false,
+    "llm_int8_skip_modules": null,
+    "llm_int8_threshold": 6.0,
+    "load_in_4bit": false,
+    "load_in_8bit": true,
+    "quant_method": "bitsandbytes"
+  },
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.1,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "task_specific_params": {
+    "text-generation": {
+      "do_sample": true,
+      "max_length": 50
+    }
+  },
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.44.0",
+  "use_cache": true,
+  "vocab_size": 50257
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 50256,
+  "eos_token_id": 50256,
+  "transformers_version": "4.44.0"
+}

logs/batch_size=4, learning_rate=0.0001, warmup_ratio=0.5/events.out.tfevents.1724044875.5f530b1cf724

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f1a85868bade499f3ab9ccbd56ed982ec6920f07e9315a45533c651351de8203
+size 28450727

logs/batch_size=4, learning_rate=0.0001, warmup_ratio=0.5/events.out.tfevents.1724053621.5f530b1cf724

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:762e29c5a66d24deec35efe39911e3fa6593c7055168acc34d0f5411927ccc3f
+size 312

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d3b465018fdd120a5bda93594ba49bc469b498ac0a0aa67d631f31f389e5171f
+size 248894656

special_tokens_map.json

@@ -0,0 +1,6 @@
+{
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "pad_token": "<|endoftext|>",
+  "unk_token": "<|endoftext|>"
+}

tokenizer_config.json

@@ -0,0 +1,20 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "50256": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|endoftext|>",
+  "model_max_length": 1024,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:91c0802635a076af8bdd8fc4456c7ef8620ce85482f8d3e21db568218b3b103d
+size 1017948104