End of training

README.md

@@ -1,83 +1,235 @@
 ---
-library_name: transformers
-license: apache-2.0
 base_model: HuggingFaceTB/SmolLM-135M
+datasets:
+- HuggingFaceFW/fineweb
+library_name: Distily
+license: creativeml-openrail-m
 tags:
 - generated_from_trainer
+- Distily
+base_model_relation: finetune
 model-index:
 - name: distily_smollm_dataset_sweep
   results: []
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
-
-# distily_smollm_dataset_sweep
 
-This model is a fine-tuned version of [HuggingFaceTB/SmolLM-135M](https://huggingface.co/HuggingFaceTB/SmolLM-135M) on an unknown dataset.
-It achieves the following results on the evaluation set:
-- Loss: 0.2647
+# Summary
 
-## Model description
+Distilled with [Distily](https://github.com/lapp0/distily) library
+using teacher model [HuggingFaceTB/SmolLM-135M](https://huggingface.co/HuggingFaceTB/SmolLM-135M)
+on dataset [HuggingFaceFW/fineweb](https://huggingface.co/datasets/HuggingFaceFW/fineweb).
 
-More information needed
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment.
 
-## Intended uses & limitations
+# Model description
 
 More information needed
 
-## Training and evaluation data
+# Intended uses & limitations
 
 More information needed
-
-## Training procedure
-
-### Training hyperparameters
-
+-->
+
+# Model Architecture:
+- **Architecture**: `LlamaForCausalLM`
+- **Total Parameters**: 81,413,568
+- **Data Type (dtype)**: torch.float32
+- **Model Size**: 0.30 GB
+
+<details>
+<summary>Student Model Details</summary>
+
+```
+LlamaForCausalLM(
+  (model): LlamaModel(
+    (embed_tokens): Embedding(49152, 576)
+    (layers): ModuleList(
+      (0-14): 15 x LlamaDecoderLayer(
+        (self_attn): LlamaSdpaAttention(
+          (q_proj): Linear(in_features=576, out_features=576, bias=False)
+          (k_proj): Linear(in_features=576, out_features=192, bias=False)
+          (v_proj): Linear(in_features=576, out_features=192, bias=False)
+          (o_proj): Linear(in_features=576, out_features=576, bias=False)
+          (rotary_emb): LlamaRotaryEmbedding()
+        )
+        (mlp): LigerSwiGLUMLP(
+          (gate_proj): Linear(in_features=576, out_features=1536, bias=False)
+          (up_proj): Linear(in_features=576, out_features=1536, bias=False)
+          (down_proj): Linear(in_features=1536, out_features=576, bias=False)
+        )
+        (input_layernorm): LigerRMSNorm((576,), eps=1e-05, offset=0.0)
+        (post_attention_layernorm): LigerRMSNorm((576,), eps=1e-05, offset=0.0)
+      )
+    )
+    (norm): LigerRMSNorm((576,), eps=1e-05, offset=0.0)
+    (rotary_emb): LlamaRotaryEmbedding()
+  )
+  (lm_head): Linear(in_features=576, out_features=49152, bias=False)
+)
+```
+
+</details>
+<br/>
+
+# Benchmark Metrics Comparison
+
+| Metric | distily_smollm_dataset_sweep/logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=20231101.en, dataset_uri=wikimedia_wikipedia, per_device_train_batch_size=8 | distily_smollm_dataset_sweep/logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=None, dataset_uri=distily_filtered_redpajama_en, per_device_train_batch_size=8 | distily_smollm_dataset_sweep/logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=sample-10BT, dataset_uri=HuggingFaceFW_fineweb, per_device_train_batch_size=8 | distily_smollm_dataset_sweep/logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=sample-10BT, dataset_uri=HuggingFaceFW_fineweb-edu, per_device_train_batch_size=8 | logs/teacher |
+| :--- | :--- | :--- | :--- | :--- | :--- |
+| tinyArc.acc_norm,none | 0.303 | 0.295 | 0.26 | 0.302 | 0.37 |
+| tinyGSM8k.exact_match,flexible-extract | 0.029 | 0.03 | 0.006 | 0.025 | 0.006 |
+| tinyGSM8k.exact_match,strict-match | 0.006 | 0.006 | 0.006 | 0.006 | 0.006 |
+| tinyHellaswag.acc_norm,none | 0.341 | 0.281 | 0.3 | 0.327 | 0.452 |
+| tinyMMLU.acc_norm,none | 0.276 | 0.281 | 0.286 | 0.31 | 0.341 |
+| tinyTruthfulQA.acc,none | 0.463 | 0.447 | 0.419 | 0.423 | 0.38 |
+| tinyWinogrande.acc_norm,none | 0.466 | 0.436 | 0.492 | 0.46 | 0.509 |
+
+# Resource Usage
+
+- Max Train VRAM Use: 13.1269 GB
+- Available VRAM: 23.4329 GB
+- GPUs: 
+  - 1x NVIDIA GeForce RTX 4090
+- CPUs: 64
+- CPU Memory: 251.7299 GB
+- CPU Memory Bandwidth: 1600 GB/s
+
+# Distillation (Teacher -> Student) Architecture Difference:
+
+- **Architecture**: `LlamaForCausalLM` -> `LlamaForCausalLM`
+- **Total Parameters**: 134,515,008 -> 81,413,568
+- **Data Type (dtype)**: torch.float32 -> torch.float32
+- **Model Size**: 0.25 GB -> 0.30 GB
+
+<details>
+<summary>Module Diff Details</summary>
+
+```diff
+--- teacher model modules
++++ student model modules
+@@ -2,7 +2,7 @@
+   (model): LlamaModel(
+     (embed_tokens): Embedding(49152, 576)
+     (layers): ModuleList(
+-      (0-29): 30 x LlamaDecoderLayer(
++      (0-14): 15 x LlamaDecoderLayer(
+         (self_attn): LlamaSdpaAttention(
+           (q_proj): Linear(in_features=576, out_features=576, bias=False)
+           (k_proj): Linear(in_features=576, out_features=192, bias=False)
+@@ -10,17 +10,16 @@
+           (o_proj): Linear(in_features=576, out_features=576, bias=False)
+           (rotary_emb): LlamaRotaryEmbedding()
+         )
+-        (mlp): LlamaMLP(
++        (mlp): LigerSwiGLUMLP(
+           (gate_proj): Linear(in_features=576, out_features=1536, bias=False)
+           (up_proj): Linear(in_features=576, out_features=1536, bias=False)
+           (down_proj): Linear(in_features=1536, out_features=576, bias=False)
+-          (act_fn): SiLU()
+         )
+-        (input_layernorm): LlamaRMSNorm((576,), eps=1e-05)
+-        (post_attention_layernorm): LlamaRMSNorm((576,), eps=1e-05)
++        (input_layernorm): LigerRMSNorm((576,), eps=1e-05, offset=0.0)
++        (post_attention_layernorm): LigerRMSNorm((576,), eps=1e-05, offset=0.0)
+       )
+     )
+-    (norm): LlamaRMSNorm((576,), eps=1e-05)
++    (norm): LigerRMSNorm((576,), eps=1e-05, offset=0.0)
+     (rotary_emb): LlamaRotaryEmbedding()
+   )
+   (lm_head): Linear(in_features=576, out_features=49152, bias=False)
+
+```
+
+</details>
+<br/>
+
+# Train Dataset
+Trained on 501,164,413 tokens from the [HuggingFaceFW/fineweb](https://huggingface.co/datasets/HuggingFaceFW/fineweb) dataset.
+
+- Num Samples: `998,000`
+- Subset: `sample-10BT`
+- Split: `train`
+
+
+# Training Objective
+
+```
+DistillationObjective(
+    logits_loss_component=LossComponent(
+        weight=1,
+        loss_fn='kl'
+    ),
+    hs_loss_component=LossComponent(
+        weight=0
+    ),
+    attn_loss_component=LossComponent(
+        weight=0
+    )
+)
+```
+
+# Hyperparameters
 The following hyperparameters were used during training:
-- learning_rate: 0.0001
-- train_batch_size: 8
-- eval_batch_size: 4
-- seed: 42
-- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
-- lr_scheduler_type: polynomial
-- lr_scheduler_warmup_ratio: 0.1
-- num_epochs: 1.0
-
-### Training results
-
-| Training Loss | Epoch  | Step   | Validation Loss |
-|:-------------:|:------:|:------:|:---------------:|
-| No log        | 0      | 0      | 18.8388         |
-| 1.2041        | 0.0401 | 5000   | 1.1584          |
-| 0.7528        | 0.0802 | 10000  | 0.7396          |
-| 0.5961        | 0.1202 | 15000  | 0.6070          |
-| 0.5023        | 0.1603 | 20000  | 0.5307          |
-| 0.4706        | 0.2004 | 25000  | 0.4836          |
-| 0.4605        | 0.2405 | 30000  | 0.4512          |
-| 0.417         | 0.2806 | 35000  | 0.4251          |
-| 0.4027        | 0.3206 | 40000  | 0.4071          |
-| 0.3693        | 0.3607 | 45000  | 0.3898          |
-| 0.3745        | 0.4008 | 50000  | 0.3759          |
-| 0.3652        | 0.4409 | 55000  | 0.3632          |
-| 0.3537        | 0.4810 | 60000  | 0.3529          |
-| 0.3665        | 0.5210 | 65000  | 0.3440          |
-| 0.3177        | 0.5611 | 70000  | 0.3346          |
-| 0.3102        | 0.6012 | 75000  | 0.3269          |
-| 0.3023        | 0.6413 | 80000  | 0.3198          |
-| 0.3076        | 0.6814 | 85000  | 0.3125          |
-| 0.3388        | 0.7214 | 90000  | 0.3062          |
-| 0.298         | 0.7615 | 95000  | 0.3003          |
-| 0.3052        | 0.8016 | 100000 | 0.2941          |
-| 0.2678        | 0.8417 | 105000 | 0.2880          |
-| 0.2684        | 0.8818 | 110000 | 0.2824          |
-| 0.274         | 0.9218 | 115000 | 0.2764          |
-| 0.2647        | 0.9619 | 120000 | 0.2706          |
-
-
-### Framework versions
 
+<details>
+<summary>Expand</summary>
+
+- learning_rate: `0.0001`
+- train_batch_size: `8`
+- eval_batch_size: `4`
+- seed: `42`
+- optimizer: `Adam with betas=(0.9,0.999) and epsilon=1e-08`
+- lr_scheduler_type: `polynomial`
+- lr_scheduler_warmup_ratio: `0.1`
+- num_epochs: `1.0`
+- distillation_objective: `DistillationObjective(
+    logits_loss_component=LossComponent(
+        weight=1,
+        loss_fn='kl'
+    ),
+    hs_loss_component=LossComponent(
+        weight=0
+    ),
+    attn_loss_component=LossComponent(
+        weight=0
+    )
+)`
+- lr_scheduler: `<torch.optim.lr_scheduler.LambdaLR object at 0x7205cc5db070>`
+- student_model_name_or_path: `None`
+- student_config_name_or_path: `None`
+- student_model_config: `{'num_hidden_layers': 15}`
+- reinitialize_weights: `None`
+- copy_teacher_modules: `[('lm_head', False)]`
+- student_model_as_bitnet: `False`
+- student_use_liger_kernel: `True`
+- teacher_model_name_or_path: `HuggingFaceTB/SmolLM-135M`
+- teacher_load_in_8bit: `False`
+- teacher_load_in_4bit: `False`
+- dataset_uri: `HuggingFaceFW/fineweb`
+- dataset_subset: `sample-10BT`
+- dataset_split: `train`
+- dataset_column_name: `text`
+- dataset_sample_size: `1000000`
+- dataset_max_seq_length: `1024`
+- dataset_test_size: `0.002`
+- dataset_shuffle: `False`
+- dataset_shuffle_seed: `42`
+- dataset_trust_remote_code: `False`
+- gradient_accumulation_steps: `1`
+- weight_decay: `0.0`
+- max_grad_norm: `1.0`
+- warmup_ratio: `0.1`
+- warmup_steps: `0`
+- gradient_checkpointing: `True`
+
+</details>
+<br/>
+
+
+# Framework Versions
+- Distily 0.5.0
 - Transformers 4.45.0.dev0
 - Pytorch 2.5.0.dev20240910+cu121
 - Datasets 2.21.0
-- Tokenizers 0.19.1

benchmarks.shelve.bak

@@ -2,3 +2,4 @@
 'distily_smollm_dataset_sweep/logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=20231101.en, dataset_uri=wikimedia_wikipedia, per_device_train_batch_size=8', (512, 448)
 'distily_smollm_dataset_sweep/logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=None, dataset_uri=distily_filtered_redpajama_en, per_device_train_batch_size=8', (1024, 448)
 'distily_smollm_dataset_sweep/logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=sample-10BT, dataset_uri=HuggingFaceFW_fineweb-edu, per_device_train_batch_size=8', (1536, 448)
+'distily_smollm_dataset_sweep/logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=sample-10BT, dataset_uri=HuggingFaceFW_fineweb, per_device_train_batch_size=8', (2048, 448)

benchmarks.shelve.dir

@@ -2,3 +2,4 @@
 'distily_smollm_dataset_sweep/logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=20231101.en, dataset_uri=wikimedia_wikipedia, per_device_train_batch_size=8', (512, 448)
 'distily_smollm_dataset_sweep/logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=None, dataset_uri=distily_filtered_redpajama_en, per_device_train_batch_size=8', (1024, 448)
 'distily_smollm_dataset_sweep/logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=sample-10BT, dataset_uri=HuggingFaceFW_fineweb-edu, per_device_train_batch_size=8', (1536, 448)
+'distily_smollm_dataset_sweep/logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=sample-10BT, dataset_uri=HuggingFaceFW_fineweb, per_device_train_batch_size=8', (2048, 448)

logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=20231101.en, dataset_uri=wikimedia_wikipedia, per_device_train_batch_size=8/events.out.tfevents.1727245509.1c1a426a2fee

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:817ca7d0896bd3aeb6fb1bcfd3a6da536f2658490279cbd43e920fbebbfa9838
+size 562

logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=None, dataset_uri=distily_filtered_redpajama_en, per_device_train_batch_size=8/events.out.tfevents.1727245509.1c1a426a2fee

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4c773e5e524db38b7421c140c09ff664a7b9123e19eb26cd1eadbe3b1687862e
+size 562

logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=sample-10BT, dataset_uri=HuggingFaceFW_fineweb, per_device_train_batch_size=8/events.out.tfevents.1727245069.1c1a426a2fee

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eb5b7721e4ed99470b12fad6a02ed811da840bdb71a83d6efc6eb1d4fb74c02c
+size 529

logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=sample-10BT, dataset_uri=HuggingFaceFW_fineweb, per_device_train_batch_size=8/events.out.tfevents.1727245509.1c1a426a2fee

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:603de8f730ea9dd7ad40f010b156c997fba29094effbafe9d8034cad531eed90
+size 562

logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=sample-10BT, dataset_uri=HuggingFaceFW_fineweb-edu, per_device_train_batch_size=8/events.out.tfevents.1727245509.1c1a426a2fee

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9c16ab5b45b959065ee903ba6c05e11f35df910dddfeb6a9224d7b50e5a06822
+size 562

tokenizer.json

@@ -1,19 +1,7 @@
 {
   "version": "1.0",
-  "truncation": {
-    "direction": "Right",
-    "max_length": 1023,
-    "strategy": "LongestFirst",
-    "stride": 0
-  },
-  "padding": {
-    "strategy": "BatchLongest",
-    "direction": "Right",
-    "pad_to_multiple_of": null,
-    "pad_id": 0,
-    "pad_type_id": 0,
-    "pad_token": "<|endoftext|>"
-  },
+  "truncation": null,
+  "padding": null,
   "added_tokens": [
     {
       "id": 0,