Training in progress, step 35000

README.md

@@ -1,7 +1,7 @@
 ---
 base_model: gpt2
 datasets:
-- Skylion007/openwebtext
+- distily/filtered_redpajama_multilingual
 library_name: Distily
 license: creativeml-openrail-m
 tags:
@@ -18,7 +18,7 @@ model-index:
 
 Distilled with [Distily](https://github.com/lapp0/distily) library
 using teacher model [gpt2](https://huggingface.co/gpt2)
-on dataset [Skylion007/openwebtext](https://huggingface.co/datasets/Skylion007/openwebtext).
+on dataset [distily/filtered_redpajama_multilingual](https://huggingface.co/datasets/distily/filtered_redpajama_multilingual).
 
 <!-- This model card has been generated automatically according to the information the Trainer had access to. You
 should probably proofread and complete it, then remove this comment.
@@ -78,13 +78,13 @@ GPT2LMHeadModel(
 
 # Resource Usage
 
-- Max Train VRAM Use: 15.7138 GB
-- Available VRAM: 23.4329 GB
+- Max Train VRAM Use: 15.7151 GB
+- Available VRAM: 23.6497 GB
 - GPUs: 
   - 1x NVIDIA GeForce RTX 4090
-- CPUs: 64
-- CPU Memory: 251.7299 GB
-- CPU Memory Bandwidth: 1600 GB/s
+- CPUs: 28
+- CPU Memory: 62.6429 GB
+- CPU Memory Bandwidth: 700 GB/s
 
 # Distillation (Teacher -> Student) Architecture Difference:
 
@@ -115,7 +115,7 @@ GPT2LMHeadModel(
 <br/>
 
 # Train Dataset
-Trained on 695,373,010 tokens from the [Skylion007/openwebtext](https://huggingface.co/datasets/Skylion007/openwebtext) dataset.
+Trained on 662,561,960 tokens from the [distily/filtered_redpajama_multilingual](https://huggingface.co/datasets/distily/filtered_redpajama_multilingual) dataset.
 
 - Num Samples: `998,000`
 - Subset: `None`
@@ -172,7 +172,7 @@ The following hyperparameters were used during training:
         projector='orthogonal'
     )
 )`
-- lr_scheduler: `<torch.optim.lr_scheduler.LambdaLR object at 0x7a0465b66050>`
+- lr_scheduler: `<torch.optim.lr_scheduler.LambdaLR object at 0x7fde6c1b6bf0>`
 - student_model_name_or_path: `None`
 - student_config_name_or_path: `distilbert/distilgpt2`
 - student_model_config: `None`
@@ -182,15 +182,15 @@ The following hyperparameters were used during training:
 - teacher_model_name_or_path: `gpt2`
 - teacher_load_in_8bit: `False`
 - teacher_load_in_4bit: `False`
-- dataset_uri: `Skylion007/openwebtext`
+- dataset_uri: `distily/filtered_redpajama_multilingual`
 - dataset_subset: `None`
 - dataset_split: `train`
 - dataset_column_name: `text`
 - dataset_sample_size: `1000000`
 - dataset_test_size: `0.002`
-- dataset_shuffle: `True`
+- dataset_shuffle: `False`
 - dataset_shuffle_seed: `42`
-- dataset_trust_remote_code: `True`
+- dataset_trust_remote_code: `False`
 - gradient_accumulation_steps: `1`
 - weight_decay: `0.0`
 - max_grad_norm: `1.0`

logs/dataset_split=train, dataset_subset=None, dataset_uri=distily_filtered_redpajama_en, lr_scheduler_kwargs=None, lr_scheduler_type=constant/events.out.tfevents.1725843275.32a7ca12a2ec

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:712a6914691a800a89dbedb1eb805f069051779ee0635f6511e6aca0ed8d771f
-size 956393
+oid sha256:2d6819ea1c78aa11044807f8b1c6a644a9ce1304b51bf19a7c456595fbed4c96
+size 1116179

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c439f688f07b573f6efb9b1c6ee9efea4014f81cb236ec25046bc8d8eccb9254
+oid sha256:aa3e9b7b109082c44feea718c92cb807eb6c8e090a5540fecc6b9ce2c6a7fb5b
 size 163832792

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:96663156563d1fc2fc2b925f5af317875cf0078e0f7b2631846e89b66ae41d9a
-size 5624
+oid sha256:b1a1ed16452d9f3422f0969ffcdb2f0c367a26c0cf92de987cd1213ef49c0169
+size 5496