Training in progress, epoch 5

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f328f655a6c8b38df202a4de4e669604f27ad31b0420d0d6742110d15af58f36
+oid sha256:375524c82c9d525ae16e95e88d9547a0a25234b55abc7cc1fcfb584a5cdd0f77
 size 3141646744

runs/Feb01_17-58-13_ip-26-0-165-24/events.out.tfevents.1706810328.ip-26-0-165-24.239318.0

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0603591350ae6d59bd20161e6fbc71c11f9404d1bf38d86d0e0cf98885ef67f2
-size 22693
+oid sha256:7c09ad97b4dd6b45192f7e3d74981fdd53f715a299ca319a89c6899d59b64d50
+size 27557

wandb/run-20240201_175850-i93q0p12/files/config.yaml

@@ -80,6 +80,26 @@ _wandb:
       5: 1
       6:
       - 1
+    - 1: train/train_runtime
+      5: 1
+      6:
+      - 1
+    - 1: train/train_samples_per_second
+      5: 1
+      6:
+      - 1
+    - 1: train/train_steps_per_second
+      5: 1
+      6:
+      - 1
+    - 1: train/total_flos
+      5: 1
+      6:
+      - 1
+    - 1: train/train_loss
+      5: 1
+      6:
+      - 1
 vocab_size:
   desc: null
   value: 32000

wandb/run-20240201_175850-i93q0p12/files/output.log

@@ -1363,3 +1363,322 @@
 
 
  81%|██████████████████████████████████████████████████████████████▌              | 1110/1365 [1:09:33<15:23,  3.62s/it]
+
+
+
+
+
+
+
+
+
+
+ 82%|███████████████████████████████████████████████████████████████▏             | 1120/1365 [1:10:09<14:41,  3.60s/it]
+
+
+
+
+
+
+
+
+
+
+ 83%|███████████████████████████████████████████████████████████████▋             | 1130/1365 [1:10:45<14:04,  3.59s/it]
+
+
+
+
+
+
+
+
+
+
+ 84%|████████████████████████████████████████████████████████████████▎            | 1140/1365 [1:11:21<13:28,  3.59s/it]
+
+
+
+
+
+
+
+
+
+
+ 84%|████████████████████████████████████████████████████████████████▊            | 1150/1365 [1:11:57<13:04,  3.65s/it]
+
+
+
+
+
+
+
+
+
+
+ 85%|█████████████████████████████████████████████████████████████████▍           | 1160/1365 [1:12:33<12:16,  3.59s/it]
+
+
+
+
+
+
+
+
+
+
+ 86%|██████████████████████████████████████████████████████████████████           | 1170/1365 [1:13:09<11:39,  3.59s/it]
+
+
+
+
+
+
+
+
+
+
+ 86%|██████████████████████████████████████████████████████████████████▌          | 1180/1365 [1:13:45<11:05,  3.60s/it]
+
+
+
+
+
+
+
+
+
+
+ 87%|███████████████████████████████████████████████████████████████████▏         | 1190/1365 [1:14:21<10:29,  3.60s/it]
+
+
+
+
+
+
+
+
+
+
+ 88%|███████████████████████████████████████████████████████████████████▋         | 1200/1365 [1:14:57<09:52,  3.59s/it]
+
+
+
+
+
+
+
+
+
+
+ 89%|████████████████████████████████████████████████████████████████████▎        | 1210/1365 [1:15:33<09:16,  3.59s/it]
+
+
+
+
+
+
+
+
+
+
+ 89%|████████████████████████████████████████████████████████████████████▊        | 1220/1365 [1:16:09<08:40,  3.59s/it]
+
+
+
+
+
+
+
+
+
+
+ 90%|█████████████████████████████████████████████████████████████████████▍       | 1230/1365 [1:16:45<08:04,  3.59s/it]
+
+
+
+
+
+
+
+
+
+
+ 91%|█████████████████████████████████████████████████████████████████████▉       | 1240/1365 [1:17:21<07:28,  3.59s/it]
+
+
+
+
+
+
+
+
+
+
+ 92%|██████████████████████████████████████████████████████████████████████▌      | 1250/1365 [1:17:57<06:59,  3.64s/it]
+
+
+
+
+
+
+
+
+
+
+ 92%|███████████████████████████████████████████████████████████████████████      | 1260/1365 [1:18:34<06:23,  3.66s/it]
+
+
+
+
+
+
+
+
+
+
+ 93%|███████████████████████████████████████████████████████████████████████▋     | 1270/1365 [1:19:10<05:43,  3.61s/it]
+
+
+
+
+
+
+
+
+
+
+ 94%|████████████████████████████████████████████████████████████████████████▏    | 1280/1365 [1:19:46<05:06,  3.61s/it]
+
+
+
+
+
+
+
+
+
+
+ 95%|█████████████████████████████���██████████████████████████████████████████▊    | 1290/1365 [1:20:22<04:30,  3.61s/it]
+
+
+
+
+
+
+
+
+
+
+ 95%|█████████████████████████████████████████████████████████████████████████▎   | 1300/1365 [1:20:58<03:54,  3.60s/it]
+
+
+
+
+
+
+
+
+
+
+ 96%|█████████████████████████████████████████████████████████████████████████▉   | 1310/1365 [1:21:34<03:18,  3.61s/it]
+
+
+
+
+
+
+
+
+
+
+ 97%|██████████████████████████████████████████████████████████████████████████▍  | 1320/1365 [1:22:10<02:42,  3.60s/it]
+
+
+
+
+
+
+
+
+
+
+ 97%|███████████████████████████████████████████████████████████████████████████  | 1330/1365 [1:22:46<02:06,  3.60s/it]
+
+
+
+
+
+
+
+
+
+
+ 98%|███████████████████████████████████████████████████████████████████████████▌ | 1340/1365 [1:23:22<01:29,  3.60s/it]
+
+
+
+
+
+
+
+
+
+
+ 99%|████████████████████████████████████████████████████████████████████████████▏| 1350/1365 [1:23:58<00:53,  3.60s/it]
+
+
+
+
+
+
+
+
+
+
+100%|████████████████████████████████████████████████████████████████████████████▋| 1360/1365 [1:24:35<00:18,  3.68s/it]
+
+
+
+
+100%|█████████████████████████████████████████████████████████████████████████████| 1365/1365 [1:24:52<00:00,  3.58s/it][INFO|trainer.py:3166] 2024-02-01 19:23:48,314 >> ***** Running Evaluation *****
+[INFO|trainer.py:3168] 2024-02-01 19:23:48,314 >>   Num examples = 15431
+[INFO|trainer.py:3171] 2024-02-01 19:23:48,314 >>   Batch size = 32
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+[INFO|configuration_utils.py:483] 2024-02-01 19:24:20,057 >> Configuration saved in ./tmp-checkpoint-1365/config.json
+[INFO|configuration_utils.py:594] 2024-02-01 19:24:20,059 >> Configuration saved in ./tmp-checkpoint-1365/generation_config.json
+{'eval_loss': 2.118281126022339, 'eval_runtime': 30.6172, 'eval_samples_per_second': 503.999, 'eval_steps_per_second': 1.992, 'epoch': 5.0}
+[INFO|modeling_utils.py:2382] 2024-02-01 19:24:23,364 >> Model weights saved in ./tmp-checkpoint-1365/pytorch_model.bin
+[INFO|tokenization_utils_base.py:2432] 2024-02-01 19:24:23,383 >> tokenizer config file saved in ./tmp-checkpoint-1365/tokenizer_config.json
+[INFO|tokenization_utils_base.py:2441] 2024-02-01 19:24:23,385 >> Special tokens file saved in ./tmp-checkpoint-1365/special_tokens_map.json
+/fsx/sanchit/miniconda3/envs/venv/lib/python3.11/site-packages/torch/nn/modules/module.py:1879: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.
+  warnings.warn(
+[2024-02-01 19:24:23,449] [INFO] [logging.py:96:log_dist] [Rank 0] [Torch] Checkpoint global_step1365 is about to be saved!
+[2024-02-01 19:24:23,462] [INFO] [logging.py:96:log_dist] [Rank 0] Saving model checkpoint: ./tmp-checkpoint-1365/global_step1365/zero_pp_rank_0_mp_rank_00_model_states.pt
+[2024-02-01 19:24:23,462] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./tmp-checkpoint-1365/global_step1365/zero_pp_rank_0_mp_rank_00_model_states.pt...
+[2024-02-01 19:24:23,467] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./tmp-checkpoint-1365/global_step1365/zero_pp_rank_0_mp_rank_00_model_states.pt.
+[2024-02-01 19:24:23,476] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./tmp-checkpoint-1365/global_step1365/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt...
+[2024-02-01 19:24:27,289] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./tmp-checkpoint-1365/global_step1365/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt.
+[2024-02-01 19:24:27,298] [INFO] [engine.py:3393:_save_zero_checkpoint] zero checkpoint saved ./tmp-checkpoint-1365/global_step1365/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt
+[2024-02-01 19:24:27,349] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step1365 is ready now!
+[INFO|tokenization_utils_base.py:2432] 2024-02-01 19:24:29,644 >> tokenizer config file saved in ./tokenizer_config.json
+[INFO|tokenization_utils_base.py:2441] 2024-02-01 19:24:29,647 >> Special tokens file saved in ./special_tokens_map.json
+[INFO|trainer.py:2979] 2024-02-01 19:24:29,677 >> Deleting older checkpoint [checkpoint-1092] due to args.save_total_limit
+[INFO|trainer.py:1947] 2024-02-01 19:24:29,709 >>
+Training completed. Do not forget to share your model on huggingface.co/models =)
+100%|█████████████████████████████████████████████████████████████████████████████| 1365/1365 [1:25:34<00:00,  3.76s/it]
+[INFO|trainer.py:3614] 2024-02-01 19:24:29,900 >> Waiting for the current checkpoint push to be finished, this might take a couple of minutes.

wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json

@@ -1 +1 @@
-{"train/loss": 2.0771, "train/learning_rate": 3.080259523093675e-05, "train/epoch": 4.07, "train/global_step": 1110, "_timestamp": 1706814509.1468732, "_runtime": 4179.062906265259, "_step": 115, "eval/loss": 2.1637322902679443, "eval/runtime": 30.5067, "eval/samples_per_second": 505.824, "eval/steps_per_second": 2.0}
+{"train/loss": 2.0369, "train/learning_rate": 1.2271520073786623e-08, "train/epoch": 5.0, "train/global_step": 1365, "_timestamp": 1706815469.711052, "_runtime": 5139.627084970474, "_step": 142, "eval/loss": 2.118281126022339, "eval/runtime": 30.6172, "eval/samples_per_second": 503.999, "eval/steps_per_second": 1.992, "train/train_runtime": 5141.5129, "train/train_samples_per_second": 135.588, "train/train_steps_per_second": 0.265, "train/total_flos": 457285168005120.0, "train/train_loss": 3.477488596011431}

wandb/run-20240201_175850-i93q0p12/run-i93q0p12.wandb

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:86fca87b3c839ac0a9a2afa910e3c01e4a90376cc9fd40b08d41240fbed65ed7
-size 1212557
+oid sha256:83bb2becd0a41bef6a83fb4db8cc052bae33199ac7e7fd2c198b5018f4037fd9
+size 1540113