Training in progress, step 1500

.gitattributes

@@ -41,3 +41,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 .venv/lib/python3.9/site-packages/torch/lib/libtorch_python.so filter=lfs diff=lfs merge=lfs -text
 wandb/run-20220130_021614-39ybnrhl/run-39ybnrhl.wandb filter=lfs diff=lfs merge=lfs -text
 wandb/run-20220131_140404-gjg8nz5t/run-gjg8nz5t.wandb filter=lfs diff=lfs merge=lfs -text
+wandb/run-20220202_140930-21azk117/run-21azk117.wandb filter=lfs diff=lfs merge=lfs -text

pytorch_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ea8b6080f2824e9bfcaf111587189727b4ec250735d10c647a6a751e6ca08a2e
+oid sha256:8e61630a066ed903764be5e04f1d15edee6a37f2edeea3df20714f3aea1b4300
 size 1262083569

wandb/run-20220202_140930-21azk117/files/output.log

@@ -1099,3 +1099,554 @@ Several commits (2) will be pushed upstream.
 Configuration saved in ./checkpoint-1000/config.json
 Model weights saved in ./checkpoint-1000/pytorch_model.bin
 Configuration saved in ./checkpoint-1000/preprocessor_config.json
+Configuration saved in ./preprocessor_config.json
+Deleting older checkpoint [checkpoint-500] due to args.save_total_limit
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 27%|█████████████████████████████████████████████▎                                                                                                                       | 1099/4000 [3:06:33<7:53:42,  9.80s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 30%|█████████████████████████████████████████████████▌                                                                                                                   | 1200/4000 [3:22:50<5:45:11,  7.40s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 32%|█████████████████████████████████████████████████████▌                                                                                                               | 1299/4000 [3:39:02<7:18:41,  9.75s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 35%|█████████████████████████████████████████████████████████▋                                                                                                           | 1400/4000 [3:55:22<5:26:38,  7.54s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 37%|█████████████████████████████████████████████████████████████▊                                                                                                       | 1499/4000 [4:11:31<6:45:09,  9.72s/it]
+ 38%|█████████████████████████████████████████████████████████████▉                                                                                                       | 1500/4000 [4:11:45<7:42:19, 11.10s/it]The following columns in the evaluation set  don't have a corresponding argument in `Wav2Vec2ForCTC.forward` and have been ignored: input_length.
+***** Running Evaluation *****
+  Num examples = 2609
+  Batch size = 72
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 37/37 [02:13<00:00,  2.82s/it]
+Configuration saved in ./checkpoint-1500/config.json
+Model weights saved in ./checkpoint-1500/pytorch_model.bin
+Configuration saved in ./checkpoint-1500/preprocessor_config.json
+Configuration saved in ./preprocessor_config.json
+Adding files tracked by Git LFS: ['wandb/run-20220202_140930-21azk117/run-21azk117.wandb']. This may take a bit of time if the files are large.