Training in progress, step 1500

.gitattributes

@@ -26,3 +26,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zstandard filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 wandb/run-20220202_014046-gq6zg0t4/run-gq6zg0t4.wandb filter=lfs diff=lfs merge=lfs -text
+wandb/run-20220202_141009-3kd8yz7g/run-3kd8yz7g.wandb filter=lfs diff=lfs merge=lfs -text

pytorch_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1fe3426704c894363230c264455ce3271686dadff1412f140819c731972310f3
+oid sha256:cb791a344df29954d34fb1faefb94ab52d0ad76ac26122345cda10c6b2dd4e37
 size 1262083569

wandb/run-20220202_141009-3kd8yz7g/files/output.log

@@ -1095,3 +1095,554 @@ Configuration saved in ./preprocessor_config.json
 Configuration saved in ./checkpoint-1000/config.json
 Model weights saved in ./checkpoint-1000/pytorch_model.bin
 Configuration saved in ./checkpoint-1000/preprocessor_config.json
+Configuration saved in ./preprocessor_config.json
+Deleting older checkpoint [checkpoint-500] due to args.save_total_limit
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 28%|█████████████████████████████████████████████▍                                                                                                                       | 1100/4000 [3:05:47<8:42:44, 10.82s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 30%|█████████████████████████████████████████████████▌                                                                                                                   | 1200/4000 [3:21:52<6:01:51,  7.75s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 32%|█████████████████████████████████████████████████████▌                                                                                                               | 1299/4000 [3:38:02<7:16:04,  9.69s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 35%|█████████████████████████████████████████████████████████▋                                                                                                           | 1400/4000 [3:54:15<5:26:18,  7.53s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 37%|█████████████████████████████████████████████████████████████▊                                                                                                       | 1499/4000 [4:10:23<6:42:43,  9.66s/it]
+ 38%|█████████████████████████████████████████████████████████████▉                                                                                                       | 1500/4000 [4:10:37<7:33:58, 10.90s/it]The following columns in the evaluation set  don't have a corresponding argument in `Wav2Vec2ForCTC.forward` and have been ignored: input_length.
+***** Running Evaluation *****
+  Num examples = 2613
+  Batch size = 72
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Configuration saved in ./checkpoint-1500/config.json
+{'eval_loss': 0.4445520341396332, 'eval_wer': 0.4220136951559726, 'eval_runtime': 137.9299, 'eval_samples_per_second': 18.944, 'eval_steps_per_second': 0.268, 'epoch': 37.5}
+Model weights saved in ./checkpoint-1500/pytorch_model.bin
+Configuration saved in ./checkpoint-1500/preprocessor_config.json
+Configuration saved in ./preprocessor_config.json
+02/02/2022 18:23:56 - WARNING - huggingface_hub.repository - Adding files tracked by Git LFS: ['wandb/run-20220202_141009-3kd8yz7g/run-3kd8yz7g.wandb']. This may take a bit of time if the files are large.