Hugging Face's logo

rlaorrn
/
working

Automatic Speech Recognition
Transformers
TensorBoard
Safetensors
Korean
whisper
hf-asr-leaderboard
Generated from Trainer
Inference Endpoints
Model card Files Metrics Community
rlaorrn commited on
Commit
9760614
•
1 Parent(s): 44d0014

Training in progress, step 2500

Browse files
Files changed (6) hide show
  1. runs/May25_17-49-41_8f1fad5fe1d2/events.out.tfevents.1716659410.8f1fad5fe1d2.34.0 +2 -2
  2. wandb/debug-internal.log +143 -0
  3. wandb/run-20240525_175010-8ah63pdc/files/output.log +4 -0
  4. wandb/run-20240525_175010-8ah63pdc/files/wandb-summary.json +1 -1
  5. wandb/run-20240525_175010-8ah63pdc/logs/debug-internal.log +143 -0
  6. wandb/run-20240525_175010-8ah63pdc/run-8ah63pdc.wandb +0 -0
runs/May25_17-49-41_8f1fad5fe1d2/events.out.tfevents.1716659410.8f1fad5fe1d2.34.0 CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:b4e61fae32c8777f6d12d747d8df89fdc18a6596e24b24816331ad3ca41b5c9a
3
- size 8435
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3817c607e56e5666cc6c45ca24339806a711867cabc3ba782be7bac6220dc095
3
+ size 8964
wandb/debug-internal.log CHANGED
@@ -670,3 +670,146 @@ subprocess.TimeoutExpired: Command '['conda', 'env', 'export']' timed out after
670
  2024-05-25 18:12:02,083 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
671
  2024-05-25 18:12:07,213 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
672
  2024-05-25 18:12:11,503 DEBUG SenderThread:225 [sender.py:send():379] send: stats
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
670
  2024-05-25 18:12:02,083 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
671
  2024-05-25 18:12:07,213 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
672
  2024-05-25 18:12:11,503 DEBUG SenderThread:225 [sender.py:send():379] send: stats
673
+ 2024-05-25 18:12:12,504 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
674
+ 2024-05-25 18:12:16,474 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
675
+ 2024-05-25 18:12:17,057 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
676
+ 2024-05-25 18:12:17,057 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
677
+ 2024-05-25 18:12:18,221 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
678
+ 2024-05-25 18:12:23,222 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
679
+ 2024-05-25 18:12:28,223 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
680
+ 2024-05-25 18:12:31,474 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
681
+ 2024-05-25 18:12:32,056 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
682
+ 2024-05-25 18:12:32,057 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
683
+ 2024-05-25 18:12:34,162 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
684
+ 2024-05-25 18:12:39,163 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
685
+ 2024-05-25 18:12:41,504 DEBUG SenderThread:225 [sender.py:send():379] send: stats
686
+ 2024-05-25 18:12:44,505 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
687
+ 2024-05-25 18:12:46,475 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
688
+ 2024-05-25 18:12:47,057 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
689
+ 2024-05-25 18:12:47,057 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
690
+ 2024-05-25 18:12:50,180 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
691
+ 2024-05-25 18:12:55,182 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
692
+ 2024-05-25 18:13:00,182 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
693
+ 2024-05-25 18:13:01,475 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
694
+ 2024-05-25 18:13:02,057 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
695
+ 2024-05-25 18:13:02,058 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
696
+ 2024-05-25 18:13:05,220 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
697
+ 2024-05-25 18:13:10,221 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
698
+ 2024-05-25 18:13:11,505 DEBUG SenderThread:225 [sender.py:send():379] send: stats
699
+ 2024-05-25 18:13:15,506 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
700
+ 2024-05-25 18:13:16,475 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
701
+ 2024-05-25 18:13:17,057 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
702
+ 2024-05-25 18:13:17,057 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
703
+ 2024-05-25 18:13:21,126 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
704
+ 2024-05-25 18:13:26,127 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
705
+ 2024-05-25 18:13:31,128 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
706
+ 2024-05-25 18:13:31,475 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
707
+ 2024-05-25 18:13:32,370 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
708
+ 2024-05-25 18:13:32,371 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
709
+ 2024-05-25 18:13:36,501 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
710
+ 2024-05-25 18:13:41,502 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
711
+ 2024-05-25 18:13:41,505 DEBUG SenderThread:225 [sender.py:send():379] send: stats
712
+ 2024-05-25 18:13:46,475 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
713
+ 2024-05-25 18:13:47,329 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
714
+ 2024-05-25 18:13:47,329 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
715
+ 2024-05-25 18:13:47,467 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
716
+ 2024-05-25 18:13:52,469 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
717
+ 2024-05-25 18:13:57,469 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
718
+ 2024-05-25 18:14:01,753 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
719
+ 2024-05-25 18:14:02,520 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
720
+ 2024-05-25 18:14:02,521 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
721
+ 2024-05-25 18:14:02,596 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
722
+ 2024-05-25 18:14:07,597 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
723
+ 2024-05-25 18:14:11,506 DEBUG SenderThread:225 [sender.py:send():379] send: stats
724
+ 2024-05-25 18:14:13,507 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
725
+ 2024-05-25 18:14:16,713 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
726
+ 2024-05-25 18:14:17,484 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
727
+ 2024-05-25 18:14:17,484 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
728
+ 2024-05-25 18:14:18,588 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
729
+ 2024-05-25 18:14:23,589 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
730
+ 2024-05-25 18:14:28,590 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
731
+ 2024-05-25 18:14:31,713 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
732
+ 2024-05-25 18:14:32,745 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
733
+ 2024-05-25 18:14:32,745 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
734
+ 2024-05-25 18:14:33,913 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
735
+ 2024-05-25 18:14:38,914 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
736
+ 2024-05-25 18:14:41,507 DEBUG SenderThread:225 [sender.py:send():379] send: stats
737
+ 2024-05-25 18:14:44,508 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
738
+ 2024-05-25 18:14:46,713 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
739
+ 2024-05-25 18:14:47,719 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
740
+ 2024-05-25 18:14:47,720 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
741
+ 2024-05-25 18:14:49,875 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
742
+ 2024-05-25 18:14:54,876 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
743
+ 2024-05-25 18:14:59,877 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
744
+ 2024-05-25 18:15:01,823 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
745
+ 2024-05-25 18:15:02,959 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
746
+ 2024-05-25 18:15:02,960 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
747
+ 2024-05-25 18:15:05,097 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
748
+ 2024-05-25 18:15:10,098 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
749
+ 2024-05-25 18:15:11,508 DEBUG SenderThread:225 [sender.py:send():379] send: stats
750
+ 2024-05-25 18:15:15,509 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
751
+ 2024-05-25 18:15:16,932 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
752
+ 2024-05-25 18:15:17,938 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
753
+ 2024-05-25 18:15:17,939 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
754
+ 2024-05-25 18:15:21,102 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
755
+ 2024-05-25 18:15:26,103 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
756
+ 2024-05-25 18:15:31,104 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
757
+ 2024-05-25 18:15:32,150 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
758
+ 2024-05-25 18:15:32,939 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
759
+ 2024-05-25 18:15:32,939 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
760
+ 2024-05-25 18:15:36,109 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
761
+ 2024-05-25 18:15:41,110 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
762
+ 2024-05-25 18:15:41,509 DEBUG SenderThread:225 [sender.py:send():379] send: stats
763
+ 2024-05-25 18:15:46,510 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
764
+ 2024-05-25 18:15:47,449 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
765
+ 2024-05-25 18:15:47,939 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
766
+ 2024-05-25 18:15:47,939 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
767
+ 2024-05-25 18:15:52,027 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
768
+ 2024-05-25 18:15:57,028 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
769
+ 2024-05-25 18:16:02,029 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
770
+ 2024-05-25 18:16:02,535 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
771
+ 2024-05-25 18:16:02,939 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
772
+ 2024-05-25 18:16:02,939 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
773
+ 2024-05-25 18:16:07,070 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
774
+ 2024-05-25 18:16:11,510 DEBUG SenderThread:225 [sender.py:send():379] send: stats
775
+ 2024-05-25 18:16:12,511 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
776
+ 2024-05-25 18:16:17,511 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
777
+ 2024-05-25 18:16:17,579 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
778
+ 2024-05-25 18:16:17,939 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
779
+ 2024-05-25 18:16:17,939 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
780
+ 2024-05-25 18:16:23,026 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
781
+ 2024-05-25 18:16:28,027 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
782
+ 2024-05-25 18:16:32,578 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
783
+ 2024-05-25 18:16:32,939 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
784
+ 2024-05-25 18:16:32,940 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
785
+ 2024-05-25 18:16:34,016 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
786
+ 2024-05-25 18:16:39,017 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
787
+ 2024-05-25 18:16:41,511 DEBUG SenderThread:225 [sender.py:send():379] send: stats
788
+ 2024-05-25 18:16:44,512 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
789
+ 2024-05-25 18:16:47,823 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
790
+ 2024-05-25 18:16:47,940 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
791
+ 2024-05-25 18:16:47,940 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
792
+ 2024-05-25 18:16:50,089 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
793
+ 2024-05-25 18:16:55,090 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
794
+ 2024-05-25 18:17:00,091 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
795
+ 2024-05-25 18:17:02,793 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
796
+ 2024-05-25 18:17:03,070 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
797
+ 2024-05-25 18:17:03,071 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
798
+ 2024-05-25 18:17:04,434 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: partial_history
799
+ 2024-05-25 18:17:04,435 DEBUG SenderThread:225 [sender.py:send():379] send: history
800
+ 2024-05-25 18:17:04,435 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: summary_record
801
+ 2024-05-25 18:17:04,436 INFO SenderThread:225 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
802
+ 2024-05-25 18:17:05,021 INFO Thread-12 :225 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240525_175010-8ah63pdc/files/wandb-summary.json
803
+ 2024-05-25 18:17:05,436 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
804
+ 2024-05-25 18:17:10,438 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
805
+ 2024-05-25 18:17:11,511 DEBUG SenderThread:225 [sender.py:send():379] send: stats
806
+ 2024-05-25 18:17:14,255 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: partial_history
807
+ 2024-05-25 18:17:14,256 DEBUG SenderThread:225 [sender.py:send():379] send: history
808
+ 2024-05-25 18:17:14,257 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: summary_record
809
+ 2024-05-25 18:17:14,257 INFO SenderThread:225 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
810
+ 2024-05-25 18:17:15,025 INFO Thread-12 :225 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240525_175010-8ah63pdc/files/wandb-summary.json
811
+ 2024-05-25 18:17:16,258 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
812
+ 2024-05-25 18:17:17,793 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
813
+ 2024-05-25 18:17:18,026 INFO Thread-12 :225 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240525_175010-8ah63pdc/files/output.log
814
+ 2024-05-25 18:17:18,098 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
815
+ 2024-05-25 18:17:18,099 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
wandb/run-20240525_175010-8ah63pdc/files/output.log CHANGED
@@ -16,4 +16,8 @@
16
  warnings.warn(
17
  /opt/conda/lib/python3.10/site-packages/transformers/models/wav2vec2/processing_wav2vec2.py:156: UserWarning: `as_target_processor` is deprecated and will be removed in v5 of Transformers. You can process your labels by using the argument `text` of the regular `__call__` method (either in the same call as your audio inputs, or in a separate call.
18
  warnings.warn(
 
 
 
 
19
  /opt/conda/lib/python3.10/site-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.
 
16
  warnings.warn(
17
  /opt/conda/lib/python3.10/site-packages/transformers/models/wav2vec2/processing_wav2vec2.py:156: UserWarning: `as_target_processor` is deprecated and will be removed in v5 of Transformers. You can process your labels by using the argument `text` of the regular `__call__` method (either in the same call as your audio inputs, or in a separate call.
18
  warnings.warn(
19
+ /opt/conda/lib/python3.10/site-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.
20
+ warnings.warn(
21
+ /opt/conda/lib/python3.10/site-packages/transformers/models/wav2vec2/processing_wav2vec2.py:156: UserWarning: `as_target_processor` is deprecated and will be removed in v5 of Transformers. You can process your labels by using the argument `text` of the regular `__call__` method (either in the same call as your audio inputs, or in a separate call.
22
+ warnings.warn(
23
  /opt/conda/lib/python3.10/site-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.
wandb/run-20240525_175010-8ah63pdc/files/wandb-summary.json CHANGED
@@ -1 +1 @@
1
- {"train/loss": 0.0, "train/grad_norm": NaN, "train/learning_rate": 1.3e-06, "train/epoch": 22.73, "train/global_step": 2000, "_timestamp": 1716660714.1692965, "_runtime": 1303.7295565605164, "_step": 7, "eval/loss": NaN, "eval/wer": 1.0, "eval/runtime": 9.9108, "eval/samples_per_second": 20.18, "eval/steps_per_second": 2.522}
 
1
+ {"train/loss": 0.0, "train/grad_norm": NaN, "train/learning_rate": 1.3e-06, "train/epoch": 28.41, "train/global_step": 2500, "_timestamp": 1716661034.2546089, "_runtime": 1623.814868927002, "_step": 9, "eval/loss": NaN, "eval/wer": 1.0, "eval/runtime": 9.817, "eval/samples_per_second": 20.373, "eval/steps_per_second": 2.547}
wandb/run-20240525_175010-8ah63pdc/logs/debug-internal.log CHANGED
@@ -670,3 +670,146 @@ subprocess.TimeoutExpired: Command '['conda', 'env', 'export']' timed out after
670
  2024-05-25 18:12:02,083 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
671
  2024-05-25 18:12:07,213 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
672
  2024-05-25 18:12:11,503 DEBUG SenderThread:225 [sender.py:send():379] send: stats
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
670
  2024-05-25 18:12:02,083 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
671
  2024-05-25 18:12:07,213 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
672
  2024-05-25 18:12:11,503 DEBUG SenderThread:225 [sender.py:send():379] send: stats
673
+ 2024-05-25 18:12:12,504 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
674
+ 2024-05-25 18:12:16,474 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
675
+ 2024-05-25 18:12:17,057 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
676
+ 2024-05-25 18:12:17,057 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
677
+ 2024-05-25 18:12:18,221 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
678
+ 2024-05-25 18:12:23,222 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
679
+ 2024-05-25 18:12:28,223 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
680
+ 2024-05-25 18:12:31,474 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
681
+ 2024-05-25 18:12:32,056 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
682
+ 2024-05-25 18:12:32,057 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
683
+ 2024-05-25 18:12:34,162 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
684
+ 2024-05-25 18:12:39,163 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
685
+ 2024-05-25 18:12:41,504 DEBUG SenderThread:225 [sender.py:send():379] send: stats
686
+ 2024-05-25 18:12:44,505 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
687
+ 2024-05-25 18:12:46,475 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
688
+ 2024-05-25 18:12:47,057 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
689
+ 2024-05-25 18:12:47,057 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
690
+ 2024-05-25 18:12:50,180 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
691
+ 2024-05-25 18:12:55,182 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
692
+ 2024-05-25 18:13:00,182 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
693
+ 2024-05-25 18:13:01,475 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
694
+ 2024-05-25 18:13:02,057 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
695
+ 2024-05-25 18:13:02,058 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
696
+ 2024-05-25 18:13:05,220 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
697
+ 2024-05-25 18:13:10,221 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
698
+ 2024-05-25 18:13:11,505 DEBUG SenderThread:225 [sender.py:send():379] send: stats
699
+ 2024-05-25 18:13:15,506 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
700
+ 2024-05-25 18:13:16,475 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
701
+ 2024-05-25 18:13:17,057 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
702
+ 2024-05-25 18:13:17,057 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
703
+ 2024-05-25 18:13:21,126 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
704
+ 2024-05-25 18:13:26,127 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
705
+ 2024-05-25 18:13:31,128 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
706
+ 2024-05-25 18:13:31,475 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
707
+ 2024-05-25 18:13:32,370 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
708
+ 2024-05-25 18:13:32,371 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
709
+ 2024-05-25 18:13:36,501 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
710
+ 2024-05-25 18:13:41,502 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
711
+ 2024-05-25 18:13:41,505 DEBUG SenderThread:225 [sender.py:send():379] send: stats
712
+ 2024-05-25 18:13:46,475 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
713
+ 2024-05-25 18:13:47,329 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
714
+ 2024-05-25 18:13:47,329 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
715
+ 2024-05-25 18:13:47,467 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
716
+ 2024-05-25 18:13:52,469 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
717
+ 2024-05-25 18:13:57,469 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
718
+ 2024-05-25 18:14:01,753 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
719
+ 2024-05-25 18:14:02,520 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
720
+ 2024-05-25 18:14:02,521 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
721
+ 2024-05-25 18:14:02,596 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
722
+ 2024-05-25 18:14:07,597 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
723
+ 2024-05-25 18:14:11,506 DEBUG SenderThread:225 [sender.py:send():379] send: stats
724
+ 2024-05-25 18:14:13,507 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
725
+ 2024-05-25 18:14:16,713 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
726
+ 2024-05-25 18:14:17,484 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
727
+ 2024-05-25 18:14:17,484 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
728
+ 2024-05-25 18:14:18,588 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
729
+ 2024-05-25 18:14:23,589 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
730
+ 2024-05-25 18:14:28,590 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
731
+ 2024-05-25 18:14:31,713 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
732
+ 2024-05-25 18:14:32,745 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
733
+ 2024-05-25 18:14:32,745 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
734
+ 2024-05-25 18:14:33,913 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
735
+ 2024-05-25 18:14:38,914 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
736
+ 2024-05-25 18:14:41,507 DEBUG SenderThread:225 [sender.py:send():379] send: stats
737
+ 2024-05-25 18:14:44,508 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
738
+ 2024-05-25 18:14:46,713 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
739
+ 2024-05-25 18:14:47,719 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
740
+ 2024-05-25 18:14:47,720 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
741
+ 2024-05-25 18:14:49,875 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
742
+ 2024-05-25 18:14:54,876 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
743
+ 2024-05-25 18:14:59,877 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
744
+ 2024-05-25 18:15:01,823 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
745
+ 2024-05-25 18:15:02,959 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
746
+ 2024-05-25 18:15:02,960 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
747
+ 2024-05-25 18:15:05,097 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
748
+ 2024-05-25 18:15:10,098 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
749
+ 2024-05-25 18:15:11,508 DEBUG SenderThread:225 [sender.py:send():379] send: stats
750
+ 2024-05-25 18:15:15,509 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
751
+ 2024-05-25 18:15:16,932 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
752
+ 2024-05-25 18:15:17,938 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
753
+ 2024-05-25 18:15:17,939 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
754
+ 2024-05-25 18:15:21,102 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
755
+ 2024-05-25 18:15:26,103 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
756
+ 2024-05-25 18:15:31,104 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
757
+ 2024-05-25 18:15:32,150 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
758
+ 2024-05-25 18:15:32,939 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
759
+ 2024-05-25 18:15:32,939 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
760
+ 2024-05-25 18:15:36,109 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
761
+ 2024-05-25 18:15:41,110 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
762
+ 2024-05-25 18:15:41,509 DEBUG SenderThread:225 [sender.py:send():379] send: stats
763
+ 2024-05-25 18:15:46,510 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
764
+ 2024-05-25 18:15:47,449 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
765
+ 2024-05-25 18:15:47,939 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
766
+ 2024-05-25 18:15:47,939 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
767
+ 2024-05-25 18:15:52,027 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
768
+ 2024-05-25 18:15:57,028 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
769
+ 2024-05-25 18:16:02,029 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
770
+ 2024-05-25 18:16:02,535 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
771
+ 2024-05-25 18:16:02,939 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
772
+ 2024-05-25 18:16:02,939 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
773
+ 2024-05-25 18:16:07,070 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
774
+ 2024-05-25 18:16:11,510 DEBUG SenderThread:225 [sender.py:send():379] send: stats
775
+ 2024-05-25 18:16:12,511 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
776
+ 2024-05-25 18:16:17,511 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
777
+ 2024-05-25 18:16:17,579 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
778
+ 2024-05-25 18:16:17,939 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
779
+ 2024-05-25 18:16:17,939 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
780
+ 2024-05-25 18:16:23,026 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
781
+ 2024-05-25 18:16:28,027 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
782
+ 2024-05-25 18:16:32,578 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
783
+ 2024-05-25 18:16:32,939 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
784
+ 2024-05-25 18:16:32,940 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
785
+ 2024-05-25 18:16:34,016 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
786
+ 2024-05-25 18:16:39,017 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
787
+ 2024-05-25 18:16:41,511 DEBUG SenderThread:225 [sender.py:send():379] send: stats
788
+ 2024-05-25 18:16:44,512 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
789
+ 2024-05-25 18:16:47,823 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
790
+ 2024-05-25 18:16:47,940 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
791
+ 2024-05-25 18:16:47,940 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
792
+ 2024-05-25 18:16:50,089 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
793
+ 2024-05-25 18:16:55,090 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
794
+ 2024-05-25 18:17:00,091 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
795
+ 2024-05-25 18:17:02,793 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
796
+ 2024-05-25 18:17:03,070 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
797
+ 2024-05-25 18:17:03,071 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
798
+ 2024-05-25 18:17:04,434 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: partial_history
799
+ 2024-05-25 18:17:04,435 DEBUG SenderThread:225 [sender.py:send():379] send: history
800
+ 2024-05-25 18:17:04,435 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: summary_record
801
+ 2024-05-25 18:17:04,436 INFO SenderThread:225 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
802
+ 2024-05-25 18:17:05,021 INFO Thread-12 :225 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240525_175010-8ah63pdc/files/wandb-summary.json
803
+ 2024-05-25 18:17:05,436 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
804
+ 2024-05-25 18:17:10,438 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
805
+ 2024-05-25 18:17:11,511 DEBUG SenderThread:225 [sender.py:send():379] send: stats
806
+ 2024-05-25 18:17:14,255 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: partial_history
807
+ 2024-05-25 18:17:14,256 DEBUG SenderThread:225 [sender.py:send():379] send: history
808
+ 2024-05-25 18:17:14,257 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: summary_record
809
+ 2024-05-25 18:17:14,257 INFO SenderThread:225 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
810
+ 2024-05-25 18:17:15,025 INFO Thread-12 :225 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240525_175010-8ah63pdc/files/wandb-summary.json
811
+ 2024-05-25 18:17:16,258 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report
812
+ 2024-05-25 18:17:17,793 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages
813
+ 2024-05-25 18:17:18,026 INFO Thread-12 :225 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240525_175010-8ah63pdc/files/output.log
814
+ 2024-05-25 18:17:18,098 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status
815
+ 2024-05-25 18:17:18,099 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status
wandb/run-20240525_175010-8ah63pdc/run-8ah63pdc.wandb CHANGED
Binary files a/wandb/run-20240525_175010-8ah63pdc/run-8ah63pdc.wandb and b/wandb/run-20240525_175010-8ah63pdc/run-8ah63pdc.wandb differ