2022-03-23 17:15:39,798 INFO MainThread:16634 [internal.py:wandb_internal():92] W&B internal server running at pid: 16634, started at: 2022-03-23 17:15:39.798196 2022-03-23 17:15:39,829 INFO WriterThread:16634 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/run-5b3vz6lx.wandb 2022-03-23 17:15:39,830 DEBUG SenderThread:16634 [sender.py:send():235] send: header 2022-03-23 17:15:39,830 DEBUG SenderThread:16634 [sender.py:send():235] send: run 2022-03-23 17:15:40,142 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: check_version 2022-03-23 17:15:40,142 INFO SenderThread:16634 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files 2022-03-23 17:15:40,143 INFO SenderThread:16634 [sender.py:_start_run_threads():812] run started: 5b3vz6lx with start time 1648055739 2022-03-23 17:15:40,143 DEBUG SenderThread:16634 [sender.py:send():235] send: summary 2022-03-23 17:15:40,143 INFO SenderThread:16634 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-03-23 17:15:40,143 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: check_version 2022-03-23 17:15:40,202 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: run_start 2022-03-23 17:15:41,144 INFO Thread-7 :16634 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/wandb-summary.json 2022-03-23 17:15:42,639 DEBUG HandlerThread:16634 [meta.py:__init__():37] meta init 2022-03-23 17:15:42,639 DEBUG HandlerThread:16634 [meta.py:__init__():51] meta init done 2022-03-23 17:15:42,639 DEBUG HandlerThread:16634 [meta.py:probe():211] probe 2022-03-23 17:15:42,641 DEBUG HandlerThread:16634 [meta.py:_setup_git():201] setup git 2022-03-23 17:15:42,676 DEBUG HandlerThread:16634 [meta.py:_setup_git():208] setup git done 2022-03-23 17:15:42,676 DEBUG HandlerThread:16634 [meta.py:_save_pip():55] save pip 2022-03-23 17:15:42,677 DEBUG HandlerThread:16634 [meta.py:_save_pip():69] save pip done 2022-03-23 17:15:42,677 DEBUG HandlerThread:16634 [meta.py:probe():249] probe done 2022-03-23 17:15:42,680 DEBUG SenderThread:16634 [sender.py:send():235] send: files 2022-03-23 17:15:42,681 INFO SenderThread:16634 [sender.py:_save_file():947] saving file wandb-metadata.json with policy now 2022-03-23 17:15:42,686 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status 2022-03-23 17:15:42,687 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: stop_status 2022-03-23 17:15:42,740 DEBUG SenderThread:16634 [sender.py:send():235] send: telemetry 2022-03-23 17:15:43,147 INFO Thread-7 :16634 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/requirements.txt 2022-03-23 17:15:43,147 INFO Thread-7 :16634 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/output.log 2022-03-23 17:15:43,147 INFO Thread-7 :16634 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/wandb-metadata.json 2022-03-23 17:15:43,573 INFO Thread-11 :16634 [upload_job.py:push():137] Uploaded file /tmp/tmpcgnu3sa6wandb/1lmj2vet-wandb-metadata.json 2022-03-23 17:15:45,148 INFO Thread-7 :16634 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/output.log 2022-03-23 17:15:57,742 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status 2022-03-23 17:15:57,742 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: stop_status 2022-03-23 17:16:01,154 INFO Thread-7 :16634 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/output.log 2022-03-23 17:16:05,156 INFO Thread-7 :16634 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/output.log 2022-03-23 17:16:09,158 INFO Thread-7 :16634 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/output.log 2022-03-23 17:16:10,158 INFO Thread-7 :16634 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/config.yaml 2022-03-23 17:16:10,726 DEBUG SenderThread:16634 [sender.py:send():235] send: stats 2022-03-23 17:16:12,777 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status 2022-03-23 17:16:12,777 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: stop_status 2022-03-23 17:16:13,159 INFO Thread-7 :16634 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/output.log 2022-03-23 17:16:17,161 INFO Thread-7 :16634 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/output.log 2022-03-23 17:16:27,852 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status 2022-03-23 17:16:27,852 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: stop_status 2022-03-23 17:16:40,804 DEBUG SenderThread:16634 [sender.py:send():235] send: stats 2022-03-23 17:16:42,916 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status 2022-03-23 17:16:42,916 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: stop_status 2022-03-23 17:16:57,968 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status 2022-03-23 17:16:57,968 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: stop_status 2022-03-23 17:17:10,883 DEBUG SenderThread:16634 [sender.py:send():235] send: stats 2022-03-23 17:17:13,013 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status 2022-03-23 17:17:13,013 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: stop_status 2022-03-23 17:17:28,060 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status 2022-03-23 17:17:28,060 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: stop_status 2022-03-23 17:17:40,963 DEBUG SenderThread:16634 [sender.py:send():235] send: stats 2022-03-23 17:17:43,134 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status 2022-03-23 17:17:43,134 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: stop_status 2022-03-23 17:17:58,179 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status 2022-03-23 17:17:58,179 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: stop_status 2022-03-23 17:18:11,043 DEBUG SenderThread:16634 [sender.py:send():235] send: stats 2022-03-23 17:18:13,217 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status 2022-03-23 17:18:13,217 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: stop_status 2022-03-23 17:18:28,336 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status 2022-03-23 17:18:28,337 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: stop_status 2022-03-23 17:18:41,123 DEBUG SenderThread:16634 [sender.py:send():235] send: stats 2022-03-23 17:18:43,408 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status 2022-03-23 17:18:43,408 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: stop_status 2022-03-23 17:18:58,446 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status 2022-03-23 17:18:58,447 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: stop_status 2022-03-23 17:19:11,201 DEBUG SenderThread:16634 [sender.py:send():235] send: stats 2022-03-23 17:19:13,488 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status 2022-03-23 17:19:13,489 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: stop_status 2022-03-23 17:19:28,533 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status 2022-03-23 17:19:28,533 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: stop_status 2022-03-23 17:19:37,253 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: partial_history 2022-03-23 17:19:37,254 DEBUG SenderThread:16634 [sender.py:send():235] send: telemetry 2022-03-23 17:19:38,243 INFO Thread-7 :16634 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/output.log 2022-03-23 17:19:39,594 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: poll_exit 2022-03-23 17:19:39,594 DEBUG SenderThread:16634 [sender.py:send():235] send: exit 2022-03-23 17:19:39,595 INFO SenderThread:16634 [sender.py:send_exit():371] handling exit code: 1 2022-03-23 17:19:39,596 INFO SenderThread:16634 [sender.py:send_exit():373] handling runtime: 239 2022-03-23 17:19:39,596 INFO SenderThread:16634 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-03-23 17:19:39,597 INFO SenderThread:16634 [sender.py:send_exit():379] send defer 2022-03-23 17:19:39,597 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: poll_exit 2022-03-23 17:19:39,597 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: defer 2022-03-23 17:19:39,598 INFO HandlerThread:16634 [handler.py:handle_request_defer():164] handle defer: 0 2022-03-23 17:19:39,598 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: defer 2022-03-23 17:19:39,598 INFO SenderThread:16634 [sender.py:send_request_defer():388] handle sender defer: 0 2022-03-23 17:19:39,598 INFO SenderThread:16634 [sender.py:transition_state():392] send defer: 1 2022-03-23 17:19:39,598 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: defer 2022-03-23 17:19:39,598 INFO HandlerThread:16634 [handler.py:handle_request_defer():164] handle defer: 1 2022-03-23 17:19:39,673 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: defer 2022-03-23 17:19:39,673 INFO SenderThread:16634 [sender.py:send_request_defer():388] handle sender defer: 1 2022-03-23 17:19:39,673 INFO SenderThread:16634 [sender.py:transition_state():392] send defer: 2 2022-03-23 17:19:39,673 DEBUG SenderThread:16634 [sender.py:send():235] send: stats 2022-03-23 17:19:39,674 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: defer 2022-03-23 17:19:39,674 INFO HandlerThread:16634 [handler.py:handle_request_defer():164] handle defer: 2 2022-03-23 17:19:39,674 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: defer 2022-03-23 17:19:39,674 INFO SenderThread:16634 [sender.py:send_request_defer():388] handle sender defer: 2 2022-03-23 17:19:39,674 INFO SenderThread:16634 [sender.py:transition_state():392] send defer: 3 2022-03-23 17:19:39,674 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: defer 2022-03-23 17:19:39,674 INFO HandlerThread:16634 [handler.py:handle_request_defer():164] handle defer: 3 2022-03-23 17:19:39,674 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: defer 2022-03-23 17:19:39,675 INFO SenderThread:16634 [sender.py:send_request_defer():388] handle sender defer: 3 2022-03-23 17:19:39,675 INFO SenderThread:16634 [sender.py:transition_state():392] send defer: 4 2022-03-23 17:19:39,675 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: defer 2022-03-23 17:19:39,675 INFO HandlerThread:16634 [handler.py:handle_request_defer():164] handle defer: 4 2022-03-23 17:19:39,675 DEBUG SenderThread:16634 [sender.py:send():235] send: summary 2022-03-23 17:19:39,675 INFO SenderThread:16634 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-03-23 17:19:39,676 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: defer 2022-03-23 17:19:39,676 INFO SenderThread:16634 [sender.py:send_request_defer():388] handle sender defer: 4 2022-03-23 17:19:39,676 INFO SenderThread:16634 [sender.py:transition_state():392] send defer: 5 2022-03-23 17:19:39,676 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: defer 2022-03-23 17:19:39,676 INFO HandlerThread:16634 [handler.py:handle_request_defer():164] handle defer: 5 2022-03-23 17:19:39,676 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: defer 2022-03-23 17:19:39,676 INFO SenderThread:16634 [sender.py:send_request_defer():388] handle sender defer: 5 2022-03-23 17:19:39,699 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: poll_exit 2022-03-23 17:19:39,863 INFO SenderThread:16634 [sender.py:transition_state():392] send defer: 6 2022-03-23 17:19:39,863 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: poll_exit 2022-03-23 17:19:39,864 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: defer 2022-03-23 17:19:39,864 INFO HandlerThread:16634 [handler.py:handle_request_defer():164] handle defer: 6 2022-03-23 17:19:39,864 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: defer 2022-03-23 17:19:39,864 INFO SenderThread:16634 [sender.py:send_request_defer():388] handle sender defer: 6 2022-03-23 17:19:39,864 INFO SenderThread:16634 [dir_watcher.py:finish():283] shutting down directory watcher 2022-03-23 17:19:39,965 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: poll_exit 2022-03-23 17:19:40,244 INFO SenderThread:16634 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/output.log 2022-03-23 17:19:40,244 INFO SenderThread:16634 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/config.yaml 2022-03-23 17:19:40,244 INFO SenderThread:16634 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/wandb-summary.json 2022-03-23 17:19:40,244 INFO SenderThread:16634 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files 2022-03-23 17:19:40,245 INFO SenderThread:16634 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/requirements.txt requirements.txt 2022-03-23 17:19:40,245 INFO SenderThread:16634 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/output.log output.log 2022-03-23 17:19:40,245 INFO SenderThread:16634 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/wandb-metadata.json wandb-metadata.json 2022-03-23 17:19:40,245 INFO SenderThread:16634 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/config.yaml config.yaml 2022-03-23 17:19:40,245 INFO SenderThread:16634 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/wandb-summary.json wandb-summary.json 2022-03-23 17:19:40,245 INFO SenderThread:16634 [sender.py:transition_state():392] send defer: 7 2022-03-23 17:19:40,246 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: poll_exit 2022-03-23 17:19:40,247 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: defer 2022-03-23 17:19:40,247 INFO HandlerThread:16634 [handler.py:handle_request_defer():164] handle defer: 7 2022-03-23 17:19:40,250 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: defer 2022-03-23 17:19:40,250 INFO SenderThread:16634 [sender.py:send_request_defer():388] handle sender defer: 7 2022-03-23 17:19:40,250 INFO SenderThread:16634 [file_pusher.py:finish():145] shutting down file pusher 2022-03-23 17:19:40,348 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: poll_exit 2022-03-23 17:19:40,349 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: poll_exit 2022-03-23 17:19:40,450 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: poll_exit 2022-03-23 17:19:40,450 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: poll_exit 2022-03-23 17:19:40,518 INFO Thread-14 :16634 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/config.yaml 2022-03-23 17:19:40,552 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: poll_exit 2022-03-23 17:19:40,552 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: poll_exit 2022-03-23 17:19:40,561 INFO Thread-12 :16634 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/requirements.txt 2022-03-23 17:19:40,570 INFO Thread-13 :16634 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/output.log 2022-03-23 17:19:40,589 INFO Thread-15 :16634 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/wandb-summary.json 2022-03-23 17:19:40,654 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: poll_exit 2022-03-23 17:19:40,654 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: poll_exit 2022-03-23 17:19:40,755 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: poll_exit 2022-03-23 17:19:40,756 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: poll_exit 2022-03-23 17:19:40,789 INFO Thread-6 :16634 [sender.py:transition_state():392] send defer: 8 2022-03-23 17:19:40,790 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: defer 2022-03-23 17:19:40,790 INFO HandlerThread:16634 [handler.py:handle_request_defer():164] handle defer: 8 2022-03-23 17:19:40,790 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: defer 2022-03-23 17:19:40,790 INFO SenderThread:16634 [sender.py:send_request_defer():388] handle sender defer: 8 2022-03-23 17:19:40,857 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: poll_exit 2022-03-23 17:19:40,915 INFO SenderThread:16634 [sender.py:transition_state():392] send defer: 9 2022-03-23 17:19:40,915 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: poll_exit 2022-03-23 17:19:40,915 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: defer 2022-03-23 17:19:40,916 INFO HandlerThread:16634 [handler.py:handle_request_defer():164] handle defer: 9 2022-03-23 17:19:40,916 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: defer 2022-03-23 17:19:40,916 INFO SenderThread:16634 [sender.py:send_request_defer():388] handle sender defer: 9 2022-03-23 17:19:40,916 INFO SenderThread:16634 [sender.py:transition_state():392] send defer: 10 2022-03-23 17:19:40,916 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: defer 2022-03-23 17:19:40,916 INFO HandlerThread:16634 [handler.py:handle_request_defer():164] handle defer: 10 2022-03-23 17:19:40,917 DEBUG SenderThread:16634 [sender.py:send():235] send: final 2022-03-23 17:19:40,917 DEBUG SenderThread:16634 [sender.py:send():235] send: footer 2022-03-23 17:19:40,917 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: defer 2022-03-23 17:19:40,917 INFO SenderThread:16634 [sender.py:send_request_defer():388] handle sender defer: 10 2022-03-23 17:19:41,017 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: poll_exit 2022-03-23 17:19:41,017 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: poll_exit 2022-03-23 17:19:41,017 INFO SenderThread:16634 [file_pusher.py:join():150] waiting for file pusher 2022-03-23 17:19:41,172 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: sampled_history 2022-03-23 17:19:41,173 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: get_summary 2022-03-23 17:19:41,174 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: shutdown 2022-03-23 17:19:41,174 INFO HandlerThread:16634 [handler.py:finish():778] shutting down handler 2022-03-23 17:19:41,917 INFO WriterThread:16634 [datastore.py:close():281] close: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/run-5b3vz6lx.wandb 2022-03-23 17:19:42,071 INFO SenderThread:16634 [sender.py:finish():1078] shutting down sender 2022-03-23 17:19:42,071 INFO SenderThread:16634 [file_pusher.py:finish():145] shutting down file pusher 2022-03-23 17:19:42,071 INFO SenderThread:16634 [file_pusher.py:join():150] waiting for file pusher 2022-03-23 17:19:42,110 INFO MainThread:16634 [internal.py:handle_exit():82] Internal process exited