2022-03-24 10:38:21,870 INFO    MainThread:94821 [internal.py:wandb_internal():92] W&B internal server running at pid: 94821, started at: 2022-03-24 10:38:21.870480
2022-03-24 10:38:21,897 INFO    WriterThread:94821 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/run-m1uuspzx.wandb
2022-03-24 10:38:21,898 DEBUG   SenderThread:94821 [sender.py:send():235] send: header
2022-03-24 10:38:21,898 DEBUG   SenderThread:94821 [sender.py:send():235] send: run
2022-03-24 10:38:21,964 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: check_version
2022-03-24 10:38:21,965 INFO    SenderThread:94821 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files
2022-03-24 10:38:21,965 INFO    SenderThread:94821 [sender.py:_start_run_threads():812] run started: m1uuspzx with start time 1648118301
2022-03-24 10:38:21,965 DEBUG   SenderThread:94821 [sender.py:send():235] send: summary
2022-03-24 10:38:21,965 INFO    SenderThread:94821 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-03-24 10:38:21,965 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: check_version
2022-03-24 10:38:22,024 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: run_start
2022-03-24 10:38:23,028 INFO    Thread-7  :94821 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/wandb-summary.json
2022-03-24 10:38:24,159 DEBUG   HandlerThread:94821 [meta.py:__init__():37] meta init
2022-03-24 10:38:24,159 DEBUG   HandlerThread:94821 [meta.py:__init__():51] meta init done
2022-03-24 10:38:24,159 DEBUG   HandlerThread:94821 [meta.py:probe():211] probe
2022-03-24 10:38:24,160 DEBUG   HandlerThread:94821 [meta.py:_setup_git():201] setup git
2022-03-24 10:38:24,193 DEBUG   HandlerThread:94821 [meta.py:_setup_git():208] setup git done
2022-03-24 10:38:24,193 DEBUG   HandlerThread:94821 [meta.py:_save_pip():55] save pip
2022-03-24 10:38:24,193 DEBUG   HandlerThread:94821 [meta.py:_save_pip():69] save pip done
2022-03-24 10:38:24,193 DEBUG   HandlerThread:94821 [meta.py:probe():249] probe done
2022-03-24 10:38:24,197 DEBUG   SenderThread:94821 [sender.py:send():235] send: files
2022-03-24 10:38:24,197 INFO    SenderThread:94821 [sender.py:_save_file():947] saving file wandb-metadata.json with policy now
2022-03-24 10:38:24,202 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:38:24,203 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:38:24,253 DEBUG   SenderThread:94821 [sender.py:send():235] send: telemetry
2022-03-24 10:38:24,436 INFO    Thread-11 :94821 [upload_job.py:push():137] Uploaded file /tmp/tmp7e9jep7fwandb/2kq1u3tk-wandb-metadata.json
2022-03-24 10:38:25,080 INFO    Thread-7  :94821 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/wandb-metadata.json
2022-03-24 10:38:25,081 INFO    Thread-7  :94821 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/output.log
2022-03-24 10:38:25,081 INFO    Thread-7  :94821 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/requirements.txt
2022-03-24 10:38:27,081 INFO    Thread-7  :94821 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/output.log
2022-03-24 10:38:39,255 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:38:39,255 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:38:43,087 INFO    Thread-7  :94821 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/output.log
2022-03-24 10:38:47,089 INFO    Thread-7  :94821 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/output.log
2022-03-24 10:38:49,090 INFO    Thread-7  :94821 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/output.log
2022-03-24 10:38:52,237 DEBUG   SenderThread:94821 [sender.py:send():235] send: stats
2022-03-24 10:38:53,091 INFO    Thread-7  :94821 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/output.log
2022-03-24 10:38:53,092 INFO    Thread-7  :94821 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/config.yaml
2022-03-24 10:38:54,291 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:38:54,291 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:38:57,093 INFO    Thread-7  :94821 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/output.log
2022-03-24 10:39:09,352 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:39:09,353 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:39:22,309 DEBUG   SenderThread:94821 [sender.py:send():235] send: stats
2022-03-24 10:39:24,449 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:39:24,449 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:39:39,531 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:39:39,531 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:39:52,382 DEBUG   SenderThread:94821 [sender.py:send():235] send: stats
2022-03-24 10:39:54,601 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:39:54,602 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:40:09,640 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:40:09,640 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:40:22,457 DEBUG   SenderThread:94821 [sender.py:send():235] send: stats
2022-03-24 10:40:24,688 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:40:24,688 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:40:39,754 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:40:39,759 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:40:52,532 DEBUG   SenderThread:94821 [sender.py:send():235] send: stats
2022-03-24 10:40:54,817 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:40:54,817 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:41:09,853 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:41:09,854 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:41:22,604 DEBUG   SenderThread:94821 [sender.py:send():235] send: stats
2022-03-24 10:41:24,889 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:41:24,890 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:41:39,927 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:41:39,927 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:41:52,678 DEBUG   SenderThread:94821 [sender.py:send():235] send: stats
2022-03-24 10:41:54,965 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:41:54,965 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:42:10,012 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:42:10,013 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:42:22,756 DEBUG   SenderThread:94821 [sender.py:send():235] send: stats
2022-03-24 10:42:25,051 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:42:25,051 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:42:40,092 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:42:40,093 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:42:52,828 DEBUG   SenderThread:94821 [sender.py:send():235] send: stats
2022-03-24 10:42:55,293 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:42:55,294 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:42:56,754 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: partial_history
2022-03-24 10:42:56,754 DEBUG   SenderThread:94821 [sender.py:send():235] send: telemetry
2022-03-24 10:42:58,184 INFO    Thread-7  :94821 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/output.log
2022-03-24 10:42:59,214 DEBUG   SenderThread:94821 [sender.py:send():235] send: exit
2022-03-24 10:42:59,215 INFO    SenderThread:94821 [sender.py:send_exit():371] handling exit code: 1
2022-03-24 10:42:59,217 INFO    SenderThread:94821 [sender.py:send_exit():373] handling runtime: 277
2022-03-24 10:42:59,217 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-24 10:42:59,218 INFO    SenderThread:94821 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-03-24 10:42:59,218 INFO    SenderThread:94821 [sender.py:send_exit():379] send defer
2022-03-24 10:42:59,218 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: poll_exit
2022-03-24 10:42:59,219 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: defer
2022-03-24 10:42:59,219 INFO    HandlerThread:94821 [handler.py:handle_request_defer():164] handle defer: 0
2022-03-24 10:42:59,220 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: defer
2022-03-24 10:42:59,220 INFO    SenderThread:94821 [sender.py:send_request_defer():388] handle sender defer: 0
2022-03-24 10:42:59,220 INFO    SenderThread:94821 [sender.py:transition_state():392] send defer: 1
2022-03-24 10:42:59,220 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: defer
2022-03-24 10:42:59,220 INFO    HandlerThread:94821 [handler.py:handle_request_defer():164] handle defer: 1
2022-03-24 10:42:59,244 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: defer
2022-03-24 10:42:59,245 INFO    SenderThread:94821 [sender.py:send_request_defer():388] handle sender defer: 1
2022-03-24 10:42:59,245 INFO    SenderThread:94821 [sender.py:transition_state():392] send defer: 2
2022-03-24 10:42:59,245 DEBUG   SenderThread:94821 [sender.py:send():235] send: stats
2022-03-24 10:42:59,245 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: defer
2022-03-24 10:42:59,245 INFO    HandlerThread:94821 [handler.py:handle_request_defer():164] handle defer: 2
2022-03-24 10:42:59,246 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: defer
2022-03-24 10:42:59,246 INFO    SenderThread:94821 [sender.py:send_request_defer():388] handle sender defer: 2
2022-03-24 10:42:59,246 INFO    SenderThread:94821 [sender.py:transition_state():392] send defer: 3
2022-03-24 10:42:59,246 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: defer
2022-03-24 10:42:59,246 INFO    HandlerThread:94821 [handler.py:handle_request_defer():164] handle defer: 3
2022-03-24 10:42:59,246 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: defer
2022-03-24 10:42:59,246 INFO    SenderThread:94821 [sender.py:send_request_defer():388] handle sender defer: 3
2022-03-24 10:42:59,246 INFO    SenderThread:94821 [sender.py:transition_state():392] send defer: 4
2022-03-24 10:42:59,247 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: defer
2022-03-24 10:42:59,247 INFO    HandlerThread:94821 [handler.py:handle_request_defer():164] handle defer: 4
2022-03-24 10:42:59,247 DEBUG   SenderThread:94821 [sender.py:send():235] send: summary
2022-03-24 10:42:59,247 INFO    SenderThread:94821 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-03-24 10:42:59,247 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: defer
2022-03-24 10:42:59,247 INFO    SenderThread:94821 [sender.py:send_request_defer():388] handle sender defer: 4
2022-03-24 10:42:59,248 INFO    SenderThread:94821 [sender.py:transition_state():392] send defer: 5
2022-03-24 10:42:59,248 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: defer
2022-03-24 10:42:59,248 INFO    HandlerThread:94821 [handler.py:handle_request_defer():164] handle defer: 5
2022-03-24 10:42:59,248 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: defer
2022-03-24 10:42:59,248 INFO    SenderThread:94821 [sender.py:send_request_defer():388] handle sender defer: 5
2022-03-24 10:42:59,321 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-24 10:42:59,347 INFO    SenderThread:94821 [sender.py:transition_state():392] send defer: 6
2022-03-24 10:42:59,348 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: poll_exit
2022-03-24 10:42:59,348 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: defer
2022-03-24 10:42:59,348 INFO    HandlerThread:94821 [handler.py:handle_request_defer():164] handle defer: 6
2022-03-24 10:42:59,348 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: defer
2022-03-24 10:42:59,349 INFO    SenderThread:94821 [sender.py:send_request_defer():388] handle sender defer: 6
2022-03-24 10:42:59,349 INFO    SenderThread:94821 [dir_watcher.py:finish():283] shutting down directory watcher
2022-03-24 10:42:59,449 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-24 10:43:00,184 INFO    Thread-7  :94821 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/output.log
2022-03-24 10:43:00,185 INFO    SenderThread:94821 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/wandb-summary.json
2022-03-24 10:43:00,185 INFO    SenderThread:94821 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/config.yaml
2022-03-24 10:43:00,186 INFO    SenderThread:94821 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files
2022-03-24 10:43:00,186 INFO    SenderThread:94821 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/requirements.txt requirements.txt
2022-03-24 10:43:00,186 INFO    SenderThread:94821 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/output.log output.log
2022-03-24 10:43:00,186 INFO    SenderThread:94821 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/wandb-metadata.json wandb-metadata.json
2022-03-24 10:43:00,186 INFO    SenderThread:94821 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/config.yaml config.yaml
2022-03-24 10:43:00,192 INFO    SenderThread:94821 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/wandb-summary.json wandb-summary.json
2022-03-24 10:43:00,193 INFO    SenderThread:94821 [sender.py:transition_state():392] send defer: 7
2022-03-24 10:43:00,193 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: poll_exit
2022-03-24 10:43:00,199 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: defer
2022-03-24 10:43:00,199 INFO    HandlerThread:94821 [handler.py:handle_request_defer():164] handle defer: 7
2022-03-24 10:43:00,199 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: defer
2022-03-24 10:43:00,200 INFO    SenderThread:94821 [sender.py:send_request_defer():388] handle sender defer: 7
2022-03-24 10:43:00,200 INFO    SenderThread:94821 [file_pusher.py:finish():145] shutting down file pusher
2022-03-24 10:43:00,295 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-24 10:43:00,295 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: poll_exit
2022-03-24 10:43:00,397 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-24 10:43:00,397 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: poll_exit
2022-03-24 10:43:00,453 INFO    Thread-14 :94821 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/config.yaml
2022-03-24 10:43:00,465 INFO    Thread-12 :94821 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/requirements.txt
2022-03-24 10:43:00,471 INFO    Thread-13 :94821 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/output.log
2022-03-24 10:43:00,499 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-24 10:43:00,499 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: poll_exit
2022-03-24 10:43:00,562 INFO    Thread-15 :94821 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/wandb-summary.json
2022-03-24 10:43:00,600 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-24 10:43:00,601 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: poll_exit
2022-03-24 10:43:00,702 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-24 10:43:00,703 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: poll_exit
2022-03-24 10:43:00,762 INFO    Thread-6  :94821 [sender.py:transition_state():392] send defer: 8
2022-03-24 10:43:00,763 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: defer
2022-03-24 10:43:00,763 INFO    HandlerThread:94821 [handler.py:handle_request_defer():164] handle defer: 8
2022-03-24 10:43:00,763 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: defer
2022-03-24 10:43:00,763 INFO    SenderThread:94821 [sender.py:send_request_defer():388] handle sender defer: 8
2022-03-24 10:43:00,804 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-24 10:43:00,864 INFO    SenderThread:94821 [sender.py:transition_state():392] send defer: 9
2022-03-24 10:43:00,864 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: poll_exit
2022-03-24 10:43:00,865 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: defer
2022-03-24 10:43:00,865 INFO    HandlerThread:94821 [handler.py:handle_request_defer():164] handle defer: 9
2022-03-24 10:43:00,865 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: defer
2022-03-24 10:43:00,865 INFO    SenderThread:94821 [sender.py:send_request_defer():388] handle sender defer: 9
2022-03-24 10:43:00,865 INFO    SenderThread:94821 [sender.py:transition_state():392] send defer: 10
2022-03-24 10:43:00,866 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: defer
2022-03-24 10:43:00,866 INFO    HandlerThread:94821 [handler.py:handle_request_defer():164] handle defer: 10
2022-03-24 10:43:00,866 DEBUG   SenderThread:94821 [sender.py:send():235] send: final
2022-03-24 10:43:00,866 DEBUG   SenderThread:94821 [sender.py:send():235] send: footer
2022-03-24 10:43:00,866 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: defer
2022-03-24 10:43:00,866 INFO    SenderThread:94821 [sender.py:send_request_defer():388] handle sender defer: 10
2022-03-24 10:43:00,966 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-24 10:43:00,966 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: poll_exit
2022-03-24 10:43:00,966 INFO    SenderThread:94821 [file_pusher.py:join():150] waiting for file pusher
2022-03-24 10:43:01,129 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: sampled_history
2022-03-24 10:43:01,130 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: get_summary
2022-03-24 10:43:01,131 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: shutdown
2022-03-24 10:43:01,131 INFO    HandlerThread:94821 [handler.py:finish():778] shutting down handler
2022-03-24 10:43:01,866 INFO    WriterThread:94821 [datastore.py:close():281] close: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/run-m1uuspzx.wandb
2022-03-24 10:43:02,028 INFO    SenderThread:94821 [sender.py:finish():1078] shutting down sender
2022-03-24 10:43:02,028 INFO    SenderThread:94821 [file_pusher.py:finish():145] shutting down file pusher
2022-03-24 10:43:02,028 INFO    SenderThread:94821 [file_pusher.py:join():150] waiting for file pusher
2022-03-24 10:43:02,031 INFO    MainThread:94821 [internal.py:handle_exit():82] Internal process exited