2022-03-23 17:07:03,443 INFO    MainThread:14495 [internal.py:wandb_internal():92] W&B internal server running at pid: 14495, started at: 2022-03-23 17:07:03.443406
2022-03-23 17:07:03,469 INFO    WriterThread:14495 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_170702-w50ub6ac/run-w50ub6ac.wandb
2022-03-23 17:07:03,471 DEBUG   SenderThread:14495 [sender.py:send():235] send: header
2022-03-23 17:07:03,471 DEBUG   SenderThread:14495 [sender.py:send():235] send: run
2022-03-23 17:07:03,629 DEBUG   HandlerThread:14495 [handler.py:handle_request():141] handle_request: check_version
2022-03-23 17:07:03,630 INFO    SenderThread:14495 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_170702-w50ub6ac/files
2022-03-23 17:07:03,630 INFO    SenderThread:14495 [sender.py:_start_run_threads():812] run started: w50ub6ac with start time 1648055222
2022-03-23 17:07:03,630 DEBUG   SenderThread:14495 [sender.py:send():235] send: summary
2022-03-23 17:07:03,630 INFO    SenderThread:14495 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-03-23 17:07:03,630 DEBUG   SenderThread:14495 [sender.py:send_request():249] send_request: check_version
2022-03-23 17:07:03,706 DEBUG   HandlerThread:14495 [handler.py:handle_request():141] handle_request: run_start
2022-03-23 17:07:04,633 INFO    Thread-7  :14495 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_170702-w50ub6ac/files/wandb-summary.json
2022-03-23 17:07:06,080 DEBUG   HandlerThread:14495 [meta.py:__init__():37] meta init
2022-03-23 17:07:06,080 DEBUG   HandlerThread:14495 [meta.py:__init__():51] meta init done
2022-03-23 17:07:06,080 DEBUG   HandlerThread:14495 [meta.py:probe():211] probe
2022-03-23 17:07:06,081 DEBUG   HandlerThread:14495 [meta.py:_setup_git():201] setup git
2022-03-23 17:07:06,114 DEBUG   HandlerThread:14495 [meta.py:_setup_git():208] setup git done
2022-03-23 17:07:06,114 DEBUG   HandlerThread:14495 [meta.py:_save_pip():55] save pip
2022-03-23 17:07:06,115 DEBUG   HandlerThread:14495 [meta.py:_save_pip():69] save pip done
2022-03-23 17:07:06,115 DEBUG   HandlerThread:14495 [meta.py:probe():249] probe done
2022-03-23 17:07:06,118 DEBUG   SenderThread:14495 [sender.py:send():235] send: files
2022-03-23 17:07:06,118 INFO    SenderThread:14495 [sender.py:_save_file():947] saving file wandb-metadata.json with policy now
2022-03-23 17:07:06,125 DEBUG   HandlerThread:14495 [handler.py:handle_request():141] handle_request: stop_status
2022-03-23 17:07:06,125 DEBUG   SenderThread:14495 [sender.py:send_request():249] send_request: stop_status
2022-03-23 17:07:06,199 DEBUG   SenderThread:14495 [sender.py:send():235] send: telemetry
2022-03-23 17:07:06,451 INFO    Thread-11 :14495 [upload_job.py:push():137] Uploaded file /tmp/tmpvkf_ouoiwandb/2u99pz4k-wandb-metadata.json
2022-03-23 17:07:06,634 INFO    Thread-7  :14495 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_170702-w50ub6ac/files/requirements.txt
2022-03-23 17:07:06,635 INFO    Thread-7  :14495 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_170702-w50ub6ac/files/wandb-metadata.json
2022-03-23 17:07:06,635 INFO    Thread-7  :14495 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_170702-w50ub6ac/files/output.log
2022-03-23 17:07:08,635 INFO    Thread-7  :14495 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_170702-w50ub6ac/files/output.log
2022-03-23 17:07:21,201 DEBUG   HandlerThread:14495 [handler.py:handle_request():141] handle_request: stop_status
2022-03-23 17:07:21,201 DEBUG   SenderThread:14495 [sender.py:send_request():249] send_request: stop_status
2022-03-23 17:07:24,642 INFO    Thread-7  :14495 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_170702-w50ub6ac/files/output.log
2022-03-23 17:07:28,643 INFO    Thread-7  :14495 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_170702-w50ub6ac/files/output.log
2022-03-23 17:07:30,644 INFO    Thread-7  :14495 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_170702-w50ub6ac/files/output.log
2022-03-23 17:07:34,165 DEBUG   SenderThread:14495 [sender.py:send():235] send: stats
2022-03-23 17:07:34,646 INFO    Thread-7  :14495 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_170702-w50ub6ac/files/config.yaml
2022-03-23 17:07:34,646 INFO    Thread-7  :14495 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_170702-w50ub6ac/files/output.log
2022-03-23 17:07:36,413 DEBUG   HandlerThread:14495 [handler.py:handle_request():141] handle_request: stop_status
2022-03-23 17:07:36,414 DEBUG   SenderThread:14495 [sender.py:send_request():249] send_request: stop_status
2022-03-23 17:07:38,648 INFO    Thread-7  :14495 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_170702-w50ub6ac/files/output.log
2022-03-23 17:07:51,584 DEBUG   HandlerThread:14495 [handler.py:handle_request():141] handle_request: stop_status
2022-03-23 17:07:51,584 DEBUG   SenderThread:14495 [sender.py:send_request():249] send_request: stop_status
2022-03-23 17:07:53,609 DEBUG   HandlerThread:14495 [handler.py:handle_request():141] handle_request: partial_history
2022-03-23 17:07:53,610 DEBUG   SenderThread:14495 [sender.py:send():235] send: telemetry
2022-03-23 17:07:54,654 INFO    Thread-7  :14495 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_170702-w50ub6ac/files/output.log
2022-03-23 17:07:55,871 DEBUG   HandlerThread:14495 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-23 17:07:55,872 DEBUG   SenderThread:14495 [sender.py:send():235] send: exit
2022-03-23 17:07:55,872 INFO    SenderThread:14495 [sender.py:send_exit():371] handling exit code: 1
2022-03-23 17:07:55,872 INFO    SenderThread:14495 [sender.py:send_exit():373] handling runtime: 52
2022-03-23 17:07:55,873 INFO    SenderThread:14495 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-03-23 17:07:55,873 INFO    SenderThread:14495 [sender.py:send_exit():379] send defer
2022-03-23 17:07:55,873 DEBUG   SenderThread:14495 [sender.py:send_request():249] send_request: poll_exit
2022-03-23 17:07:55,873 DEBUG   HandlerThread:14495 [handler.py:handle_request():141] handle_request: defer
2022-03-23 17:07:55,874 INFO    HandlerThread:14495 [handler.py:handle_request_defer():164] handle defer: 0
2022-03-23 17:07:55,874 DEBUG   SenderThread:14495 [sender.py:send_request():249] send_request: defer
2022-03-23 17:07:55,874 INFO    SenderThread:14495 [sender.py:send_request_defer():388] handle sender defer: 0
2022-03-23 17:07:55,874 INFO    SenderThread:14495 [sender.py:transition_state():392] send defer: 1
2022-03-23 17:07:55,874 DEBUG   HandlerThread:14495 [handler.py:handle_request():141] handle_request: defer
2022-03-23 17:07:55,874 INFO    HandlerThread:14495 [handler.py:handle_request_defer():164] handle defer: 1
2022-03-23 17:07:55,918 DEBUG   SenderThread:14495 [sender.py:send_request():249] send_request: defer
2022-03-23 17:07:55,918 INFO    SenderThread:14495 [sender.py:send_request_defer():388] handle sender defer: 1
2022-03-23 17:07:55,918 INFO    SenderThread:14495 [sender.py:transition_state():392] send defer: 2
2022-03-23 17:07:55,918 DEBUG   SenderThread:14495 [sender.py:send():235] send: stats
2022-03-23 17:07:55,918 DEBUG   HandlerThread:14495 [handler.py:handle_request():141] handle_request: defer
2022-03-23 17:07:55,918 INFO    HandlerThread:14495 [handler.py:handle_request_defer():164] handle defer: 2
2022-03-23 17:07:55,919 DEBUG   SenderThread:14495 [sender.py:send_request():249] send_request: defer
2022-03-23 17:07:55,919 INFO    SenderThread:14495 [sender.py:send_request_defer():388] handle sender defer: 2
2022-03-23 17:07:55,919 INFO    SenderThread:14495 [sender.py:transition_state():392] send defer: 3
2022-03-23 17:07:55,919 DEBUG   HandlerThread:14495 [handler.py:handle_request():141] handle_request: defer
2022-03-23 17:07:55,919 INFO    HandlerThread:14495 [handler.py:handle_request_defer():164] handle defer: 3
2022-03-23 17:07:55,919 DEBUG   SenderThread:14495 [sender.py:send_request():249] send_request: defer
2022-03-23 17:07:55,919 INFO    SenderThread:14495 [sender.py:send_request_defer():388] handle sender defer: 3
2022-03-23 17:07:55,919 INFO    SenderThread:14495 [sender.py:transition_state():392] send defer: 4
2022-03-23 17:07:55,919 DEBUG   HandlerThread:14495 [handler.py:handle_request():141] handle_request: defer
2022-03-23 17:07:55,920 INFO    HandlerThread:14495 [handler.py:handle_request_defer():164] handle defer: 4
2022-03-23 17:07:55,920 DEBUG   SenderThread:14495 [sender.py:send():235] send: summary
2022-03-23 17:07:55,920 INFO    SenderThread:14495 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-03-23 17:07:55,920 DEBUG   SenderThread:14495 [sender.py:send_request():249] send_request: defer
2022-03-23 17:07:55,920 INFO    SenderThread:14495 [sender.py:send_request_defer():388] handle sender defer: 4
2022-03-23 17:07:55,920 INFO    SenderThread:14495 [sender.py:transition_state():392] send defer: 5
2022-03-23 17:07:55,921 DEBUG   HandlerThread:14495 [handler.py:handle_request():141] handle_request: defer
2022-03-23 17:07:55,921 INFO    HandlerThread:14495 [handler.py:handle_request_defer():164] handle defer: 5
2022-03-23 17:07:55,921 DEBUG   SenderThread:14495 [sender.py:send_request():249] send_request: defer
2022-03-23 17:07:55,921 INFO    SenderThread:14495 [sender.py:send_request_defer():388] handle sender defer: 5
2022-03-23 17:07:55,975 DEBUG   HandlerThread:14495 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-23 17:07:55,996 INFO    SenderThread:14495 [sender.py:transition_state():392] send defer: 6
2022-03-23 17:07:55,996 DEBUG   SenderThread:14495 [sender.py:send_request():249] send_request: poll_exit
2022-03-23 17:07:55,996 DEBUG   HandlerThread:14495 [handler.py:handle_request():141] handle_request: defer
2022-03-23 17:07:55,996 INFO    HandlerThread:14495 [handler.py:handle_request_defer():164] handle defer: 6
2022-03-23 17:07:55,997 DEBUG   SenderThread:14495 [sender.py:send_request():249] send_request: defer
2022-03-23 17:07:55,997 INFO    SenderThread:14495 [sender.py:send_request_defer():388] handle sender defer: 6
2022-03-23 17:07:55,997 INFO    SenderThread:14495 [dir_watcher.py:finish():283] shutting down directory watcher
2022-03-23 17:07:56,097 DEBUG   HandlerThread:14495 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-23 17:07:56,655 INFO    Thread-7  :14495 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_170702-w50ub6ac/files/output.log
2022-03-23 17:07:56,656 INFO    SenderThread:14495 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_170702-w50ub6ac/files/config.yaml
2022-03-23 17:07:56,656 INFO    SenderThread:14495 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_170702-w50ub6ac/files/wandb-summary.json
2022-03-23 17:07:56,656 INFO    SenderThread:14495 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_170702-w50ub6ac/files
2022-03-23 17:07:56,656 INFO    SenderThread:14495 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_170702-w50ub6ac/files/requirements.txt requirements.txt
2022-03-23 17:07:56,656 INFO    SenderThread:14495 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_170702-w50ub6ac/files/output.log output.log
2022-03-23 17:07:56,656 INFO    SenderThread:14495 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_170702-w50ub6ac/files/wandb-metadata.json wandb-metadata.json
2022-03-23 17:07:56,656 INFO    SenderThread:14495 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_170702-w50ub6ac/files/config.yaml config.yaml
2022-03-23 17:07:56,657 INFO    SenderThread:14495 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_170702-w50ub6ac/files/wandb-summary.json wandb-summary.json
2022-03-23 17:07:56,657 INFO    SenderThread:14495 [sender.py:transition_state():392] send defer: 7
2022-03-23 17:07:56,657 DEBUG   SenderThread:14495 [sender.py:send_request():249] send_request: poll_exit
2022-03-23 17:07:56,658 DEBUG   HandlerThread:14495 [handler.py:handle_request():141] handle_request: defer
2022-03-23 17:07:56,658 INFO    HandlerThread:14495 [handler.py:handle_request_defer():164] handle defer: 7
2022-03-23 17:07:56,658 DEBUG   SenderThread:14495 [sender.py:send_request():249] send_request: defer
2022-03-23 17:07:56,658 INFO    SenderThread:14495 [sender.py:send_request_defer():388] handle sender defer: 7
2022-03-23 17:07:56,658 INFO    SenderThread:14495 [file_pusher.py:finish():145] shutting down file pusher
2022-03-23 17:07:56,759 DEBUG   HandlerThread:14495 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-23 17:07:56,759 DEBUG   SenderThread:14495 [sender.py:send_request():249] send_request: poll_exit
2022-03-23 17:07:56,861 DEBUG   HandlerThread:14495 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-23 17:07:56,861 DEBUG   SenderThread:14495 [sender.py:send_request():249] send_request: poll_exit
2022-03-23 17:07:56,916 INFO    Thread-13 :14495 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_170702-w50ub6ac/files/output.log
2022-03-23 17:07:56,924 INFO    Thread-14 :14495 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_170702-w50ub6ac/files/config.yaml
2022-03-23 17:07:56,944 INFO    Thread-15 :14495 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_170702-w50ub6ac/files/wandb-summary.json
2022-03-23 17:07:56,948 INFO    Thread-12 :14495 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_170702-w50ub6ac/files/requirements.txt
2022-03-23 17:07:56,963 DEBUG   HandlerThread:14495 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-23 17:07:56,963 DEBUG   SenderThread:14495 [sender.py:send_request():249] send_request: poll_exit
2022-03-23 17:07:57,064 DEBUG   HandlerThread:14495 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-23 17:07:57,065 DEBUG   SenderThread:14495 [sender.py:send_request():249] send_request: poll_exit
2022-03-23 17:07:57,148 INFO    Thread-6  :14495 [sender.py:transition_state():392] send defer: 8
2022-03-23 17:07:57,149 DEBUG   HandlerThread:14495 [handler.py:handle_request():141] handle_request: defer
2022-03-23 17:07:57,149 INFO    HandlerThread:14495 [handler.py:handle_request_defer():164] handle defer: 8
2022-03-23 17:07:57,149 DEBUG   SenderThread:14495 [sender.py:send_request():249] send_request: defer
2022-03-23 17:07:57,149 INFO    SenderThread:14495 [sender.py:send_request_defer():388] handle sender defer: 8
2022-03-23 17:07:57,166 DEBUG   HandlerThread:14495 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-23 17:07:57,212 INFO    SenderThread:14495 [sender.py:transition_state():392] send defer: 9
2022-03-23 17:07:57,212 DEBUG   SenderThread:14495 [sender.py:send_request():249] send_request: poll_exit
2022-03-23 17:07:57,212 DEBUG   HandlerThread:14495 [handler.py:handle_request():141] handle_request: defer
2022-03-23 17:07:57,212 INFO    HandlerThread:14495 [handler.py:handle_request_defer():164] handle defer: 9
2022-03-23 17:07:57,213 DEBUG   SenderThread:14495 [sender.py:send_request():249] send_request: defer
2022-03-23 17:07:57,213 INFO    SenderThread:14495 [sender.py:send_request_defer():388] handle sender defer: 9
2022-03-23 17:07:57,213 INFO    SenderThread:14495 [sender.py:transition_state():392] send defer: 10
2022-03-23 17:07:57,213 DEBUG   SenderThread:14495 [sender.py:send():235] send: final
2022-03-23 17:07:57,213 DEBUG   SenderThread:14495 [sender.py:send():235] send: footer
2022-03-23 17:07:57,213 DEBUG   HandlerThread:14495 [handler.py:handle_request():141] handle_request: defer
2022-03-23 17:07:57,214 INFO    HandlerThread:14495 [handler.py:handle_request_defer():164] handle defer: 10
2022-03-23 17:07:57,214 DEBUG   SenderThread:14495 [sender.py:send_request():249] send_request: defer
2022-03-23 17:07:57,214 INFO    SenderThread:14495 [sender.py:send_request_defer():388] handle sender defer: 10
2022-03-23 17:07:57,313 DEBUG   HandlerThread:14495 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-23 17:07:57,314 DEBUG   SenderThread:14495 [sender.py:send_request():249] send_request: poll_exit
2022-03-23 17:07:57,314 INFO    SenderThread:14495 [file_pusher.py:join():150] waiting for file pusher
2022-03-23 17:07:57,507 DEBUG   HandlerThread:14495 [handler.py:handle_request():141] handle_request: sampled_history
2022-03-23 17:07:57,508 DEBUG   HandlerThread:14495 [handler.py:handle_request():141] handle_request: get_summary
2022-03-23 17:07:57,509 DEBUG   HandlerThread:14495 [handler.py:handle_request():141] handle_request: shutdown
2022-03-23 17:07:57,509 INFO    HandlerThread:14495 [handler.py:finish():778] shutting down handler
2022-03-23 17:07:58,214 INFO    WriterThread:14495 [datastore.py:close():281] close: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_170702-w50ub6ac/run-w50ub6ac.wandb
2022-03-23 17:07:58,406 INFO    SenderThread:14495 [sender.py:finish():1078] shutting down sender
2022-03-23 17:07:58,406 INFO    SenderThread:14495 [file_pusher.py:finish():145] shutting down file pusher
2022-03-23 17:07:58,406 INFO    SenderThread:14495 [file_pusher.py:join():150] waiting for file pusher
2022-03-23 17:07:58,409 INFO    MainThread:14495 [internal.py:handle_exit():82] Internal process exited