2022-05-05 17:00:26,145 INFO    MainThread:70464 [internal.py:wandb_internal():89] W&B internal server running at pid: 70464, started at: 2022-05-05 17:00:26.145556
2022-05-05 17:00:26,148 DEBUG   HandlerThread:70464 [handler.py:handle_request():131] handle_request: check_version
2022-05-05 17:00:26,148 INFO    WriterThread:70464 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_170025-58ade1nh/run-58ade1nh.wandb
2022-05-05 17:00:26,149 DEBUG   SenderThread:70464 [sender.py:send():235] send: header
2022-05-05 17:00:26,150 DEBUG   SenderThread:70464 [sender.py:send_request():249] send_request: check_version
2022-05-05 17:00:26,216 DEBUG   SenderThread:70464 [sender.py:send():235] send: run
2022-05-05 17:00:26,335 INFO    SenderThread:70464 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_170025-58ade1nh/files
2022-05-05 17:00:26,335 INFO    SenderThread:70464 [sender.py:_start_run_threads():809] run started: 58ade1nh with start time 1651770025
2022-05-05 17:00:26,335 DEBUG   SenderThread:70464 [sender.py:send():235] send: summary
2022-05-05 17:00:26,335 INFO    SenderThread:70464 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-05-05 17:00:26,336 DEBUG   HandlerThread:70464 [handler.py:handle_request():131] handle_request: run_start
2022-05-05 17:00:27,340 INFO    Thread-8  :70464 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_170025-58ade1nh/files/wandb-summary.json
2022-05-05 17:00:28,697 DEBUG   HandlerThread:70464 [meta.py:__init__():36] meta init
2022-05-05 17:00:28,697 DEBUG   HandlerThread:70464 [meta.py:__init__():50] meta init done
2022-05-05 17:00:28,697 DEBUG   HandlerThread:70464 [meta.py:probe():210] probe
2022-05-05 17:00:28,704 DEBUG   HandlerThread:70464 [meta.py:_setup_git():200] setup git
2022-05-05 17:00:28,734 DEBUG   HandlerThread:70464 [meta.py:_setup_git():207] setup git done
2022-05-05 17:00:28,734 DEBUG   HandlerThread:70464 [meta.py:_save_pip():54] save pip
2022-05-05 17:00:28,735 DEBUG   HandlerThread:70464 [meta.py:_save_pip():68] save pip done
2022-05-05 17:00:28,735 DEBUG   HandlerThread:70464 [meta.py:probe():248] probe done
2022-05-05 17:00:28,738 DEBUG   SenderThread:70464 [sender.py:send():235] send: files
2022-05-05 17:00:28,739 INFO    SenderThread:70464 [sender.py:_save_file():944] saving file wandb-metadata.json with policy now
2022-05-05 17:00:28,744 DEBUG   HandlerThread:70464 [handler.py:handle_request():131] handle_request: stop_status
2022-05-05 17:00:28,745 DEBUG   SenderThread:70464 [sender.py:send_request():249] send_request: stop_status
2022-05-05 17:00:28,776 DEBUG   SenderThread:70464 [sender.py:send():235] send: config
2022-05-05 17:00:28,778 DEBUG   SenderThread:70464 [sender.py:send():235] send: metric
2022-05-05 17:00:28,778 DEBUG   SenderThread:70464 [sender.py:send():235] send: metric
2022-05-05 17:00:28,778 WARNING SenderThread:70464 [sender.py:send_metric():902] Seen metric with glob (shouldnt happen)
2022-05-05 17:00:29,125 INFO    Thread-11 :70464 [upload_job.py:push():137] Uploaded file /tmp/tmp9dpang11wandb/2m49egg5-wandb-metadata.json
2022-05-05 17:00:29,346 INFO    Thread-8  :70464 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_170025-58ade1nh/files/output.log
2022-05-05 17:00:29,346 INFO    Thread-8  :70464 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_170025-58ade1nh/files/wandb-metadata.json
2022-05-05 17:00:29,347 INFO    Thread-8  :70464 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_170025-58ade1nh/files/requirements.txt
2022-05-05 17:00:31,347 INFO    Thread-8  :70464 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_170025-58ade1nh/files/output.log
2022-05-05 17:00:32,881 DEBUG   SenderThread:70464 [sender.py:send():235] send: metric
2022-05-05 17:00:32,881 DEBUG   SenderThread:70464 [sender.py:send():235] send: metric
2022-05-05 17:00:32,881 DEBUG   SenderThread:70464 [sender.py:send():235] send: metric
2022-05-05 17:00:32,882 DEBUG   SenderThread:70464 [sender.py:send():235] send: metric
2022-05-05 17:00:32,882 DEBUG   SenderThread:70464 [sender.py:send():235] send: metric
2022-05-05 17:00:32,882 DEBUG   SenderThread:70464 [sender.py:send():235] send: metric
2022-05-05 17:00:32,882 DEBUG   SenderThread:70464 [sender.py:send():235] send: history
2022-05-05 17:00:32,882 DEBUG   SenderThread:70464 [sender.py:send():235] send: summary
2022-05-05 17:00:32,882 INFO    SenderThread:70464 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-05-05 17:00:33,348 INFO    Thread-8  :70464 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_170025-58ade1nh/files/output.log
2022-05-05 17:00:33,348 INFO    Thread-8  :70464 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_170025-58ade1nh/files/wandb-summary.json
2022-05-05 17:00:35,349 INFO    Thread-8  :70464 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_170025-58ade1nh/files/output.log
2022-05-05 17:00:43,783 DEBUG   HandlerThread:70464 [handler.py:handle_request():131] handle_request: stop_status
2022-05-05 17:00:43,836 DEBUG   SenderThread:70464 [sender.py:send_request():249] send_request: stop_status
2022-05-05 17:00:47,354 INFO    Thread-8  :70464 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_170025-58ade1nh/files/output.log
2022-05-05 17:00:58,871 DEBUG   HandlerThread:70464 [handler.py:handle_request():131] handle_request: stop_status
2022-05-05 17:00:59,359 INFO    Thread-8  :70464 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_170025-58ade1nh/files/output.log
2022-05-05 17:00:59,416 DEBUG   SenderThread:70464 [sender.py:send():235] send: stats
2022-05-05 17:00:59,416 DEBUG   SenderThread:70464 [sender.py:send_request():249] send_request: stop_status
2022-05-05 17:01:00,359 INFO    Thread-8  :70464 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_170025-58ade1nh/files/config.yaml
2022-05-05 17:01:14,463 DEBUG   HandlerThread:70464 [handler.py:handle_request():131] handle_request: stop_status
2022-05-05 17:01:14,463 DEBUG   SenderThread:70464 [sender.py:send_request():249] send_request: stop_status
2022-05-05 17:01:25,368 INFO    Thread-8  :70464 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_170025-58ade1nh/files/output.log
2022-05-05 17:01:27,369 INFO    Thread-8  :70464 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_170025-58ade1nh/files/output.log
2022-05-05 17:01:27,443 DEBUG   SenderThread:70464 [sender.py:send():235] send: stats
2022-05-05 17:01:29,370 INFO    Thread-8  :70464 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_170025-58ade1nh/files/output.log
2022-05-05 17:01:29,511 DEBUG   HandlerThread:70464 [handler.py:handle_request():131] handle_request: stop_status
2022-05-05 17:01:29,511 DEBUG   SenderThread:70464 [sender.py:send_request():249] send_request: stop_status
2022-05-05 17:01:39,375 INFO    Thread-8  :70464 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_170025-58ade1nh/files/output.log
2022-05-05 17:01:41,376 INFO    Thread-8  :70464 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_170025-58ade1nh/files/output.log
2022-05-05 17:01:44,544 DEBUG   HandlerThread:70464 [handler.py:handle_request():131] handle_request: stop_status
2022-05-05 17:01:44,544 DEBUG   SenderThread:70464 [sender.py:send_request():249] send_request: stop_status
2022-05-05 17:01:53,381 INFO    Thread-8  :70464 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_170025-58ade1nh/files/output.log
2022-05-05 17:01:57,797 DEBUG   SenderThread:70464 [sender.py:send():235] send: stats
2022-05-05 17:01:59,579 DEBUG   HandlerThread:70464 [handler.py:handle_request():131] handle_request: stop_status
2022-05-05 17:01:59,580 DEBUG   SenderThread:70464 [sender.py:send_request():249] send_request: stop_status
2022-05-05 17:02:14,621 DEBUG   HandlerThread:70464 [handler.py:handle_request():131] handle_request: stop_status
2022-05-05 17:02:14,621 DEBUG   SenderThread:70464 [sender.py:send_request():249] send_request: stop_status
2022-05-05 17:02:20,391 INFO    Thread-8  :70464 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_170025-58ade1nh/files/output.log
2022-05-05 17:02:24,393 INFO    Thread-8  :70464 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_170025-58ade1nh/files/output.log
2022-05-05 17:02:24,630 DEBUG   HandlerThread:70464 [handler.py:handle_request():131] handle_request: poll_exit
2022-05-05 17:02:24,631 DEBUG   SenderThread:70464 [sender.py:send():235] send: telemetry
2022-05-05 17:02:24,631 DEBUG   SenderThread:70464 [sender.py:send():235] send: exit
2022-05-05 17:02:24,631 INFO    SenderThread:70464 [sender.py:send_exit():371] handling exit code: 1
2022-05-05 17:02:24,631 INFO    SenderThread:70464 [sender.py:send_exit():373] handling runtime: 118
2022-05-05 17:02:24,632 INFO    SenderThread:70464 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-05-05 17:02:24,632 INFO    SenderThread:70464 [sender.py:send_exit():379] send defer
2022-05-05 17:02:24,632 DEBUG   SenderThread:70464 [sender.py:send_request():249] send_request: poll_exit
2022-05-05 17:02:24,632 DEBUG   HandlerThread:70464 [handler.py:handle_request():131] handle_request: defer
2022-05-05 17:02:24,632 INFO    HandlerThread:70464 [handler.py:handle_request_defer():154] handle defer: 0
2022-05-05 17:02:24,633 DEBUG   SenderThread:70464 [sender.py:send_request():249] send_request: defer
2022-05-05 17:02:24,633 INFO    SenderThread:70464 [sender.py:send_request_defer():388] handle sender defer: 0
2022-05-05 17:02:24,633 INFO    SenderThread:70464 [sender.py:transition_state():392] send defer: 1
2022-05-05 17:02:24,633 DEBUG   HandlerThread:70464 [handler.py:handle_request():131] handle_request: defer
2022-05-05 17:02:24,633 INFO    HandlerThread:70464 [handler.py:handle_request_defer():154] handle defer: 1
2022-05-05 17:02:24,718 DEBUG   SenderThread:70464 [sender.py:send_request():249] send_request: defer
2022-05-05 17:02:24,719 INFO    SenderThread:70464 [sender.py:send_request_defer():388] handle sender defer: 1
2022-05-05 17:02:24,719 INFO    SenderThread:70464 [sender.py:transition_state():392] send defer: 2
2022-05-05 17:02:24,719 DEBUG   SenderThread:70464 [sender.py:send():235] send: stats
2022-05-05 17:02:24,720 DEBUG   HandlerThread:70464 [handler.py:handle_request():131] handle_request: defer
2022-05-05 17:02:24,720 INFO    HandlerThread:70464 [handler.py:handle_request_defer():154] handle defer: 2
2022-05-05 17:02:24,720 DEBUG   SenderThread:70464 [sender.py:send_request():249] send_request: defer
2022-05-05 17:02:24,720 INFO    SenderThread:70464 [sender.py:send_request_defer():388] handle sender defer: 2
2022-05-05 17:02:24,720 INFO    SenderThread:70464 [sender.py:transition_state():392] send defer: 3
2022-05-05 17:02:24,720 DEBUG   HandlerThread:70464 [handler.py:handle_request():131] handle_request: defer
2022-05-05 17:02:24,720 INFO    HandlerThread:70464 [handler.py:handle_request_defer():154] handle defer: 3
2022-05-05 17:02:24,720 DEBUG   SenderThread:70464 [sender.py:send():235] send: summary
2022-05-05 17:02:24,721 INFO    SenderThread:70464 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-05-05 17:02:24,721 DEBUG   SenderThread:70464 [sender.py:send_request():249] send_request: defer
2022-05-05 17:02:24,721 INFO    SenderThread:70464 [sender.py:send_request_defer():388] handle sender defer: 3
2022-05-05 17:02:24,721 INFO    SenderThread:70464 [sender.py:transition_state():392] send defer: 4
2022-05-05 17:02:24,721 DEBUG   HandlerThread:70464 [handler.py:handle_request():131] handle_request: defer
2022-05-05 17:02:24,721 INFO    HandlerThread:70464 [handler.py:handle_request_defer():154] handle defer: 4
2022-05-05 17:02:24,722 DEBUG   SenderThread:70464 [sender.py:send_request():249] send_request: defer
2022-05-05 17:02:24,722 INFO    SenderThread:70464 [sender.py:send_request_defer():388] handle sender defer: 4
2022-05-05 17:02:24,734 DEBUG   HandlerThread:70464 [handler.py:handle_request():131] handle_request: poll_exit
2022-05-05 17:02:24,823 INFO    SenderThread:70464 [sender.py:transition_state():392] send defer: 5
2022-05-05 17:02:24,823 DEBUG   SenderThread:70464 [sender.py:send_request():249] send_request: poll_exit
2022-05-05 17:02:24,824 DEBUG   HandlerThread:70464 [handler.py:handle_request():131] handle_request: defer
2022-05-05 17:02:24,824 INFO    HandlerThread:70464 [handler.py:handle_request_defer():154] handle defer: 5
2022-05-05 17:02:24,824 DEBUG   SenderThread:70464 [sender.py:send_request():249] send_request: defer
2022-05-05 17:02:24,824 INFO    SenderThread:70464 [sender.py:send_request_defer():388] handle sender defer: 5
2022-05-05 17:02:24,824 INFO    SenderThread:70464 [dir_watcher.py:finish():283] shutting down directory watcher
2022-05-05 17:02:24,925 DEBUG   HandlerThread:70464 [handler.py:handle_request():131] handle_request: poll_exit
2022-05-05 17:02:25,394 INFO    Thread-8  :70464 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_170025-58ade1nh/files/output.log
2022-05-05 17:02:25,394 INFO    SenderThread:70464 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_170025-58ade1nh/files/config.yaml
2022-05-05 17:02:25,394 INFO    SenderThread:70464 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_170025-58ade1nh/files/wandb-summary.json
2022-05-05 17:02:25,395 INFO    SenderThread:70464 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_170025-58ade1nh/files
2022-05-05 17:02:25,395 INFO    SenderThread:70464 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_170025-58ade1nh/files/wandb-metadata.json wandb-metadata.json
2022-05-05 17:02:25,395 INFO    SenderThread:70464 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_170025-58ade1nh/files/output.log output.log
2022-05-05 17:02:25,395 INFO    SenderThread:70464 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_170025-58ade1nh/files/wandb-summary.json wandb-summary.json
2022-05-05 17:02:25,395 INFO    SenderThread:70464 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_170025-58ade1nh/files/requirements.txt requirements.txt
2022-05-05 17:02:25,395 INFO    SenderThread:70464 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_170025-58ade1nh/files/config.yaml config.yaml
2022-05-05 17:02:25,396 INFO    SenderThread:70464 [sender.py:transition_state():392] send defer: 6
2022-05-05 17:02:25,396 DEBUG   SenderThread:70464 [sender.py:send_request():249] send_request: poll_exit
2022-05-05 17:02:25,401 DEBUG   HandlerThread:70464 [handler.py:handle_request():131] handle_request: defer
2022-05-05 17:02:25,401 INFO    HandlerThread:70464 [handler.py:handle_request_defer():154] handle defer: 6
2022-05-05 17:02:25,402 DEBUG   SenderThread:70464 [sender.py:send_request():249] send_request: defer
2022-05-05 17:02:25,405 INFO    SenderThread:70464 [sender.py:send_request_defer():388] handle sender defer: 6
2022-05-05 17:02:25,408 INFO    SenderThread:70464 [file_pusher.py:finish():177] shutting down file pusher
2022-05-05 17:02:25,503 DEBUG   HandlerThread:70464 [handler.py:handle_request():131] handle_request: poll_exit
2022-05-05 17:02:25,504 DEBUG   SenderThread:70464 [sender.py:send_request():249] send_request: poll_exit
2022-05-05 17:02:25,606 DEBUG   HandlerThread:70464 [handler.py:handle_request():131] handle_request: poll_exit
2022-05-05 17:02:25,606 DEBUG   SenderThread:70464 [sender.py:send_request():249] send_request: poll_exit
2022-05-05 17:02:25,640 INFO    Thread-12 :70464 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_170025-58ade1nh/files/output.log
2022-05-05 17:02:25,660 INFO    Thread-15 :70464 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_170025-58ade1nh/files/config.yaml
2022-05-05 17:02:25,676 INFO    Thread-13 :70464 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_170025-58ade1nh/files/wandb-summary.json
2022-05-05 17:02:25,708 DEBUG   HandlerThread:70464 [handler.py:handle_request():131] handle_request: poll_exit
2022-05-05 17:02:25,708 DEBUG   SenderThread:70464 [sender.py:send_request():249] send_request: poll_exit
2022-05-05 17:02:25,762 INFO    Thread-14 :70464 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_170025-58ade1nh/files/requirements.txt
2022-05-05 17:02:25,810 DEBUG   HandlerThread:70464 [handler.py:handle_request():131] handle_request: poll_exit
2022-05-05 17:02:25,810 DEBUG   SenderThread:70464 [sender.py:send_request():249] send_request: poll_exit
2022-05-05 17:02:25,912 DEBUG   HandlerThread:70464 [handler.py:handle_request():131] handle_request: poll_exit
2022-05-05 17:02:25,913 DEBUG   SenderThread:70464 [sender.py:send_request():249] send_request: poll_exit
2022-05-05 17:02:25,963 INFO    Thread-7  :70464 [sender.py:transition_state():392] send defer: 7
2022-05-05 17:02:25,963 DEBUG   HandlerThread:70464 [handler.py:handle_request():131] handle_request: defer
2022-05-05 17:02:25,963 INFO    HandlerThread:70464 [handler.py:handle_request_defer():154] handle defer: 7
2022-05-05 17:02:25,964 DEBUG   SenderThread:70464 [sender.py:send_request():249] send_request: defer
2022-05-05 17:02:25,964 INFO    SenderThread:70464 [sender.py:send_request_defer():388] handle sender defer: 7
2022-05-05 17:02:26,016 DEBUG   HandlerThread:70464 [handler.py:handle_request():131] handle_request: poll_exit
2022-05-05 17:02:26,121 INFO    SenderThread:70464 [sender.py:transition_state():392] send defer: 8
2022-05-05 17:02:26,122 DEBUG   SenderThread:70464 [sender.py:send_request():249] send_request: poll_exit
2022-05-05 17:02:26,122 DEBUG   HandlerThread:70464 [handler.py:handle_request():131] handle_request: defer
2022-05-05 17:02:26,122 INFO    HandlerThread:70464 [handler.py:handle_request_defer():154] handle defer: 8
2022-05-05 17:02:26,123 DEBUG   SenderThread:70464 [sender.py:send_request():249] send_request: defer
2022-05-05 17:02:26,123 INFO    SenderThread:70464 [sender.py:send_request_defer():388] handle sender defer: 8
2022-05-05 17:02:26,123 INFO    SenderThread:70464 [sender.py:transition_state():392] send defer: 9
2022-05-05 17:02:26,123 DEBUG   SenderThread:70464 [sender.py:send():235] send: final
2022-05-05 17:02:26,123 DEBUG   SenderThread:70464 [sender.py:send():235] send: footer
2022-05-05 17:02:26,123 DEBUG   HandlerThread:70464 [handler.py:handle_request():131] handle_request: defer
2022-05-05 17:02:26,124 INFO    HandlerThread:70464 [handler.py:handle_request_defer():154] handle defer: 9
2022-05-05 17:02:26,124 DEBUG   SenderThread:70464 [sender.py:send_request():249] send_request: defer
2022-05-05 17:02:26,124 INFO    SenderThread:70464 [sender.py:send_request_defer():388] handle sender defer: 9
2022-05-05 17:02:26,224 DEBUG   HandlerThread:70464 [handler.py:handle_request():131] handle_request: poll_exit
2022-05-05 17:02:26,224 DEBUG   SenderThread:70464 [sender.py:send_request():249] send_request: poll_exit
2022-05-05 17:02:26,224 INFO    SenderThread:70464 [file_pusher.py:join():182] waiting for file pusher
2022-05-05 17:02:26,281 DEBUG   HandlerThread:70464 [handler.py:handle_request():131] handle_request: get_summary
2022-05-05 17:02:26,282 DEBUG   HandlerThread:70464 [handler.py:handle_request():131] handle_request: sampled_history
2022-05-05 17:02:26,283 DEBUG   HandlerThread:70464 [handler.py:handle_request():131] handle_request: shutdown
2022-05-05 17:02:26,283 INFO    HandlerThread:70464 [handler.py:finish():739] shutting down handler
2022-05-05 17:02:27,124 INFO    WriterThread:70464 [datastore.py:close():281] close: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_170025-58ade1nh/run-58ade1nh.wandb
2022-05-05 17:02:27,280 INFO    SenderThread:70464 [sender.py:finish():1075] shutting down sender
2022-05-05 17:02:27,280 INFO    SenderThread:70464 [file_pusher.py:finish():177] shutting down file pusher
2022-05-05 17:02:27,281 INFO    SenderThread:70464 [file_pusher.py:join():182] waiting for file pusher
2022-05-05 17:02:27,283 INFO    MainThread:70464 [internal.py:handle_exit():79] Internal process exited