sanchit-gandhi's picture
Training in progress, step 500
21aa7a2
2022-05-05 16:49:03,416 INFO MainThread:67587 [internal.py:wandb_internal():89] W&B internal server running at pid: 67587, started at: 2022-05-05 16:49:03.416613
2022-05-05 16:49:03,418 DEBUG HandlerThread:67587 [handler.py:handle_request():131] handle_request: check_version
2022-05-05 16:49:03,419 INFO WriterThread:67587 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en/wandb/run-20220505_164902-lr7oonrp/run-lr7oonrp.wandb
2022-05-05 16:49:03,420 DEBUG SenderThread:67587 [sender.py:send():235] send: header
2022-05-05 16:49:03,420 DEBUG SenderThread:67587 [sender.py:send_request():249] send_request: check_version
2022-05-05 16:49:03,488 DEBUG SenderThread:67587 [sender.py:send():235] send: run
2022-05-05 16:49:03,569 INFO SenderThread:67587 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en/wandb/run-20220505_164902-lr7oonrp/files
2022-05-05 16:49:03,569 INFO SenderThread:67587 [sender.py:_start_run_threads():809] run started: lr7oonrp with start time 1651769342
2022-05-05 16:49:03,569 DEBUG SenderThread:67587 [sender.py:send():235] send: summary
2022-05-05 16:49:03,569 INFO SenderThread:67587 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-05-05 16:49:03,570 DEBUG HandlerThread:67587 [handler.py:handle_request():131] handle_request: run_start
2022-05-05 16:49:04,571 INFO Thread-8 :67587 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en/wandb/run-20220505_164902-lr7oonrp/files/wandb-summary.json
2022-05-05 16:49:06,020 DEBUG HandlerThread:67587 [meta.py:__init__():36] meta init
2022-05-05 16:49:06,021 DEBUG HandlerThread:67587 [meta.py:__init__():50] meta init done
2022-05-05 16:49:06,021 DEBUG HandlerThread:67587 [meta.py:probe():210] probe
2022-05-05 16:49:06,027 DEBUG HandlerThread:67587 [meta.py:_setup_git():200] setup git
2022-05-05 16:49:06,056 DEBUG HandlerThread:67587 [meta.py:_setup_git():207] setup git done
2022-05-05 16:49:06,056 DEBUG HandlerThread:67587 [meta.py:_save_pip():54] save pip
2022-05-05 16:49:06,057 DEBUG HandlerThread:67587 [meta.py:_save_pip():68] save pip done
2022-05-05 16:49:06,057 DEBUG HandlerThread:67587 [meta.py:probe():248] probe done
2022-05-05 16:49:06,060 DEBUG SenderThread:67587 [sender.py:send():235] send: files
2022-05-05 16:49:06,060 INFO SenderThread:67587 [sender.py:_save_file():944] saving file wandb-metadata.json with policy now
2022-05-05 16:49:06,067 DEBUG HandlerThread:67587 [handler.py:handle_request():131] handle_request: stop_status
2022-05-05 16:49:06,067 DEBUG SenderThread:67587 [sender.py:send_request():249] send_request: stop_status
2022-05-05 16:49:06,159 DEBUG SenderThread:67587 [sender.py:send():235] send: config
2022-05-05 16:49:06,159 DEBUG SenderThread:67587 [sender.py:send():235] send: metric
2022-05-05 16:49:06,160 DEBUG SenderThread:67587 [sender.py:send():235] send: metric
2022-05-05 16:49:06,160 WARNING SenderThread:67587 [sender.py:send_metric():902] Seen metric with glob (shouldnt happen)
2022-05-05 16:49:06,426 INFO Thread-11 :67587 [upload_job.py:push():137] Uploaded file /tmp/tmpfcxm9u9mwandb/2tf3wo9d-wandb-metadata.json
2022-05-05 16:49:06,573 INFO Thread-8 :67587 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en/wandb/run-20220505_164902-lr7oonrp/files/output.log
2022-05-05 16:49:06,573 INFO Thread-8 :67587 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en/wandb/run-20220505_164902-lr7oonrp/files/requirements.txt
2022-05-05 16:49:06,574 INFO Thread-8 :67587 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en/wandb/run-20220505_164902-lr7oonrp/files/wandb-metadata.json
2022-05-05 16:49:08,574 INFO Thread-8 :67587 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en/wandb/run-20220505_164902-lr7oonrp/files/output.log
2022-05-05 16:49:10,575 INFO Thread-8 :67587 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en/wandb/run-20220505_164902-lr7oonrp/files/output.log
2022-05-05 16:49:10,670 DEBUG SenderThread:67587 [sender.py:send():235] send: metric
2022-05-05 16:49:10,671 DEBUG SenderThread:67587 [sender.py:send():235] send: metric
2022-05-05 16:49:10,671 DEBUG SenderThread:67587 [sender.py:send():235] send: metric
2022-05-05 16:49:10,671 DEBUG SenderThread:67587 [sender.py:send():235] send: metric
2022-05-05 16:49:10,671 DEBUG SenderThread:67587 [sender.py:send():235] send: metric
2022-05-05 16:49:10,671 DEBUG SenderThread:67587 [sender.py:send():235] send: metric
2022-05-05 16:49:10,671 DEBUG SenderThread:67587 [sender.py:send():235] send: history
2022-05-05 16:49:10,671 DEBUG SenderThread:67587 [sender.py:send():235] send: summary
2022-05-05 16:49:10,672 INFO SenderThread:67587 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-05-05 16:49:11,575 INFO Thread-8 :67587 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en/wandb/run-20220505_164902-lr7oonrp/files/wandb-summary.json
2022-05-05 16:49:12,575 INFO Thread-8 :67587 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en/wandb/run-20220505_164902-lr7oonrp/files/output.log
2022-05-05 16:49:21,171 DEBUG HandlerThread:67587 [handler.py:handle_request():131] handle_request: stop_status
2022-05-05 16:49:21,172 DEBUG SenderThread:67587 [sender.py:send_request():249] send_request: stop_status
2022-05-05 16:49:26,581 INFO Thread-8 :67587 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en/wandb/run-20220505_164902-lr7oonrp/files/output.log
2022-05-05 16:49:34,433 DEBUG SenderThread:67587 [sender.py:send():235] send: stats
2022-05-05 16:49:34,583 INFO Thread-8 :67587 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en/wandb/run-20220505_164902-lr7oonrp/files/config.yaml
2022-05-05 16:49:36,219 DEBUG HandlerThread:67587 [handler.py:handle_request():131] handle_request: stop_status
2022-05-05 16:49:36,220 DEBUG SenderThread:67587 [sender.py:send_request():249] send_request: stop_status
2022-05-05 16:49:42,586 INFO Thread-8 :67587 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en/wandb/run-20220505_164902-lr7oonrp/files/output.log
2022-05-05 16:49:51,252 DEBUG HandlerThread:67587 [handler.py:handle_request():131] handle_request: stop_status
2022-05-05 16:49:51,253 DEBUG SenderThread:67587 [sender.py:send_request():249] send_request: stop_status
2022-05-05 16:50:04,812 DEBUG SenderThread:67587 [sender.py:send():235] send: stats
2022-05-05 16:50:06,304 DEBUG HandlerThread:67587 [handler.py:handle_request():131] handle_request: stop_status
2022-05-05 16:50:06,305 DEBUG SenderThread:67587 [sender.py:send_request():249] send_request: stop_status
2022-05-05 16:50:10,596 INFO Thread-8 :67587 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en/wandb/run-20220505_164902-lr7oonrp/files/output.log
2022-05-05 16:50:14,597 INFO Thread-8 :67587 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en/wandb/run-20220505_164902-lr7oonrp/files/output.log
2022-05-05 16:50:21,345 DEBUG HandlerThread:67587 [handler.py:handle_request():131] handle_request: stop_status
2022-05-05 16:50:21,346 DEBUG SenderThread:67587 [sender.py:send_request():249] send_request: stop_status
2022-05-05 16:50:24,601 INFO Thread-8 :67587 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en/wandb/run-20220505_164902-lr7oonrp/files/output.log
2022-05-05 16:50:26,601 INFO Thread-8 :67587 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en/wandb/run-20220505_164902-lr7oonrp/files/output.log
2022-05-05 16:50:35,204 DEBUG SenderThread:67587 [sender.py:send():235] send: stats
2022-05-05 16:50:36,391 DEBUG HandlerThread:67587 [handler.py:handle_request():131] handle_request: stop_status
2022-05-05 16:50:36,391 DEBUG SenderThread:67587 [sender.py:send_request():249] send_request: stop_status
2022-05-05 16:50:40,607 INFO Thread-8 :67587 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en/wandb/run-20220505_164902-lr7oonrp/files/output.log
2022-05-05 16:50:51,424 DEBUG HandlerThread:67587 [handler.py:handle_request():131] handle_request: stop_status
2022-05-05 16:50:51,425 DEBUG SenderThread:67587 [sender.py:send_request():249] send_request: stop_status
2022-05-05 16:51:05,586 DEBUG SenderThread:67587 [sender.py:send():235] send: stats
2022-05-05 16:51:06,466 DEBUG HandlerThread:67587 [handler.py:handle_request():131] handle_request: stop_status
2022-05-05 16:51:06,467 DEBUG SenderThread:67587 [sender.py:send_request():249] send_request: stop_status
2022-05-05 16:51:12,618 INFO Thread-8 :67587 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en/wandb/run-20220505_164902-lr7oonrp/files/output.log
2022-05-05 16:51:16,619 INFO Thread-8 :67587 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en/wandb/run-20220505_164902-lr7oonrp/files/output.log
2022-05-05 16:51:18,214 DEBUG HandlerThread:67587 [handler.py:handle_request():131] handle_request: poll_exit
2022-05-05 16:51:18,214 DEBUG SenderThread:67587 [sender.py:send():235] send: telemetry
2022-05-05 16:51:18,214 DEBUG SenderThread:67587 [sender.py:send():235] send: exit
2022-05-05 16:51:18,215 INFO SenderThread:67587 [sender.py:send_exit():371] handling exit code: 1
2022-05-05 16:51:18,215 INFO SenderThread:67587 [sender.py:send_exit():373] handling runtime: 134
2022-05-05 16:51:18,215 INFO SenderThread:67587 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-05-05 16:51:18,215 INFO SenderThread:67587 [sender.py:send_exit():379] send defer
2022-05-05 16:51:18,216 DEBUG SenderThread:67587 [sender.py:send_request():249] send_request: poll_exit
2022-05-05 16:51:18,216 DEBUG HandlerThread:67587 [handler.py:handle_request():131] handle_request: defer
2022-05-05 16:51:18,216 INFO HandlerThread:67587 [handler.py:handle_request_defer():154] handle defer: 0
2022-05-05 16:51:18,217 DEBUG SenderThread:67587 [sender.py:send_request():249] send_request: defer
2022-05-05 16:51:18,217 INFO SenderThread:67587 [sender.py:send_request_defer():388] handle sender defer: 0
2022-05-05 16:51:18,217 INFO SenderThread:67587 [sender.py:transition_state():392] send defer: 1
2022-05-05 16:51:18,217 DEBUG HandlerThread:67587 [handler.py:handle_request():131] handle_request: defer
2022-05-05 16:51:18,217 INFO HandlerThread:67587 [handler.py:handle_request_defer():154] handle defer: 1
2022-05-05 16:51:18,266 DEBUG SenderThread:67587 [sender.py:send_request():249] send_request: defer
2022-05-05 16:51:18,266 INFO SenderThread:67587 [sender.py:send_request_defer():388] handle sender defer: 1
2022-05-05 16:51:18,266 INFO SenderThread:67587 [sender.py:transition_state():392] send defer: 2
2022-05-05 16:51:18,266 DEBUG SenderThread:67587 [sender.py:send():235] send: stats
2022-05-05 16:51:18,267 DEBUG HandlerThread:67587 [handler.py:handle_request():131] handle_request: defer
2022-05-05 16:51:18,267 INFO HandlerThread:67587 [handler.py:handle_request_defer():154] handle defer: 2
2022-05-05 16:51:18,267 DEBUG SenderThread:67587 [sender.py:send_request():249] send_request: defer
2022-05-05 16:51:18,268 INFO SenderThread:67587 [sender.py:send_request_defer():388] handle sender defer: 2
2022-05-05 16:51:18,268 INFO SenderThread:67587 [sender.py:transition_state():392] send defer: 3
2022-05-05 16:51:18,268 DEBUG HandlerThread:67587 [handler.py:handle_request():131] handle_request: defer
2022-05-05 16:51:18,268 INFO HandlerThread:67587 [handler.py:handle_request_defer():154] handle defer: 3
2022-05-05 16:51:18,268 DEBUG SenderThread:67587 [sender.py:send():235] send: summary
2022-05-05 16:51:18,269 INFO SenderThread:67587 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-05-05 16:51:18,269 DEBUG SenderThread:67587 [sender.py:send_request():249] send_request: defer
2022-05-05 16:51:18,270 INFO SenderThread:67587 [sender.py:send_request_defer():388] handle sender defer: 3
2022-05-05 16:51:18,270 INFO SenderThread:67587 [sender.py:transition_state():392] send defer: 4
2022-05-05 16:51:18,270 DEBUG HandlerThread:67587 [handler.py:handle_request():131] handle_request: defer
2022-05-05 16:51:18,270 INFO HandlerThread:67587 [handler.py:handle_request_defer():154] handle defer: 4
2022-05-05 16:51:18,270 DEBUG SenderThread:67587 [sender.py:send_request():249] send_request: defer
2022-05-05 16:51:18,270 INFO SenderThread:67587 [sender.py:send_request_defer():388] handle sender defer: 4
2022-05-05 16:51:18,318 DEBUG HandlerThread:67587 [handler.py:handle_request():131] handle_request: poll_exit
2022-05-05 16:51:18,620 INFO Thread-8 :67587 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en/wandb/run-20220505_164902-lr7oonrp/files/output.log
2022-05-05 16:51:18,620 INFO Thread-8 :67587 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en/wandb/run-20220505_164902-lr7oonrp/files/wandb-summary.json
2022-05-05 16:51:18,806 INFO SenderThread:67587 [sender.py:transition_state():392] send defer: 5
2022-05-05 16:51:18,807 DEBUG SenderThread:67587 [sender.py:send_request():249] send_request: poll_exit
2022-05-05 16:51:18,807 DEBUG HandlerThread:67587 [handler.py:handle_request():131] handle_request: defer
2022-05-05 16:51:18,808 INFO HandlerThread:67587 [handler.py:handle_request_defer():154] handle defer: 5
2022-05-05 16:51:18,808 DEBUG SenderThread:67587 [sender.py:send_request():249] send_request: defer
2022-05-05 16:51:18,808 INFO SenderThread:67587 [sender.py:send_request_defer():388] handle sender defer: 5
2022-05-05 16:51:18,808 INFO SenderThread:67587 [dir_watcher.py:finish():283] shutting down directory watcher
2022-05-05 16:51:18,909 DEBUG HandlerThread:67587 [handler.py:handle_request():131] handle_request: poll_exit
2022-05-05 16:51:19,621 INFO Thread-8 :67587 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en/wandb/run-20220505_164902-lr7oonrp/files/config.yaml
2022-05-05 16:51:19,621 INFO SenderThread:67587 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en/wandb/run-20220505_164902-lr7oonrp/files
2022-05-05 16:51:19,621 INFO SenderThread:67587 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en/wandb/run-20220505_164902-lr7oonrp/files/wandb-metadata.json wandb-metadata.json
2022-05-05 16:51:19,621 INFO SenderThread:67587 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en/wandb/run-20220505_164902-lr7oonrp/files/output.log output.log
2022-05-05 16:51:19,622 INFO SenderThread:67587 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en/wandb/run-20220505_164902-lr7oonrp/files/wandb-summary.json wandb-summary.json
2022-05-05 16:51:19,622 INFO SenderThread:67587 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en/wandb/run-20220505_164902-lr7oonrp/files/requirements.txt requirements.txt
2022-05-05 16:51:19,625 INFO SenderThread:67587 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en/wandb/run-20220505_164902-lr7oonrp/files/config.yaml config.yaml
2022-05-05 16:51:19,625 INFO SenderThread:67587 [sender.py:transition_state():392] send defer: 6
2022-05-05 16:51:19,626 DEBUG SenderThread:67587 [sender.py:send_request():249] send_request: poll_exit
2022-05-05 16:51:19,629 DEBUG HandlerThread:67587 [handler.py:handle_request():131] handle_request: defer
2022-05-05 16:51:19,629 INFO HandlerThread:67587 [handler.py:handle_request_defer():154] handle defer: 6
2022-05-05 16:51:19,629 DEBUG SenderThread:67587 [sender.py:send_request():249] send_request: defer
2022-05-05 16:51:19,629 INFO SenderThread:67587 [sender.py:send_request_defer():388] handle sender defer: 6
2022-05-05 16:51:19,629 INFO SenderThread:67587 [file_pusher.py:finish():177] shutting down file pusher
2022-05-05 16:51:19,727 DEBUG HandlerThread:67587 [handler.py:handle_request():131] handle_request: poll_exit
2022-05-05 16:51:19,728 DEBUG SenderThread:67587 [sender.py:send_request():249] send_request: poll_exit
2022-05-05 16:51:19,830 DEBUG HandlerThread:67587 [handler.py:handle_request():131] handle_request: poll_exit
2022-05-05 16:51:19,830 DEBUG SenderThread:67587 [sender.py:send_request():249] send_request: poll_exit
2022-05-05 16:51:19,891 INFO Thread-14 :67587 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en/wandb/run-20220505_164902-lr7oonrp/files/requirements.txt
2022-05-05 16:51:19,902 INFO Thread-12 :67587 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en/wandb/run-20220505_164902-lr7oonrp/files/output.log
2022-05-05 16:51:19,909 INFO Thread-13 :67587 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en/wandb/run-20220505_164902-lr7oonrp/files/wandb-summary.json
2022-05-05 16:51:19,933 INFO Thread-15 :67587 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en/wandb/run-20220505_164902-lr7oonrp/files/config.yaml
2022-05-05 16:51:19,934 DEBUG HandlerThread:67587 [handler.py:handle_request():131] handle_request: poll_exit
2022-05-05 16:51:19,934 DEBUG SenderThread:67587 [sender.py:send_request():249] send_request: poll_exit
2022-05-05 16:51:20,036 DEBUG HandlerThread:67587 [handler.py:handle_request():131] handle_request: poll_exit
2022-05-05 16:51:20,037 DEBUG SenderThread:67587 [sender.py:send_request():249] send_request: poll_exit
2022-05-05 16:51:20,134 INFO Thread-7 :67587 [sender.py:transition_state():392] send defer: 7
2022-05-05 16:51:20,135 DEBUG HandlerThread:67587 [handler.py:handle_request():131] handle_request: defer
2022-05-05 16:51:20,135 INFO HandlerThread:67587 [handler.py:handle_request_defer():154] handle defer: 7
2022-05-05 16:51:20,135 DEBUG SenderThread:67587 [sender.py:send_request():249] send_request: defer
2022-05-05 16:51:20,135 INFO SenderThread:67587 [sender.py:send_request_defer():388] handle sender defer: 7
2022-05-05 16:51:20,139 DEBUG HandlerThread:67587 [handler.py:handle_request():131] handle_request: poll_exit
2022-05-05 16:51:20,296 INFO SenderThread:67587 [sender.py:transition_state():392] send defer: 8
2022-05-05 16:51:20,296 DEBUG SenderThread:67587 [sender.py:send_request():249] send_request: poll_exit
2022-05-05 16:51:20,297 DEBUG HandlerThread:67587 [handler.py:handle_request():131] handle_request: defer
2022-05-05 16:51:20,297 INFO HandlerThread:67587 [handler.py:handle_request_defer():154] handle defer: 8
2022-05-05 16:51:20,297 DEBUG SenderThread:67587 [sender.py:send_request():249] send_request: defer
2022-05-05 16:51:20,297 INFO SenderThread:67587 [sender.py:send_request_defer():388] handle sender defer: 8
2022-05-05 16:51:20,297 INFO SenderThread:67587 [sender.py:transition_state():392] send defer: 9
2022-05-05 16:51:20,298 DEBUG HandlerThread:67587 [handler.py:handle_request():131] handle_request: defer
2022-05-05 16:51:20,298 DEBUG SenderThread:67587 [sender.py:send():235] send: final
2022-05-05 16:51:20,298 INFO HandlerThread:67587 [handler.py:handle_request_defer():154] handle defer: 9
2022-05-05 16:51:20,298 DEBUG SenderThread:67587 [sender.py:send():235] send: footer
2022-05-05 16:51:20,299 DEBUG SenderThread:67587 [sender.py:send_request():249] send_request: defer
2022-05-05 16:51:20,299 INFO SenderThread:67587 [sender.py:send_request_defer():388] handle sender defer: 9
2022-05-05 16:51:20,398 DEBUG HandlerThread:67587 [handler.py:handle_request():131] handle_request: poll_exit
2022-05-05 16:51:20,398 DEBUG SenderThread:67587 [sender.py:send_request():249] send_request: poll_exit
2022-05-05 16:51:20,398 INFO SenderThread:67587 [file_pusher.py:join():182] waiting for file pusher
2022-05-05 16:51:21,569 DEBUG HandlerThread:67587 [handler.py:handle_request():131] handle_request: get_summary
2022-05-05 16:51:21,570 DEBUG HandlerThread:67587 [handler.py:handle_request():131] handle_request: sampled_history
2022-05-05 16:51:21,571 DEBUG HandlerThread:67587 [handler.py:handle_request():131] handle_request: shutdown
2022-05-05 16:51:21,571 INFO HandlerThread:67587 [handler.py:finish():739] shutting down handler
2022-05-05 16:51:22,298 INFO WriterThread:67587 [datastore.py:close():281] close: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en/wandb/run-20220505_164902-lr7oonrp/run-lr7oonrp.wandb
2022-05-05 16:51:22,568 INFO SenderThread:67587 [sender.py:finish():1075] shutting down sender
2022-05-05 16:51:22,568 INFO SenderThread:67587 [file_pusher.py:finish():177] shutting down file pusher
2022-05-05 16:51:22,568 INFO SenderThread:67587 [file_pusher.py:join():182] waiting for file pusher
2022-05-05 16:51:22,570 INFO MainThread:67587 [internal.py:handle_exit():79] Internal process exited