sanchit-gandhi's picture
Training in progress, step 500
998e66d
2022-05-05 17:32:29,076 INFO MainThread:74054 [internal.py:wandb_internal():89] W&B internal server running at pid: 74054, started at: 2022-05-05 17:32:29.057409
2022-05-05 17:32:29,079 DEBUG HandlerThread:74054 [handler.py:handle_request():131] handle_request: check_version
2022-05-05 17:32:29,079 INFO WriterThread:74054 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_173227-3sxltsfz/run-3sxltsfz.wandb
2022-05-05 17:32:29,080 DEBUG SenderThread:74054 [sender.py:send():235] send: header
2022-05-05 17:32:29,080 DEBUG SenderThread:74054 [sender.py:send_request():249] send_request: check_version
2022-05-05 17:32:29,146 DEBUG SenderThread:74054 [sender.py:send():235] send: run
2022-05-05 17:32:29,231 INFO SenderThread:74054 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_173227-3sxltsfz/files
2022-05-05 17:32:29,231 INFO SenderThread:74054 [sender.py:_start_run_threads():809] run started: 3sxltsfz with start time 1651771948
2022-05-05 17:32:29,232 DEBUG SenderThread:74054 [sender.py:send():235] send: summary
2022-05-05 17:32:29,232 INFO SenderThread:74054 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-05-05 17:32:29,233 DEBUG HandlerThread:74054 [handler.py:handle_request():131] handle_request: run_start
2022-05-05 17:32:30,235 INFO Thread-8 :74054 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_173227-3sxltsfz/files/wandb-summary.json
2022-05-05 17:32:31,660 DEBUG HandlerThread:74054 [meta.py:__init__():36] meta init
2022-05-05 17:32:31,660 DEBUG HandlerThread:74054 [meta.py:__init__():50] meta init done
2022-05-05 17:32:31,660 DEBUG HandlerThread:74054 [meta.py:probe():210] probe
2022-05-05 17:32:31,667 DEBUG HandlerThread:74054 [meta.py:_setup_git():200] setup git
2022-05-05 17:32:31,699 DEBUG HandlerThread:74054 [meta.py:_setup_git():207] setup git done
2022-05-05 17:32:31,699 DEBUG HandlerThread:74054 [meta.py:_save_pip():54] save pip
2022-05-05 17:32:31,700 DEBUG HandlerThread:74054 [meta.py:_save_pip():68] save pip done
2022-05-05 17:32:31,700 DEBUG HandlerThread:74054 [meta.py:probe():248] probe done
2022-05-05 17:32:31,703 DEBUG SenderThread:74054 [sender.py:send():235] send: files
2022-05-05 17:32:31,704 INFO SenderThread:74054 [sender.py:_save_file():944] saving file wandb-metadata.json with policy now
2022-05-05 17:32:31,710 DEBUG HandlerThread:74054 [handler.py:handle_request():131] handle_request: stop_status
2022-05-05 17:32:31,710 DEBUG SenderThread:74054 [sender.py:send_request():249] send_request: stop_status
2022-05-05 17:32:31,746 DEBUG SenderThread:74054 [sender.py:send():235] send: config
2022-05-05 17:32:31,748 DEBUG SenderThread:74054 [sender.py:send():235] send: metric
2022-05-05 17:32:31,748 DEBUG SenderThread:74054 [sender.py:send():235] send: metric
2022-05-05 17:32:31,748 WARNING SenderThread:74054 [sender.py:send_metric():902] Seen metric with glob (shouldnt happen)
2022-05-05 17:32:31,997 INFO Thread-11 :74054 [upload_job.py:push():137] Uploaded file /tmp/tmpynm7cz5wwandb/10lu44ms-wandb-metadata.json
2022-05-05 17:32:32,235 INFO Thread-8 :74054 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_173227-3sxltsfz/files/requirements.txt
2022-05-05 17:32:32,236 INFO Thread-8 :74054 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_173227-3sxltsfz/files/wandb-metadata.json
2022-05-05 17:32:32,236 INFO Thread-8 :74054 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_173227-3sxltsfz/files/output.log
2022-05-05 17:32:34,236 INFO Thread-8 :74054 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_173227-3sxltsfz/files/output.log
2022-05-05 17:32:36,000 DEBUG SenderThread:74054 [sender.py:send():235] send: metric
2022-05-05 17:32:36,000 DEBUG SenderThread:74054 [sender.py:send():235] send: metric
2022-05-05 17:32:36,000 DEBUG SenderThread:74054 [sender.py:send():235] send: metric
2022-05-05 17:32:36,000 DEBUG SenderThread:74054 [sender.py:send():235] send: metric
2022-05-05 17:32:36,000 DEBUG SenderThread:74054 [sender.py:send():235] send: metric
2022-05-05 17:32:36,000 DEBUG SenderThread:74054 [sender.py:send():235] send: metric
2022-05-05 17:32:36,000 DEBUG SenderThread:74054 [sender.py:send():235] send: history
2022-05-05 17:32:36,001 DEBUG SenderThread:74054 [sender.py:send():235] send: summary
2022-05-05 17:32:36,003 INFO SenderThread:74054 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-05-05 17:32:36,237 INFO Thread-8 :74054 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_173227-3sxltsfz/files/wandb-summary.json
2022-05-05 17:32:38,238 INFO Thread-8 :74054 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_173227-3sxltsfz/files/output.log
2022-05-05 17:32:46,770 DEBUG HandlerThread:74054 [handler.py:handle_request():131] handle_request: stop_status
2022-05-05 17:32:46,770 DEBUG SenderThread:74054 [sender.py:send_request():249] send_request: stop_status
2022-05-05 17:32:56,246 INFO Thread-8 :74054 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_173227-3sxltsfz/files/output.log
2022-05-05 17:33:00,098 DEBUG SenderThread:74054 [sender.py:send():235] send: stats
2022-05-05 17:33:00,247 INFO Thread-8 :74054 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_173227-3sxltsfz/files/config.yaml
2022-05-05 17:33:01,807 DEBUG HandlerThread:74054 [handler.py:handle_request():131] handle_request: stop_status
2022-05-05 17:33:01,807 DEBUG SenderThread:74054 [sender.py:send_request():249] send_request: stop_status
2022-05-05 17:33:08,250 INFO Thread-8 :74054 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_173227-3sxltsfz/files/output.log
2022-05-05 17:33:16,850 DEBUG HandlerThread:74054 [handler.py:handle_request():131] handle_request: stop_status
2022-05-05 17:33:16,850 DEBUG SenderThread:74054 [sender.py:send_request():249] send_request: stop_status
2022-05-05 17:33:30,499 DEBUG SenderThread:74054 [sender.py:send():235] send: stats
2022-05-05 17:33:31,886 DEBUG HandlerThread:74054 [handler.py:handle_request():131] handle_request: stop_status
2022-05-05 17:33:31,886 DEBUG SenderThread:74054 [sender.py:send_request():249] send_request: stop_status
2022-05-05 17:33:34,206 WARNING MainThread:74054 [internal.py:is_dead():380] Internal process exiting, parent pid 73809 disappeared
2022-05-05 17:33:34,206 ERROR MainThread:74054 [internal.py:wandb_internal():148] Internal process shutdown.
2022-05-05 17:33:34,500 INFO WriterThread:74054 [datastore.py:close():281] close: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_173227-3sxltsfz/run-3sxltsfz.wandb
2022-05-05 17:33:34,890 INFO HandlerThread:74054 [handler.py:finish():739] shutting down handler
2022-05-05 17:33:34,921 INFO SenderThread:74054 [sender.py:finish():1075] shutting down sender
2022-05-05 17:33:34,921 INFO SenderThread:74054 [dir_watcher.py:finish():283] shutting down directory watcher
2022-05-05 17:33:35,256 INFO SenderThread:74054 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_173227-3sxltsfz/files
2022-05-05 17:33:35,256 INFO SenderThread:74054 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_173227-3sxltsfz/files/wandb-metadata.json wandb-metadata.json
2022-05-05 17:33:35,256 INFO SenderThread:74054 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_173227-3sxltsfz/files/output.log output.log
2022-05-05 17:33:35,256 INFO SenderThread:74054 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_173227-3sxltsfz/files/wandb-summary.json wandb-summary.json
2022-05-05 17:33:35,257 INFO SenderThread:74054 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_173227-3sxltsfz/files/requirements.txt requirements.txt
2022-05-05 17:33:35,257 INFO SenderThread:74054 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_173227-3sxltsfz/files/config.yaml config.yaml
2022-05-05 17:33:35,257 INFO SenderThread:74054 [file_pusher.py:finish():177] shutting down file pusher
2022-05-05 17:33:35,257 INFO SenderThread:74054 [file_pusher.py:join():182] waiting for file pusher
2022-05-05 17:33:35,532 INFO Thread-15 :74054 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_173227-3sxltsfz/files/config.yaml
2022-05-05 17:33:35,548 INFO Thread-12 :74054 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_173227-3sxltsfz/files/output.log
2022-05-05 17:33:35,562 INFO Thread-13 :74054 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_173227-3sxltsfz/files/wandb-summary.json
2022-05-05 17:33:35,629 INFO Thread-14 :74054 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/xtreme_s_xlsr_2_bart_covost2_fr_en_2/wandb/run-20220505_173227-3sxltsfz/files/requirements.txt
2022-05-05 17:33:36,318 INFO MainThread:74054 [internal.py:handle_exit():79] Internal process exited