2022-03-02 08:48:34,041 INFO MainThread:254137 [internal.py:wandb_internal():89] W&B internal server running at pid: 254137, started at: 2022-03-02 08:48:34.041581 2022-03-02 08:48:34,043 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: check_version 2022-03-02 08:48:34,044 INFO WriterThread:254137 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/run-2m6jtwtj.wandb 2022-03-02 08:48:34,045 DEBUG SenderThread:254137 [sender.py:send():235] send: header 2022-03-02 08:48:34,046 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: check_version 2022-03-02 08:48:34,116 DEBUG SenderThread:254137 [sender.py:send():235] send: run 2022-03-02 08:48:34,222 INFO SenderThread:254137 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files 2022-03-02 08:48:34,222 INFO SenderThread:254137 [sender.py:_start_run_threads():809] run started: 2m6jtwtj with start time 1646210913 2022-03-02 08:48:34,222 DEBUG SenderThread:254137 [sender.py:send():235] send: summary 2022-03-02 08:48:34,222 INFO SenderThread:254137 [sender.py:_save_file():944] saving file wandb-summary.json with policy end 2022-03-02 08:48:34,223 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: run_start 2022-03-02 08:48:34,227 DEBUG HandlerThread:254137 [meta.py:__init__():36] meta init 2022-03-02 08:48:34,227 DEBUG HandlerThread:254137 [meta.py:__init__():50] meta init done 2022-03-02 08:48:34,227 DEBUG HandlerThread:254137 [meta.py:probe():210] probe 2022-03-02 08:48:34,234 DEBUG HandlerThread:254137 [meta.py:_setup_git():200] setup git 2022-03-02 08:48:34,248 DEBUG HandlerThread:254137 [meta.py:_setup_git():207] setup git done 2022-03-02 08:48:34,248 DEBUG HandlerThread:254137 [meta.py:_save_pip():54] save pip 2022-03-02 08:48:34,249 DEBUG HandlerThread:254137 [meta.py:_save_pip():68] save pip done 2022-03-02 08:48:34,249 DEBUG HandlerThread:254137 [meta.py:probe():248] probe done 2022-03-02 08:48:34,326 DEBUG SenderThread:254137 [sender.py:send():235] send: files 2022-03-02 08:48:34,327 INFO SenderThread:254137 [sender.py:_save_file():944] saving file wandb-metadata.json with policy now 2022-03-02 08:48:34,331 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: stop_status 2022-03-02 08:48:34,331 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: stop_status 2022-03-02 08:48:34,417 DEBUG SenderThread:254137 [sender.py:send():235] send: config 2022-03-02 08:48:34,418 DEBUG SenderThread:254137 [sender.py:send():235] send: metric 2022-03-02 08:48:34,418 DEBUG SenderThread:254137 [sender.py:send():235] send: metric 2022-03-02 08:48:34,418 WARNING SenderThread:254137 [sender.py:send_metric():902] Seen metric with glob (shouldnt happen) 2022-03-02 08:48:34,653 INFO Thread-11 :254137 [upload_job.py:push():137] Uploaded file /tmp/tmpvy5l0hy6wandb/q5wa9252-wandb-metadata.json 2022-03-02 08:48:35,224 INFO Thread-8 :254137 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/requirements.txt 2022-03-02 08:48:35,224 INFO Thread-8 :254137 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/wandb-summary.json 2022-03-02 08:48:35,224 INFO Thread-8 :254137 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/output.log 2022-03-02 08:48:35,224 INFO Thread-8 :254137 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/wandb-metadata.json 2022-03-02 08:48:37,223 INFO Thread-8 :254137 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/output.log 2022-03-02 08:48:41,224 INFO Thread-8 :254137 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/output.log 2022-03-02 08:48:42,671 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: poll_exit 2022-03-02 08:48:42,671 DEBUG SenderThread:254137 [sender.py:send():235] send: telemetry 2022-03-02 08:48:42,671 DEBUG SenderThread:254137 [sender.py:send():235] send: exit 2022-03-02 08:48:42,671 INFO SenderThread:254137 [sender.py:send_exit():371] handling exit code: 1 2022-03-02 08:48:42,671 INFO SenderThread:254137 [sender.py:send_exit():373] handling runtime: 8 2022-03-02 08:48:42,672 INFO SenderThread:254137 [sender.py:_save_file():944] saving file wandb-summary.json with policy end 2022-03-02 08:48:42,672 INFO SenderThread:254137 [sender.py:send_exit():379] send defer 2022-03-02 08:48:42,672 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: poll_exit 2022-03-02 08:48:42,672 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: defer 2022-03-02 08:48:42,673 INFO HandlerThread:254137 [handler.py:handle_request_defer():154] handle defer: 0 2022-03-02 08:48:42,673 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: defer 2022-03-02 08:48:42,673 INFO SenderThread:254137 [sender.py:send_request_defer():388] handle sender defer: 0 2022-03-02 08:48:42,673 INFO SenderThread:254137 [sender.py:transition_state():392] send defer: 1 2022-03-02 08:48:42,673 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: defer 2022-03-02 08:48:42,673 INFO HandlerThread:254137 [handler.py:handle_request_defer():154] handle defer: 1 2022-03-02 08:48:42,753 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: defer 2022-03-02 08:48:42,753 INFO SenderThread:254137 [sender.py:send_request_defer():388] handle sender defer: 1 2022-03-02 08:48:42,753 INFO SenderThread:254137 [sender.py:transition_state():392] send defer: 2 2022-03-02 08:48:42,754 DEBUG SenderThread:254137 [sender.py:send():235] send: stats 2022-03-02 08:48:42,754 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: defer 2022-03-02 08:48:42,754 INFO HandlerThread:254137 [handler.py:handle_request_defer():154] handle defer: 2 2022-03-02 08:48:42,755 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: defer 2022-03-02 08:48:42,755 INFO SenderThread:254137 [sender.py:send_request_defer():388] handle sender defer: 2 2022-03-02 08:48:42,755 INFO SenderThread:254137 [sender.py:transition_state():392] send defer: 3 2022-03-02 08:48:42,755 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: defer 2022-03-02 08:48:42,755 INFO HandlerThread:254137 [handler.py:handle_request_defer():154] handle defer: 3 2022-03-02 08:48:42,755 DEBUG SenderThread:254137 [sender.py:send():235] send: summary 2022-03-02 08:48:42,756 INFO SenderThread:254137 [sender.py:_save_file():944] saving file wandb-summary.json with policy end 2022-03-02 08:48:42,756 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: defer 2022-03-02 08:48:42,756 INFO SenderThread:254137 [sender.py:send_request_defer():388] handle sender defer: 3 2022-03-02 08:48:42,756 INFO SenderThread:254137 [sender.py:transition_state():392] send defer: 4 2022-03-02 08:48:42,756 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: defer 2022-03-02 08:48:42,756 INFO HandlerThread:254137 [handler.py:handle_request_defer():154] handle defer: 4 2022-03-02 08:48:42,756 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: defer 2022-03-02 08:48:42,756 INFO SenderThread:254137 [sender.py:send_request_defer():388] handle sender defer: 4 2022-03-02 08:48:42,774 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: poll_exit 2022-03-02 08:48:42,847 INFO SenderThread:254137 [sender.py:transition_state():392] send defer: 5 2022-03-02 08:48:42,848 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: poll_exit 2022-03-02 08:48:42,848 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: defer 2022-03-02 08:48:42,848 INFO HandlerThread:254137 [handler.py:handle_request_defer():154] handle defer: 5 2022-03-02 08:48:42,848 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: defer 2022-03-02 08:48:42,849 INFO SenderThread:254137 [sender.py:send_request_defer():388] handle sender defer: 5 2022-03-02 08:48:42,849 INFO SenderThread:254137 [dir_watcher.py:finish():283] shutting down directory watcher 2022-03-02 08:48:42,949 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: poll_exit 2022-03-02 08:48:43,225 INFO Thread-8 :254137 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/wandb-summary.json 2022-03-02 08:48:43,226 INFO SenderThread:254137 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/output.log 2022-03-02 08:48:43,227 INFO SenderThread:254137 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/config.yaml 2022-03-02 08:48:43,227 INFO SenderThread:254137 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files 2022-03-02 08:48:43,227 INFO SenderThread:254137 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/wandb-metadata.json wandb-metadata.json 2022-03-02 08:48:43,227 INFO SenderThread:254137 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/output.log output.log 2022-03-02 08:48:43,227 INFO SenderThread:254137 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/wandb-summary.json wandb-summary.json 2022-03-02 08:48:43,227 INFO SenderThread:254137 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/requirements.txt requirements.txt 2022-03-02 08:48:43,228 INFO SenderThread:254137 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/config.yaml config.yaml 2022-03-02 08:48:43,230 INFO SenderThread:254137 [sender.py:transition_state():392] send defer: 6 2022-03-02 08:48:43,231 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: poll_exit 2022-03-02 08:48:43,231 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: defer 2022-03-02 08:48:43,231 INFO HandlerThread:254137 [handler.py:handle_request_defer():154] handle defer: 6 2022-03-02 08:48:43,234 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: defer 2022-03-02 08:48:43,234 INFO SenderThread:254137 [sender.py:send_request_defer():388] handle sender defer: 6 2022-03-02 08:48:43,234 INFO SenderThread:254137 [file_pusher.py:finish():177] shutting down file pusher 2022-03-02 08:48:43,339 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: poll_exit 2022-03-02 08:48:43,339 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: poll_exit 2022-03-02 08:48:43,440 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: poll_exit 2022-03-02 08:48:43,441 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: poll_exit 2022-03-02 08:48:43,497 INFO Thread-13 :254137 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/wandb-summary.json 2022-03-02 08:48:43,498 INFO Thread-14 :254137 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/requirements.txt 2022-03-02 08:48:43,502 INFO Thread-12 :254137 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/output.log 2022-03-02 08:48:43,533 INFO Thread-15 :254137 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/config.yaml 2022-03-02 08:48:43,542 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: poll_exit 2022-03-02 08:48:43,542 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: poll_exit 2022-03-02 08:48:43,644 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: poll_exit 2022-03-02 08:48:43,644 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: poll_exit 2022-03-02 08:48:43,733 INFO Thread-7 :254137 [sender.py:transition_state():392] send defer: 7 2022-03-02 08:48:43,734 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: defer 2022-03-02 08:48:43,734 INFO HandlerThread:254137 [handler.py:handle_request_defer():154] handle defer: 7 2022-03-02 08:48:43,734 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: defer 2022-03-02 08:48:43,734 INFO SenderThread:254137 [sender.py:send_request_defer():388] handle sender defer: 7 2022-03-02 08:48:43,745 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: poll_exit 2022-03-02 08:48:43,905 INFO SenderThread:254137 [sender.py:transition_state():392] send defer: 8 2022-03-02 08:48:43,905 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: poll_exit 2022-03-02 08:48:43,906 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: defer 2022-03-02 08:48:43,906 INFO HandlerThread:254137 [handler.py:handle_request_defer():154] handle defer: 8 2022-03-02 08:48:43,906 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: defer 2022-03-02 08:48:43,906 INFO SenderThread:254137 [sender.py:send_request_defer():388] handle sender defer: 8 2022-03-02 08:48:43,906 INFO SenderThread:254137 [sender.py:transition_state():392] send defer: 9 2022-03-02 08:48:43,907 DEBUG SenderThread:254137 [sender.py:send():235] send: final 2022-03-02 08:48:43,907 DEBUG SenderThread:254137 [sender.py:send():235] send: footer 2022-03-02 08:48:43,907 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: defer 2022-03-02 08:48:43,907 INFO HandlerThread:254137 [handler.py:handle_request_defer():154] handle defer: 9 2022-03-02 08:48:43,907 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: defer 2022-03-02 08:48:43,908 INFO SenderThread:254137 [sender.py:send_request_defer():388] handle sender defer: 9 2022-03-02 08:48:44,007 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: poll_exit 2022-03-02 08:48:44,007 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: poll_exit 2022-03-02 08:48:44,007 INFO SenderThread:254137 [file_pusher.py:join():182] waiting for file pusher 2022-03-02 08:48:44,067 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: get_summary 2022-03-02 08:48:44,068 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: sampled_history 2022-03-02 08:48:44,068 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: shutdown 2022-03-02 08:48:44,068 INFO HandlerThread:254137 [handler.py:finish():739] shutting down handler 2022-03-02 08:48:44,907 INFO WriterThread:254137 [datastore.py:close():281] close: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/run-2m6jtwtj.wandb 2022-03-02 08:48:45,066 INFO SenderThread:254137 [sender.py:finish():1075] shutting down sender 2022-03-02 08:48:45,066 INFO SenderThread:254137 [file_pusher.py:finish():177] shutting down file pusher 2022-03-02 08:48:45,066 INFO SenderThread:254137 [file_pusher.py:join():182] waiting for file pusher 2022-03-02 08:48:45,068 INFO MainThread:254137 [internal.py:handle_exit():79] Internal process exited