| 2022-03-01 01:29:45,420 INFO MainThread:236851 [internal.py:wandb_internal():89] W&B internal server running at pid: 236851, started at: 2022-03-01 01:29:45.420003 | |
| 2022-03-01 01:29:45,422 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: check_version | |
| 2022-03-01 01:29:45,422 INFO WriterThread:236851 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/run-25hk0gke.wandb | |
| 2022-03-01 01:29:45,424 DEBUG SenderThread:236851 [sender.py:send():235] send: header | |
| 2022-03-01 01:29:45,424 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: check_version | |
| 2022-03-01 01:29:45,489 DEBUG SenderThread:236851 [sender.py:send():235] send: run | |
| 2022-03-01 01:29:45,576 INFO SenderThread:236851 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files | |
| 2022-03-01 01:29:45,576 INFO SenderThread:236851 [sender.py:_start_run_threads():809] run started: 25hk0gke with start time 1646098184 | |
| 2022-03-01 01:29:45,576 DEBUG SenderThread:236851 [sender.py:send():235] send: summary | |
| 2022-03-01 01:29:45,576 INFO SenderThread:236851 [sender.py:_save_file():944] saving file wandb-summary.json with policy end | |
| 2022-03-01 01:29:45,577 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: run_start | |
| 2022-03-01 01:29:45,582 DEBUG HandlerThread:236851 [meta.py:__init__():36] meta init | |
| 2022-03-01 01:29:45,582 DEBUG HandlerThread:236851 [meta.py:__init__():50] meta init done | |
| 2022-03-01 01:29:45,582 DEBUG HandlerThread:236851 [meta.py:probe():210] probe | |
| 2022-03-01 01:29:45,588 DEBUG HandlerThread:236851 [meta.py:_setup_git():200] setup git | |
| 2022-03-01 01:29:45,603 DEBUG HandlerThread:236851 [meta.py:_setup_git():207] setup git done | |
| 2022-03-01 01:29:45,603 DEBUG HandlerThread:236851 [meta.py:_save_pip():54] save pip | |
| 2022-03-01 01:29:45,604 DEBUG HandlerThread:236851 [meta.py:_save_pip():68] save pip done | |
| 2022-03-01 01:29:45,604 DEBUG HandlerThread:236851 [meta.py:probe():248] probe done | |
| 2022-03-01 01:29:45,684 DEBUG SenderThread:236851 [sender.py:send():235] send: files | |
| 2022-03-01 01:29:45,684 INFO SenderThread:236851 [sender.py:_save_file():944] saving file wandb-metadata.json with policy now | |
| 2022-03-01 01:29:45,689 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: stop_status | |
| 2022-03-01 01:29:45,690 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: stop_status | |
| 2022-03-01 01:29:45,729 DEBUG SenderThread:236851 [sender.py:send():235] send: config | |
| 2022-03-01 01:29:45,731 DEBUG SenderThread:236851 [sender.py:send():235] send: metric | |
| 2022-03-01 01:29:45,731 DEBUG SenderThread:236851 [sender.py:send():235] send: metric | |
| 2022-03-01 01:29:45,731 WARNING SenderThread:236851 [sender.py:send_metric():902] Seen metric with glob (shouldnt happen) | |
| 2022-03-01 01:29:45,969 INFO Thread-11 :236851 [upload_job.py:push():137] Uploaded file /tmp/tmpdzeo2dm2wandb/3813kelu-wandb-metadata.json | |
| 2022-03-01 01:29:46,578 INFO Thread-8 :236851 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/requirements.txt | |
| 2022-03-01 01:29:46,578 INFO Thread-8 :236851 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/wandb-summary.json | |
| 2022-03-01 01:29:46,578 INFO Thread-8 :236851 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/wandb-metadata.json | |
| 2022-03-01 01:29:46,578 INFO Thread-8 :236851 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/output.log | |
| 2022-03-01 01:29:48,577 INFO Thread-8 :236851 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/output.log | |
| 2022-03-01 01:29:52,578 INFO Thread-8 :236851 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/output.log | |
| 2022-03-01 01:29:54,442 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: poll_exit | |
| 2022-03-01 01:29:54,442 DEBUG SenderThread:236851 [sender.py:send():235] send: telemetry | |
| 2022-03-01 01:29:54,443 DEBUG SenderThread:236851 [sender.py:send():235] send: exit | |
| 2022-03-01 01:29:54,443 INFO SenderThread:236851 [sender.py:send_exit():371] handling exit code: 1 | |
| 2022-03-01 01:29:54,443 INFO SenderThread:236851 [sender.py:send_exit():373] handling runtime: 8 | |
| 2022-03-01 01:29:54,443 INFO SenderThread:236851 [sender.py:_save_file():944] saving file wandb-summary.json with policy end | |
| 2022-03-01 01:29:54,443 INFO SenderThread:236851 [sender.py:send_exit():379] send defer | |
| 2022-03-01 01:29:54,444 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-03-01 01:29:54,444 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer | |
| 2022-03-01 01:29:54,444 INFO HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 0 | |
| 2022-03-01 01:29:54,444 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: defer | |
| 2022-03-01 01:29:54,445 INFO SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 0 | |
| 2022-03-01 01:29:54,445 INFO SenderThread:236851 [sender.py:transition_state():392] send defer: 1 | |
| 2022-03-01 01:29:54,445 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer | |
| 2022-03-01 01:29:54,445 INFO HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 1 | |
| 2022-03-01 01:29:54,515 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: defer | |
| 2022-03-01 01:29:54,515 INFO SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 1 | |
| 2022-03-01 01:29:54,515 INFO SenderThread:236851 [sender.py:transition_state():392] send defer: 2 | |
| 2022-03-01 01:29:54,515 DEBUG SenderThread:236851 [sender.py:send():235] send: stats | |
| 2022-03-01 01:29:54,516 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer | |
| 2022-03-01 01:29:54,516 INFO HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 2 | |
| 2022-03-01 01:29:54,516 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: defer | |
| 2022-03-01 01:29:54,516 INFO SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 2 | |
| 2022-03-01 01:29:54,516 INFO SenderThread:236851 [sender.py:transition_state():392] send defer: 3 | |
| 2022-03-01 01:29:54,517 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer | |
| 2022-03-01 01:29:54,517 INFO HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 3 | |
| 2022-03-01 01:29:54,517 DEBUG SenderThread:236851 [sender.py:send():235] send: summary | |
| 2022-03-01 01:29:54,517 INFO SenderThread:236851 [sender.py:_save_file():944] saving file wandb-summary.json with policy end | |
| 2022-03-01 01:29:54,518 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: defer | |
| 2022-03-01 01:29:54,518 INFO SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 3 | |
| 2022-03-01 01:29:54,518 INFO SenderThread:236851 [sender.py:transition_state():392] send defer: 4 | |
| 2022-03-01 01:29:54,518 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer | |
| 2022-03-01 01:29:54,518 INFO HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 4 | |
| 2022-03-01 01:29:54,518 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: defer | |
| 2022-03-01 01:29:54,518 INFO SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 4 | |
| 2022-03-01 01:29:54,545 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: poll_exit | |
| 2022-03-01 01:29:54,579 INFO Thread-8 :236851 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/wandb-summary.json | |
| 2022-03-01 01:29:54,579 INFO Thread-8 :236851 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/output.log | |
| 2022-03-01 01:29:54,803 INFO SenderThread:236851 [sender.py:transition_state():392] send defer: 5 | |
| 2022-03-01 01:29:54,803 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-03-01 01:29:54,804 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer | |
| 2022-03-01 01:29:54,804 INFO HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 5 | |
| 2022-03-01 01:29:54,804 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: defer | |
| 2022-03-01 01:29:54,804 INFO SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 5 | |
| 2022-03-01 01:29:54,804 INFO SenderThread:236851 [dir_watcher.py:finish():283] shutting down directory watcher | |
| 2022-03-01 01:29:54,905 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: poll_exit | |
| 2022-03-01 01:29:55,579 INFO Thread-8 :236851 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/config.yaml | |
| 2022-03-01 01:29:55,580 INFO SenderThread:236851 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files | |
| 2022-03-01 01:29:55,580 INFO SenderThread:236851 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/wandb-metadata.json wandb-metadata.json | |
| 2022-03-01 01:29:55,580 INFO SenderThread:236851 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/output.log output.log | |
| 2022-03-01 01:29:55,581 INFO SenderThread:236851 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/wandb-summary.json wandb-summary.json | |
| 2022-03-01 01:29:55,581 INFO SenderThread:236851 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/requirements.txt requirements.txt | |
| 2022-03-01 01:29:55,586 INFO SenderThread:236851 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/config.yaml config.yaml | |
| 2022-03-01 01:29:55,586 INFO SenderThread:236851 [sender.py:transition_state():392] send defer: 6 | |
| 2022-03-01 01:29:55,587 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-03-01 01:29:55,588 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer | |
| 2022-03-01 01:29:55,588 INFO HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 6 | |
| 2022-03-01 01:29:55,591 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: defer | |
| 2022-03-01 01:29:55,591 INFO SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 6 | |
| 2022-03-01 01:29:55,592 INFO SenderThread:236851 [file_pusher.py:finish():177] shutting down file pusher | |
| 2022-03-01 01:29:55,689 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: poll_exit | |
| 2022-03-01 01:29:55,689 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-03-01 01:29:55,791 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: poll_exit | |
| 2022-03-01 01:29:55,791 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-03-01 01:29:55,856 INFO Thread-14 :236851 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/requirements.txt | |
| 2022-03-01 01:29:55,860 INFO Thread-13 :236851 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/wandb-summary.json | |
| 2022-03-01 01:29:55,885 INFO Thread-15 :236851 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/config.yaml | |
| 2022-03-01 01:29:55,893 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: poll_exit | |
| 2022-03-01 01:29:55,893 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-03-01 01:29:55,941 INFO Thread-12 :236851 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/output.log | |
| 2022-03-01 01:29:55,994 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: poll_exit | |
| 2022-03-01 01:29:55,995 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-03-01 01:29:56,096 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: poll_exit | |
| 2022-03-01 01:29:56,096 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-03-01 01:29:56,141 INFO Thread-7 :236851 [sender.py:transition_state():392] send defer: 7 | |
| 2022-03-01 01:29:56,142 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer | |
| 2022-03-01 01:29:56,142 INFO HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 7 | |
| 2022-03-01 01:29:56,142 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: defer | |
| 2022-03-01 01:29:56,142 INFO SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 7 | |
| 2022-03-01 01:29:56,188 INFO SenderThread:236851 [sender.py:transition_state():392] send defer: 8 | |
| 2022-03-01 01:29:56,188 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer | |
| 2022-03-01 01:29:56,188 INFO HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 8 | |
| 2022-03-01 01:29:56,188 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: defer | |
| 2022-03-01 01:29:56,188 INFO SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 8 | |
| 2022-03-01 01:29:56,189 INFO SenderThread:236851 [sender.py:transition_state():392] send defer: 9 | |
| 2022-03-01 01:29:56,189 DEBUG SenderThread:236851 [sender.py:send():235] send: final | |
| 2022-03-01 01:29:56,189 DEBUG SenderThread:236851 [sender.py:send():235] send: footer | |
| 2022-03-01 01:29:56,189 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer | |
| 2022-03-01 01:29:56,189 INFO HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 9 | |
| 2022-03-01 01:29:56,190 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: defer | |
| 2022-03-01 01:29:56,190 INFO SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 9 | |
| 2022-03-01 01:29:56,197 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: poll_exit | |
| 2022-03-01 01:29:56,198 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-03-01 01:29:56,198 INFO SenderThread:236851 [file_pusher.py:join():182] waiting for file pusher | |
| 2022-03-01 01:29:56,258 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: get_summary | |
| 2022-03-01 01:29:56,259 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: sampled_history | |
| 2022-03-01 01:29:56,259 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: shutdown | |
| 2022-03-01 01:29:56,259 INFO HandlerThread:236851 [handler.py:finish():739] shutting down handler | |
| 2022-03-01 01:29:57,190 INFO WriterThread:236851 [datastore.py:close():281] close: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/run-25hk0gke.wandb | |
| 2022-03-01 01:29:57,257 INFO SenderThread:236851 [sender.py:finish():1075] shutting down sender | |
| 2022-03-01 01:29:57,257 INFO SenderThread:236851 [file_pusher.py:finish():177] shutting down file pusher | |
| 2022-03-01 01:29:57,257 INFO SenderThread:236851 [file_pusher.py:join():182] waiting for file pusher | |
| 2022-03-01 01:29:57,259 INFO MainThread:236851 [internal.py:handle_exit():79] Internal process exited | |