| 2022-03-10 21:09:51,233 INFO MainThread:1044076 [internal.py:wandb_internal():89] W&B internal server running at pid: 1044076, started at: 2022-03-10 21:09:51.233570 | |
| 2022-03-10 21:09:51,235 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: check_version | |
| 2022-03-10 21:09:51,235 INFO WriterThread:1044076 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/run-19sg19ou.wandb | |
| 2022-03-10 21:09:51,237 DEBUG SenderThread:1044076 [sender.py:send():235] send: header | |
| 2022-03-10 21:09:51,237 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: check_version | |
| 2022-03-10 21:09:51,306 DEBUG SenderThread:1044076 [sender.py:send():235] send: run | |
| 2022-03-10 21:09:51,397 INFO SenderThread:1044076 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files | |
| 2022-03-10 21:09:51,397 INFO SenderThread:1044076 [sender.py:_start_run_threads():809] run started: 19sg19ou with start time 1646946590 | |
| 2022-03-10 21:09:51,398 DEBUG SenderThread:1044076 [sender.py:send():235] send: summary | |
| 2022-03-10 21:09:51,398 INFO SenderThread:1044076 [sender.py:_save_file():944] saving file wandb-summary.json with policy end | |
| 2022-03-10 21:09:51,398 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: run_start | |
| 2022-03-10 21:09:51,402 DEBUG HandlerThread:1044076 [meta.py:__init__():36] meta init | |
| 2022-03-10 21:09:51,403 DEBUG HandlerThread:1044076 [meta.py:__init__():50] meta init done | |
| 2022-03-10 21:09:51,403 DEBUG HandlerThread:1044076 [meta.py:probe():210] probe | |
| 2022-03-10 21:09:51,410 DEBUG HandlerThread:1044076 [meta.py:_setup_git():200] setup git | |
| 2022-03-10 21:09:51,425 DEBUG HandlerThread:1044076 [meta.py:_setup_git():207] setup git done | |
| 2022-03-10 21:09:51,426 DEBUG HandlerThread:1044076 [meta.py:_save_pip():54] save pip | |
| 2022-03-10 21:09:51,426 DEBUG HandlerThread:1044076 [meta.py:_save_pip():68] save pip done | |
| 2022-03-10 21:09:51,426 DEBUG HandlerThread:1044076 [meta.py:probe():248] probe done | |
| 2022-03-10 21:09:51,505 DEBUG SenderThread:1044076 [sender.py:send():235] send: files | |
| 2022-03-10 21:09:51,505 INFO SenderThread:1044076 [sender.py:_save_file():944] saving file wandb-metadata.json with policy now | |
| 2022-03-10 21:09:51,509 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: stop_status | |
| 2022-03-10 21:09:51,510 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: stop_status | |
| 2022-03-10 21:09:51,544 DEBUG SenderThread:1044076 [sender.py:send():235] send: config | |
| 2022-03-10 21:09:51,545 DEBUG SenderThread:1044076 [sender.py:send():235] send: metric | |
| 2022-03-10 21:09:51,546 DEBUG SenderThread:1044076 [sender.py:send():235] send: metric | |
| 2022-03-10 21:09:51,546 WARNING SenderThread:1044076 [sender.py:send_metric():902] Seen metric with glob (shouldnt happen) | |
| 2022-03-10 21:09:51,790 INFO Thread-11 :1044076 [upload_job.py:push():137] Uploaded file /tmp/tmp0u4vh57hwandb/2s9kjsb8-wandb-metadata.json | |
| 2022-03-10 21:09:52,400 INFO Thread-8 :1044076 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/wandb-summary.json | |
| 2022-03-10 21:09:52,401 INFO Thread-8 :1044076 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/requirements.txt | |
| 2022-03-10 21:09:52,401 INFO Thread-8 :1044076 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/output.log | |
| 2022-03-10 21:09:52,401 INFO Thread-8 :1044076 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/wandb-metadata.json | |
| 2022-03-10 21:09:54,398 INFO Thread-8 :1044076 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/output.log | |
| 2022-03-10 21:09:58,400 INFO Thread-8 :1044076 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/output.log | |
| 2022-03-10 21:09:59,980 DEBUG SenderThread:1044076 [sender.py:send():235] send: telemetry | |
| 2022-03-10 21:09:59,980 DEBUG SenderThread:1044076 [sender.py:send():235] send: exit | |
| 2022-03-10 21:09:59,980 INFO SenderThread:1044076 [sender.py:send_exit():371] handling exit code: 1 | |
| 2022-03-10 21:09:59,981 INFO SenderThread:1044076 [sender.py:send_exit():373] handling runtime: 8 | |
| 2022-03-10 21:09:59,981 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: poll_exit | |
| 2022-03-10 21:09:59,981 INFO SenderThread:1044076 [sender.py:_save_file():944] saving file wandb-summary.json with policy end | |
| 2022-03-10 21:09:59,982 INFO SenderThread:1044076 [sender.py:send_exit():379] send defer | |
| 2022-03-10 21:09:59,982 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-03-10 21:09:59,982 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: defer | |
| 2022-03-10 21:09:59,982 INFO HandlerThread:1044076 [handler.py:handle_request_defer():154] handle defer: 0 | |
| 2022-03-10 21:09:59,983 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: defer | |
| 2022-03-10 21:09:59,983 INFO SenderThread:1044076 [sender.py:send_request_defer():388] handle sender defer: 0 | |
| 2022-03-10 21:09:59,983 INFO SenderThread:1044076 [sender.py:transition_state():392] send defer: 1 | |
| 2022-03-10 21:09:59,983 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: defer | |
| 2022-03-10 21:09:59,983 INFO HandlerThread:1044076 [handler.py:handle_request_defer():154] handle defer: 1 | |
| 2022-03-10 21:10:00,043 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: defer | |
| 2022-03-10 21:10:00,043 INFO SenderThread:1044076 [sender.py:send_request_defer():388] handle sender defer: 1 | |
| 2022-03-10 21:10:00,043 INFO SenderThread:1044076 [sender.py:transition_state():392] send defer: 2 | |
| 2022-03-10 21:10:00,043 DEBUG SenderThread:1044076 [sender.py:send():235] send: stats | |
| 2022-03-10 21:10:00,044 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: defer | |
| 2022-03-10 21:10:00,044 INFO HandlerThread:1044076 [handler.py:handle_request_defer():154] handle defer: 2 | |
| 2022-03-10 21:10:00,044 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: defer | |
| 2022-03-10 21:10:00,044 INFO SenderThread:1044076 [sender.py:send_request_defer():388] handle sender defer: 2 | |
| 2022-03-10 21:10:00,044 INFO SenderThread:1044076 [sender.py:transition_state():392] send defer: 3 | |
| 2022-03-10 21:10:00,044 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: defer | |
| 2022-03-10 21:10:00,045 INFO HandlerThread:1044076 [handler.py:handle_request_defer():154] handle defer: 3 | |
| 2022-03-10 21:10:00,045 DEBUG SenderThread:1044076 [sender.py:send():235] send: summary | |
| 2022-03-10 21:10:00,045 INFO SenderThread:1044076 [sender.py:_save_file():944] saving file wandb-summary.json with policy end | |
| 2022-03-10 21:10:00,045 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: defer | |
| 2022-03-10 21:10:00,045 INFO SenderThread:1044076 [sender.py:send_request_defer():388] handle sender defer: 3 | |
| 2022-03-10 21:10:00,045 INFO SenderThread:1044076 [sender.py:transition_state():392] send defer: 4 | |
| 2022-03-10 21:10:00,046 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: defer | |
| 2022-03-10 21:10:00,046 INFO HandlerThread:1044076 [handler.py:handle_request_defer():154] handle defer: 4 | |
| 2022-03-10 21:10:00,046 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: defer | |
| 2022-03-10 21:10:00,046 INFO SenderThread:1044076 [sender.py:send_request_defer():388] handle sender defer: 4 | |
| 2022-03-10 21:10:00,083 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: poll_exit | |
| 2022-03-10 21:10:00,172 INFO SenderThread:1044076 [sender.py:transition_state():392] send defer: 5 | |
| 2022-03-10 21:10:00,172 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-03-10 21:10:00,173 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: defer | |
| 2022-03-10 21:10:00,173 INFO HandlerThread:1044076 [handler.py:handle_request_defer():154] handle defer: 5 | |
| 2022-03-10 21:10:00,173 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: defer | |
| 2022-03-10 21:10:00,173 INFO SenderThread:1044076 [sender.py:send_request_defer():388] handle sender defer: 5 | |
| 2022-03-10 21:10:00,173 INFO SenderThread:1044076 [dir_watcher.py:finish():283] shutting down directory watcher | |
| 2022-03-10 21:10:00,274 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: poll_exit | |
| 2022-03-10 21:10:00,401 INFO Thread-8 :1044076 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/wandb-summary.json | |
| 2022-03-10 21:10:00,401 INFO SenderThread:1044076 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/config.yaml | |
| 2022-03-10 21:10:00,401 INFO SenderThread:1044076 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/output.log | |
| 2022-03-10 21:10:00,402 INFO SenderThread:1044076 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files | |
| 2022-03-10 21:10:00,402 INFO SenderThread:1044076 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/wandb-metadata.json wandb-metadata.json | |
| 2022-03-10 21:10:00,402 INFO SenderThread:1044076 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/output.log output.log | |
| 2022-03-10 21:10:00,402 INFO SenderThread:1044076 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/wandb-summary.json wandb-summary.json | |
| 2022-03-10 21:10:00,402 INFO SenderThread:1044076 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/requirements.txt requirements.txt | |
| 2022-03-10 21:10:00,405 INFO SenderThread:1044076 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/config.yaml config.yaml | |
| 2022-03-10 21:10:00,408 INFO SenderThread:1044076 [sender.py:transition_state():392] send defer: 6 | |
| 2022-03-10 21:10:00,408 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-03-10 21:10:00,414 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: defer | |
| 2022-03-10 21:10:00,414 INFO HandlerThread:1044076 [handler.py:handle_request_defer():154] handle defer: 6 | |
| 2022-03-10 21:10:00,415 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: defer | |
| 2022-03-10 21:10:00,415 INFO SenderThread:1044076 [sender.py:send_request_defer():388] handle sender defer: 6 | |
| 2022-03-10 21:10:00,415 INFO SenderThread:1044076 [file_pusher.py:finish():177] shutting down file pusher | |
| 2022-03-10 21:10:00,510 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: poll_exit | |
| 2022-03-10 21:10:00,510 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-03-10 21:10:00,612 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: poll_exit | |
| 2022-03-10 21:10:00,612 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-03-10 21:10:00,664 INFO Thread-12 :1044076 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/output.log | |
| 2022-03-10 21:10:00,672 INFO Thread-15 :1044076 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/config.yaml | |
| 2022-03-10 21:10:00,688 INFO Thread-13 :1044076 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/wandb-summary.json | |
| 2022-03-10 21:10:00,699 INFO Thread-14 :1044076 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/requirements.txt | |
| 2022-03-10 21:10:00,713 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: poll_exit | |
| 2022-03-10 21:10:00,714 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-03-10 21:10:00,815 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: poll_exit | |
| 2022-03-10 21:10:00,815 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-03-10 21:10:00,899 INFO Thread-7 :1044076 [sender.py:transition_state():392] send defer: 7 | |
| 2022-03-10 21:10:00,900 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: defer | |
| 2022-03-10 21:10:00,900 INFO HandlerThread:1044076 [handler.py:handle_request_defer():154] handle defer: 7 | |
| 2022-03-10 21:10:00,900 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: defer | |
| 2022-03-10 21:10:00,900 INFO SenderThread:1044076 [sender.py:send_request_defer():388] handle sender defer: 7 | |
| 2022-03-10 21:10:00,916 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: poll_exit | |
| 2022-03-10 21:10:01,040 INFO SenderThread:1044076 [sender.py:transition_state():392] send defer: 8 | |
| 2022-03-10 21:10:01,040 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-03-10 21:10:01,041 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: defer | |
| 2022-03-10 21:10:01,041 INFO HandlerThread:1044076 [handler.py:handle_request_defer():154] handle defer: 8 | |
| 2022-03-10 21:10:01,041 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: defer | |
| 2022-03-10 21:10:01,041 INFO SenderThread:1044076 [sender.py:send_request_defer():388] handle sender defer: 8 | |
| 2022-03-10 21:10:01,041 INFO SenderThread:1044076 [sender.py:transition_state():392] send defer: 9 | |
| 2022-03-10 21:10:01,042 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: defer | |
| 2022-03-10 21:10:01,042 DEBUG SenderThread:1044076 [sender.py:send():235] send: final | |
| 2022-03-10 21:10:01,042 INFO HandlerThread:1044076 [handler.py:handle_request_defer():154] handle defer: 9 | |
| 2022-03-10 21:10:01,043 DEBUG SenderThread:1044076 [sender.py:send():235] send: footer | |
| 2022-03-10 21:10:01,043 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: defer | |
| 2022-03-10 21:10:01,043 INFO SenderThread:1044076 [sender.py:send_request_defer():388] handle sender defer: 9 | |
| 2022-03-10 21:10:01,142 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: poll_exit | |
| 2022-03-10 21:10:01,142 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-03-10 21:10:01,142 INFO SenderThread:1044076 [file_pusher.py:join():182] waiting for file pusher | |
| 2022-03-10 21:10:01,199 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: get_summary | |
| 2022-03-10 21:10:01,200 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: sampled_history | |
| 2022-03-10 21:10:01,201 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: shutdown | |
| 2022-03-10 21:10:01,201 INFO HandlerThread:1044076 [handler.py:finish():739] shutting down handler | |
| 2022-03-10 21:10:02,042 INFO WriterThread:1044076 [datastore.py:close():281] close: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/run-19sg19ou.wandb | |
| 2022-03-10 21:10:02,198 INFO SenderThread:1044076 [sender.py:finish():1075] shutting down sender | |
| 2022-03-10 21:10:02,198 INFO SenderThread:1044076 [file_pusher.py:finish():177] shutting down file pusher | |
| 2022-03-10 21:10:02,198 INFO SenderThread:1044076 [file_pusher.py:join():182] waiting for file pusher | |
| 2022-03-10 21:10:02,200 INFO MainThread:1044076 [internal.py:handle_exit():79] Internal process exited | |