File size: 16,996 Bytes
53a9ef4 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 |
2022-03-01 17:10:33,078 INFO MainThread:245236 [internal.py:wandb_internal():89] W&B internal server running at pid: 245236, started at: 2022-03-01 17:10:33.078632
2022-03-01 17:10:33,080 DEBUG HandlerThread:245236 [handler.py:handle_request():131] handle_request: check_version
2022-03-01 17:10:33,081 INFO WriterThread:245236 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/run-6le9s370.wandb
2022-03-01 17:10:33,082 DEBUG SenderThread:245236 [sender.py:send():235] send: header
2022-03-01 17:10:33,082 DEBUG SenderThread:245236 [sender.py:send_request():249] send_request: check_version
2022-03-01 17:10:33,155 DEBUG SenderThread:245236 [sender.py:send():235] send: run
2022-03-01 17:10:33,265 INFO SenderThread:245236 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files
2022-03-01 17:10:33,265 INFO SenderThread:245236 [sender.py:_start_run_threads():809] run started: 6le9s370 with start time 1646154632
2022-03-01 17:10:33,265 DEBUG SenderThread:245236 [sender.py:send():235] send: summary
2022-03-01 17:10:33,265 INFO SenderThread:245236 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 17:10:33,266 DEBUG HandlerThread:245236 [handler.py:handle_request():131] handle_request: run_start
2022-03-01 17:10:33,270 DEBUG HandlerThread:245236 [meta.py:__init__():36] meta init
2022-03-01 17:10:33,270 DEBUG HandlerThread:245236 [meta.py:__init__():50] meta init done
2022-03-01 17:10:33,270 DEBUG HandlerThread:245236 [meta.py:probe():210] probe
2022-03-01 17:10:33,276 DEBUG HandlerThread:245236 [meta.py:_setup_git():200] setup git
2022-03-01 17:10:33,290 DEBUG HandlerThread:245236 [meta.py:_setup_git():207] setup git done
2022-03-01 17:10:33,290 DEBUG HandlerThread:245236 [meta.py:_save_pip():54] save pip
2022-03-01 17:10:33,291 DEBUG HandlerThread:245236 [meta.py:_save_pip():68] save pip done
2022-03-01 17:10:33,291 DEBUG HandlerThread:245236 [meta.py:probe():248] probe done
2022-03-01 17:10:33,365 DEBUG SenderThread:245236 [sender.py:send():235] send: files
2022-03-01 17:10:33,365 INFO SenderThread:245236 [sender.py:_save_file():944] saving file wandb-metadata.json with policy now
2022-03-01 17:10:33,370 DEBUG HandlerThread:245236 [handler.py:handle_request():131] handle_request: stop_status
2022-03-01 17:10:33,370 DEBUG SenderThread:245236 [sender.py:send_request():249] send_request: stop_status
2022-03-01 17:10:33,410 DEBUG SenderThread:245236 [sender.py:send():235] send: config
2022-03-01 17:10:33,411 DEBUG SenderThread:245236 [sender.py:send():235] send: metric
2022-03-01 17:10:33,411 DEBUG SenderThread:245236 [sender.py:send():235] send: metric
2022-03-01 17:10:33,411 WARNING SenderThread:245236 [sender.py:send_metric():902] Seen metric with glob (shouldnt happen)
2022-03-01 17:10:33,654 INFO Thread-11 :245236 [upload_job.py:push():137] Uploaded file /tmp/tmpimoy9n47wandb/3840sgse-wandb-metadata.json
2022-03-01 17:10:34,267 INFO Thread-8 :245236 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files/wandb-summary.json
2022-03-01 17:10:34,267 INFO Thread-8 :245236 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files/requirements.txt
2022-03-01 17:10:34,268 INFO Thread-8 :245236 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files/output.log
2022-03-01 17:10:34,268 INFO Thread-8 :245236 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files/wandb-metadata.json
2022-03-01 17:10:36,266 INFO Thread-8 :245236 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files/output.log
2022-03-01 17:10:40,268 INFO Thread-8 :245236 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files/output.log
2022-03-01 17:10:42,126 DEBUG HandlerThread:245236 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:10:42,126 DEBUG SenderThread:245236 [sender.py:send():235] send: telemetry
2022-03-01 17:10:42,127 DEBUG SenderThread:245236 [sender.py:send():235] send: exit
2022-03-01 17:10:42,127 INFO SenderThread:245236 [sender.py:send_exit():371] handling exit code: 1
2022-03-01 17:10:42,127 INFO SenderThread:245236 [sender.py:send_exit():373] handling runtime: 8
2022-03-01 17:10:42,127 INFO SenderThread:245236 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 17:10:42,127 INFO SenderThread:245236 [sender.py:send_exit():379] send defer
2022-03-01 17:10:42,127 DEBUG SenderThread:245236 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:10:42,128 DEBUG HandlerThread:245236 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:10:42,128 INFO HandlerThread:245236 [handler.py:handle_request_defer():154] handle defer: 0
2022-03-01 17:10:42,128 DEBUG SenderThread:245236 [sender.py:send_request():249] send_request: defer
2022-03-01 17:10:42,128 INFO SenderThread:245236 [sender.py:send_request_defer():388] handle sender defer: 0
2022-03-01 17:10:42,128 INFO SenderThread:245236 [sender.py:transition_state():392] send defer: 1
2022-03-01 17:10:42,128 DEBUG HandlerThread:245236 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:10:42,128 INFO HandlerThread:245236 [handler.py:handle_request_defer():154] handle defer: 1
2022-03-01 17:10:42,190 DEBUG SenderThread:245236 [sender.py:send_request():249] send_request: defer
2022-03-01 17:10:42,190 INFO SenderThread:245236 [sender.py:send_request_defer():388] handle sender defer: 1
2022-03-01 17:10:42,190 INFO SenderThread:245236 [sender.py:transition_state():392] send defer: 2
2022-03-01 17:10:42,190 DEBUG SenderThread:245236 [sender.py:send():235] send: stats
2022-03-01 17:10:42,191 DEBUG HandlerThread:245236 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:10:42,191 INFO HandlerThread:245236 [handler.py:handle_request_defer():154] handle defer: 2
2022-03-01 17:10:42,191 DEBUG SenderThread:245236 [sender.py:send_request():249] send_request: defer
2022-03-01 17:10:42,191 INFO SenderThread:245236 [sender.py:send_request_defer():388] handle sender defer: 2
2022-03-01 17:10:42,191 INFO SenderThread:245236 [sender.py:transition_state():392] send defer: 3
2022-03-01 17:10:42,191 DEBUG HandlerThread:245236 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:10:42,191 INFO HandlerThread:245236 [handler.py:handle_request_defer():154] handle defer: 3
2022-03-01 17:10:42,191 DEBUG SenderThread:245236 [sender.py:send():235] send: summary
2022-03-01 17:10:42,192 INFO SenderThread:245236 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 17:10:42,192 DEBUG SenderThread:245236 [sender.py:send_request():249] send_request: defer
2022-03-01 17:10:42,192 INFO SenderThread:245236 [sender.py:send_request_defer():388] handle sender defer: 3
2022-03-01 17:10:42,192 INFO SenderThread:245236 [sender.py:transition_state():392] send defer: 4
2022-03-01 17:10:42,192 DEBUG HandlerThread:245236 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:10:42,192 INFO HandlerThread:245236 [handler.py:handle_request_defer():154] handle defer: 4
2022-03-01 17:10:42,192 DEBUG SenderThread:245236 [sender.py:send_request():249] send_request: defer
2022-03-01 17:10:42,193 INFO SenderThread:245236 [sender.py:send_request_defer():388] handle sender defer: 4
2022-03-01 17:10:42,229 DEBUG HandlerThread:245236 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:10:42,301 INFO Thread-8 :245236 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files/wandb-summary.json
2022-03-01 17:10:42,301 INFO Thread-8 :245236 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files/output.log
2022-03-01 17:10:42,302 INFO SenderThread:245236 [sender.py:transition_state():392] send defer: 5
2022-03-01 17:10:42,302 DEBUG SenderThread:245236 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:10:42,302 DEBUG HandlerThread:245236 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:10:42,302 INFO HandlerThread:245236 [handler.py:handle_request_defer():154] handle defer: 5
2022-03-01 17:10:42,303 DEBUG SenderThread:245236 [sender.py:send_request():249] send_request: defer
2022-03-01 17:10:42,303 INFO SenderThread:245236 [sender.py:send_request_defer():388] handle sender defer: 5
2022-03-01 17:10:42,303 INFO SenderThread:245236 [dir_watcher.py:finish():283] shutting down directory watcher
2022-03-01 17:10:42,403 DEBUG HandlerThread:245236 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:10:43,302 INFO Thread-8 :245236 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files/config.yaml
2022-03-01 17:10:43,302 INFO SenderThread:245236 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files
2022-03-01 17:10:43,303 INFO SenderThread:245236 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files/wandb-metadata.json wandb-metadata.json
2022-03-01 17:10:43,303 INFO SenderThread:245236 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files/output.log output.log
2022-03-01 17:10:43,303 INFO SenderThread:245236 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files/wandb-summary.json wandb-summary.json
2022-03-01 17:10:43,303 INFO SenderThread:245236 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files/requirements.txt requirements.txt
2022-03-01 17:10:43,303 INFO SenderThread:245236 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files/config.yaml config.yaml
2022-03-01 17:10:43,312 INFO SenderThread:245236 [sender.py:transition_state():392] send defer: 6
2022-03-01 17:10:43,312 DEBUG SenderThread:245236 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:10:43,316 DEBUG HandlerThread:245236 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:10:43,316 INFO HandlerThread:245236 [handler.py:handle_request_defer():154] handle defer: 6
2022-03-01 17:10:43,316 DEBUG SenderThread:245236 [sender.py:send_request():249] send_request: defer
2022-03-01 17:10:43,316 INFO SenderThread:245236 [sender.py:send_request_defer():388] handle sender defer: 6
2022-03-01 17:10:43,317 INFO SenderThread:245236 [file_pusher.py:finish():177] shutting down file pusher
2022-03-01 17:10:43,414 DEBUG HandlerThread:245236 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:10:43,414 DEBUG SenderThread:245236 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:10:43,516 DEBUG HandlerThread:245236 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:10:43,516 DEBUG SenderThread:245236 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:10:43,561 INFO Thread-12 :245236 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files/output.log
2022-03-01 17:10:43,580 INFO Thread-13 :245236 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files/wandb-summary.json
2022-03-01 17:10:43,581 INFO Thread-14 :245236 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files/requirements.txt
2022-03-01 17:10:43,598 INFO Thread-15 :245236 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/files/config.yaml
2022-03-01 17:10:43,617 DEBUG HandlerThread:245236 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:10:43,617 DEBUG SenderThread:245236 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:10:43,719 DEBUG HandlerThread:245236 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:10:43,719 DEBUG SenderThread:245236 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:10:43,799 INFO Thread-7 :245236 [sender.py:transition_state():392] send defer: 7
2022-03-01 17:10:43,799 DEBUG HandlerThread:245236 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:10:43,799 INFO HandlerThread:245236 [handler.py:handle_request_defer():154] handle defer: 7
2022-03-01 17:10:43,799 DEBUG SenderThread:245236 [sender.py:send_request():249] send_request: defer
2022-03-01 17:10:43,799 INFO SenderThread:245236 [sender.py:send_request_defer():388] handle sender defer: 7
2022-03-01 17:10:43,820 DEBUG HandlerThread:245236 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:10:43,839 INFO SenderThread:245236 [sender.py:transition_state():392] send defer: 8
2022-03-01 17:10:43,840 DEBUG SenderThread:245236 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:10:43,840 DEBUG HandlerThread:245236 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:10:43,840 INFO HandlerThread:245236 [handler.py:handle_request_defer():154] handle defer: 8
2022-03-01 17:10:43,840 DEBUG SenderThread:245236 [sender.py:send_request():249] send_request: defer
2022-03-01 17:10:43,840 INFO SenderThread:245236 [sender.py:send_request_defer():388] handle sender defer: 8
2022-03-01 17:10:43,841 INFO SenderThread:245236 [sender.py:transition_state():392] send defer: 9
2022-03-01 17:10:43,841 DEBUG SenderThread:245236 [sender.py:send():235] send: final
2022-03-01 17:10:43,841 DEBUG HandlerThread:245236 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:10:43,842 DEBUG SenderThread:245236 [sender.py:send():235] send: footer
2022-03-01 17:10:43,842 INFO HandlerThread:245236 [handler.py:handle_request_defer():154] handle defer: 9
2022-03-01 17:10:43,842 DEBUG SenderThread:245236 [sender.py:send_request():249] send_request: defer
2022-03-01 17:10:43,842 INFO SenderThread:245236 [sender.py:send_request_defer():388] handle sender defer: 9
2022-03-01 17:10:43,941 DEBUG HandlerThread:245236 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:10:43,942 DEBUG SenderThread:245236 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:10:43,942 INFO SenderThread:245236 [file_pusher.py:join():182] waiting for file pusher
2022-03-01 17:10:43,995 DEBUG HandlerThread:245236 [handler.py:handle_request():131] handle_request: get_summary
2022-03-01 17:10:43,996 DEBUG HandlerThread:245236 [handler.py:handle_request():131] handle_request: sampled_history
2022-03-01 17:10:43,997 DEBUG HandlerThread:245236 [handler.py:handle_request():131] handle_request: shutdown
2022-03-01 17:10:43,997 INFO HandlerThread:245236 [handler.py:finish():739] shutting down handler
2022-03-01 17:10:44,842 INFO WriterThread:245236 [datastore.py:close():281] close: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171032-6le9s370/run-6le9s370.wandb
2022-03-01 17:10:44,994 INFO SenderThread:245236 [sender.py:finish():1075] shutting down sender
2022-03-01 17:10:44,995 INFO SenderThread:245236 [file_pusher.py:finish():177] shutting down file pusher
2022-03-01 17:10:44,995 INFO SenderThread:245236 [file_pusher.py:join():182] waiting for file pusher
2022-03-01 17:10:44,997 INFO MainThread:245236 [internal.py:handle_exit():79] Internal process exited
|