File size: 17,000 Bytes
fc53a59 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 |
2022-03-02 08:48:34,041 INFO MainThread:254137 [internal.py:wandb_internal():89] W&B internal server running at pid: 254137, started at: 2022-03-02 08:48:34.041581
2022-03-02 08:48:34,043 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: check_version
2022-03-02 08:48:34,044 INFO WriterThread:254137 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/run-2m6jtwtj.wandb
2022-03-02 08:48:34,045 DEBUG SenderThread:254137 [sender.py:send():235] send: header
2022-03-02 08:48:34,046 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: check_version
2022-03-02 08:48:34,116 DEBUG SenderThread:254137 [sender.py:send():235] send: run
2022-03-02 08:48:34,222 INFO SenderThread:254137 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files
2022-03-02 08:48:34,222 INFO SenderThread:254137 [sender.py:_start_run_threads():809] run started: 2m6jtwtj with start time 1646210913
2022-03-02 08:48:34,222 DEBUG SenderThread:254137 [sender.py:send():235] send: summary
2022-03-02 08:48:34,222 INFO SenderThread:254137 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-02 08:48:34,223 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: run_start
2022-03-02 08:48:34,227 DEBUG HandlerThread:254137 [meta.py:__init__():36] meta init
2022-03-02 08:48:34,227 DEBUG HandlerThread:254137 [meta.py:__init__():50] meta init done
2022-03-02 08:48:34,227 DEBUG HandlerThread:254137 [meta.py:probe():210] probe
2022-03-02 08:48:34,234 DEBUG HandlerThread:254137 [meta.py:_setup_git():200] setup git
2022-03-02 08:48:34,248 DEBUG HandlerThread:254137 [meta.py:_setup_git():207] setup git done
2022-03-02 08:48:34,248 DEBUG HandlerThread:254137 [meta.py:_save_pip():54] save pip
2022-03-02 08:48:34,249 DEBUG HandlerThread:254137 [meta.py:_save_pip():68] save pip done
2022-03-02 08:48:34,249 DEBUG HandlerThread:254137 [meta.py:probe():248] probe done
2022-03-02 08:48:34,326 DEBUG SenderThread:254137 [sender.py:send():235] send: files
2022-03-02 08:48:34,327 INFO SenderThread:254137 [sender.py:_save_file():944] saving file wandb-metadata.json with policy now
2022-03-02 08:48:34,331 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: stop_status
2022-03-02 08:48:34,331 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: stop_status
2022-03-02 08:48:34,417 DEBUG SenderThread:254137 [sender.py:send():235] send: config
2022-03-02 08:48:34,418 DEBUG SenderThread:254137 [sender.py:send():235] send: metric
2022-03-02 08:48:34,418 DEBUG SenderThread:254137 [sender.py:send():235] send: metric
2022-03-02 08:48:34,418 WARNING SenderThread:254137 [sender.py:send_metric():902] Seen metric with glob (shouldnt happen)
2022-03-02 08:48:34,653 INFO Thread-11 :254137 [upload_job.py:push():137] Uploaded file /tmp/tmpvy5l0hy6wandb/q5wa9252-wandb-metadata.json
2022-03-02 08:48:35,224 INFO Thread-8 :254137 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/requirements.txt
2022-03-02 08:48:35,224 INFO Thread-8 :254137 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/wandb-summary.json
2022-03-02 08:48:35,224 INFO Thread-8 :254137 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/output.log
2022-03-02 08:48:35,224 INFO Thread-8 :254137 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/wandb-metadata.json
2022-03-02 08:48:37,223 INFO Thread-8 :254137 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/output.log
2022-03-02 08:48:41,224 INFO Thread-8 :254137 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/output.log
2022-03-02 08:48:42,671 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:48:42,671 DEBUG SenderThread:254137 [sender.py:send():235] send: telemetry
2022-03-02 08:48:42,671 DEBUG SenderThread:254137 [sender.py:send():235] send: exit
2022-03-02 08:48:42,671 INFO SenderThread:254137 [sender.py:send_exit():371] handling exit code: 1
2022-03-02 08:48:42,671 INFO SenderThread:254137 [sender.py:send_exit():373] handling runtime: 8
2022-03-02 08:48:42,672 INFO SenderThread:254137 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-02 08:48:42,672 INFO SenderThread:254137 [sender.py:send_exit():379] send defer
2022-03-02 08:48:42,672 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:48:42,672 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:48:42,673 INFO HandlerThread:254137 [handler.py:handle_request_defer():154] handle defer: 0
2022-03-02 08:48:42,673 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: defer
2022-03-02 08:48:42,673 INFO SenderThread:254137 [sender.py:send_request_defer():388] handle sender defer: 0
2022-03-02 08:48:42,673 INFO SenderThread:254137 [sender.py:transition_state():392] send defer: 1
2022-03-02 08:48:42,673 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:48:42,673 INFO HandlerThread:254137 [handler.py:handle_request_defer():154] handle defer: 1
2022-03-02 08:48:42,753 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: defer
2022-03-02 08:48:42,753 INFO SenderThread:254137 [sender.py:send_request_defer():388] handle sender defer: 1
2022-03-02 08:48:42,753 INFO SenderThread:254137 [sender.py:transition_state():392] send defer: 2
2022-03-02 08:48:42,754 DEBUG SenderThread:254137 [sender.py:send():235] send: stats
2022-03-02 08:48:42,754 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:48:42,754 INFO HandlerThread:254137 [handler.py:handle_request_defer():154] handle defer: 2
2022-03-02 08:48:42,755 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: defer
2022-03-02 08:48:42,755 INFO SenderThread:254137 [sender.py:send_request_defer():388] handle sender defer: 2
2022-03-02 08:48:42,755 INFO SenderThread:254137 [sender.py:transition_state():392] send defer: 3
2022-03-02 08:48:42,755 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:48:42,755 INFO HandlerThread:254137 [handler.py:handle_request_defer():154] handle defer: 3
2022-03-02 08:48:42,755 DEBUG SenderThread:254137 [sender.py:send():235] send: summary
2022-03-02 08:48:42,756 INFO SenderThread:254137 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-02 08:48:42,756 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: defer
2022-03-02 08:48:42,756 INFO SenderThread:254137 [sender.py:send_request_defer():388] handle sender defer: 3
2022-03-02 08:48:42,756 INFO SenderThread:254137 [sender.py:transition_state():392] send defer: 4
2022-03-02 08:48:42,756 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:48:42,756 INFO HandlerThread:254137 [handler.py:handle_request_defer():154] handle defer: 4
2022-03-02 08:48:42,756 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: defer
2022-03-02 08:48:42,756 INFO SenderThread:254137 [sender.py:send_request_defer():388] handle sender defer: 4
2022-03-02 08:48:42,774 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:48:42,847 INFO SenderThread:254137 [sender.py:transition_state():392] send defer: 5
2022-03-02 08:48:42,848 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:48:42,848 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:48:42,848 INFO HandlerThread:254137 [handler.py:handle_request_defer():154] handle defer: 5
2022-03-02 08:48:42,848 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: defer
2022-03-02 08:48:42,849 INFO SenderThread:254137 [sender.py:send_request_defer():388] handle sender defer: 5
2022-03-02 08:48:42,849 INFO SenderThread:254137 [dir_watcher.py:finish():283] shutting down directory watcher
2022-03-02 08:48:42,949 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:48:43,225 INFO Thread-8 :254137 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/wandb-summary.json
2022-03-02 08:48:43,226 INFO SenderThread:254137 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/output.log
2022-03-02 08:48:43,227 INFO SenderThread:254137 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/config.yaml
2022-03-02 08:48:43,227 INFO SenderThread:254137 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files
2022-03-02 08:48:43,227 INFO SenderThread:254137 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/wandb-metadata.json wandb-metadata.json
2022-03-02 08:48:43,227 INFO SenderThread:254137 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/output.log output.log
2022-03-02 08:48:43,227 INFO SenderThread:254137 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/wandb-summary.json wandb-summary.json
2022-03-02 08:48:43,227 INFO SenderThread:254137 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/requirements.txt requirements.txt
2022-03-02 08:48:43,228 INFO SenderThread:254137 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/config.yaml config.yaml
2022-03-02 08:48:43,230 INFO SenderThread:254137 [sender.py:transition_state():392] send defer: 6
2022-03-02 08:48:43,231 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:48:43,231 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:48:43,231 INFO HandlerThread:254137 [handler.py:handle_request_defer():154] handle defer: 6
2022-03-02 08:48:43,234 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: defer
2022-03-02 08:48:43,234 INFO SenderThread:254137 [sender.py:send_request_defer():388] handle sender defer: 6
2022-03-02 08:48:43,234 INFO SenderThread:254137 [file_pusher.py:finish():177] shutting down file pusher
2022-03-02 08:48:43,339 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:48:43,339 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:48:43,440 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:48:43,441 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:48:43,497 INFO Thread-13 :254137 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/wandb-summary.json
2022-03-02 08:48:43,498 INFO Thread-14 :254137 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/requirements.txt
2022-03-02 08:48:43,502 INFO Thread-12 :254137 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/output.log
2022-03-02 08:48:43,533 INFO Thread-15 :254137 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/files/config.yaml
2022-03-02 08:48:43,542 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:48:43,542 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:48:43,644 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:48:43,644 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:48:43,733 INFO Thread-7 :254137 [sender.py:transition_state():392] send defer: 7
2022-03-02 08:48:43,734 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:48:43,734 INFO HandlerThread:254137 [handler.py:handle_request_defer():154] handle defer: 7
2022-03-02 08:48:43,734 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: defer
2022-03-02 08:48:43,734 INFO SenderThread:254137 [sender.py:send_request_defer():388] handle sender defer: 7
2022-03-02 08:48:43,745 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:48:43,905 INFO SenderThread:254137 [sender.py:transition_state():392] send defer: 8
2022-03-02 08:48:43,905 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:48:43,906 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:48:43,906 INFO HandlerThread:254137 [handler.py:handle_request_defer():154] handle defer: 8
2022-03-02 08:48:43,906 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: defer
2022-03-02 08:48:43,906 INFO SenderThread:254137 [sender.py:send_request_defer():388] handle sender defer: 8
2022-03-02 08:48:43,906 INFO SenderThread:254137 [sender.py:transition_state():392] send defer: 9
2022-03-02 08:48:43,907 DEBUG SenderThread:254137 [sender.py:send():235] send: final
2022-03-02 08:48:43,907 DEBUG SenderThread:254137 [sender.py:send():235] send: footer
2022-03-02 08:48:43,907 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:48:43,907 INFO HandlerThread:254137 [handler.py:handle_request_defer():154] handle defer: 9
2022-03-02 08:48:43,907 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: defer
2022-03-02 08:48:43,908 INFO SenderThread:254137 [sender.py:send_request_defer():388] handle sender defer: 9
2022-03-02 08:48:44,007 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:48:44,007 DEBUG SenderThread:254137 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:48:44,007 INFO SenderThread:254137 [file_pusher.py:join():182] waiting for file pusher
2022-03-02 08:48:44,067 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: get_summary
2022-03-02 08:48:44,068 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: sampled_history
2022-03-02 08:48:44,068 DEBUG HandlerThread:254137 [handler.py:handle_request():131] handle_request: shutdown
2022-03-02 08:48:44,068 INFO HandlerThread:254137 [handler.py:finish():739] shutting down handler
2022-03-02 08:48:44,907 INFO WriterThread:254137 [datastore.py:close():281] close: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_084833-2m6jtwtj/run-2m6jtwtj.wandb
2022-03-02 08:48:45,066 INFO SenderThread:254137 [sender.py:finish():1075] shutting down sender
2022-03-02 08:48:45,066 INFO SenderThread:254137 [file_pusher.py:finish():177] shutting down file pusher
2022-03-02 08:48:45,066 INFO SenderThread:254137 [file_pusher.py:join():182] waiting for file pusher
2022-03-02 08:48:45,068 INFO MainThread:254137 [internal.py:handle_exit():79] Internal process exited
|