File size: 17,440 Bytes
fc53a59 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 |
2022-03-02 08:50:01,203 INFO MainThread:254314 [internal.py:wandb_internal():89] W&B internal server running at pid: 254314, started at: 2022-03-02 08:50:01.203327
2022-03-02 08:50:01,205 INFO WriterThread:254314 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/run-3jbwcde6.wandb
2022-03-02 08:50:01,206 DEBUG HandlerThread:254314 [handler.py:handle_request():131] handle_request: check_version
2022-03-02 08:50:01,207 DEBUG SenderThread:254314 [sender.py:send():235] send: header
2022-03-02 08:50:01,207 DEBUG SenderThread:254314 [sender.py:send_request():249] send_request: check_version
2022-03-02 08:50:01,273 DEBUG SenderThread:254314 [sender.py:send():235] send: run
2022-03-02 08:50:01,361 INFO SenderThread:254314 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files
2022-03-02 08:50:01,361 INFO SenderThread:254314 [sender.py:_start_run_threads():809] run started: 3jbwcde6 with start time 1646211000
2022-03-02 08:50:01,361 DEBUG SenderThread:254314 [sender.py:send():235] send: summary
2022-03-02 08:50:01,361 INFO SenderThread:254314 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-02 08:50:01,362 DEBUG HandlerThread:254314 [handler.py:handle_request():131] handle_request: run_start
2022-03-02 08:50:01,366 DEBUG HandlerThread:254314 [meta.py:__init__():36] meta init
2022-03-02 08:50:01,367 DEBUG HandlerThread:254314 [meta.py:__init__():50] meta init done
2022-03-02 08:50:01,367 DEBUG HandlerThread:254314 [meta.py:probe():210] probe
2022-03-02 08:50:01,373 DEBUG HandlerThread:254314 [meta.py:_setup_git():200] setup git
2022-03-02 08:50:01,387 DEBUG HandlerThread:254314 [meta.py:_setup_git():207] setup git done
2022-03-02 08:50:01,388 DEBUG HandlerThread:254314 [meta.py:_save_pip():54] save pip
2022-03-02 08:50:01,388 DEBUG HandlerThread:254314 [meta.py:_save_pip():68] save pip done
2022-03-02 08:50:01,388 DEBUG HandlerThread:254314 [meta.py:probe():248] probe done
2022-03-02 08:50:01,468 DEBUG SenderThread:254314 [sender.py:send():235] send: files
2022-03-02 08:50:01,468 INFO SenderThread:254314 [sender.py:_save_file():944] saving file wandb-metadata.json with policy now
2022-03-02 08:50:01,473 DEBUG HandlerThread:254314 [handler.py:handle_request():131] handle_request: stop_status
2022-03-02 08:50:01,473 DEBUG SenderThread:254314 [sender.py:send_request():249] send_request: stop_status
2022-03-02 08:50:01,520 DEBUG SenderThread:254314 [sender.py:send():235] send: config
2022-03-02 08:50:01,521 DEBUG SenderThread:254314 [sender.py:send():235] send: metric
2022-03-02 08:50:01,522 DEBUG SenderThread:254314 [sender.py:send():235] send: metric
2022-03-02 08:50:01,522 WARNING SenderThread:254314 [sender.py:send_metric():902] Seen metric with glob (shouldnt happen)
2022-03-02 08:50:01,741 INFO Thread-11 :254314 [upload_job.py:push():137] Uploaded file /tmp/tmpfcch1qofwandb/3oinsdv0-wandb-metadata.json
2022-03-02 08:50:02,363 INFO Thread-8 :254314 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files/wandb-metadata.json
2022-03-02 08:50:02,363 INFO Thread-8 :254314 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files/wandb-summary.json
2022-03-02 08:50:02,363 INFO Thread-8 :254314 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files/output.log
2022-03-02 08:50:02,363 INFO Thread-8 :254314 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files/requirements.txt
2022-03-02 08:50:04,362 INFO Thread-8 :254314 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files/output.log
2022-03-02 08:50:08,364 INFO Thread-8 :254314 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files/output.log
2022-03-02 08:50:09,886 DEBUG HandlerThread:254314 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:50:09,886 DEBUG SenderThread:254314 [sender.py:send():235] send: telemetry
2022-03-02 08:50:09,887 DEBUG SenderThread:254314 [sender.py:send():235] send: exit
2022-03-02 08:50:09,887 INFO SenderThread:254314 [sender.py:send_exit():371] handling exit code: 1
2022-03-02 08:50:09,887 INFO SenderThread:254314 [sender.py:send_exit():373] handling runtime: 8
2022-03-02 08:50:09,887 INFO SenderThread:254314 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-02 08:50:09,887 INFO SenderThread:254314 [sender.py:send_exit():379] send defer
2022-03-02 08:50:09,888 DEBUG SenderThread:254314 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:50:09,888 DEBUG HandlerThread:254314 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:50:09,888 INFO HandlerThread:254314 [handler.py:handle_request_defer():154] handle defer: 0
2022-03-02 08:50:09,888 DEBUG SenderThread:254314 [sender.py:send_request():249] send_request: defer
2022-03-02 08:50:09,888 INFO SenderThread:254314 [sender.py:send_request_defer():388] handle sender defer: 0
2022-03-02 08:50:09,889 INFO SenderThread:254314 [sender.py:transition_state():392] send defer: 1
2022-03-02 08:50:09,889 DEBUG HandlerThread:254314 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:50:09,889 INFO HandlerThread:254314 [handler.py:handle_request_defer():154] handle defer: 1
2022-03-02 08:50:09,992 DEBUG SenderThread:254314 [sender.py:send_request():249] send_request: defer
2022-03-02 08:50:09,992 DEBUG HandlerThread:254314 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:50:09,992 INFO SenderThread:254314 [sender.py:send_request_defer():388] handle sender defer: 1
2022-03-02 08:50:09,993 INFO SenderThread:254314 [sender.py:transition_state():392] send defer: 2
2022-03-02 08:50:09,993 DEBUG SenderThread:254314 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:50:09,993 DEBUG SenderThread:254314 [sender.py:send():235] send: stats
2022-03-02 08:50:09,994 DEBUG HandlerThread:254314 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:50:09,994 INFO HandlerThread:254314 [handler.py:handle_request_defer():154] handle defer: 2
2022-03-02 08:50:09,994 DEBUG SenderThread:254314 [sender.py:send_request():249] send_request: defer
2022-03-02 08:50:09,995 INFO SenderThread:254314 [sender.py:send_request_defer():388] handle sender defer: 2
2022-03-02 08:50:09,995 INFO SenderThread:254314 [sender.py:transition_state():392] send defer: 3
2022-03-02 08:50:09,995 DEBUG HandlerThread:254314 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:50:09,995 INFO HandlerThread:254314 [handler.py:handle_request_defer():154] handle defer: 3
2022-03-02 08:50:09,995 DEBUG SenderThread:254314 [sender.py:send():235] send: summary
2022-03-02 08:50:09,995 INFO SenderThread:254314 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-02 08:50:09,996 DEBUG SenderThread:254314 [sender.py:send_request():249] send_request: defer
2022-03-02 08:50:09,996 INFO SenderThread:254314 [sender.py:send_request_defer():388] handle sender defer: 3
2022-03-02 08:50:09,996 INFO SenderThread:254314 [sender.py:transition_state():392] send defer: 4
2022-03-02 08:50:09,996 DEBUG HandlerThread:254314 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:50:09,996 INFO HandlerThread:254314 [handler.py:handle_request_defer():154] handle defer: 4
2022-03-02 08:50:09,996 DEBUG SenderThread:254314 [sender.py:send_request():249] send_request: defer
2022-03-02 08:50:09,996 INFO SenderThread:254314 [sender.py:send_request_defer():388] handle sender defer: 4
2022-03-02 08:50:10,100 DEBUG HandlerThread:254314 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:50:10,126 INFO SenderThread:254314 [sender.py:transition_state():392] send defer: 5
2022-03-02 08:50:10,126 DEBUG SenderThread:254314 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:50:10,127 DEBUG HandlerThread:254314 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:50:10,127 INFO HandlerThread:254314 [handler.py:handle_request_defer():154] handle defer: 5
2022-03-02 08:50:10,127 DEBUG SenderThread:254314 [sender.py:send_request():249] send_request: defer
2022-03-02 08:50:10,127 INFO SenderThread:254314 [sender.py:send_request_defer():388] handle sender defer: 5
2022-03-02 08:50:10,127 INFO SenderThread:254314 [dir_watcher.py:finish():283] shutting down directory watcher
2022-03-02 08:50:10,228 DEBUG HandlerThread:254314 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:50:10,364 INFO Thread-8 :254314 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files/wandb-summary.json
2022-03-02 08:50:10,365 INFO SenderThread:254314 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files/output.log
2022-03-02 08:50:10,366 INFO SenderThread:254314 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files/config.yaml
2022-03-02 08:50:10,366 INFO SenderThread:254314 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files
2022-03-02 08:50:10,366 INFO SenderThread:254314 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files/wandb-metadata.json wandb-metadata.json
2022-03-02 08:50:10,366 INFO SenderThread:254314 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files/output.log output.log
2022-03-02 08:50:10,366 INFO SenderThread:254314 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files/wandb-summary.json wandb-summary.json
2022-03-02 08:50:10,367 INFO SenderThread:254314 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files/requirements.txt requirements.txt
2022-03-02 08:50:10,372 INFO SenderThread:254314 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files/config.yaml config.yaml
2022-03-02 08:50:10,375 INFO SenderThread:254314 [sender.py:transition_state():392] send defer: 6
2022-03-02 08:50:10,376 DEBUG SenderThread:254314 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:50:10,376 DEBUG HandlerThread:254314 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:50:10,377 INFO HandlerThread:254314 [handler.py:handle_request_defer():154] handle defer: 6
2022-03-02 08:50:10,379 DEBUG SenderThread:254314 [sender.py:send_request():249] send_request: defer
2022-03-02 08:50:10,379 INFO SenderThread:254314 [sender.py:send_request_defer():388] handle sender defer: 6
2022-03-02 08:50:10,379 INFO SenderThread:254314 [file_pusher.py:finish():177] shutting down file pusher
2022-03-02 08:50:10,477 DEBUG HandlerThread:254314 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:50:10,478 DEBUG SenderThread:254314 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:50:10,579 DEBUG HandlerThread:254314 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:50:10,579 DEBUG SenderThread:254314 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:50:10,641 INFO Thread-12 :254314 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files/output.log
2022-03-02 08:50:10,642 INFO Thread-14 :254314 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files/requirements.txt
2022-03-02 08:50:10,656 INFO Thread-15 :254314 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files/config.yaml
2022-03-02 08:50:10,681 DEBUG HandlerThread:254314 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:50:10,681 DEBUG SenderThread:254314 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:50:10,700 INFO Thread-13 :254314 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/files/wandb-summary.json
2022-03-02 08:50:10,782 DEBUG HandlerThread:254314 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:50:10,782 DEBUG SenderThread:254314 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:50:10,884 DEBUG HandlerThread:254314 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:50:10,884 DEBUG SenderThread:254314 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:50:10,901 INFO Thread-7 :254314 [sender.py:transition_state():392] send defer: 7
2022-03-02 08:50:10,901 DEBUG HandlerThread:254314 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:50:10,901 INFO HandlerThread:254314 [handler.py:handle_request_defer():154] handle defer: 7
2022-03-02 08:50:10,902 DEBUG SenderThread:254314 [sender.py:send_request():249] send_request: defer
2022-03-02 08:50:10,902 INFO SenderThread:254314 [sender.py:send_request_defer():388] handle sender defer: 7
2022-03-02 08:50:10,985 DEBUG HandlerThread:254314 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:50:11,046 INFO SenderThread:254314 [sender.py:transition_state():392] send defer: 8
2022-03-02 08:50:11,047 DEBUG SenderThread:254314 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:50:11,047 DEBUG HandlerThread:254314 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:50:11,047 INFO HandlerThread:254314 [handler.py:handle_request_defer():154] handle defer: 8
2022-03-02 08:50:11,047 DEBUG SenderThread:254314 [sender.py:send_request():249] send_request: defer
2022-03-02 08:50:11,047 INFO SenderThread:254314 [sender.py:send_request_defer():388] handle sender defer: 8
2022-03-02 08:50:11,047 INFO SenderThread:254314 [sender.py:transition_state():392] send defer: 9
2022-03-02 08:50:11,048 DEBUG SenderThread:254314 [sender.py:send():235] send: final
2022-03-02 08:50:11,048 DEBUG SenderThread:254314 [sender.py:send():235] send: footer
2022-03-02 08:50:11,048 DEBUG HandlerThread:254314 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:50:11,048 INFO HandlerThread:254314 [handler.py:handle_request_defer():154] handle defer: 9
2022-03-02 08:50:11,049 DEBUG SenderThread:254314 [sender.py:send_request():249] send_request: defer
2022-03-02 08:50:11,049 INFO SenderThread:254314 [sender.py:send_request_defer():388] handle sender defer: 9
2022-03-02 08:50:11,148 DEBUG HandlerThread:254314 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:50:11,148 DEBUG SenderThread:254314 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:50:11,148 INFO SenderThread:254314 [file_pusher.py:join():182] waiting for file pusher
2022-03-02 08:50:11,233 DEBUG HandlerThread:254314 [handler.py:handle_request():131] handle_request: get_summary
2022-03-02 08:50:11,233 DEBUG HandlerThread:254314 [handler.py:handle_request():131] handle_request: sampled_history
2022-03-02 08:50:11,234 DEBUG HandlerThread:254314 [handler.py:handle_request():131] handle_request: shutdown
2022-03-02 08:50:11,234 INFO HandlerThread:254314 [handler.py:finish():739] shutting down handler
2022-03-02 08:50:12,049 INFO WriterThread:254314 [datastore.py:close():281] close: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085000-3jbwcde6/run-3jbwcde6.wandb
2022-03-02 08:50:12,232 INFO SenderThread:254314 [sender.py:finish():1075] shutting down sender
2022-03-02 08:50:12,232 INFO SenderThread:254314 [file_pusher.py:finish():177] shutting down file pusher
2022-03-02 08:50:12,232 INFO SenderThread:254314 [file_pusher.py:join():182] waiting for file pusher
2022-03-02 08:50:12,234 INFO MainThread:254314 [internal.py:handle_exit():79] Internal process exited
|