File size: 17,220 Bytes
a5afdd5 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 |
2022-03-01 01:28:17,477 INFO MainThread:236675 [internal.py:wandb_internal():89] W&B internal server running at pid: 236675, started at: 2022-03-01 01:28:17.477147
2022-03-01 01:28:17,479 DEBUG HandlerThread:236675 [handler.py:handle_request():131] handle_request: check_version
2022-03-01 01:28:17,479 INFO WriterThread:236675 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/run-3bh80nzn.wandb
2022-03-01 01:28:17,481 DEBUG SenderThread:236675 [sender.py:send():235] send: header
2022-03-01 01:28:17,481 DEBUG SenderThread:236675 [sender.py:send_request():249] send_request: check_version
2022-03-01 01:28:17,547 DEBUG SenderThread:236675 [sender.py:send():235] send: run
2022-03-01 01:28:17,636 INFO SenderThread:236675 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files
2022-03-01 01:28:17,636 INFO SenderThread:236675 [sender.py:_start_run_threads():809] run started: 3bh80nzn with start time 1646098096
2022-03-01 01:28:17,636 DEBUG SenderThread:236675 [sender.py:send():235] send: summary
2022-03-01 01:28:17,636 INFO SenderThread:236675 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 01:28:17,637 DEBUG HandlerThread:236675 [handler.py:handle_request():131] handle_request: run_start
2022-03-01 01:28:17,641 DEBUG HandlerThread:236675 [meta.py:__init__():36] meta init
2022-03-01 01:28:17,641 DEBUG HandlerThread:236675 [meta.py:__init__():50] meta init done
2022-03-01 01:28:17,642 DEBUG HandlerThread:236675 [meta.py:probe():210] probe
2022-03-01 01:28:17,648 DEBUG HandlerThread:236675 [meta.py:_setup_git():200] setup git
2022-03-01 01:28:17,663 DEBUG HandlerThread:236675 [meta.py:_setup_git():207] setup git done
2022-03-01 01:28:17,663 DEBUG HandlerThread:236675 [meta.py:_save_pip():54] save pip
2022-03-01 01:28:17,663 DEBUG HandlerThread:236675 [meta.py:_save_pip():68] save pip done
2022-03-01 01:28:17,664 DEBUG HandlerThread:236675 [meta.py:probe():248] probe done
2022-03-01 01:28:17,744 DEBUG SenderThread:236675 [sender.py:send():235] send: files
2022-03-01 01:28:17,744 INFO SenderThread:236675 [sender.py:_save_file():944] saving file wandb-metadata.json with policy now
2022-03-01 01:28:17,749 DEBUG HandlerThread:236675 [handler.py:handle_request():131] handle_request: stop_status
2022-03-01 01:28:17,749 DEBUG SenderThread:236675 [sender.py:send_request():249] send_request: stop_status
2022-03-01 01:28:17,783 DEBUG SenderThread:236675 [sender.py:send():235] send: config
2022-03-01 01:28:17,784 DEBUG SenderThread:236675 [sender.py:send():235] send: metric
2022-03-01 01:28:17,784 DEBUG SenderThread:236675 [sender.py:send():235] send: metric
2022-03-01 01:28:17,784 WARNING SenderThread:236675 [sender.py:send_metric():902] Seen metric with glob (shouldnt happen)
2022-03-01 01:28:18,011 INFO Thread-11 :236675 [upload_job.py:push():137] Uploaded file /tmp/tmpb_wel_s6wandb/1ng2jof2-wandb-metadata.json
2022-03-01 01:28:18,638 INFO Thread-8 :236675 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files/wandb-metadata.json
2022-03-01 01:28:18,638 INFO Thread-8 :236675 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files/requirements.txt
2022-03-01 01:28:18,638 INFO Thread-8 :236675 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files/wandb-summary.json
2022-03-01 01:28:18,638 INFO Thread-8 :236675 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files/output.log
2022-03-01 01:28:20,637 INFO Thread-8 :236675 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files/output.log
2022-03-01 01:28:24,639 INFO Thread-8 :236675 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files/output.log
2022-03-01 01:28:25,989 DEBUG SenderThread:236675 [sender.py:send():235] send: telemetry
2022-03-01 01:28:25,989 DEBUG SenderThread:236675 [sender.py:send():235] send: exit
2022-03-01 01:28:25,989 INFO SenderThread:236675 [sender.py:send_exit():371] handling exit code: 1
2022-03-01 01:28:25,989 INFO SenderThread:236675 [sender.py:send_exit():373] handling runtime: 8
2022-03-01 01:28:25,990 DEBUG HandlerThread:236675 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:28:25,990 INFO SenderThread:236675 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 01:28:25,991 INFO SenderThread:236675 [sender.py:send_exit():379] send defer
2022-03-01 01:28:25,991 DEBUG SenderThread:236675 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:28:25,991 DEBUG HandlerThread:236675 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:28:25,991 INFO HandlerThread:236675 [handler.py:handle_request_defer():154] handle defer: 0
2022-03-01 01:28:25,992 DEBUG SenderThread:236675 [sender.py:send_request():249] send_request: defer
2022-03-01 01:28:25,992 INFO SenderThread:236675 [sender.py:send_request_defer():388] handle sender defer: 0
2022-03-01 01:28:25,992 INFO SenderThread:236675 [sender.py:transition_state():392] send defer: 1
2022-03-01 01:28:25,992 DEBUG HandlerThread:236675 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:28:25,992 INFO HandlerThread:236675 [handler.py:handle_request_defer():154] handle defer: 1
2022-03-01 01:28:26,073 DEBUG SenderThread:236675 [sender.py:send_request():249] send_request: defer
2022-03-01 01:28:26,073 INFO SenderThread:236675 [sender.py:send_request_defer():388] handle sender defer: 1
2022-03-01 01:28:26,073 INFO SenderThread:236675 [sender.py:transition_state():392] send defer: 2
2022-03-01 01:28:26,074 DEBUG HandlerThread:236675 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:28:26,074 INFO HandlerThread:236675 [handler.py:handle_request_defer():154] handle defer: 2
2022-03-01 01:28:26,074 DEBUG SenderThread:236675 [sender.py:send():235] send: stats
2022-03-01 01:28:26,075 DEBUG SenderThread:236675 [sender.py:send_request():249] send_request: defer
2022-03-01 01:28:26,075 INFO SenderThread:236675 [sender.py:send_request_defer():388] handle sender defer: 2
2022-03-01 01:28:26,075 INFO SenderThread:236675 [sender.py:transition_state():392] send defer: 3
2022-03-01 01:28:26,075 DEBUG HandlerThread:236675 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:28:26,075 INFO HandlerThread:236675 [handler.py:handle_request_defer():154] handle defer: 3
2022-03-01 01:28:26,075 DEBUG SenderThread:236675 [sender.py:send():235] send: summary
2022-03-01 01:28:26,076 INFO SenderThread:236675 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 01:28:26,076 DEBUG SenderThread:236675 [sender.py:send_request():249] send_request: defer
2022-03-01 01:28:26,076 INFO SenderThread:236675 [sender.py:send_request_defer():388] handle sender defer: 3
2022-03-01 01:28:26,076 INFO SenderThread:236675 [sender.py:transition_state():392] send defer: 4
2022-03-01 01:28:26,076 DEBUG HandlerThread:236675 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:28:26,076 INFO HandlerThread:236675 [handler.py:handle_request_defer():154] handle defer: 4
2022-03-01 01:28:26,077 DEBUG SenderThread:236675 [sender.py:send_request():249] send_request: defer
2022-03-01 01:28:26,077 INFO SenderThread:236675 [sender.py:send_request_defer():388] handle sender defer: 4
2022-03-01 01:28:26,093 DEBUG HandlerThread:236675 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:28:26,179 INFO SenderThread:236675 [sender.py:transition_state():392] send defer: 5
2022-03-01 01:28:26,179 DEBUG SenderThread:236675 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:28:26,180 DEBUG HandlerThread:236675 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:28:26,180 INFO HandlerThread:236675 [handler.py:handle_request_defer():154] handle defer: 5
2022-03-01 01:28:26,180 DEBUG SenderThread:236675 [sender.py:send_request():249] send_request: defer
2022-03-01 01:28:26,180 INFO SenderThread:236675 [sender.py:send_request_defer():388] handle sender defer: 5
2022-03-01 01:28:26,180 INFO SenderThread:236675 [dir_watcher.py:finish():283] shutting down directory watcher
2022-03-01 01:28:26,281 DEBUG HandlerThread:236675 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:28:26,639 INFO Thread-8 :236675 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files/wandb-summary.json
2022-03-01 01:28:26,640 INFO SenderThread:236675 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files/output.log
2022-03-01 01:28:26,640 INFO SenderThread:236675 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files/config.yaml
2022-03-01 01:28:26,640 INFO SenderThread:236675 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files
2022-03-01 01:28:26,641 INFO SenderThread:236675 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files/wandb-metadata.json wandb-metadata.json
2022-03-01 01:28:26,641 INFO SenderThread:236675 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files/output.log output.log
2022-03-01 01:28:26,641 INFO SenderThread:236675 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files/wandb-summary.json wandb-summary.json
2022-03-01 01:28:26,644 INFO SenderThread:236675 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files/requirements.txt requirements.txt
2022-03-01 01:28:26,644 INFO SenderThread:236675 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files/config.yaml config.yaml
2022-03-01 01:28:26,644 INFO SenderThread:236675 [sender.py:transition_state():392] send defer: 6
2022-03-01 01:28:26,645 DEBUG SenderThread:236675 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:28:26,648 DEBUG HandlerThread:236675 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:28:26,651 INFO HandlerThread:236675 [handler.py:handle_request_defer():154] handle defer: 6
2022-03-01 01:28:26,653 DEBUG SenderThread:236675 [sender.py:send_request():249] send_request: defer
2022-03-01 01:28:26,653 INFO SenderThread:236675 [sender.py:send_request_defer():388] handle sender defer: 6
2022-03-01 01:28:26,654 INFO SenderThread:236675 [file_pusher.py:finish():177] shutting down file pusher
2022-03-01 01:28:26,752 DEBUG HandlerThread:236675 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:28:26,752 DEBUG SenderThread:236675 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:28:26,854 DEBUG HandlerThread:236675 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:28:26,854 DEBUG SenderThread:236675 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:28:26,905 INFO Thread-14 :236675 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files/requirements.txt
2022-03-01 01:28:26,912 INFO Thread-13 :236675 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files/wandb-summary.json
2022-03-01 01:28:26,932 INFO Thread-12 :236675 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files/output.log
2022-03-01 01:28:26,955 DEBUG HandlerThread:236675 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:28:26,955 DEBUG SenderThread:236675 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:28:26,961 INFO Thread-15 :236675 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/files/config.yaml
2022-03-01 01:28:27,057 DEBUG HandlerThread:236675 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:28:27,057 DEBUG SenderThread:236675 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:28:27,158 DEBUG HandlerThread:236675 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:28:27,158 DEBUG SenderThread:236675 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:28:27,162 INFO Thread-7 :236675 [sender.py:transition_state():392] send defer: 7
2022-03-01 01:28:27,162 DEBUG HandlerThread:236675 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:28:27,162 INFO HandlerThread:236675 [handler.py:handle_request_defer():154] handle defer: 7
2022-03-01 01:28:27,162 DEBUG SenderThread:236675 [sender.py:send_request():249] send_request: defer
2022-03-01 01:28:27,162 INFO SenderThread:236675 [sender.py:send_request_defer():388] handle sender defer: 7
2022-03-01 01:28:27,259 DEBUG HandlerThread:236675 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:28:28,140 INFO SenderThread:236675 [sender.py:transition_state():392] send defer: 8
2022-03-01 01:28:28,140 DEBUG SenderThread:236675 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:28:28,141 DEBUG HandlerThread:236675 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:28:28,141 INFO HandlerThread:236675 [handler.py:handle_request_defer():154] handle defer: 8
2022-03-01 01:28:28,141 DEBUG SenderThread:236675 [sender.py:send_request():249] send_request: defer
2022-03-01 01:28:28,141 INFO SenderThread:236675 [sender.py:send_request_defer():388] handle sender defer: 8
2022-03-01 01:28:28,141 INFO SenderThread:236675 [sender.py:transition_state():392] send defer: 9
2022-03-01 01:28:28,142 DEBUG SenderThread:236675 [sender.py:send():235] send: final
2022-03-01 01:28:28,142 DEBUG HandlerThread:236675 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:28:28,142 INFO HandlerThread:236675 [handler.py:handle_request_defer():154] handle defer: 9
2022-03-01 01:28:28,142 DEBUG SenderThread:236675 [sender.py:send():235] send: footer
2022-03-01 01:28:28,142 DEBUG SenderThread:236675 [sender.py:send_request():249] send_request: defer
2022-03-01 01:28:28,142 INFO SenderThread:236675 [sender.py:send_request_defer():388] handle sender defer: 9
2022-03-01 01:28:28,241 DEBUG HandlerThread:236675 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:28:28,242 DEBUG SenderThread:236675 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:28:28,242 INFO SenderThread:236675 [file_pusher.py:join():182] waiting for file pusher
2022-03-01 01:28:28,300 DEBUG HandlerThread:236675 [handler.py:handle_request():131] handle_request: get_summary
2022-03-01 01:28:28,301 DEBUG HandlerThread:236675 [handler.py:handle_request():131] handle_request: sampled_history
2022-03-01 01:28:28,302 DEBUG HandlerThread:236675 [handler.py:handle_request():131] handle_request: shutdown
2022-03-01 01:28:28,302 INFO HandlerThread:236675 [handler.py:finish():739] shutting down handler
2022-03-01 01:28:29,142 INFO WriterThread:236675 [datastore.py:close():281] close: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012816-3bh80nzn/run-3bh80nzn.wandb
2022-03-01 01:28:29,299 INFO SenderThread:236675 [sender.py:finish():1075] shutting down sender
2022-03-01 01:28:29,300 INFO SenderThread:236675 [file_pusher.py:finish():177] shutting down file pusher
2022-03-01 01:28:29,300 INFO SenderThread:236675 [file_pusher.py:join():182] waiting for file pusher
2022-03-01 01:28:29,302 INFO MainThread:236675 [internal.py:handle_exit():79] Internal process exited
|