File size: 17,229 Bytes
bcb2332 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 |
2022-03-03 00:42:16,953 INFO MainThread:267379 [internal.py:wandb_internal():89] W&B internal server running at pid: 267379, started at: 2022-03-03 00:42:16.952980
2022-03-03 00:42:16,956 DEBUG HandlerThread:267379 [handler.py:handle_request():131] handle_request: check_version
2022-03-03 00:42:16,956 INFO WriterThread:267379 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/run-2ttcbe3s.wandb
2022-03-03 00:42:16,957 DEBUG SenderThread:267379 [sender.py:send():235] send: header
2022-03-03 00:42:16,957 DEBUG SenderThread:267379 [sender.py:send_request():249] send_request: check_version
2022-03-03 00:42:17,024 DEBUG SenderThread:267379 [sender.py:send():235] send: run
2022-03-03 00:42:17,144 INFO SenderThread:267379 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files
2022-03-03 00:42:17,144 INFO SenderThread:267379 [sender.py:_start_run_threads():809] run started: 2ttcbe3s with start time 1646268136
2022-03-03 00:42:17,145 DEBUG SenderThread:267379 [sender.py:send():235] send: summary
2022-03-03 00:42:17,145 INFO SenderThread:267379 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-03 00:42:17,145 DEBUG HandlerThread:267379 [handler.py:handle_request():131] handle_request: run_start
2022-03-03 00:42:17,150 DEBUG HandlerThread:267379 [meta.py:__init__():36] meta init
2022-03-03 00:42:17,150 DEBUG HandlerThread:267379 [meta.py:__init__():50] meta init done
2022-03-03 00:42:17,150 DEBUG HandlerThread:267379 [meta.py:probe():210] probe
2022-03-03 00:42:17,156 DEBUG HandlerThread:267379 [meta.py:_setup_git():200] setup git
2022-03-03 00:42:17,172 DEBUG HandlerThread:267379 [meta.py:_setup_git():207] setup git done
2022-03-03 00:42:17,172 DEBUG HandlerThread:267379 [meta.py:_save_pip():54] save pip
2022-03-03 00:42:17,173 DEBUG HandlerThread:267379 [meta.py:_save_pip():68] save pip done
2022-03-03 00:42:17,173 DEBUG HandlerThread:267379 [meta.py:probe():248] probe done
2022-03-03 00:42:17,254 DEBUG SenderThread:267379 [sender.py:send():235] send: files
2022-03-03 00:42:17,254 INFO SenderThread:267379 [sender.py:_save_file():944] saving file wandb-metadata.json with policy now
2022-03-03 00:42:17,259 DEBUG HandlerThread:267379 [handler.py:handle_request():131] handle_request: stop_status
2022-03-03 00:42:17,259 DEBUG SenderThread:267379 [sender.py:send_request():249] send_request: stop_status
2022-03-03 00:42:17,295 DEBUG SenderThread:267379 [sender.py:send():235] send: config
2022-03-03 00:42:17,296 DEBUG SenderThread:267379 [sender.py:send():235] send: metric
2022-03-03 00:42:17,296 DEBUG SenderThread:267379 [sender.py:send():235] send: metric
2022-03-03 00:42:17,296 WARNING SenderThread:267379 [sender.py:send_metric():902] Seen metric with glob (shouldnt happen)
2022-03-03 00:42:17,551 INFO Thread-11 :267379 [upload_job.py:push():137] Uploaded file /tmp/tmpamllbtqswandb/2k16no24-wandb-metadata.json
2022-03-03 00:42:18,148 INFO Thread-8 :267379 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/wandb-metadata.json
2022-03-03 00:42:18,148 INFO Thread-8 :267379 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/wandb-summary.json
2022-03-03 00:42:18,148 INFO Thread-8 :267379 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/requirements.txt
2022-03-03 00:42:18,148 INFO Thread-8 :267379 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/output.log
2022-03-03 00:42:20,146 INFO Thread-8 :267379 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/output.log
2022-03-03 00:42:24,147 INFO Thread-8 :267379 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/output.log
2022-03-03 00:42:26,115 DEBUG HandlerThread:267379 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:42:26,115 DEBUG SenderThread:267379 [sender.py:send():235] send: telemetry
2022-03-03 00:42:26,115 DEBUG SenderThread:267379 [sender.py:send():235] send: exit
2022-03-03 00:42:26,116 INFO SenderThread:267379 [sender.py:send_exit():371] handling exit code: 1
2022-03-03 00:42:26,116 INFO SenderThread:267379 [sender.py:send_exit():373] handling runtime: 8
2022-03-03 00:42:26,116 INFO SenderThread:267379 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-03 00:42:26,116 INFO SenderThread:267379 [sender.py:send_exit():379] send defer
2022-03-03 00:42:26,116 DEBUG SenderThread:267379 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:42:26,117 DEBUG HandlerThread:267379 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:42:26,117 INFO HandlerThread:267379 [handler.py:handle_request_defer():154] handle defer: 0
2022-03-03 00:42:26,117 DEBUG SenderThread:267379 [sender.py:send_request():249] send_request: defer
2022-03-03 00:42:26,117 INFO SenderThread:267379 [sender.py:send_request_defer():388] handle sender defer: 0
2022-03-03 00:42:26,117 INFO SenderThread:267379 [sender.py:transition_state():392] send defer: 1
2022-03-03 00:42:26,118 DEBUG HandlerThread:267379 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:42:26,118 INFO HandlerThread:267379 [handler.py:handle_request_defer():154] handle defer: 1
2022-03-03 00:42:26,148 INFO Thread-8 :267379 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/wandb-summary.json
2022-03-03 00:42:26,148 INFO Thread-8 :267379 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/output.log
2022-03-03 00:42:26,201 DEBUG SenderThread:267379 [sender.py:send_request():249] send_request: defer
2022-03-03 00:42:26,201 INFO SenderThread:267379 [sender.py:send_request_defer():388] handle sender defer: 1
2022-03-03 00:42:26,201 INFO SenderThread:267379 [sender.py:transition_state():392] send defer: 2
2022-03-03 00:42:26,202 DEBUG SenderThread:267379 [sender.py:send():235] send: stats
2022-03-03 00:42:26,202 DEBUG HandlerThread:267379 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:42:26,203 INFO HandlerThread:267379 [handler.py:handle_request_defer():154] handle defer: 2
2022-03-03 00:42:26,203 DEBUG SenderThread:267379 [sender.py:send_request():249] send_request: defer
2022-03-03 00:42:26,203 INFO SenderThread:267379 [sender.py:send_request_defer():388] handle sender defer: 2
2022-03-03 00:42:26,203 INFO SenderThread:267379 [sender.py:transition_state():392] send defer: 3
2022-03-03 00:42:26,203 DEBUG HandlerThread:267379 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:42:26,203 INFO HandlerThread:267379 [handler.py:handle_request_defer():154] handle defer: 3
2022-03-03 00:42:26,203 DEBUG SenderThread:267379 [sender.py:send():235] send: summary
2022-03-03 00:42:26,204 INFO SenderThread:267379 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-03 00:42:26,204 DEBUG SenderThread:267379 [sender.py:send_request():249] send_request: defer
2022-03-03 00:42:26,204 INFO SenderThread:267379 [sender.py:send_request_defer():388] handle sender defer: 3
2022-03-03 00:42:26,204 INFO SenderThread:267379 [sender.py:transition_state():392] send defer: 4
2022-03-03 00:42:26,204 DEBUG HandlerThread:267379 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:42:26,205 INFO HandlerThread:267379 [handler.py:handle_request_defer():154] handle defer: 4
2022-03-03 00:42:26,205 DEBUG SenderThread:267379 [sender.py:send_request():249] send_request: defer
2022-03-03 00:42:26,205 INFO SenderThread:267379 [sender.py:send_request_defer():388] handle sender defer: 4
2022-03-03 00:42:26,218 DEBUG HandlerThread:267379 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:42:26,313 INFO SenderThread:267379 [sender.py:transition_state():392] send defer: 5
2022-03-03 00:42:26,313 DEBUG SenderThread:267379 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:42:26,314 DEBUG HandlerThread:267379 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:42:26,314 INFO HandlerThread:267379 [handler.py:handle_request_defer():154] handle defer: 5
2022-03-03 00:42:26,314 DEBUG SenderThread:267379 [sender.py:send_request():249] send_request: defer
2022-03-03 00:42:26,315 INFO SenderThread:267379 [sender.py:send_request_defer():388] handle sender defer: 5
2022-03-03 00:42:26,315 INFO SenderThread:267379 [dir_watcher.py:finish():283] shutting down directory watcher
2022-03-03 00:42:26,415 DEBUG HandlerThread:267379 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:42:27,148 INFO Thread-8 :267379 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/wandb-summary.json
2022-03-03 00:42:27,149 INFO SenderThread:267379 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/config.yaml
2022-03-03 00:42:27,149 INFO SenderThread:267379 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files
2022-03-03 00:42:27,150 INFO SenderThread:267379 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/wandb-metadata.json wandb-metadata.json
2022-03-03 00:42:27,150 INFO SenderThread:267379 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/output.log output.log
2022-03-03 00:42:27,150 INFO SenderThread:267379 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/wandb-summary.json wandb-summary.json
2022-03-03 00:42:27,152 INFO SenderThread:267379 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/requirements.txt requirements.txt
2022-03-03 00:42:27,156 INFO SenderThread:267379 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/config.yaml config.yaml
2022-03-03 00:42:27,156 INFO SenderThread:267379 [sender.py:transition_state():392] send defer: 6
2022-03-03 00:42:27,156 DEBUG SenderThread:267379 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:42:27,157 DEBUG HandlerThread:267379 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:42:27,157 INFO HandlerThread:267379 [handler.py:handle_request_defer():154] handle defer: 6
2022-03-03 00:42:27,160 DEBUG SenderThread:267379 [sender.py:send_request():249] send_request: defer
2022-03-03 00:42:27,160 INFO SenderThread:267379 [sender.py:send_request_defer():388] handle sender defer: 6
2022-03-03 00:42:27,160 INFO SenderThread:267379 [file_pusher.py:finish():177] shutting down file pusher
2022-03-03 00:42:27,258 DEBUG HandlerThread:267379 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:42:27,258 DEBUG SenderThread:267379 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:42:27,360 DEBUG HandlerThread:267379 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:42:27,360 DEBUG SenderThread:267379 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:42:27,434 INFO Thread-12 :267379 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/output.log
2022-03-03 00:42:27,457 INFO Thread-13 :267379 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/wandb-summary.json
2022-03-03 00:42:27,460 INFO Thread-15 :267379 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/config.yaml
2022-03-03 00:42:27,462 INFO Thread-14 :267379 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/files/requirements.txt
2022-03-03 00:42:27,462 DEBUG HandlerThread:267379 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:42:27,463 DEBUG SenderThread:267379 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:42:27,565 DEBUG HandlerThread:267379 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:42:27,565 DEBUG SenderThread:267379 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:42:27,663 INFO Thread-7 :267379 [sender.py:transition_state():392] send defer: 7
2022-03-03 00:42:27,664 DEBUG HandlerThread:267379 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:42:27,664 INFO HandlerThread:267379 [handler.py:handle_request_defer():154] handle defer: 7
2022-03-03 00:42:27,664 DEBUG SenderThread:267379 [sender.py:send_request():249] send_request: defer
2022-03-03 00:42:27,664 INFO SenderThread:267379 [sender.py:send_request_defer():388] handle sender defer: 7
2022-03-03 00:42:27,667 DEBUG HandlerThread:267379 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:42:27,834 INFO SenderThread:267379 [sender.py:transition_state():392] send defer: 8
2022-03-03 00:42:27,834 DEBUG SenderThread:267379 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:42:27,835 DEBUG HandlerThread:267379 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:42:27,835 INFO HandlerThread:267379 [handler.py:handle_request_defer():154] handle defer: 8
2022-03-03 00:42:27,835 DEBUG SenderThread:267379 [sender.py:send_request():249] send_request: defer
2022-03-03 00:42:27,835 INFO SenderThread:267379 [sender.py:send_request_defer():388] handle sender defer: 8
2022-03-03 00:42:27,835 INFO SenderThread:267379 [sender.py:transition_state():392] send defer: 9
2022-03-03 00:42:27,836 DEBUG HandlerThread:267379 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:42:27,836 INFO HandlerThread:267379 [handler.py:handle_request_defer():154] handle defer: 9
2022-03-03 00:42:27,836 DEBUG SenderThread:267379 [sender.py:send():235] send: final
2022-03-03 00:42:27,836 DEBUG SenderThread:267379 [sender.py:send():235] send: footer
2022-03-03 00:42:27,836 DEBUG SenderThread:267379 [sender.py:send_request():249] send_request: defer
2022-03-03 00:42:27,836 INFO SenderThread:267379 [sender.py:send_request_defer():388] handle sender defer: 9
2022-03-03 00:42:27,936 DEBUG HandlerThread:267379 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:42:27,937 DEBUG SenderThread:267379 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:42:27,937 INFO SenderThread:267379 [file_pusher.py:join():182] waiting for file pusher
2022-03-03 00:42:28,000 DEBUG HandlerThread:267379 [handler.py:handle_request():131] handle_request: get_summary
2022-03-03 00:42:28,001 DEBUG HandlerThread:267379 [handler.py:handle_request():131] handle_request: sampled_history
2022-03-03 00:42:28,002 DEBUG HandlerThread:267379 [handler.py:handle_request():131] handle_request: shutdown
2022-03-03 00:42:28,002 INFO HandlerThread:267379 [handler.py:finish():739] shutting down handler
2022-03-03 00:42:28,836 INFO WriterThread:267379 [datastore.py:close():281] close: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004215-2ttcbe3s/run-2ttcbe3s.wandb
2022-03-03 00:42:28,999 INFO SenderThread:267379 [sender.py:finish():1075] shutting down sender
2022-03-03 00:42:28,999 INFO SenderThread:267379 [file_pusher.py:finish():177] shutting down file pusher
2022-03-03 00:42:28,999 INFO SenderThread:267379 [file_pusher.py:join():182] waiting for file pusher
2022-03-03 00:42:29,001 INFO MainThread:267379 [internal.py:handle_exit():79] Internal process exited
|