File size: 17,000 Bytes
53a9ef4 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 |
2022-03-01 17:13:29,716 INFO MainThread:245590 [internal.py:wandb_internal():89] W&B internal server running at pid: 245590, started at: 2022-03-01 17:13:29.715995
2022-03-01 17:13:29,718 DEBUG HandlerThread:245590 [handler.py:handle_request():131] handle_request: check_version
2022-03-01 17:13:29,718 INFO WriterThread:245590 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/run-194tzn4x.wandb
2022-03-01 17:13:29,720 DEBUG SenderThread:245590 [sender.py:send():235] send: header
2022-03-01 17:13:29,720 DEBUG SenderThread:245590 [sender.py:send_request():249] send_request: check_version
2022-03-01 17:13:29,786 DEBUG SenderThread:245590 [sender.py:send():235] send: run
2022-03-01 17:13:29,877 INFO SenderThread:245590 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files
2022-03-01 17:13:29,877 INFO SenderThread:245590 [sender.py:_start_run_threads():809] run started: 194tzn4x with start time 1646154808
2022-03-01 17:13:29,877 DEBUG SenderThread:245590 [sender.py:send():235] send: summary
2022-03-01 17:13:29,877 INFO SenderThread:245590 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 17:13:29,878 DEBUG HandlerThread:245590 [handler.py:handle_request():131] handle_request: run_start
2022-03-01 17:13:29,882 DEBUG HandlerThread:245590 [meta.py:__init__():36] meta init
2022-03-01 17:13:29,882 DEBUG HandlerThread:245590 [meta.py:__init__():50] meta init done
2022-03-01 17:13:29,882 DEBUG HandlerThread:245590 [meta.py:probe():210] probe
2022-03-01 17:13:29,889 DEBUG HandlerThread:245590 [meta.py:_setup_git():200] setup git
2022-03-01 17:13:29,904 DEBUG HandlerThread:245590 [meta.py:_setup_git():207] setup git done
2022-03-01 17:13:29,904 DEBUG HandlerThread:245590 [meta.py:_save_pip():54] save pip
2022-03-01 17:13:29,904 DEBUG HandlerThread:245590 [meta.py:_save_pip():68] save pip done
2022-03-01 17:13:29,905 DEBUG HandlerThread:245590 [meta.py:probe():248] probe done
2022-03-01 17:13:29,980 DEBUG SenderThread:245590 [sender.py:send():235] send: files
2022-03-01 17:13:29,980 INFO SenderThread:245590 [sender.py:_save_file():944] saving file wandb-metadata.json with policy now
2022-03-01 17:13:29,985 DEBUG HandlerThread:245590 [handler.py:handle_request():131] handle_request: stop_status
2022-03-01 17:13:29,985 DEBUG SenderThread:245590 [sender.py:send_request():249] send_request: stop_status
2022-03-01 17:13:30,015 DEBUG SenderThread:245590 [sender.py:send():235] send: config
2022-03-01 17:13:30,016 DEBUG SenderThread:245590 [sender.py:send():235] send: metric
2022-03-01 17:13:30,016 DEBUG SenderThread:245590 [sender.py:send():235] send: metric
2022-03-01 17:13:30,016 WARNING SenderThread:245590 [sender.py:send_metric():902] Seen metric with glob (shouldnt happen)
2022-03-01 17:13:30,247 INFO Thread-11 :245590 [upload_job.py:push():137] Uploaded file /tmp/tmpqzj81002wandb/2csiwq7p-wandb-metadata.json
2022-03-01 17:13:30,878 INFO Thread-8 :245590 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files/wandb-summary.json
2022-03-01 17:13:30,879 INFO Thread-8 :245590 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files/requirements.txt
2022-03-01 17:13:30,879 INFO Thread-8 :245590 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files/wandb-metadata.json
2022-03-01 17:13:30,879 INFO Thread-8 :245590 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files/output.log
2022-03-01 17:13:32,877 INFO Thread-8 :245590 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files/output.log
2022-03-01 17:13:36,879 INFO Thread-8 :245590 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files/output.log
2022-03-01 17:13:38,372 DEBUG HandlerThread:245590 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:13:38,373 DEBUG SenderThread:245590 [sender.py:send():235] send: telemetry
2022-03-01 17:13:38,373 DEBUG SenderThread:245590 [sender.py:send():235] send: exit
2022-03-01 17:13:38,373 INFO SenderThread:245590 [sender.py:send_exit():371] handling exit code: 1
2022-03-01 17:13:38,373 INFO SenderThread:245590 [sender.py:send_exit():373] handling runtime: 8
2022-03-01 17:13:38,373 INFO SenderThread:245590 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 17:13:38,374 INFO SenderThread:245590 [sender.py:send_exit():379] send defer
2022-03-01 17:13:38,374 DEBUG SenderThread:245590 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:13:38,374 DEBUG HandlerThread:245590 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:13:38,374 INFO HandlerThread:245590 [handler.py:handle_request_defer():154] handle defer: 0
2022-03-01 17:13:38,375 DEBUG SenderThread:245590 [sender.py:send_request():249] send_request: defer
2022-03-01 17:13:38,375 INFO SenderThread:245590 [sender.py:send_request_defer():388] handle sender defer: 0
2022-03-01 17:13:38,375 INFO SenderThread:245590 [sender.py:transition_state():392] send defer: 1
2022-03-01 17:13:38,375 DEBUG HandlerThread:245590 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:13:38,375 INFO HandlerThread:245590 [handler.py:handle_request_defer():154] handle defer: 1
2022-03-01 17:13:38,414 DEBUG SenderThread:245590 [sender.py:send_request():249] send_request: defer
2022-03-01 17:13:38,414 INFO SenderThread:245590 [sender.py:send_request_defer():388] handle sender defer: 1
2022-03-01 17:13:38,414 INFO SenderThread:245590 [sender.py:transition_state():392] send defer: 2
2022-03-01 17:13:38,414 DEBUG SenderThread:245590 [sender.py:send():235] send: stats
2022-03-01 17:13:38,415 DEBUG HandlerThread:245590 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:13:38,415 INFO HandlerThread:245590 [handler.py:handle_request_defer():154] handle defer: 2
2022-03-01 17:13:38,415 DEBUG SenderThread:245590 [sender.py:send_request():249] send_request: defer
2022-03-01 17:13:38,415 INFO SenderThread:245590 [sender.py:send_request_defer():388] handle sender defer: 2
2022-03-01 17:13:38,415 INFO SenderThread:245590 [sender.py:transition_state():392] send defer: 3
2022-03-01 17:13:38,415 DEBUG HandlerThread:245590 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:13:38,415 INFO HandlerThread:245590 [handler.py:handle_request_defer():154] handle defer: 3
2022-03-01 17:13:38,416 DEBUG SenderThread:245590 [sender.py:send():235] send: summary
2022-03-01 17:13:38,416 INFO SenderThread:245590 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 17:13:38,416 DEBUG SenderThread:245590 [sender.py:send_request():249] send_request: defer
2022-03-01 17:13:38,416 INFO SenderThread:245590 [sender.py:send_request_defer():388] handle sender defer: 3
2022-03-01 17:13:38,416 INFO SenderThread:245590 [sender.py:transition_state():392] send defer: 4
2022-03-01 17:13:38,416 DEBUG HandlerThread:245590 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:13:38,417 INFO HandlerThread:245590 [handler.py:handle_request_defer():154] handle defer: 4
2022-03-01 17:13:38,417 DEBUG SenderThread:245590 [sender.py:send_request():249] send_request: defer
2022-03-01 17:13:38,417 INFO SenderThread:245590 [sender.py:send_request_defer():388] handle sender defer: 4
2022-03-01 17:13:38,475 DEBUG HandlerThread:245590 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:13:38,539 INFO SenderThread:245590 [sender.py:transition_state():392] send defer: 5
2022-03-01 17:13:38,539 DEBUG SenderThread:245590 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:13:38,539 DEBUG HandlerThread:245590 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:13:38,539 INFO HandlerThread:245590 [handler.py:handle_request_defer():154] handle defer: 5
2022-03-01 17:13:38,540 DEBUG SenderThread:245590 [sender.py:send_request():249] send_request: defer
2022-03-01 17:13:38,540 INFO SenderThread:245590 [sender.py:send_request_defer():388] handle sender defer: 5
2022-03-01 17:13:38,540 INFO SenderThread:245590 [dir_watcher.py:finish():283] shutting down directory watcher
2022-03-01 17:13:38,640 DEBUG HandlerThread:245590 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:13:38,880 INFO Thread-8 :245590 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files/wandb-summary.json
2022-03-01 17:13:38,880 INFO SenderThread:245590 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files/config.yaml
2022-03-01 17:13:38,880 INFO SenderThread:245590 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files/output.log
2022-03-01 17:13:38,880 INFO SenderThread:245590 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files
2022-03-01 17:13:38,881 INFO SenderThread:245590 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files/wandb-metadata.json wandb-metadata.json
2022-03-01 17:13:38,881 INFO SenderThread:245590 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files/output.log output.log
2022-03-01 17:13:38,881 INFO SenderThread:245590 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files/wandb-summary.json wandb-summary.json
2022-03-01 17:13:38,881 INFO SenderThread:245590 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files/requirements.txt requirements.txt
2022-03-01 17:13:38,881 INFO SenderThread:245590 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files/config.yaml config.yaml
2022-03-01 17:13:38,884 INFO SenderThread:245590 [sender.py:transition_state():392] send defer: 6
2022-03-01 17:13:38,884 DEBUG SenderThread:245590 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:13:38,887 DEBUG HandlerThread:245590 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:13:38,887 INFO HandlerThread:245590 [handler.py:handle_request_defer():154] handle defer: 6
2022-03-01 17:13:38,888 DEBUG SenderThread:245590 [sender.py:send_request():249] send_request: defer
2022-03-01 17:13:38,888 INFO SenderThread:245590 [sender.py:send_request_defer():388] handle sender defer: 6
2022-03-01 17:13:38,888 INFO SenderThread:245590 [file_pusher.py:finish():177] shutting down file pusher
2022-03-01 17:13:38,988 DEBUG HandlerThread:245590 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:13:38,988 DEBUG SenderThread:245590 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:13:39,090 DEBUG HandlerThread:245590 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:13:39,090 DEBUG SenderThread:245590 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:13:39,137 INFO Thread-12 :245590 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files/output.log
2022-03-01 17:13:39,153 INFO Thread-13 :245590 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files/wandb-summary.json
2022-03-01 17:13:39,162 INFO Thread-14 :245590 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files/requirements.txt
2022-03-01 17:13:39,176 INFO Thread-15 :245590 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/files/config.yaml
2022-03-01 17:13:39,191 DEBUG HandlerThread:245590 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:13:39,192 DEBUG SenderThread:245590 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:13:39,293 DEBUG HandlerThread:245590 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:13:39,293 DEBUG SenderThread:245590 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:13:39,376 INFO Thread-7 :245590 [sender.py:transition_state():392] send defer: 7
2022-03-01 17:13:39,377 DEBUG HandlerThread:245590 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:13:39,377 INFO HandlerThread:245590 [handler.py:handle_request_defer():154] handle defer: 7
2022-03-01 17:13:39,377 DEBUG SenderThread:245590 [sender.py:send_request():249] send_request: defer
2022-03-01 17:13:39,377 INFO SenderThread:245590 [sender.py:send_request_defer():388] handle sender defer: 7
2022-03-01 17:13:39,394 DEBUG HandlerThread:245590 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:13:39,524 INFO SenderThread:245590 [sender.py:transition_state():392] send defer: 8
2022-03-01 17:13:39,524 DEBUG SenderThread:245590 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:13:39,524 DEBUG HandlerThread:245590 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:13:39,524 INFO HandlerThread:245590 [handler.py:handle_request_defer():154] handle defer: 8
2022-03-01 17:13:39,524 DEBUG SenderThread:245590 [sender.py:send_request():249] send_request: defer
2022-03-01 17:13:39,525 INFO SenderThread:245590 [sender.py:send_request_defer():388] handle sender defer: 8
2022-03-01 17:13:39,525 INFO SenderThread:245590 [sender.py:transition_state():392] send defer: 9
2022-03-01 17:13:39,525 DEBUG HandlerThread:245590 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:13:39,525 DEBUG SenderThread:245590 [sender.py:send():235] send: final
2022-03-01 17:13:39,526 INFO HandlerThread:245590 [handler.py:handle_request_defer():154] handle defer: 9
2022-03-01 17:13:39,526 DEBUG SenderThread:245590 [sender.py:send():235] send: footer
2022-03-01 17:13:39,526 DEBUG SenderThread:245590 [sender.py:send_request():249] send_request: defer
2022-03-01 17:13:39,526 INFO SenderThread:245590 [sender.py:send_request_defer():388] handle sender defer: 9
2022-03-01 17:13:39,625 DEBUG HandlerThread:245590 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:13:39,625 DEBUG SenderThread:245590 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:13:39,625 INFO SenderThread:245590 [file_pusher.py:join():182] waiting for file pusher
2022-03-01 17:13:39,694 DEBUG HandlerThread:245590 [handler.py:handle_request():131] handle_request: get_summary
2022-03-01 17:13:39,694 DEBUG HandlerThread:245590 [handler.py:handle_request():131] handle_request: sampled_history
2022-03-01 17:13:39,695 DEBUG HandlerThread:245590 [handler.py:handle_request():131] handle_request: shutdown
2022-03-01 17:13:39,695 INFO HandlerThread:245590 [handler.py:finish():739] shutting down handler
2022-03-01 17:13:40,526 INFO WriterThread:245590 [datastore.py:close():281] close: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171328-194tzn4x/run-194tzn4x.wandb
2022-03-01 17:13:40,693 INFO SenderThread:245590 [sender.py:finish():1075] shutting down sender
2022-03-01 17:13:40,693 INFO SenderThread:245590 [file_pusher.py:finish():177] shutting down file pusher
2022-03-01 17:13:40,693 INFO SenderThread:245590 [file_pusher.py:join():182] waiting for file pusher
2022-03-01 17:13:40,695 INFO MainThread:245590 [internal.py:handle_exit():79] Internal process exited
|