File size: 17,663 Bytes
bcb2332 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 |
2022-03-03 00:40:40,587 INFO MainThread:267195 [internal.py:wandb_internal():89] W&B internal server running at pid: 267195, started at: 2022-03-03 00:40:40.586996
2022-03-03 00:40:40,590 INFO WriterThread:267195 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/run-1a18zsso.wandb
2022-03-03 00:40:40,590 DEBUG HandlerThread:267195 [handler.py:handle_request():131] handle_request: check_version
2022-03-03 00:40:40,591 DEBUG SenderThread:267195 [sender.py:send():235] send: header
2022-03-03 00:40:40,591 DEBUG SenderThread:267195 [sender.py:send_request():249] send_request: check_version
2022-03-03 00:40:40,668 DEBUG SenderThread:267195 [sender.py:send():235] send: run
2022-03-03 00:40:40,791 INFO SenderThread:267195 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files
2022-03-03 00:40:40,791 INFO SenderThread:267195 [sender.py:_start_run_threads():809] run started: 1a18zsso with start time 1646268039
2022-03-03 00:40:40,791 DEBUG SenderThread:267195 [sender.py:send():235] send: summary
2022-03-03 00:40:40,791 INFO SenderThread:267195 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-03 00:40:40,793 DEBUG HandlerThread:267195 [handler.py:handle_request():131] handle_request: run_start
2022-03-03 00:40:40,798 DEBUG HandlerThread:267195 [meta.py:__init__():36] meta init
2022-03-03 00:40:40,798 DEBUG HandlerThread:267195 [meta.py:__init__():50] meta init done
2022-03-03 00:40:40,798 DEBUG HandlerThread:267195 [meta.py:probe():210] probe
2022-03-03 00:40:40,805 DEBUG HandlerThread:267195 [meta.py:_setup_git():200] setup git
2022-03-03 00:40:40,821 DEBUG HandlerThread:267195 [meta.py:_setup_git():207] setup git done
2022-03-03 00:40:40,822 DEBUG HandlerThread:267195 [meta.py:_save_pip():54] save pip
2022-03-03 00:40:40,823 DEBUG HandlerThread:267195 [meta.py:_save_pip():68] save pip done
2022-03-03 00:40:40,823 DEBUG HandlerThread:267195 [meta.py:probe():248] probe done
2022-03-03 00:40:40,946 DEBUG SenderThread:267195 [sender.py:send():235] send: files
2022-03-03 00:40:40,946 INFO SenderThread:267195 [sender.py:_save_file():944] saving file wandb-metadata.json with policy now
2022-03-03 00:40:40,951 DEBUG HandlerThread:267195 [handler.py:handle_request():131] handle_request: stop_status
2022-03-03 00:40:40,952 DEBUG SenderThread:267195 [sender.py:send_request():249] send_request: stop_status
2022-03-03 00:40:40,999 DEBUG SenderThread:267195 [sender.py:send():235] send: config
2022-03-03 00:40:41,000 DEBUG SenderThread:267195 [sender.py:send():235] send: metric
2022-03-03 00:40:41,000 DEBUG SenderThread:267195 [sender.py:send():235] send: metric
2022-03-03 00:40:41,000 WARNING SenderThread:267195 [sender.py:send_metric():902] Seen metric with glob (shouldnt happen)
2022-03-03 00:40:41,240 INFO Thread-11 :267195 [upload_job.py:push():137] Uploaded file /tmp/tmp9mhkg11ywandb/2vus7xrr-wandb-metadata.json
2022-03-03 00:40:41,793 INFO Thread-8 :267195 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/output.log
2022-03-03 00:40:41,793 INFO Thread-8 :267195 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/wandb-metadata.json
2022-03-03 00:40:41,794 INFO Thread-8 :267195 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/requirements.txt
2022-03-03 00:40:41,794 INFO Thread-8 :267195 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/wandb-summary.json
2022-03-03 00:40:43,792 INFO Thread-8 :267195 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/output.log
2022-03-03 00:40:47,793 INFO Thread-8 :267195 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/output.log
2022-03-03 00:40:49,794 INFO Thread-8 :267195 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/output.log
2022-03-03 00:40:49,872 DEBUG SenderThread:267195 [sender.py:send():235] send: telemetry
2022-03-03 00:40:49,872 DEBUG SenderThread:267195 [sender.py:send():235] send: exit
2022-03-03 00:40:49,872 DEBUG HandlerThread:267195 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:40:49,873 INFO SenderThread:267195 [sender.py:send_exit():371] handling exit code: 1
2022-03-03 00:40:49,873 INFO SenderThread:267195 [sender.py:send_exit():373] handling runtime: 9
2022-03-03 00:40:49,873 INFO SenderThread:267195 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-03 00:40:49,873 INFO SenderThread:267195 [sender.py:send_exit():379] send defer
2022-03-03 00:40:49,874 DEBUG SenderThread:267195 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:40:49,874 DEBUG HandlerThread:267195 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:40:49,875 INFO HandlerThread:267195 [handler.py:handle_request_defer():154] handle defer: 0
2022-03-03 00:40:49,875 DEBUG SenderThread:267195 [sender.py:send_request():249] send_request: defer
2022-03-03 00:40:49,875 INFO SenderThread:267195 [sender.py:send_request_defer():388] handle sender defer: 0
2022-03-03 00:40:49,875 INFO SenderThread:267195 [sender.py:transition_state():392] send defer: 1
2022-03-03 00:40:49,875 DEBUG HandlerThread:267195 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:40:49,876 INFO HandlerThread:267195 [handler.py:handle_request_defer():154] handle defer: 1
2022-03-03 00:40:49,994 DEBUG HandlerThread:267195 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:40:49,994 DEBUG SenderThread:267195 [sender.py:send_request():249] send_request: defer
2022-03-03 00:40:49,994 INFO SenderThread:267195 [sender.py:send_request_defer():388] handle sender defer: 1
2022-03-03 00:40:49,994 INFO SenderThread:267195 [sender.py:transition_state():392] send defer: 2
2022-03-03 00:40:49,995 DEBUG SenderThread:267195 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:40:49,995 DEBUG SenderThread:267195 [sender.py:send():235] send: stats
2022-03-03 00:40:49,996 DEBUG HandlerThread:267195 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:40:49,996 INFO HandlerThread:267195 [handler.py:handle_request_defer():154] handle defer: 2
2022-03-03 00:40:49,996 DEBUG SenderThread:267195 [sender.py:send_request():249] send_request: defer
2022-03-03 00:40:49,996 INFO SenderThread:267195 [sender.py:send_request_defer():388] handle sender defer: 2
2022-03-03 00:40:49,996 INFO SenderThread:267195 [sender.py:transition_state():392] send defer: 3
2022-03-03 00:40:49,996 DEBUG HandlerThread:267195 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:40:49,997 INFO HandlerThread:267195 [handler.py:handle_request_defer():154] handle defer: 3
2022-03-03 00:40:49,997 DEBUG SenderThread:267195 [sender.py:send():235] send: summary
2022-03-03 00:40:49,998 INFO SenderThread:267195 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-03 00:40:49,998 DEBUG SenderThread:267195 [sender.py:send_request():249] send_request: defer
2022-03-03 00:40:49,998 INFO SenderThread:267195 [sender.py:send_request_defer():388] handle sender defer: 3
2022-03-03 00:40:49,998 INFO SenderThread:267195 [sender.py:transition_state():392] send defer: 4
2022-03-03 00:40:49,998 DEBUG HandlerThread:267195 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:40:49,998 INFO HandlerThread:267195 [handler.py:handle_request_defer():154] handle defer: 4
2022-03-03 00:40:49,998 DEBUG SenderThread:267195 [sender.py:send_request():249] send_request: defer
2022-03-03 00:40:49,998 INFO SenderThread:267195 [sender.py:send_request_defer():388] handle sender defer: 4
2022-03-03 00:40:50,102 DEBUG HandlerThread:267195 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:40:50,128 INFO SenderThread:267195 [sender.py:transition_state():392] send defer: 5
2022-03-03 00:40:50,128 DEBUG SenderThread:267195 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:40:50,129 DEBUG HandlerThread:267195 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:40:50,129 INFO HandlerThread:267195 [handler.py:handle_request_defer():154] handle defer: 5
2022-03-03 00:40:50,129 DEBUG SenderThread:267195 [sender.py:send_request():249] send_request: defer
2022-03-03 00:40:50,130 INFO SenderThread:267195 [sender.py:send_request_defer():388] handle sender defer: 5
2022-03-03 00:40:50,130 INFO SenderThread:267195 [dir_watcher.py:finish():283] shutting down directory watcher
2022-03-03 00:40:50,231 DEBUG HandlerThread:267195 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:40:50,795 INFO Thread-8 :267195 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/output.log
2022-03-03 00:40:50,795 INFO SenderThread:267195 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/wandb-summary.json
2022-03-03 00:40:50,796 INFO SenderThread:267195 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/config.yaml
2022-03-03 00:40:50,796 INFO SenderThread:267195 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files
2022-03-03 00:40:50,796 INFO SenderThread:267195 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/wandb-metadata.json wandb-metadata.json
2022-03-03 00:40:50,796 INFO SenderThread:267195 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/output.log output.log
2022-03-03 00:40:50,797 INFO SenderThread:267195 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/wandb-summary.json wandb-summary.json
2022-03-03 00:40:50,800 INFO SenderThread:267195 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/requirements.txt requirements.txt
2022-03-03 00:40:50,803 INFO SenderThread:267195 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/config.yaml config.yaml
2022-03-03 00:40:50,807 INFO SenderThread:267195 [sender.py:transition_state():392] send defer: 6
2022-03-03 00:40:50,807 DEBUG SenderThread:267195 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:40:50,808 DEBUG HandlerThread:267195 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:40:50,809 INFO HandlerThread:267195 [handler.py:handle_request_defer():154] handle defer: 6
2022-03-03 00:40:50,812 DEBUG SenderThread:267195 [sender.py:send_request():249] send_request: defer
2022-03-03 00:40:50,813 INFO SenderThread:267195 [sender.py:send_request_defer():388] handle sender defer: 6
2022-03-03 00:40:50,813 INFO SenderThread:267195 [file_pusher.py:finish():177] shutting down file pusher
2022-03-03 00:40:50,910 DEBUG HandlerThread:267195 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:40:50,911 DEBUG SenderThread:267195 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:40:51,013 DEBUG HandlerThread:267195 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:40:51,013 DEBUG SenderThread:267195 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:40:51,068 INFO Thread-15 :267195 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/config.yaml
2022-03-03 00:40:51,071 INFO Thread-13 :267195 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/wandb-summary.json
2022-03-03 00:40:51,116 INFO Thread-12 :267195 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/output.log
2022-03-03 00:40:51,117 DEBUG HandlerThread:267195 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:40:51,117 DEBUG SenderThread:267195 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:40:51,172 INFO Thread-14 :267195 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/files/requirements.txt
2022-03-03 00:40:51,219 DEBUG HandlerThread:267195 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:40:51,219 DEBUG SenderThread:267195 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:40:51,321 DEBUG HandlerThread:267195 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:40:51,322 DEBUG SenderThread:267195 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:40:51,372 INFO Thread-7 :267195 [sender.py:transition_state():392] send defer: 7
2022-03-03 00:40:51,373 DEBUG HandlerThread:267195 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:40:51,373 INFO HandlerThread:267195 [handler.py:handle_request_defer():154] handle defer: 7
2022-03-03 00:40:51,373 DEBUG SenderThread:267195 [sender.py:send_request():249] send_request: defer
2022-03-03 00:40:51,373 INFO SenderThread:267195 [sender.py:send_request_defer():388] handle sender defer: 7
2022-03-03 00:40:51,424 DEBUG HandlerThread:267195 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:40:51,429 INFO SenderThread:267195 [sender.py:transition_state():392] send defer: 8
2022-03-03 00:40:51,429 DEBUG SenderThread:267195 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:40:51,430 DEBUG HandlerThread:267195 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:40:51,430 INFO HandlerThread:267195 [handler.py:handle_request_defer():154] handle defer: 8
2022-03-03 00:40:51,431 DEBUG SenderThread:267195 [sender.py:send_request():249] send_request: defer
2022-03-03 00:40:51,431 INFO SenderThread:267195 [sender.py:send_request_defer():388] handle sender defer: 8
2022-03-03 00:40:51,431 INFO SenderThread:267195 [sender.py:transition_state():392] send defer: 9
2022-03-03 00:40:51,431 DEBUG SenderThread:267195 [sender.py:send():235] send: final
2022-03-03 00:40:51,432 DEBUG SenderThread:267195 [sender.py:send():235] send: footer
2022-03-03 00:40:51,432 DEBUG HandlerThread:267195 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:40:51,432 INFO HandlerThread:267195 [handler.py:handle_request_defer():154] handle defer: 9
2022-03-03 00:40:51,432 DEBUG SenderThread:267195 [sender.py:send_request():249] send_request: defer
2022-03-03 00:40:51,433 INFO SenderThread:267195 [sender.py:send_request_defer():388] handle sender defer: 9
2022-03-03 00:40:51,531 DEBUG HandlerThread:267195 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:40:51,532 DEBUG SenderThread:267195 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:40:51,532 INFO SenderThread:267195 [file_pusher.py:join():182] waiting for file pusher
2022-03-03 00:40:51,631 DEBUG HandlerThread:267195 [handler.py:handle_request():131] handle_request: get_summary
2022-03-03 00:40:51,632 DEBUG HandlerThread:267195 [handler.py:handle_request():131] handle_request: sampled_history
2022-03-03 00:40:51,633 DEBUG HandlerThread:267195 [handler.py:handle_request():131] handle_request: shutdown
2022-03-03 00:40:51,633 INFO HandlerThread:267195 [handler.py:finish():739] shutting down handler
2022-03-03 00:40:52,432 INFO WriterThread:267195 [datastore.py:close():281] close: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004039-1a18zsso/run-1a18zsso.wandb
2022-03-03 00:40:52,630 INFO SenderThread:267195 [sender.py:finish():1075] shutting down sender
2022-03-03 00:40:52,630 INFO SenderThread:267195 [file_pusher.py:finish():177] shutting down file pusher
2022-03-03 00:40:52,630 INFO SenderThread:267195 [file_pusher.py:join():182] waiting for file pusher
2022-03-03 00:40:52,632 INFO MainThread:267195 [internal.py:handle_exit():79] Internal process exited
|