File size: 17,220 Bytes
fc53a59 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 |
2022-03-02 08:51:28,518 INFO MainThread:254490 [internal.py:wandb_internal():89] W&B internal server running at pid: 254490, started at: 2022-03-02 08:51:28.518100
2022-03-02 08:51:28,520 DEBUG HandlerThread:254490 [handler.py:handle_request():131] handle_request: check_version
2022-03-02 08:51:28,520 INFO WriterThread:254490 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/run-8m3km5vl.wandb
2022-03-02 08:51:28,522 DEBUG SenderThread:254490 [sender.py:send():235] send: header
2022-03-02 08:51:28,522 DEBUG SenderThread:254490 [sender.py:send_request():249] send_request: check_version
2022-03-02 08:51:28,588 DEBUG SenderThread:254490 [sender.py:send():235] send: run
2022-03-02 08:51:28,683 INFO SenderThread:254490 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files
2022-03-02 08:51:28,683 INFO SenderThread:254490 [sender.py:_start_run_threads():809] run started: 8m3km5vl with start time 1646211087
2022-03-02 08:51:28,683 DEBUG SenderThread:254490 [sender.py:send():235] send: summary
2022-03-02 08:51:28,683 INFO SenderThread:254490 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-02 08:51:28,684 DEBUG HandlerThread:254490 [handler.py:handle_request():131] handle_request: run_start
2022-03-02 08:51:28,689 DEBUG HandlerThread:254490 [meta.py:__init__():36] meta init
2022-03-02 08:51:28,689 DEBUG HandlerThread:254490 [meta.py:__init__():50] meta init done
2022-03-02 08:51:28,689 DEBUG HandlerThread:254490 [meta.py:probe():210] probe
2022-03-02 08:51:28,695 DEBUG HandlerThread:254490 [meta.py:_setup_git():200] setup git
2022-03-02 08:51:28,710 DEBUG HandlerThread:254490 [meta.py:_setup_git():207] setup git done
2022-03-02 08:51:28,710 DEBUG HandlerThread:254490 [meta.py:_save_pip():54] save pip
2022-03-02 08:51:28,711 DEBUG HandlerThread:254490 [meta.py:_save_pip():68] save pip done
2022-03-02 08:51:28,711 DEBUG HandlerThread:254490 [meta.py:probe():248] probe done
2022-03-02 08:51:28,787 DEBUG SenderThread:254490 [sender.py:send():235] send: files
2022-03-02 08:51:28,787 INFO SenderThread:254490 [sender.py:_save_file():944] saving file wandb-metadata.json with policy now
2022-03-02 08:51:28,792 DEBUG HandlerThread:254490 [handler.py:handle_request():131] handle_request: stop_status
2022-03-02 08:51:28,792 DEBUG SenderThread:254490 [sender.py:send_request():249] send_request: stop_status
2022-03-02 08:51:28,825 DEBUG SenderThread:254490 [sender.py:send():235] send: config
2022-03-02 08:51:28,826 DEBUG SenderThread:254490 [sender.py:send():235] send: metric
2022-03-02 08:51:28,826 DEBUG SenderThread:254490 [sender.py:send():235] send: metric
2022-03-02 08:51:28,826 WARNING SenderThread:254490 [sender.py:send_metric():902] Seen metric with glob (shouldnt happen)
2022-03-02 08:51:29,051 INFO Thread-11 :254490 [upload_job.py:push():137] Uploaded file /tmp/tmp4hagpn04wandb/h47718pd-wandb-metadata.json
2022-03-02 08:51:29,685 INFO Thread-8 :254490 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files/wandb-metadata.json
2022-03-02 08:51:29,685 INFO Thread-8 :254490 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files/wandb-summary.json
2022-03-02 08:51:29,685 INFO Thread-8 :254490 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files/output.log
2022-03-02 08:51:29,685 INFO Thread-8 :254490 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files/requirements.txt
2022-03-02 08:51:31,684 INFO Thread-8 :254490 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files/output.log
2022-03-02 08:51:35,685 INFO Thread-8 :254490 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files/output.log
2022-03-02 08:51:37,089 DEBUG SenderThread:254490 [sender.py:send():235] send: telemetry
2022-03-02 08:51:37,090 DEBUG HandlerThread:254490 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:51:37,090 DEBUG SenderThread:254490 [sender.py:send():235] send: exit
2022-03-02 08:51:37,090 INFO SenderThread:254490 [sender.py:send_exit():371] handling exit code: 1
2022-03-02 08:51:37,091 INFO SenderThread:254490 [sender.py:send_exit():373] handling runtime: 8
2022-03-02 08:51:37,091 INFO SenderThread:254490 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-02 08:51:37,091 INFO SenderThread:254490 [sender.py:send_exit():379] send defer
2022-03-02 08:51:37,091 DEBUG SenderThread:254490 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:51:37,092 DEBUG HandlerThread:254490 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:51:37,092 INFO HandlerThread:254490 [handler.py:handle_request_defer():154] handle defer: 0
2022-03-02 08:51:37,092 DEBUG SenderThread:254490 [sender.py:send_request():249] send_request: defer
2022-03-02 08:51:37,092 INFO SenderThread:254490 [sender.py:send_request_defer():388] handle sender defer: 0
2022-03-02 08:51:37,092 INFO SenderThread:254490 [sender.py:transition_state():392] send defer: 1
2022-03-02 08:51:37,092 DEBUG HandlerThread:254490 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:51:37,092 INFO HandlerThread:254490 [handler.py:handle_request_defer():154] handle defer: 1
2022-03-02 08:51:37,120 DEBUG SenderThread:254490 [sender.py:send_request():249] send_request: defer
2022-03-02 08:51:37,120 INFO SenderThread:254490 [sender.py:send_request_defer():388] handle sender defer: 1
2022-03-02 08:51:37,120 INFO SenderThread:254490 [sender.py:transition_state():392] send defer: 2
2022-03-02 08:51:37,120 DEBUG SenderThread:254490 [sender.py:send():235] send: stats
2022-03-02 08:51:37,121 DEBUG HandlerThread:254490 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:51:37,121 INFO HandlerThread:254490 [handler.py:handle_request_defer():154] handle defer: 2
2022-03-02 08:51:37,121 DEBUG SenderThread:254490 [sender.py:send_request():249] send_request: defer
2022-03-02 08:51:37,121 INFO SenderThread:254490 [sender.py:send_request_defer():388] handle sender defer: 2
2022-03-02 08:51:37,121 INFO SenderThread:254490 [sender.py:transition_state():392] send defer: 3
2022-03-02 08:51:37,121 DEBUG HandlerThread:254490 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:51:37,121 INFO HandlerThread:254490 [handler.py:handle_request_defer():154] handle defer: 3
2022-03-02 08:51:37,122 DEBUG SenderThread:254490 [sender.py:send():235] send: summary
2022-03-02 08:51:37,122 INFO SenderThread:254490 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-02 08:51:37,122 DEBUG SenderThread:254490 [sender.py:send_request():249] send_request: defer
2022-03-02 08:51:37,122 INFO SenderThread:254490 [sender.py:send_request_defer():388] handle sender defer: 3
2022-03-02 08:51:37,122 INFO SenderThread:254490 [sender.py:transition_state():392] send defer: 4
2022-03-02 08:51:37,123 DEBUG HandlerThread:254490 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:51:37,123 INFO HandlerThread:254490 [handler.py:handle_request_defer():154] handle defer: 4
2022-03-02 08:51:37,123 DEBUG SenderThread:254490 [sender.py:send_request():249] send_request: defer
2022-03-02 08:51:37,123 INFO SenderThread:254490 [sender.py:send_request_defer():388] handle sender defer: 4
2022-03-02 08:51:37,198 DEBUG HandlerThread:254490 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:51:37,219 INFO SenderThread:254490 [sender.py:transition_state():392] send defer: 5
2022-03-02 08:51:37,219 DEBUG SenderThread:254490 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:51:37,220 DEBUG HandlerThread:254490 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:51:37,220 INFO HandlerThread:254490 [handler.py:handle_request_defer():154] handle defer: 5
2022-03-02 08:51:37,220 DEBUG SenderThread:254490 [sender.py:send_request():249] send_request: defer
2022-03-02 08:51:37,220 INFO SenderThread:254490 [sender.py:send_request_defer():388] handle sender defer: 5
2022-03-02 08:51:37,220 INFO SenderThread:254490 [dir_watcher.py:finish():283] shutting down directory watcher
2022-03-02 08:51:37,321 DEBUG HandlerThread:254490 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:51:37,686 INFO Thread-8 :254490 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files/wandb-summary.json
2022-03-02 08:51:37,687 INFO SenderThread:254490 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files/config.yaml
2022-03-02 08:51:37,687 INFO SenderThread:254490 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files/output.log
2022-03-02 08:51:37,687 INFO SenderThread:254490 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files
2022-03-02 08:51:37,687 INFO SenderThread:254490 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files/wandb-metadata.json wandb-metadata.json
2022-03-02 08:51:37,687 INFO SenderThread:254490 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files/output.log output.log
2022-03-02 08:51:37,687 INFO SenderThread:254490 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files/wandb-summary.json wandb-summary.json
2022-03-02 08:51:37,691 INFO SenderThread:254490 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files/requirements.txt requirements.txt
2022-03-02 08:51:37,691 INFO SenderThread:254490 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files/config.yaml config.yaml
2022-03-02 08:51:37,694 INFO SenderThread:254490 [sender.py:transition_state():392] send defer: 6
2022-03-02 08:51:37,694 DEBUG SenderThread:254490 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:51:37,700 DEBUG HandlerThread:254490 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:51:37,700 INFO HandlerThread:254490 [handler.py:handle_request_defer():154] handle defer: 6
2022-03-02 08:51:37,701 DEBUG SenderThread:254490 [sender.py:send_request():249] send_request: defer
2022-03-02 08:51:37,701 INFO SenderThread:254490 [sender.py:send_request_defer():388] handle sender defer: 6
2022-03-02 08:51:37,701 INFO SenderThread:254490 [file_pusher.py:finish():177] shutting down file pusher
2022-03-02 08:51:37,803 DEBUG HandlerThread:254490 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:51:37,803 DEBUG SenderThread:254490 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:51:37,905 DEBUG HandlerThread:254490 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:51:37,905 DEBUG SenderThread:254490 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:51:37,949 INFO Thread-15 :254490 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files/config.yaml
2022-03-02 08:51:37,966 INFO Thread-12 :254490 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files/output.log
2022-03-02 08:51:37,975 INFO Thread-14 :254490 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files/requirements.txt
2022-03-02 08:51:38,007 DEBUG HandlerThread:254490 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:51:38,007 DEBUG SenderThread:254490 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:51:38,029 INFO Thread-13 :254490 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/files/wandb-summary.json
2022-03-02 08:51:38,108 DEBUG HandlerThread:254490 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:51:38,109 DEBUG SenderThread:254490 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:51:38,210 DEBUG HandlerThread:254490 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:51:38,210 DEBUG SenderThread:254490 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:51:38,229 INFO Thread-7 :254490 [sender.py:transition_state():392] send defer: 7
2022-03-02 08:51:38,230 DEBUG HandlerThread:254490 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:51:38,230 INFO HandlerThread:254490 [handler.py:handle_request_defer():154] handle defer: 7
2022-03-02 08:51:38,230 DEBUG SenderThread:254490 [sender.py:send_request():249] send_request: defer
2022-03-02 08:51:38,230 INFO SenderThread:254490 [sender.py:send_request_defer():388] handle sender defer: 7
2022-03-02 08:51:38,311 DEBUG HandlerThread:254490 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:51:38,350 INFO SenderThread:254490 [sender.py:transition_state():392] send defer: 8
2022-03-02 08:51:38,350 DEBUG SenderThread:254490 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:51:38,350 DEBUG HandlerThread:254490 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:51:38,351 INFO HandlerThread:254490 [handler.py:handle_request_defer():154] handle defer: 8
2022-03-02 08:51:38,351 DEBUG SenderThread:254490 [sender.py:send_request():249] send_request: defer
2022-03-02 08:51:38,351 INFO SenderThread:254490 [sender.py:send_request_defer():388] handle sender defer: 8
2022-03-02 08:51:38,351 INFO SenderThread:254490 [sender.py:transition_state():392] send defer: 9
2022-03-02 08:51:38,352 DEBUG HandlerThread:254490 [handler.py:handle_request():131] handle_request: defer
2022-03-02 08:51:38,352 INFO HandlerThread:254490 [handler.py:handle_request_defer():154] handle defer: 9
2022-03-02 08:51:38,352 DEBUG SenderThread:254490 [sender.py:send():235] send: final
2022-03-02 08:51:38,352 DEBUG SenderThread:254490 [sender.py:send():235] send: footer
2022-03-02 08:51:38,352 DEBUG SenderThread:254490 [sender.py:send_request():249] send_request: defer
2022-03-02 08:51:38,352 INFO SenderThread:254490 [sender.py:send_request_defer():388] handle sender defer: 9
2022-03-02 08:51:38,451 DEBUG HandlerThread:254490 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-02 08:51:38,452 DEBUG SenderThread:254490 [sender.py:send_request():249] send_request: poll_exit
2022-03-02 08:51:38,452 INFO SenderThread:254490 [file_pusher.py:join():182] waiting for file pusher
2022-03-02 08:51:38,532 DEBUG HandlerThread:254490 [handler.py:handle_request():131] handle_request: get_summary
2022-03-02 08:51:38,533 DEBUG HandlerThread:254490 [handler.py:handle_request():131] handle_request: sampled_history
2022-03-02 08:51:38,534 DEBUG HandlerThread:254490 [handler.py:handle_request():131] handle_request: shutdown
2022-03-02 08:51:38,534 INFO HandlerThread:254490 [handler.py:finish():739] shutting down handler
2022-03-02 08:51:39,352 INFO WriterThread:254490 [datastore.py:close():281] close: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_085127-8m3km5vl/run-8m3km5vl.wandb
2022-03-02 08:51:39,532 INFO SenderThread:254490 [sender.py:finish():1075] shutting down sender
2022-03-02 08:51:39,532 INFO SenderThread:254490 [file_pusher.py:finish():177] shutting down file pusher
2022-03-02 08:51:39,532 INFO SenderThread:254490 [file_pusher.py:join():182] waiting for file pusher
2022-03-02 08:51:39,534 INFO MainThread:254490 [internal.py:handle_exit():79] Internal process exited
|