File size: 17,439 Bytes
bcb2332 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 |
2022-03-03 00:43:48,671 INFO MainThread:267559 [internal.py:wandb_internal():89] W&B internal server running at pid: 267559, started at: 2022-03-03 00:43:48.670697
2022-03-03 00:43:48,673 DEBUG HandlerThread:267559 [handler.py:handle_request():131] handle_request: check_version
2022-03-03 00:43:48,674 INFO WriterThread:267559 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/run-13qlgnoe.wandb
2022-03-03 00:43:48,675 DEBUG SenderThread:267559 [sender.py:send():235] send: header
2022-03-03 00:43:48,675 DEBUG SenderThread:267559 [sender.py:send_request():249] send_request: check_version
2022-03-03 00:43:48,744 DEBUG SenderThread:267559 [sender.py:send():235] send: run
2022-03-03 00:43:48,863 INFO SenderThread:267559 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files
2022-03-03 00:43:48,863 INFO SenderThread:267559 [sender.py:_start_run_threads():809] run started: 13qlgnoe with start time 1646268227
2022-03-03 00:43:48,863 DEBUG SenderThread:267559 [sender.py:send():235] send: summary
2022-03-03 00:43:48,863 INFO SenderThread:267559 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-03 00:43:48,864 DEBUG HandlerThread:267559 [handler.py:handle_request():131] handle_request: run_start
2022-03-03 00:43:48,868 DEBUG HandlerThread:267559 [meta.py:__init__():36] meta init
2022-03-03 00:43:48,869 DEBUG HandlerThread:267559 [meta.py:__init__():50] meta init done
2022-03-03 00:43:48,869 DEBUG HandlerThread:267559 [meta.py:probe():210] probe
2022-03-03 00:43:48,875 DEBUG HandlerThread:267559 [meta.py:_setup_git():200] setup git
2022-03-03 00:43:48,891 DEBUG HandlerThread:267559 [meta.py:_setup_git():207] setup git done
2022-03-03 00:43:48,891 DEBUG HandlerThread:267559 [meta.py:_save_pip():54] save pip
2022-03-03 00:43:48,892 DEBUG HandlerThread:267559 [meta.py:_save_pip():68] save pip done
2022-03-03 00:43:48,892 DEBUG HandlerThread:267559 [meta.py:probe():248] probe done
2022-03-03 00:43:48,974 DEBUG SenderThread:267559 [sender.py:send():235] send: files
2022-03-03 00:43:48,974 INFO SenderThread:267559 [sender.py:_save_file():944] saving file wandb-metadata.json with policy now
2022-03-03 00:43:48,979 DEBUG HandlerThread:267559 [handler.py:handle_request():131] handle_request: stop_status
2022-03-03 00:43:48,980 DEBUG SenderThread:267559 [sender.py:send_request():249] send_request: stop_status
2022-03-03 00:43:49,032 DEBUG SenderThread:267559 [sender.py:send():235] send: config
2022-03-03 00:43:49,033 DEBUG SenderThread:267559 [sender.py:send():235] send: metric
2022-03-03 00:43:49,033 DEBUG SenderThread:267559 [sender.py:send():235] send: metric
2022-03-03 00:43:49,033 WARNING SenderThread:267559 [sender.py:send_metric():902] Seen metric with glob (shouldnt happen)
2022-03-03 00:43:49,247 INFO Thread-11 :267559 [upload_job.py:push():137] Uploaded file /tmp/tmps9bhl1w0wandb/kc71lad7-wandb-metadata.json
2022-03-03 00:43:49,865 INFO Thread-8 :267559 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/output.log
2022-03-03 00:43:49,865 INFO Thread-8 :267559 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/wandb-metadata.json
2022-03-03 00:43:49,865 INFO Thread-8 :267559 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/wandb-summary.json
2022-03-03 00:43:49,865 INFO Thread-8 :267559 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/requirements.txt
2022-03-03 00:43:51,864 INFO Thread-8 :267559 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/output.log
2022-03-03 00:43:55,866 INFO Thread-8 :267559 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/output.log
2022-03-03 00:43:57,866 INFO Thread-8 :267559 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/output.log
2022-03-03 00:43:58,509 DEBUG HandlerThread:267559 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:43:58,509 DEBUG SenderThread:267559 [sender.py:send():235] send: telemetry
2022-03-03 00:43:58,510 DEBUG SenderThread:267559 [sender.py:send():235] send: exit
2022-03-03 00:43:58,510 INFO SenderThread:267559 [sender.py:send_exit():371] handling exit code: 1
2022-03-03 00:43:58,510 INFO SenderThread:267559 [sender.py:send_exit():373] handling runtime: 9
2022-03-03 00:43:58,510 INFO SenderThread:267559 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-03 00:43:58,510 INFO SenderThread:267559 [sender.py:send_exit():379] send defer
2022-03-03 00:43:58,511 DEBUG SenderThread:267559 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:43:58,511 DEBUG HandlerThread:267559 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:43:58,511 INFO HandlerThread:267559 [handler.py:handle_request_defer():154] handle defer: 0
2022-03-03 00:43:58,512 DEBUG SenderThread:267559 [sender.py:send_request():249] send_request: defer
2022-03-03 00:43:58,512 INFO SenderThread:267559 [sender.py:send_request_defer():388] handle sender defer: 0
2022-03-03 00:43:58,512 INFO SenderThread:267559 [sender.py:transition_state():392] send defer: 1
2022-03-03 00:43:58,512 DEBUG HandlerThread:267559 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:43:58,512 INFO HandlerThread:267559 [handler.py:handle_request_defer():154] handle defer: 1
2022-03-03 00:43:58,627 DEBUG SenderThread:267559 [sender.py:send_request():249] send_request: defer
2022-03-03 00:43:58,627 DEBUG HandlerThread:267559 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:43:58,627 INFO SenderThread:267559 [sender.py:send_request_defer():388] handle sender defer: 1
2022-03-03 00:43:58,628 INFO SenderThread:267559 [sender.py:transition_state():392] send defer: 2
2022-03-03 00:43:58,628 DEBUG SenderThread:267559 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:43:58,628 DEBUG SenderThread:267559 [sender.py:send():235] send: stats
2022-03-03 00:43:58,629 DEBUG HandlerThread:267559 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:43:58,629 INFO HandlerThread:267559 [handler.py:handle_request_defer():154] handle defer: 2
2022-03-03 00:43:58,629 DEBUG SenderThread:267559 [sender.py:send_request():249] send_request: defer
2022-03-03 00:43:58,629 INFO SenderThread:267559 [sender.py:send_request_defer():388] handle sender defer: 2
2022-03-03 00:43:58,629 INFO SenderThread:267559 [sender.py:transition_state():392] send defer: 3
2022-03-03 00:43:58,629 DEBUG HandlerThread:267559 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:43:58,629 INFO HandlerThread:267559 [handler.py:handle_request_defer():154] handle defer: 3
2022-03-03 00:43:58,630 DEBUG SenderThread:267559 [sender.py:send():235] send: summary
2022-03-03 00:43:58,630 INFO SenderThread:267559 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-03 00:43:58,630 DEBUG SenderThread:267559 [sender.py:send_request():249] send_request: defer
2022-03-03 00:43:58,630 INFO SenderThread:267559 [sender.py:send_request_defer():388] handle sender defer: 3
2022-03-03 00:43:58,630 INFO SenderThread:267559 [sender.py:transition_state():392] send defer: 4
2022-03-03 00:43:58,631 DEBUG HandlerThread:267559 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:43:58,631 INFO HandlerThread:267559 [handler.py:handle_request_defer():154] handle defer: 4
2022-03-03 00:43:58,631 DEBUG SenderThread:267559 [sender.py:send_request():249] send_request: defer
2022-03-03 00:43:58,631 INFO SenderThread:267559 [sender.py:send_request_defer():388] handle sender defer: 4
2022-03-03 00:43:58,730 DEBUG HandlerThread:267559 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:43:58,867 INFO Thread-8 :267559 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/wandb-summary.json
2022-03-03 00:43:58,867 INFO Thread-8 :267559 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/output.log
2022-03-03 00:43:58,921 INFO SenderThread:267559 [sender.py:transition_state():392] send defer: 5
2022-03-03 00:43:58,922 DEBUG SenderThread:267559 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:43:58,922 DEBUG HandlerThread:267559 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:43:58,922 INFO HandlerThread:267559 [handler.py:handle_request_defer():154] handle defer: 5
2022-03-03 00:43:58,923 DEBUG SenderThread:267559 [sender.py:send_request():249] send_request: defer
2022-03-03 00:43:58,923 INFO SenderThread:267559 [sender.py:send_request_defer():388] handle sender defer: 5
2022-03-03 00:43:58,923 INFO SenderThread:267559 [dir_watcher.py:finish():283] shutting down directory watcher
2022-03-03 00:43:59,024 DEBUG HandlerThread:267559 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:43:59,867 INFO Thread-8 :267559 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/config.yaml
2022-03-03 00:43:59,868 INFO SenderThread:267559 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files
2022-03-03 00:43:59,868 INFO SenderThread:267559 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/wandb-metadata.json wandb-metadata.json
2022-03-03 00:43:59,869 INFO SenderThread:267559 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/output.log output.log
2022-03-03 00:43:59,869 INFO SenderThread:267559 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/wandb-summary.json wandb-summary.json
2022-03-03 00:43:59,869 INFO SenderThread:267559 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/requirements.txt requirements.txt
2022-03-03 00:43:59,869 INFO SenderThread:267559 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/config.yaml config.yaml
2022-03-03 00:43:59,873 INFO SenderThread:267559 [sender.py:transition_state():392] send defer: 6
2022-03-03 00:43:59,876 DEBUG SenderThread:267559 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:43:59,882 DEBUG HandlerThread:267559 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:43:59,882 INFO HandlerThread:267559 [handler.py:handle_request_defer():154] handle defer: 6
2022-03-03 00:43:59,884 DEBUG SenderThread:267559 [sender.py:send_request():249] send_request: defer
2022-03-03 00:43:59,884 INFO SenderThread:267559 [sender.py:send_request_defer():388] handle sender defer: 6
2022-03-03 00:43:59,884 INFO SenderThread:267559 [file_pusher.py:finish():177] shutting down file pusher
2022-03-03 00:43:59,977 DEBUG HandlerThread:267559 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:43:59,978 DEBUG SenderThread:267559 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:44:00,080 DEBUG HandlerThread:267559 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:44:00,080 DEBUG SenderThread:267559 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:44:00,168 INFO Thread-15 :267559 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/config.yaml
2022-03-03 00:44:00,179 INFO Thread-12 :267559 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/output.log
2022-03-03 00:44:00,181 INFO Thread-13 :267559 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/wandb-summary.json
2022-03-03 00:44:00,182 DEBUG HandlerThread:267559 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:44:00,182 DEBUG SenderThread:267559 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:44:00,186 INFO Thread-14 :267559 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/files/requirements.txt
2022-03-03 00:44:00,284 DEBUG HandlerThread:267559 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:44:00,284 DEBUG SenderThread:267559 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:44:00,386 DEBUG HandlerThread:267559 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:44:00,386 DEBUG SenderThread:267559 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:44:00,386 INFO Thread-7 :267559 [sender.py:transition_state():392] send defer: 7
2022-03-03 00:44:00,387 DEBUG HandlerThread:267559 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:44:00,387 INFO HandlerThread:267559 [handler.py:handle_request_defer():154] handle defer: 7
2022-03-03 00:44:00,387 DEBUG SenderThread:267559 [sender.py:send_request():249] send_request: defer
2022-03-03 00:44:00,387 INFO SenderThread:267559 [sender.py:send_request_defer():388] handle sender defer: 7
2022-03-03 00:44:00,423 INFO SenderThread:267559 [sender.py:transition_state():392] send defer: 8
2022-03-03 00:44:00,423 DEBUG HandlerThread:267559 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:44:00,423 INFO HandlerThread:267559 [handler.py:handle_request_defer():154] handle defer: 8
2022-03-03 00:44:00,423 DEBUG SenderThread:267559 [sender.py:send_request():249] send_request: defer
2022-03-03 00:44:00,424 INFO SenderThread:267559 [sender.py:send_request_defer():388] handle sender defer: 8
2022-03-03 00:44:00,424 INFO SenderThread:267559 [sender.py:transition_state():392] send defer: 9
2022-03-03 00:44:00,424 DEBUG SenderThread:267559 [sender.py:send():235] send: final
2022-03-03 00:44:00,425 DEBUG SenderThread:267559 [sender.py:send():235] send: footer
2022-03-03 00:44:00,425 DEBUG HandlerThread:267559 [handler.py:handle_request():131] handle_request: defer
2022-03-03 00:44:00,425 INFO HandlerThread:267559 [handler.py:handle_request_defer():154] handle defer: 9
2022-03-03 00:44:00,425 DEBUG SenderThread:267559 [sender.py:send_request():249] send_request: defer
2022-03-03 00:44:00,425 INFO SenderThread:267559 [sender.py:send_request_defer():388] handle sender defer: 9
2022-03-03 00:44:00,488 DEBUG HandlerThread:267559 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-03 00:44:00,489 DEBUG SenderThread:267559 [sender.py:send_request():249] send_request: poll_exit
2022-03-03 00:44:00,489 INFO SenderThread:267559 [file_pusher.py:join():182] waiting for file pusher
2022-03-03 00:44:00,551 DEBUG HandlerThread:267559 [handler.py:handle_request():131] handle_request: get_summary
2022-03-03 00:44:00,552 DEBUG HandlerThread:267559 [handler.py:handle_request():131] handle_request: sampled_history
2022-03-03 00:44:00,552 DEBUG HandlerThread:267559 [handler.py:handle_request():131] handle_request: shutdown
2022-03-03 00:44:00,552 INFO HandlerThread:267559 [handler.py:finish():739] shutting down handler
2022-03-03 00:44:01,425 INFO WriterThread:267559 [datastore.py:close():281] close: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220303_004347-13qlgnoe/run-13qlgnoe.wandb
2022-03-03 00:44:01,549 INFO SenderThread:267559 [sender.py:finish():1075] shutting down sender
2022-03-03 00:44:01,550 INFO SenderThread:267559 [file_pusher.py:finish():177] shutting down file pusher
2022-03-03 00:44:01,550 INFO SenderThread:267559 [file_pusher.py:join():182] waiting for file pusher
2022-03-03 00:44:01,552 INFO MainThread:267559 [internal.py:handle_exit():79] Internal process exited
|