File size: 17,449 Bytes
53a9ef4 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 |
2022-03-01 17:12:01,466 INFO MainThread:245412 [internal.py:wandb_internal():89] W&B internal server running at pid: 245412, started at: 2022-03-01 17:12:01.466387
2022-03-01 17:12:01,468 DEBUG HandlerThread:245412 [handler.py:handle_request():131] handle_request: check_version
2022-03-01 17:12:01,469 INFO WriterThread:245412 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/run-3h52ha0p.wandb
2022-03-01 17:12:01,470 DEBUG SenderThread:245412 [sender.py:send():235] send: header
2022-03-01 17:12:01,470 DEBUG SenderThread:245412 [sender.py:send_request():249] send_request: check_version
2022-03-01 17:12:01,535 DEBUG SenderThread:245412 [sender.py:send():235] send: run
2022-03-01 17:12:01,658 INFO SenderThread:245412 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files
2022-03-01 17:12:01,658 INFO SenderThread:245412 [sender.py:_start_run_threads():809] run started: 3h52ha0p with start time 1646154720
2022-03-01 17:12:01,658 DEBUG SenderThread:245412 [sender.py:send():235] send: summary
2022-03-01 17:12:01,658 INFO SenderThread:245412 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 17:12:01,659 DEBUG HandlerThread:245412 [handler.py:handle_request():131] handle_request: run_start
2022-03-01 17:12:01,663 DEBUG HandlerThread:245412 [meta.py:__init__():36] meta init
2022-03-01 17:12:01,663 DEBUG HandlerThread:245412 [meta.py:__init__():50] meta init done
2022-03-01 17:12:01,663 DEBUG HandlerThread:245412 [meta.py:probe():210] probe
2022-03-01 17:12:01,669 DEBUG HandlerThread:245412 [meta.py:_setup_git():200] setup git
2022-03-01 17:12:01,684 DEBUG HandlerThread:245412 [meta.py:_setup_git():207] setup git done
2022-03-01 17:12:01,684 DEBUG HandlerThread:245412 [meta.py:_save_pip():54] save pip
2022-03-01 17:12:01,684 DEBUG HandlerThread:245412 [meta.py:_save_pip():68] save pip done
2022-03-01 17:12:01,685 DEBUG HandlerThread:245412 [meta.py:probe():248] probe done
2022-03-01 17:12:01,758 DEBUG SenderThread:245412 [sender.py:send():235] send: files
2022-03-01 17:12:01,758 INFO SenderThread:245412 [sender.py:_save_file():944] saving file wandb-metadata.json with policy now
2022-03-01 17:12:01,762 DEBUG HandlerThread:245412 [handler.py:handle_request():131] handle_request: stop_status
2022-03-01 17:12:01,763 DEBUG SenderThread:245412 [sender.py:send_request():249] send_request: stop_status
2022-03-01 17:12:01,796 DEBUG SenderThread:245412 [sender.py:send():235] send: config
2022-03-01 17:12:01,797 DEBUG SenderThread:245412 [sender.py:send():235] send: metric
2022-03-01 17:12:01,797 DEBUG SenderThread:245412 [sender.py:send():235] send: metric
2022-03-01 17:12:01,797 WARNING SenderThread:245412 [sender.py:send_metric():902] Seen metric with glob (shouldnt happen)
2022-03-01 17:12:02,036 INFO Thread-11 :245412 [upload_job.py:push():137] Uploaded file /tmp/tmpt38fg238wandb/2am5wtm4-wandb-metadata.json
2022-03-01 17:12:02,659 INFO Thread-8 :245412 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/requirements.txt
2022-03-01 17:12:02,660 INFO Thread-8 :245412 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/output.log
2022-03-01 17:12:02,660 INFO Thread-8 :245412 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/wandb-metadata.json
2022-03-01 17:12:02,660 INFO Thread-8 :245412 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/wandb-summary.json
2022-03-01 17:12:04,659 INFO Thread-8 :245412 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/output.log
2022-03-01 17:12:08,660 INFO Thread-8 :245412 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/output.log
2022-03-01 17:12:10,572 DEBUG HandlerThread:245412 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:12:10,572 DEBUG SenderThread:245412 [sender.py:send():235] send: telemetry
2022-03-01 17:12:10,573 DEBUG SenderThread:245412 [sender.py:send():235] send: exit
2022-03-01 17:12:10,573 INFO SenderThread:245412 [sender.py:send_exit():371] handling exit code: 1
2022-03-01 17:12:10,573 INFO SenderThread:245412 [sender.py:send_exit():373] handling runtime: 8
2022-03-01 17:12:10,573 INFO SenderThread:245412 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 17:12:10,573 INFO SenderThread:245412 [sender.py:send_exit():379] send defer
2022-03-01 17:12:10,574 DEBUG SenderThread:245412 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:12:10,574 DEBUG HandlerThread:245412 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:12:10,574 INFO HandlerThread:245412 [handler.py:handle_request_defer():154] handle defer: 0
2022-03-01 17:12:10,574 DEBUG SenderThread:245412 [sender.py:send_request():249] send_request: defer
2022-03-01 17:12:10,574 INFO SenderThread:245412 [sender.py:send_request_defer():388] handle sender defer: 0
2022-03-01 17:12:10,575 INFO SenderThread:245412 [sender.py:transition_state():392] send defer: 1
2022-03-01 17:12:10,575 DEBUG HandlerThread:245412 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:12:10,575 INFO HandlerThread:245412 [handler.py:handle_request_defer():154] handle defer: 1
2022-03-01 17:12:10,661 INFO Thread-8 :245412 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/output.log
2022-03-01 17:12:10,661 INFO Thread-8 :245412 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/wandb-summary.json
2022-03-01 17:12:10,684 DEBUG SenderThread:245412 [sender.py:send_request():249] send_request: defer
2022-03-01 17:12:10,684 INFO SenderThread:245412 [sender.py:send_request_defer():388] handle sender defer: 1
2022-03-01 17:12:10,684 INFO SenderThread:245412 [sender.py:transition_state():392] send defer: 2
2022-03-01 17:12:10,684 DEBUG HandlerThread:245412 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:12:10,685 DEBUG SenderThread:245412 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:12:10,685 DEBUG SenderThread:245412 [sender.py:send():235] send: stats
2022-03-01 17:12:10,685 DEBUG HandlerThread:245412 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:12:10,686 INFO HandlerThread:245412 [handler.py:handle_request_defer():154] handle defer: 2
2022-03-01 17:12:10,686 DEBUG SenderThread:245412 [sender.py:send_request():249] send_request: defer
2022-03-01 17:12:10,686 INFO SenderThread:245412 [sender.py:send_request_defer():388] handle sender defer: 2
2022-03-01 17:12:10,686 INFO SenderThread:245412 [sender.py:transition_state():392] send defer: 3
2022-03-01 17:12:10,686 DEBUG HandlerThread:245412 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:12:10,686 INFO HandlerThread:245412 [handler.py:handle_request_defer():154] handle defer: 3
2022-03-01 17:12:10,686 DEBUG SenderThread:245412 [sender.py:send():235] send: summary
2022-03-01 17:12:10,687 INFO SenderThread:245412 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 17:12:10,687 DEBUG SenderThread:245412 [sender.py:send_request():249] send_request: defer
2022-03-01 17:12:10,687 INFO SenderThread:245412 [sender.py:send_request_defer():388] handle sender defer: 3
2022-03-01 17:12:10,687 INFO SenderThread:245412 [sender.py:transition_state():392] send defer: 4
2022-03-01 17:12:10,687 DEBUG HandlerThread:245412 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:12:10,687 INFO HandlerThread:245412 [handler.py:handle_request_defer():154] handle defer: 4
2022-03-01 17:12:10,687 DEBUG SenderThread:245412 [sender.py:send_request():249] send_request: defer
2022-03-01 17:12:10,688 INFO SenderThread:245412 [sender.py:send_request_defer():388] handle sender defer: 4
2022-03-01 17:12:10,791 DEBUG HandlerThread:245412 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:12:10,821 INFO SenderThread:245412 [sender.py:transition_state():392] send defer: 5
2022-03-01 17:12:10,821 DEBUG SenderThread:245412 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:12:10,822 DEBUG HandlerThread:245412 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:12:10,822 INFO HandlerThread:245412 [handler.py:handle_request_defer():154] handle defer: 5
2022-03-01 17:12:10,822 DEBUG SenderThread:245412 [sender.py:send_request():249] send_request: defer
2022-03-01 17:12:10,822 INFO SenderThread:245412 [sender.py:send_request_defer():388] handle sender defer: 5
2022-03-01 17:12:10,822 INFO SenderThread:245412 [dir_watcher.py:finish():283] shutting down directory watcher
2022-03-01 17:12:10,923 DEBUG HandlerThread:245412 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:12:11,662 INFO Thread-8 :245412 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/config.yaml
2022-03-01 17:12:11,662 INFO SenderThread:245412 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/wandb-summary.json
2022-03-01 17:12:11,662 INFO SenderThread:245412 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files
2022-03-01 17:12:11,663 INFO SenderThread:245412 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/wandb-metadata.json wandb-metadata.json
2022-03-01 17:12:11,663 INFO SenderThread:245412 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/output.log output.log
2022-03-01 17:12:11,663 INFO SenderThread:245412 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/wandb-summary.json wandb-summary.json
2022-03-01 17:12:11,663 INFO SenderThread:245412 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/requirements.txt requirements.txt
2022-03-01 17:12:11,663 INFO SenderThread:245412 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/config.yaml config.yaml
2022-03-01 17:12:11,666 INFO SenderThread:245412 [sender.py:transition_state():392] send defer: 6
2022-03-01 17:12:11,666 DEBUG SenderThread:245412 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:12:11,672 DEBUG HandlerThread:245412 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:12:11,673 INFO HandlerThread:245412 [handler.py:handle_request_defer():154] handle defer: 6
2022-03-01 17:12:11,673 DEBUG SenderThread:245412 [sender.py:send_request():249] send_request: defer
2022-03-01 17:12:11,673 INFO SenderThread:245412 [sender.py:send_request_defer():388] handle sender defer: 6
2022-03-01 17:12:11,674 INFO SenderThread:245412 [file_pusher.py:finish():177] shutting down file pusher
2022-03-01 17:12:11,768 DEBUG HandlerThread:245412 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:12:11,768 DEBUG SenderThread:245412 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:12:11,869 DEBUG HandlerThread:245412 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:12:11,869 DEBUG SenderThread:245412 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:12:11,967 INFO Thread-12 :245412 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/output.log
2022-03-01 17:12:11,970 INFO Thread-14 :245412 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/requirements.txt
2022-03-01 17:12:11,971 DEBUG HandlerThread:245412 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:12:11,971 DEBUG SenderThread:245412 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:12:11,976 INFO Thread-13 :245412 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/wandb-summary.json
2022-03-01 17:12:11,995 INFO Thread-15 :245412 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/files/config.yaml
2022-03-01 17:12:12,072 DEBUG HandlerThread:245412 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:12:12,073 DEBUG SenderThread:245412 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:12:12,174 DEBUG HandlerThread:245412 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:12:12,174 DEBUG SenderThread:245412 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:12:12,196 INFO Thread-7 :245412 [sender.py:transition_state():392] send defer: 7
2022-03-01 17:12:12,196 DEBUG HandlerThread:245412 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:12:12,196 INFO HandlerThread:245412 [handler.py:handle_request_defer():154] handle defer: 7
2022-03-01 17:12:12,196 DEBUG SenderThread:245412 [sender.py:send_request():249] send_request: defer
2022-03-01 17:12:12,196 INFO SenderThread:245412 [sender.py:send_request_defer():388] handle sender defer: 7
2022-03-01 17:12:12,243 INFO SenderThread:245412 [sender.py:transition_state():392] send defer: 8
2022-03-01 17:12:12,244 DEBUG HandlerThread:245412 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:12:12,244 INFO HandlerThread:245412 [handler.py:handle_request_defer():154] handle defer: 8
2022-03-01 17:12:12,244 DEBUG SenderThread:245412 [sender.py:send_request():249] send_request: defer
2022-03-01 17:12:12,244 INFO SenderThread:245412 [sender.py:send_request_defer():388] handle sender defer: 8
2022-03-01 17:12:12,244 INFO SenderThread:245412 [sender.py:transition_state():392] send defer: 9
2022-03-01 17:12:12,245 DEBUG HandlerThread:245412 [handler.py:handle_request():131] handle_request: defer
2022-03-01 17:12:12,245 INFO HandlerThread:245412 [handler.py:handle_request_defer():154] handle defer: 9
2022-03-01 17:12:12,245 DEBUG SenderThread:245412 [sender.py:send():235] send: final
2022-03-01 17:12:12,245 DEBUG SenderThread:245412 [sender.py:send():235] send: footer
2022-03-01 17:12:12,245 DEBUG SenderThread:245412 [sender.py:send_request():249] send_request: defer
2022-03-01 17:12:12,245 INFO SenderThread:245412 [sender.py:send_request_defer():388] handle sender defer: 9
2022-03-01 17:12:12,275 DEBUG HandlerThread:245412 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 17:12:12,275 DEBUG SenderThread:245412 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 17:12:12,276 INFO SenderThread:245412 [file_pusher.py:join():182] waiting for file pusher
2022-03-01 17:12:12,332 DEBUG HandlerThread:245412 [handler.py:handle_request():131] handle_request: get_summary
2022-03-01 17:12:12,333 DEBUG HandlerThread:245412 [handler.py:handle_request():131] handle_request: sampled_history
2022-03-01 17:12:12,334 DEBUG HandlerThread:245412 [handler.py:handle_request():131] handle_request: shutdown
2022-03-01 17:12:12,334 INFO HandlerThread:245412 [handler.py:finish():739] shutting down handler
2022-03-01 17:12:13,246 INFO WriterThread:245412 [datastore.py:close():281] close: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_171200-3h52ha0p/run-3h52ha0p.wandb
2022-03-01 17:12:13,331 INFO SenderThread:245412 [sender.py:finish():1075] shutting down sender
2022-03-01 17:12:13,331 INFO SenderThread:245412 [file_pusher.py:finish():177] shutting down file pusher
2022-03-01 17:12:13,331 INFO SenderThread:245412 [file_pusher.py:join():182] waiting for file pusher
2022-03-01 17:12:13,333 INFO MainThread:245412 [internal.py:handle_exit():79] Internal process exited
|