File size: 17,000 Bytes
a5afdd5 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 |
2022-03-01 01:26:49,894 INFO MainThread:236498 [internal.py:wandb_internal():89] W&B internal server running at pid: 236498, started at: 2022-03-01 01:26:49.893757
2022-03-01 01:26:49,896 INFO WriterThread:236498 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/run-nzue7jk0.wandb
2022-03-01 01:26:49,896 DEBUG HandlerThread:236498 [handler.py:handle_request():131] handle_request: check_version
2022-03-01 01:26:49,897 DEBUG SenderThread:236498 [sender.py:send():235] send: header
2022-03-01 01:26:49,898 DEBUG SenderThread:236498 [sender.py:send_request():249] send_request: check_version
2022-03-01 01:26:49,967 DEBUG SenderThread:236498 [sender.py:send():235] send: run
2022-03-01 01:26:50,066 INFO SenderThread:236498 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files
2022-03-01 01:26:50,066 INFO SenderThread:236498 [sender.py:_start_run_threads():809] run started: nzue7jk0 with start time 1646098009
2022-03-01 01:26:50,066 DEBUG SenderThread:236498 [sender.py:send():235] send: summary
2022-03-01 01:26:50,066 INFO SenderThread:236498 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 01:26:50,067 DEBUG HandlerThread:236498 [handler.py:handle_request():131] handle_request: run_start
2022-03-01 01:26:50,071 DEBUG HandlerThread:236498 [meta.py:__init__():36] meta init
2022-03-01 01:26:50,071 DEBUG HandlerThread:236498 [meta.py:__init__():50] meta init done
2022-03-01 01:26:50,071 DEBUG HandlerThread:236498 [meta.py:probe():210] probe
2022-03-01 01:26:50,078 DEBUG HandlerThread:236498 [meta.py:_setup_git():200] setup git
2022-03-01 01:26:50,092 DEBUG HandlerThread:236498 [meta.py:_setup_git():207] setup git done
2022-03-01 01:26:50,093 DEBUG HandlerThread:236498 [meta.py:_save_pip():54] save pip
2022-03-01 01:26:50,093 DEBUG HandlerThread:236498 [meta.py:_save_pip():68] save pip done
2022-03-01 01:26:50,093 DEBUG HandlerThread:236498 [meta.py:probe():248] probe done
2022-03-01 01:26:50,171 DEBUG SenderThread:236498 [sender.py:send():235] send: files
2022-03-01 01:26:50,171 INFO SenderThread:236498 [sender.py:_save_file():944] saving file wandb-metadata.json with policy now
2022-03-01 01:26:50,176 DEBUG HandlerThread:236498 [handler.py:handle_request():131] handle_request: stop_status
2022-03-01 01:26:50,176 DEBUG SenderThread:236498 [sender.py:send_request():249] send_request: stop_status
2022-03-01 01:26:50,211 DEBUG SenderThread:236498 [sender.py:send():235] send: config
2022-03-01 01:26:50,212 DEBUG SenderThread:236498 [sender.py:send():235] send: metric
2022-03-01 01:26:50,212 DEBUG SenderThread:236498 [sender.py:send():235] send: metric
2022-03-01 01:26:50,212 WARNING SenderThread:236498 [sender.py:send_metric():902] Seen metric with glob (shouldnt happen)
2022-03-01 01:26:50,430 INFO Thread-11 :236498 [upload_job.py:push():137] Uploaded file /tmp/tmpscakcml5wandb/3qv9ow3a-wandb-metadata.json
2022-03-01 01:26:51,068 INFO Thread-8 :236498 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files/wandb-summary.json
2022-03-01 01:26:51,068 INFO Thread-8 :236498 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files/output.log
2022-03-01 01:26:51,068 INFO Thread-8 :236498 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files/wandb-metadata.json
2022-03-01 01:26:51,068 INFO Thread-8 :236498 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files/requirements.txt
2022-03-01 01:26:53,067 INFO Thread-8 :236498 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files/output.log
2022-03-01 01:26:57,068 INFO Thread-8 :236498 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files/output.log
2022-03-01 01:26:58,513 DEBUG SenderThread:236498 [sender.py:send():235] send: telemetry
2022-03-01 01:26:58,513 DEBUG HandlerThread:236498 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:26:58,514 DEBUG SenderThread:236498 [sender.py:send():235] send: exit
2022-03-01 01:26:58,514 INFO SenderThread:236498 [sender.py:send_exit():371] handling exit code: 1
2022-03-01 01:26:58,514 INFO SenderThread:236498 [sender.py:send_exit():373] handling runtime: 8
2022-03-01 01:26:58,515 INFO SenderThread:236498 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 01:26:58,515 INFO SenderThread:236498 [sender.py:send_exit():379] send defer
2022-03-01 01:26:58,515 DEBUG SenderThread:236498 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:26:58,516 DEBUG HandlerThread:236498 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:26:58,516 INFO HandlerThread:236498 [handler.py:handle_request_defer():154] handle defer: 0
2022-03-01 01:26:58,516 DEBUG SenderThread:236498 [sender.py:send_request():249] send_request: defer
2022-03-01 01:26:58,516 INFO SenderThread:236498 [sender.py:send_request_defer():388] handle sender defer: 0
2022-03-01 01:26:58,516 INFO SenderThread:236498 [sender.py:transition_state():392] send defer: 1
2022-03-01 01:26:58,516 DEBUG HandlerThread:236498 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:26:58,516 INFO HandlerThread:236498 [handler.py:handle_request_defer():154] handle defer: 1
2022-03-01 01:26:58,601 DEBUG SenderThread:236498 [sender.py:send_request():249] send_request: defer
2022-03-01 01:26:58,601 INFO SenderThread:236498 [sender.py:send_request_defer():388] handle sender defer: 1
2022-03-01 01:26:58,601 INFO SenderThread:236498 [sender.py:transition_state():392] send defer: 2
2022-03-01 01:26:58,601 DEBUG SenderThread:236498 [sender.py:send():235] send: stats
2022-03-01 01:26:58,602 DEBUG HandlerThread:236498 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:26:58,602 INFO HandlerThread:236498 [handler.py:handle_request_defer():154] handle defer: 2
2022-03-01 01:26:58,602 DEBUG SenderThread:236498 [sender.py:send_request():249] send_request: defer
2022-03-01 01:26:58,602 INFO SenderThread:236498 [sender.py:send_request_defer():388] handle sender defer: 2
2022-03-01 01:26:58,602 INFO SenderThread:236498 [sender.py:transition_state():392] send defer: 3
2022-03-01 01:26:58,603 DEBUG HandlerThread:236498 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:26:58,603 INFO HandlerThread:236498 [handler.py:handle_request_defer():154] handle defer: 3
2022-03-01 01:26:58,603 DEBUG SenderThread:236498 [sender.py:send():235] send: summary
2022-03-01 01:26:58,603 INFO SenderThread:236498 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 01:26:58,603 DEBUG SenderThread:236498 [sender.py:send_request():249] send_request: defer
2022-03-01 01:26:58,603 INFO SenderThread:236498 [sender.py:send_request_defer():388] handle sender defer: 3
2022-03-01 01:26:58,603 INFO SenderThread:236498 [sender.py:transition_state():392] send defer: 4
2022-03-01 01:26:58,604 DEBUG HandlerThread:236498 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:26:58,604 INFO HandlerThread:236498 [handler.py:handle_request_defer():154] handle defer: 4
2022-03-01 01:26:58,604 DEBUG SenderThread:236498 [sender.py:send_request():249] send_request: defer
2022-03-01 01:26:58,604 INFO SenderThread:236498 [sender.py:send_request_defer():388] handle sender defer: 4
2022-03-01 01:26:58,617 DEBUG HandlerThread:236498 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:26:58,708 INFO SenderThread:236498 [sender.py:transition_state():392] send defer: 5
2022-03-01 01:26:58,708 DEBUG SenderThread:236498 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:26:58,709 DEBUG HandlerThread:236498 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:26:58,709 INFO HandlerThread:236498 [handler.py:handle_request_defer():154] handle defer: 5
2022-03-01 01:26:58,709 DEBUG SenderThread:236498 [sender.py:send_request():249] send_request: defer
2022-03-01 01:26:58,709 INFO SenderThread:236498 [sender.py:send_request_defer():388] handle sender defer: 5
2022-03-01 01:26:58,709 INFO SenderThread:236498 [dir_watcher.py:finish():283] shutting down directory watcher
2022-03-01 01:26:58,809 DEBUG HandlerThread:236498 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:26:59,069 INFO Thread-8 :236498 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files/config.yaml
2022-03-01 01:26:59,070 INFO SenderThread:236498 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files/output.log
2022-03-01 01:26:59,070 INFO SenderThread:236498 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files/wandb-summary.json
2022-03-01 01:26:59,070 INFO SenderThread:236498 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files
2022-03-01 01:26:59,070 INFO SenderThread:236498 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files/wandb-metadata.json wandb-metadata.json
2022-03-01 01:26:59,070 INFO SenderThread:236498 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files/output.log output.log
2022-03-01 01:26:59,070 INFO SenderThread:236498 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files/wandb-summary.json wandb-summary.json
2022-03-01 01:26:59,073 INFO SenderThread:236498 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files/requirements.txt requirements.txt
2022-03-01 01:26:59,074 INFO SenderThread:236498 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files/config.yaml config.yaml
2022-03-01 01:26:59,076 INFO SenderThread:236498 [sender.py:transition_state():392] send defer: 6
2022-03-01 01:26:59,076 DEBUG SenderThread:236498 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:26:59,077 DEBUG HandlerThread:236498 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:26:59,077 INFO HandlerThread:236498 [handler.py:handle_request_defer():154] handle defer: 6
2022-03-01 01:26:59,078 DEBUG SenderThread:236498 [sender.py:send_request():249] send_request: defer
2022-03-01 01:26:59,084 INFO SenderThread:236498 [sender.py:send_request_defer():388] handle sender defer: 6
2022-03-01 01:26:59,084 INFO SenderThread:236498 [file_pusher.py:finish():177] shutting down file pusher
2022-03-01 01:26:59,179 DEBUG HandlerThread:236498 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:26:59,179 DEBUG SenderThread:236498 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:26:59,281 DEBUG HandlerThread:236498 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:26:59,281 DEBUG SenderThread:236498 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:26:59,340 INFO Thread-15 :236498 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files/config.yaml
2022-03-01 01:26:59,342 INFO Thread-14 :236498 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files/requirements.txt
2022-03-01 01:26:59,369 INFO Thread-12 :236498 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files/output.log
2022-03-01 01:26:59,376 INFO Thread-13 :236498 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/files/wandb-summary.json
2022-03-01 01:26:59,383 DEBUG HandlerThread:236498 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:26:59,383 DEBUG SenderThread:236498 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:26:59,484 DEBUG HandlerThread:236498 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:26:59,484 DEBUG SenderThread:236498 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:26:59,577 INFO Thread-7 :236498 [sender.py:transition_state():392] send defer: 7
2022-03-01 01:26:59,577 DEBUG HandlerThread:236498 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:26:59,577 INFO HandlerThread:236498 [handler.py:handle_request_defer():154] handle defer: 7
2022-03-01 01:26:59,578 DEBUG SenderThread:236498 [sender.py:send_request():249] send_request: defer
2022-03-01 01:26:59,578 INFO SenderThread:236498 [sender.py:send_request_defer():388] handle sender defer: 7
2022-03-01 01:26:59,585 DEBUG HandlerThread:236498 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:26:59,682 INFO SenderThread:236498 [sender.py:transition_state():392] send defer: 8
2022-03-01 01:26:59,683 DEBUG SenderThread:236498 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:26:59,683 DEBUG HandlerThread:236498 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:26:59,683 INFO HandlerThread:236498 [handler.py:handle_request_defer():154] handle defer: 8
2022-03-01 01:26:59,683 DEBUG SenderThread:236498 [sender.py:send_request():249] send_request: defer
2022-03-01 01:26:59,684 INFO SenderThread:236498 [sender.py:send_request_defer():388] handle sender defer: 8
2022-03-01 01:26:59,684 INFO SenderThread:236498 [sender.py:transition_state():392] send defer: 9
2022-03-01 01:26:59,684 DEBUG HandlerThread:236498 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:26:59,684 INFO HandlerThread:236498 [handler.py:handle_request_defer():154] handle defer: 9
2022-03-01 01:26:59,684 DEBUG SenderThread:236498 [sender.py:send():235] send: final
2022-03-01 01:26:59,685 DEBUG SenderThread:236498 [sender.py:send():235] send: footer
2022-03-01 01:26:59,685 DEBUG SenderThread:236498 [sender.py:send_request():249] send_request: defer
2022-03-01 01:26:59,685 INFO SenderThread:236498 [sender.py:send_request_defer():388] handle sender defer: 9
2022-03-01 01:26:59,784 DEBUG HandlerThread:236498 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:26:59,785 DEBUG SenderThread:236498 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:26:59,785 INFO SenderThread:236498 [file_pusher.py:join():182] waiting for file pusher
2022-03-01 01:26:59,847 DEBUG HandlerThread:236498 [handler.py:handle_request():131] handle_request: get_summary
2022-03-01 01:26:59,847 DEBUG HandlerThread:236498 [handler.py:handle_request():131] handle_request: sampled_history
2022-03-01 01:26:59,848 DEBUG HandlerThread:236498 [handler.py:handle_request():131] handle_request: shutdown
2022-03-01 01:26:59,848 INFO HandlerThread:236498 [handler.py:finish():739] shutting down handler
2022-03-01 01:27:00,685 INFO WriterThread:236498 [datastore.py:close():281] close: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012648-nzue7jk0/run-nzue7jk0.wandb
2022-03-01 01:27:00,846 INFO SenderThread:236498 [sender.py:finish():1075] shutting down sender
2022-03-01 01:27:00,846 INFO SenderThread:236498 [file_pusher.py:finish():177] shutting down file pusher
2022-03-01 01:27:00,846 INFO SenderThread:236498 [file_pusher.py:join():182] waiting for file pusher
2022-03-01 01:27:00,848 INFO MainThread:236498 [internal.py:handle_exit():79] Internal process exited
|