File size: 17,050 Bytes
9e8de2e |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 |
2022-03-10 21:09:51,233 INFO MainThread:1044076 [internal.py:wandb_internal():89] W&B internal server running at pid: 1044076, started at: 2022-03-10 21:09:51.233570
2022-03-10 21:09:51,235 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: check_version
2022-03-10 21:09:51,235 INFO WriterThread:1044076 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/run-19sg19ou.wandb
2022-03-10 21:09:51,237 DEBUG SenderThread:1044076 [sender.py:send():235] send: header
2022-03-10 21:09:51,237 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: check_version
2022-03-10 21:09:51,306 DEBUG SenderThread:1044076 [sender.py:send():235] send: run
2022-03-10 21:09:51,397 INFO SenderThread:1044076 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files
2022-03-10 21:09:51,397 INFO SenderThread:1044076 [sender.py:_start_run_threads():809] run started: 19sg19ou with start time 1646946590
2022-03-10 21:09:51,398 DEBUG SenderThread:1044076 [sender.py:send():235] send: summary
2022-03-10 21:09:51,398 INFO SenderThread:1044076 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-10 21:09:51,398 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: run_start
2022-03-10 21:09:51,402 DEBUG HandlerThread:1044076 [meta.py:__init__():36] meta init
2022-03-10 21:09:51,403 DEBUG HandlerThread:1044076 [meta.py:__init__():50] meta init done
2022-03-10 21:09:51,403 DEBUG HandlerThread:1044076 [meta.py:probe():210] probe
2022-03-10 21:09:51,410 DEBUG HandlerThread:1044076 [meta.py:_setup_git():200] setup git
2022-03-10 21:09:51,425 DEBUG HandlerThread:1044076 [meta.py:_setup_git():207] setup git done
2022-03-10 21:09:51,426 DEBUG HandlerThread:1044076 [meta.py:_save_pip():54] save pip
2022-03-10 21:09:51,426 DEBUG HandlerThread:1044076 [meta.py:_save_pip():68] save pip done
2022-03-10 21:09:51,426 DEBUG HandlerThread:1044076 [meta.py:probe():248] probe done
2022-03-10 21:09:51,505 DEBUG SenderThread:1044076 [sender.py:send():235] send: files
2022-03-10 21:09:51,505 INFO SenderThread:1044076 [sender.py:_save_file():944] saving file wandb-metadata.json with policy now
2022-03-10 21:09:51,509 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: stop_status
2022-03-10 21:09:51,510 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: stop_status
2022-03-10 21:09:51,544 DEBUG SenderThread:1044076 [sender.py:send():235] send: config
2022-03-10 21:09:51,545 DEBUG SenderThread:1044076 [sender.py:send():235] send: metric
2022-03-10 21:09:51,546 DEBUG SenderThread:1044076 [sender.py:send():235] send: metric
2022-03-10 21:09:51,546 WARNING SenderThread:1044076 [sender.py:send_metric():902] Seen metric with glob (shouldnt happen)
2022-03-10 21:09:51,790 INFO Thread-11 :1044076 [upload_job.py:push():137] Uploaded file /tmp/tmp0u4vh57hwandb/2s9kjsb8-wandb-metadata.json
2022-03-10 21:09:52,400 INFO Thread-8 :1044076 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/wandb-summary.json
2022-03-10 21:09:52,401 INFO Thread-8 :1044076 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/requirements.txt
2022-03-10 21:09:52,401 INFO Thread-8 :1044076 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/output.log
2022-03-10 21:09:52,401 INFO Thread-8 :1044076 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/wandb-metadata.json
2022-03-10 21:09:54,398 INFO Thread-8 :1044076 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/output.log
2022-03-10 21:09:58,400 INFO Thread-8 :1044076 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/output.log
2022-03-10 21:09:59,980 DEBUG SenderThread:1044076 [sender.py:send():235] send: telemetry
2022-03-10 21:09:59,980 DEBUG SenderThread:1044076 [sender.py:send():235] send: exit
2022-03-10 21:09:59,980 INFO SenderThread:1044076 [sender.py:send_exit():371] handling exit code: 1
2022-03-10 21:09:59,981 INFO SenderThread:1044076 [sender.py:send_exit():373] handling runtime: 8
2022-03-10 21:09:59,981 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-10 21:09:59,981 INFO SenderThread:1044076 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-10 21:09:59,982 INFO SenderThread:1044076 [sender.py:send_exit():379] send defer
2022-03-10 21:09:59,982 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: poll_exit
2022-03-10 21:09:59,982 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: defer
2022-03-10 21:09:59,982 INFO HandlerThread:1044076 [handler.py:handle_request_defer():154] handle defer: 0
2022-03-10 21:09:59,983 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: defer
2022-03-10 21:09:59,983 INFO SenderThread:1044076 [sender.py:send_request_defer():388] handle sender defer: 0
2022-03-10 21:09:59,983 INFO SenderThread:1044076 [sender.py:transition_state():392] send defer: 1
2022-03-10 21:09:59,983 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: defer
2022-03-10 21:09:59,983 INFO HandlerThread:1044076 [handler.py:handle_request_defer():154] handle defer: 1
2022-03-10 21:10:00,043 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: defer
2022-03-10 21:10:00,043 INFO SenderThread:1044076 [sender.py:send_request_defer():388] handle sender defer: 1
2022-03-10 21:10:00,043 INFO SenderThread:1044076 [sender.py:transition_state():392] send defer: 2
2022-03-10 21:10:00,043 DEBUG SenderThread:1044076 [sender.py:send():235] send: stats
2022-03-10 21:10:00,044 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: defer
2022-03-10 21:10:00,044 INFO HandlerThread:1044076 [handler.py:handle_request_defer():154] handle defer: 2
2022-03-10 21:10:00,044 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: defer
2022-03-10 21:10:00,044 INFO SenderThread:1044076 [sender.py:send_request_defer():388] handle sender defer: 2
2022-03-10 21:10:00,044 INFO SenderThread:1044076 [sender.py:transition_state():392] send defer: 3
2022-03-10 21:10:00,044 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: defer
2022-03-10 21:10:00,045 INFO HandlerThread:1044076 [handler.py:handle_request_defer():154] handle defer: 3
2022-03-10 21:10:00,045 DEBUG SenderThread:1044076 [sender.py:send():235] send: summary
2022-03-10 21:10:00,045 INFO SenderThread:1044076 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-10 21:10:00,045 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: defer
2022-03-10 21:10:00,045 INFO SenderThread:1044076 [sender.py:send_request_defer():388] handle sender defer: 3
2022-03-10 21:10:00,045 INFO SenderThread:1044076 [sender.py:transition_state():392] send defer: 4
2022-03-10 21:10:00,046 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: defer
2022-03-10 21:10:00,046 INFO HandlerThread:1044076 [handler.py:handle_request_defer():154] handle defer: 4
2022-03-10 21:10:00,046 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: defer
2022-03-10 21:10:00,046 INFO SenderThread:1044076 [sender.py:send_request_defer():388] handle sender defer: 4
2022-03-10 21:10:00,083 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-10 21:10:00,172 INFO SenderThread:1044076 [sender.py:transition_state():392] send defer: 5
2022-03-10 21:10:00,172 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: poll_exit
2022-03-10 21:10:00,173 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: defer
2022-03-10 21:10:00,173 INFO HandlerThread:1044076 [handler.py:handle_request_defer():154] handle defer: 5
2022-03-10 21:10:00,173 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: defer
2022-03-10 21:10:00,173 INFO SenderThread:1044076 [sender.py:send_request_defer():388] handle sender defer: 5
2022-03-10 21:10:00,173 INFO SenderThread:1044076 [dir_watcher.py:finish():283] shutting down directory watcher
2022-03-10 21:10:00,274 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-10 21:10:00,401 INFO Thread-8 :1044076 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/wandb-summary.json
2022-03-10 21:10:00,401 INFO SenderThread:1044076 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/config.yaml
2022-03-10 21:10:00,401 INFO SenderThread:1044076 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/output.log
2022-03-10 21:10:00,402 INFO SenderThread:1044076 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files
2022-03-10 21:10:00,402 INFO SenderThread:1044076 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/wandb-metadata.json wandb-metadata.json
2022-03-10 21:10:00,402 INFO SenderThread:1044076 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/output.log output.log
2022-03-10 21:10:00,402 INFO SenderThread:1044076 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/wandb-summary.json wandb-summary.json
2022-03-10 21:10:00,402 INFO SenderThread:1044076 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/requirements.txt requirements.txt
2022-03-10 21:10:00,405 INFO SenderThread:1044076 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/config.yaml config.yaml
2022-03-10 21:10:00,408 INFO SenderThread:1044076 [sender.py:transition_state():392] send defer: 6
2022-03-10 21:10:00,408 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: poll_exit
2022-03-10 21:10:00,414 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: defer
2022-03-10 21:10:00,414 INFO HandlerThread:1044076 [handler.py:handle_request_defer():154] handle defer: 6
2022-03-10 21:10:00,415 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: defer
2022-03-10 21:10:00,415 INFO SenderThread:1044076 [sender.py:send_request_defer():388] handle sender defer: 6
2022-03-10 21:10:00,415 INFO SenderThread:1044076 [file_pusher.py:finish():177] shutting down file pusher
2022-03-10 21:10:00,510 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-10 21:10:00,510 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: poll_exit
2022-03-10 21:10:00,612 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-10 21:10:00,612 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: poll_exit
2022-03-10 21:10:00,664 INFO Thread-12 :1044076 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/output.log
2022-03-10 21:10:00,672 INFO Thread-15 :1044076 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/config.yaml
2022-03-10 21:10:00,688 INFO Thread-13 :1044076 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/wandb-summary.json
2022-03-10 21:10:00,699 INFO Thread-14 :1044076 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/files/requirements.txt
2022-03-10 21:10:00,713 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-10 21:10:00,714 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: poll_exit
2022-03-10 21:10:00,815 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-10 21:10:00,815 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: poll_exit
2022-03-10 21:10:00,899 INFO Thread-7 :1044076 [sender.py:transition_state():392] send defer: 7
2022-03-10 21:10:00,900 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: defer
2022-03-10 21:10:00,900 INFO HandlerThread:1044076 [handler.py:handle_request_defer():154] handle defer: 7
2022-03-10 21:10:00,900 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: defer
2022-03-10 21:10:00,900 INFO SenderThread:1044076 [sender.py:send_request_defer():388] handle sender defer: 7
2022-03-10 21:10:00,916 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-10 21:10:01,040 INFO SenderThread:1044076 [sender.py:transition_state():392] send defer: 8
2022-03-10 21:10:01,040 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: poll_exit
2022-03-10 21:10:01,041 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: defer
2022-03-10 21:10:01,041 INFO HandlerThread:1044076 [handler.py:handle_request_defer():154] handle defer: 8
2022-03-10 21:10:01,041 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: defer
2022-03-10 21:10:01,041 INFO SenderThread:1044076 [sender.py:send_request_defer():388] handle sender defer: 8
2022-03-10 21:10:01,041 INFO SenderThread:1044076 [sender.py:transition_state():392] send defer: 9
2022-03-10 21:10:01,042 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: defer
2022-03-10 21:10:01,042 DEBUG SenderThread:1044076 [sender.py:send():235] send: final
2022-03-10 21:10:01,042 INFO HandlerThread:1044076 [handler.py:handle_request_defer():154] handle defer: 9
2022-03-10 21:10:01,043 DEBUG SenderThread:1044076 [sender.py:send():235] send: footer
2022-03-10 21:10:01,043 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: defer
2022-03-10 21:10:01,043 INFO SenderThread:1044076 [sender.py:send_request_defer():388] handle sender defer: 9
2022-03-10 21:10:01,142 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-10 21:10:01,142 DEBUG SenderThread:1044076 [sender.py:send_request():249] send_request: poll_exit
2022-03-10 21:10:01,142 INFO SenderThread:1044076 [file_pusher.py:join():182] waiting for file pusher
2022-03-10 21:10:01,199 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: get_summary
2022-03-10 21:10:01,200 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: sampled_history
2022-03-10 21:10:01,201 DEBUG HandlerThread:1044076 [handler.py:handle_request():131] handle_request: shutdown
2022-03-10 21:10:01,201 INFO HandlerThread:1044076 [handler.py:finish():739] shutting down handler
2022-03-10 21:10:02,042 INFO WriterThread:1044076 [datastore.py:close():281] close: /home/sanchit_huggingface_co/wav2vec2-2-rnd-2-layer-bart/wandb/run-20220310_210950-19sg19ou/run-19sg19ou.wandb
2022-03-10 21:10:02,198 INFO SenderThread:1044076 [sender.py:finish():1075] shutting down sender
2022-03-10 21:10:02,198 INFO SenderThread:1044076 [file_pusher.py:finish():177] shutting down file pusher
2022-03-10 21:10:02,198 INFO SenderThread:1044076 [file_pusher.py:join():182] waiting for file pusher
2022-03-10 21:10:02,200 INFO MainThread:1044076 [internal.py:handle_exit():79] Internal process exited
|