flax-wav2vec2-2-bart-large-cnn-gradient-accumulation
/
wandb
/run-20220323_180211-1s0w56op
/logs
/debug-internal.log
| 2022-03-23 18:02:11,929 INFO MainThread:23223 [internal.py:wandb_internal():92] W&B internal server running at pid: 23223, started at: 2022-03-23 18:02:11.928790 | |
| 2022-03-23 18:02:11,954 INFO WriterThread:23223 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_180211-1s0w56op/run-1s0w56op.wandb | |
| 2022-03-23 18:02:11,955 DEBUG SenderThread:23223 [sender.py:send():235] send: header | |
| 2022-03-23 18:02:11,955 DEBUG SenderThread:23223 [sender.py:send():235] send: run | |
| 2022-03-23 18:02:12,124 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: check_version | |
| 2022-03-23 18:02:12,125 INFO SenderThread:23223 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_180211-1s0w56op/files | |
| 2022-03-23 18:02:12,125 INFO SenderThread:23223 [sender.py:_start_run_threads():812] run started: 1s0w56op with start time 1648058531 | |
| 2022-03-23 18:02:12,125 DEBUG SenderThread:23223 [sender.py:send():235] send: summary | |
| 2022-03-23 18:02:12,125 INFO SenderThread:23223 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-03-23 18:02:12,125 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: check_version | |
| 2022-03-23 18:02:12,185 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: run_start | |
| 2022-03-23 18:02:13,174 INFO Thread-7 :23223 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_180211-1s0w56op/files/wandb-summary.json | |
| 2022-03-23 18:02:14,291 DEBUG HandlerThread:23223 [meta.py:__init__():37] meta init | |
| 2022-03-23 18:02:14,291 DEBUG HandlerThread:23223 [meta.py:__init__():51] meta init done | |
| 2022-03-23 18:02:14,291 DEBUG HandlerThread:23223 [meta.py:probe():211] probe | |
| 2022-03-23 18:02:14,292 DEBUG HandlerThread:23223 [meta.py:_setup_git():201] setup git | |
| 2022-03-23 18:02:14,324 DEBUG HandlerThread:23223 [meta.py:_setup_git():208] setup git done | |
| 2022-03-23 18:02:14,324 DEBUG HandlerThread:23223 [meta.py:_save_pip():55] save pip | |
| 2022-03-23 18:02:14,325 DEBUG HandlerThread:23223 [meta.py:_save_pip():69] save pip done | |
| 2022-03-23 18:02:14,325 DEBUG HandlerThread:23223 [meta.py:probe():249] probe done | |
| 2022-03-23 18:02:14,328 DEBUG SenderThread:23223 [sender.py:send():235] send: files | |
| 2022-03-23 18:02:14,328 INFO SenderThread:23223 [sender.py:_save_file():947] saving file wandb-metadata.json with policy now | |
| 2022-03-23 18:02:14,333 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-03-23 18:02:14,333 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: stop_status | |
| 2022-03-23 18:02:14,386 DEBUG SenderThread:23223 [sender.py:send():235] send: telemetry | |
| 2022-03-23 18:02:14,784 INFO Thread-11 :23223 [upload_job.py:push():137] Uploaded file /tmp/tmpv76b5b10wandb/3rtj44eg-wandb-metadata.json | |
| 2022-03-23 18:02:15,209 INFO Thread-7 :23223 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_180211-1s0w56op/files/output.log | |
| 2022-03-23 18:02:15,209 INFO Thread-7 :23223 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_180211-1s0w56op/files/requirements.txt | |
| 2022-03-23 18:02:15,209 INFO Thread-7 :23223 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_180211-1s0w56op/files/wandb-metadata.json | |
| 2022-03-23 18:02:17,209 INFO Thread-7 :23223 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_180211-1s0w56op/files/output.log | |
| 2022-03-23 18:02:29,391 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-03-23 18:02:29,392 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: stop_status | |
| 2022-03-23 18:02:33,216 INFO Thread-7 :23223 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_180211-1s0w56op/files/output.log | |
| 2022-03-23 18:02:37,218 INFO Thread-7 :23223 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_180211-1s0w56op/files/output.log | |
| 2022-03-23 18:02:39,219 INFO Thread-7 :23223 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_180211-1s0w56op/files/output.log | |
| 2022-03-23 18:02:41,220 INFO Thread-7 :23223 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_180211-1s0w56op/files/output.log | |
| 2022-03-23 18:02:42,374 DEBUG SenderThread:23223 [sender.py:send():235] send: stats | |
| 2022-03-23 18:02:43,221 INFO Thread-7 :23223 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_180211-1s0w56op/files/output.log | |
| 2022-03-23 18:02:43,221 INFO Thread-7 :23223 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_180211-1s0w56op/files/config.yaml | |
| 2022-03-23 18:02:44,431 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-03-23 18:02:44,431 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: stop_status | |
| 2022-03-23 18:02:47,222 INFO Thread-7 :23223 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_180211-1s0w56op/files/output.log | |
| 2022-03-23 18:02:59,563 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-03-23 18:02:59,564 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: stop_status | |
| 2022-03-23 18:03:12,449 DEBUG SenderThread:23223 [sender.py:send():235] send: stats | |
| 2022-03-23 18:03:14,653 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-03-23 18:03:14,653 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: stop_status | |
| 2022-03-23 18:03:29,799 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-03-23 18:03:29,800 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: stop_status | |
| 2022-03-23 18:03:42,529 DEBUG SenderThread:23223 [sender.py:send():235] send: stats | |
| 2022-03-23 18:03:44,842 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-03-23 18:03:44,842 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: stop_status | |
| 2022-03-23 18:03:59,876 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-03-23 18:03:59,877 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: stop_status | |
| 2022-03-23 18:04:12,604 DEBUG SenderThread:23223 [sender.py:send():235] send: stats | |
| 2022-03-23 18:04:14,919 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-03-23 18:04:14,919 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: stop_status | |
| 2022-03-23 18:04:29,960 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-03-23 18:04:29,961 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: stop_status | |
| 2022-03-23 18:04:42,679 DEBUG SenderThread:23223 [sender.py:send():235] send: stats | |
| 2022-03-23 18:04:44,996 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-03-23 18:04:44,996 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: stop_status | |
| 2022-03-23 18:05:00,038 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-03-23 18:05:00,039 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: stop_status | |
| 2022-03-23 18:05:12,756 DEBUG SenderThread:23223 [sender.py:send():235] send: stats | |
| 2022-03-23 18:05:15,081 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-03-23 18:05:15,081 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: stop_status | |
| 2022-03-23 18:05:30,164 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-03-23 18:05:30,164 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: stop_status | |
| 2022-03-23 18:05:42,833 DEBUG SenderThread:23223 [sender.py:send():235] send: stats | |
| 2022-03-23 18:05:45,263 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-03-23 18:05:45,264 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: stop_status | |
| 2022-03-23 18:06:00,312 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-03-23 18:06:00,312 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: stop_status | |
| 2022-03-23 18:06:09,832 DEBUG SenderThread:23223 [sender.py:send():235] send: telemetry | |
| 2022-03-23 18:06:09,832 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-03-23 18:06:11,964 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-03-23 18:06:11,964 DEBUG SenderThread:23223 [sender.py:send():235] send: exit | |
| 2022-03-23 18:06:11,964 INFO SenderThread:23223 [sender.py:send_exit():371] handling exit code: 1 | |
| 2022-03-23 18:06:11,965 INFO SenderThread:23223 [sender.py:send_exit():373] handling runtime: 239 | |
| 2022-03-23 18:06:11,965 INFO SenderThread:23223 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-03-23 18:06:11,965 INFO SenderThread:23223 [sender.py:send_exit():379] send defer | |
| 2022-03-23 18:06:11,965 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-03-23 18:06:11,966 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: defer | |
| 2022-03-23 18:06:11,966 INFO HandlerThread:23223 [handler.py:handle_request_defer():164] handle defer: 0 | |
| 2022-03-23 18:06:11,966 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: defer | |
| 2022-03-23 18:06:11,966 INFO SenderThread:23223 [sender.py:send_request_defer():388] handle sender defer: 0 | |
| 2022-03-23 18:06:11,966 INFO SenderThread:23223 [sender.py:transition_state():392] send defer: 1 | |
| 2022-03-23 18:06:11,966 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: defer | |
| 2022-03-23 18:06:11,967 INFO HandlerThread:23223 [handler.py:handle_request_defer():164] handle defer: 1 | |
| 2022-03-23 18:06:12,008 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: defer | |
| 2022-03-23 18:06:12,008 INFO SenderThread:23223 [sender.py:send_request_defer():388] handle sender defer: 1 | |
| 2022-03-23 18:06:12,008 INFO SenderThread:23223 [sender.py:transition_state():392] send defer: 2 | |
| 2022-03-23 18:06:12,008 DEBUG SenderThread:23223 [sender.py:send():235] send: stats | |
| 2022-03-23 18:06:12,009 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: defer | |
| 2022-03-23 18:06:12,009 INFO HandlerThread:23223 [handler.py:handle_request_defer():164] handle defer: 2 | |
| 2022-03-23 18:06:12,009 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: defer | |
| 2022-03-23 18:06:12,009 INFO SenderThread:23223 [sender.py:send_request_defer():388] handle sender defer: 2 | |
| 2022-03-23 18:06:12,009 INFO SenderThread:23223 [sender.py:transition_state():392] send defer: 3 | |
| 2022-03-23 18:06:12,010 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: defer | |
| 2022-03-23 18:06:12,010 INFO HandlerThread:23223 [handler.py:handle_request_defer():164] handle defer: 3 | |
| 2022-03-23 18:06:12,010 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: defer | |
| 2022-03-23 18:06:12,010 INFO SenderThread:23223 [sender.py:send_request_defer():388] handle sender defer: 3 | |
| 2022-03-23 18:06:12,010 INFO SenderThread:23223 [sender.py:transition_state():392] send defer: 4 | |
| 2022-03-23 18:06:12,010 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: defer | |
| 2022-03-23 18:06:12,010 INFO HandlerThread:23223 [handler.py:handle_request_defer():164] handle defer: 4 | |
| 2022-03-23 18:06:12,010 DEBUG SenderThread:23223 [sender.py:send():235] send: summary | |
| 2022-03-23 18:06:12,010 INFO SenderThread:23223 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-03-23 18:06:12,011 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: defer | |
| 2022-03-23 18:06:12,011 INFO SenderThread:23223 [sender.py:send_request_defer():388] handle sender defer: 4 | |
| 2022-03-23 18:06:12,011 INFO SenderThread:23223 [sender.py:transition_state():392] send defer: 5 | |
| 2022-03-23 18:06:12,011 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: defer | |
| 2022-03-23 18:06:12,011 INFO HandlerThread:23223 [handler.py:handle_request_defer():164] handle defer: 5 | |
| 2022-03-23 18:06:12,011 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: defer | |
| 2022-03-23 18:06:12,011 INFO SenderThread:23223 [sender.py:send_request_defer():388] handle sender defer: 5 | |
| 2022-03-23 18:06:12,067 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-03-23 18:06:12,078 INFO SenderThread:23223 [sender.py:transition_state():392] send defer: 6 | |
| 2022-03-23 18:06:12,078 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-03-23 18:06:12,078 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: defer | |
| 2022-03-23 18:06:12,079 INFO HandlerThread:23223 [handler.py:handle_request_defer():164] handle defer: 6 | |
| 2022-03-23 18:06:12,079 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: defer | |
| 2022-03-23 18:06:12,079 INFO SenderThread:23223 [sender.py:send_request_defer():388] handle sender defer: 6 | |
| 2022-03-23 18:06:12,079 INFO SenderThread:23223 [dir_watcher.py:finish():283] shutting down directory watcher | |
| 2022-03-23 18:06:12,179 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-03-23 18:06:12,270 INFO SenderThread:23223 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_180211-1s0w56op/files/output.log | |
| 2022-03-23 18:06:12,270 INFO SenderThread:23223 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_180211-1s0w56op/files/config.yaml | |
| 2022-03-23 18:06:12,270 INFO SenderThread:23223 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_180211-1s0w56op/files/wandb-summary.json | |
| 2022-03-23 18:06:12,270 INFO SenderThread:23223 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_180211-1s0w56op/files | |
| 2022-03-23 18:06:12,270 INFO SenderThread:23223 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_180211-1s0w56op/files/requirements.txt requirements.txt | |
| 2022-03-23 18:06:12,270 INFO SenderThread:23223 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_180211-1s0w56op/files/output.log output.log | |
| 2022-03-23 18:06:12,270 INFO SenderThread:23223 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_180211-1s0w56op/files/wandb-metadata.json wandb-metadata.json | |
| 2022-03-23 18:06:12,270 INFO SenderThread:23223 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_180211-1s0w56op/files/config.yaml config.yaml | |
| 2022-03-23 18:06:12,271 INFO SenderThread:23223 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_180211-1s0w56op/files/wandb-summary.json wandb-summary.json | |
| 2022-03-23 18:06:12,271 INFO SenderThread:23223 [sender.py:transition_state():392] send defer: 7 | |
| 2022-03-23 18:06:12,271 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-03-23 18:06:12,274 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: defer | |
| 2022-03-23 18:06:12,274 INFO HandlerThread:23223 [handler.py:handle_request_defer():164] handle defer: 7 | |
| 2022-03-23 18:06:12,276 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: defer | |
| 2022-03-23 18:06:12,276 INFO SenderThread:23223 [sender.py:send_request_defer():388] handle sender defer: 7 | |
| 2022-03-23 18:06:12,276 INFO SenderThread:23223 [file_pusher.py:finish():145] shutting down file pusher | |
| 2022-03-23 18:06:12,372 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-03-23 18:06:12,373 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-03-23 18:06:12,474 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-03-23 18:06:12,474 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-03-23 18:06:12,542 INFO Thread-12 :23223 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_180211-1s0w56op/files/requirements.txt | |
| 2022-03-23 18:06:12,571 INFO Thread-13 :23223 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_180211-1s0w56op/files/output.log | |
| 2022-03-23 18:06:12,575 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-03-23 18:06:12,576 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-03-23 18:06:12,656 INFO Thread-14 :23223 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_180211-1s0w56op/files/config.yaml | |
| 2022-03-23 18:06:12,677 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-03-23 18:06:12,677 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-03-23 18:06:12,778 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-03-23 18:06:12,779 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-03-23 18:06:12,880 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-03-23 18:06:12,880 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-03-23 18:06:12,981 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-03-23 18:06:12,981 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-03-23 18:06:13,082 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-03-23 18:06:13,083 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-03-23 18:06:13,184 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-03-23 18:06:13,184 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-03-23 18:06:13,285 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-03-23 18:06:13,286 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-03-23 18:06:13,387 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-03-23 18:06:13,387 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-03-23 18:06:13,488 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-03-23 18:06:13,488 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-03-23 18:06:13,539 INFO Thread-15 :23223 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_180211-1s0w56op/files/wandb-summary.json | |
| 2022-03-23 18:06:13,590 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-03-23 18:06:13,590 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-03-23 18:06:13,691 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-03-23 18:06:13,691 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-03-23 18:06:13,739 INFO Thread-6 :23223 [sender.py:transition_state():392] send defer: 8 | |
| 2022-03-23 18:06:13,740 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: defer | |
| 2022-03-23 18:06:13,740 INFO HandlerThread:23223 [handler.py:handle_request_defer():164] handle defer: 8 | |
| 2022-03-23 18:06:13,740 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: defer | |
| 2022-03-23 18:06:13,740 INFO SenderThread:23223 [sender.py:send_request_defer():388] handle sender defer: 8 | |
| 2022-03-23 18:06:13,792 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-03-23 18:06:13,930 INFO SenderThread:23223 [sender.py:transition_state():392] send defer: 9 | |
| 2022-03-23 18:06:13,930 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-03-23 18:06:13,931 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: defer | |
| 2022-03-23 18:06:13,931 INFO HandlerThread:23223 [handler.py:handle_request_defer():164] handle defer: 9 | |
| 2022-03-23 18:06:13,931 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: defer | |
| 2022-03-23 18:06:13,931 INFO SenderThread:23223 [sender.py:send_request_defer():388] handle sender defer: 9 | |
| 2022-03-23 18:06:13,931 INFO SenderThread:23223 [sender.py:transition_state():392] send defer: 10 | |
| 2022-03-23 18:06:13,931 DEBUG SenderThread:23223 [sender.py:send():235] send: final | |
| 2022-03-23 18:06:13,931 DEBUG SenderThread:23223 [sender.py:send():235] send: footer | |
| 2022-03-23 18:06:13,932 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: defer | |
| 2022-03-23 18:06:13,932 INFO HandlerThread:23223 [handler.py:handle_request_defer():164] handle defer: 10 | |
| 2022-03-23 18:06:13,932 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: defer | |
| 2022-03-23 18:06:13,932 INFO SenderThread:23223 [sender.py:send_request_defer():388] handle sender defer: 10 | |
| 2022-03-23 18:06:14,031 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-03-23 18:06:14,032 DEBUG SenderThread:23223 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-03-23 18:06:14,032 INFO SenderThread:23223 [file_pusher.py:join():150] waiting for file pusher | |
| 2022-03-23 18:06:14,208 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: sampled_history | |
| 2022-03-23 18:06:14,208 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: get_summary | |
| 2022-03-23 18:06:14,209 DEBUG HandlerThread:23223 [handler.py:handle_request():141] handle_request: shutdown | |
| 2022-03-23 18:06:14,209 INFO HandlerThread:23223 [handler.py:finish():778] shutting down handler | |
| 2022-03-23 18:06:14,932 INFO WriterThread:23223 [datastore.py:close():281] close: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_180211-1s0w56op/run-1s0w56op.wandb | |
| 2022-03-23 18:06:15,107 INFO SenderThread:23223 [sender.py:finish():1078] shutting down sender | |
| 2022-03-23 18:06:15,107 INFO SenderThread:23223 [file_pusher.py:finish():145] shutting down file pusher | |
| 2022-03-23 18:06:15,107 INFO SenderThread:23223 [file_pusher.py:join():150] waiting for file pusher | |
| 2022-03-23 18:06:15,109 INFO MainThread:23223 [internal.py:handle_exit():82] Internal process exited | |