| 2022-05-07 18:12:33,358 INFO MainThread:32132 [internal.py:wandb_internal():92] W&B internal server running at pid: 32132, started at: 2022-05-07 18:12:33.358518 | |
| 2022-05-07 18:12:33,361 INFO WriterThread:32132 [datastore.py:open_for_write():77] open: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/run-3jt20u8h.wandb | |
| 2022-05-07 18:12:33,362 DEBUG SenderThread:32132 [sender.py:send():235] send: header | |
| 2022-05-07 18:12:33,362 DEBUG SenderThread:32132 [sender.py:send():235] send: run | |
| 2022-05-07 18:12:33,469 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: check_version | |
| 2022-05-07 18:12:33,469 INFO SenderThread:32132 [dir_watcher.py:__init__():169] watching files in: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files | |
| 2022-05-07 18:12:33,469 INFO SenderThread:32132 [sender.py:_start_run_threads():812] run started: 3jt20u8h with start time 1651947152 | |
| 2022-05-07 18:12:33,470 DEBUG SenderThread:32132 [sender.py:send():235] send: summary | |
| 2022-05-07 18:12:33,470 INFO SenderThread:32132 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 18:12:33,470 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: check_version | |
| 2022-05-07 18:12:33,533 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: run_start | |
| 2022-05-07 18:12:34,523 INFO Thread-7 :32132 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/wandb-summary.json | |
| 2022-05-07 18:12:35,648 DEBUG HandlerThread:32132 [meta.py:__init__():37] meta init | |
| 2022-05-07 18:12:35,649 DEBUG HandlerThread:32132 [meta.py:__init__():51] meta init done | |
| 2022-05-07 18:12:35,649 DEBUG HandlerThread:32132 [meta.py:probe():211] probe | |
| 2022-05-07 18:12:35,650 DEBUG HandlerThread:32132 [meta.py:_setup_git():201] setup git | |
| 2022-05-07 18:12:35,679 DEBUG HandlerThread:32132 [meta.py:_setup_git():208] setup git done | |
| 2022-05-07 18:12:35,679 DEBUG HandlerThread:32132 [meta.py:_save_pip():55] save pip | |
| 2022-05-07 18:12:35,679 DEBUG HandlerThread:32132 [meta.py:_save_pip():69] save pip done | |
| 2022-05-07 18:12:35,680 DEBUG HandlerThread:32132 [meta.py:probe():249] probe done | |
| 2022-05-07 18:12:35,683 DEBUG SenderThread:32132 [sender.py:send():235] send: files | |
| 2022-05-07 18:12:35,683 INFO SenderThread:32132 [sender.py:_save_file():947] saving file wandb-metadata.json with policy now | |
| 2022-05-07 18:12:35,688 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-05-07 18:12:35,689 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: stop_status | |
| 2022-05-07 18:12:35,803 DEBUG SenderThread:32132 [sender.py:send():235] send: telemetry | |
| 2022-05-07 18:12:36,001 INFO Thread-11 :32132 [upload_job.py:push():137] Uploaded file /tmp/tmpi1_5w_fvwandb/15jjf330-wandb-metadata.json | |
| 2022-05-07 18:12:36,561 INFO Thread-7 :32132 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:12:36,562 INFO Thread-7 :32132 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/wandb-metadata.json | |
| 2022-05-07 18:12:36,562 INFO Thread-7 :32132 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/requirements.txt | |
| 2022-05-07 18:12:38,562 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:12:40,563 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:12:50,567 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:12:50,805 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-05-07 18:12:50,805 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: stop_status | |
| 2022-05-07 18:12:52,568 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:13:03,731 DEBUG SenderThread:32132 [sender.py:send():235] send: stats | |
| 2022-05-07 18:13:04,574 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/config.yaml | |
| 2022-05-07 18:13:05,883 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-05-07 18:13:05,883 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: stop_status | |
| 2022-05-07 18:13:16,579 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:13:21,679 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-05-07 18:13:21,679 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: stop_status | |
| 2022-05-07 18:13:33,808 DEBUG SenderThread:32132 [sender.py:send():235] send: stats | |
| 2022-05-07 18:13:36,713 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-05-07 18:13:36,714 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: stop_status | |
| 2022-05-07 18:13:37,400 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:13:37,498 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:13:38,588 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:13:38,770 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:13:38,772 DEBUG SenderThread:32132 [sender.py:send():235] send: history | |
| 2022-05-07 18:13:38,773 DEBUG SenderThread:32132 [sender.py:send():235] send: summary | |
| 2022-05-07 18:13:38,773 INFO SenderThread:32132 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 18:13:38,855 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:13:39,588 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/wandb-summary.json | |
| 2022-05-07 18:13:39,926 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:13:39,928 DEBUG SenderThread:32132 [sender.py:send():235] send: history | |
| 2022-05-07 18:13:39,929 DEBUG SenderThread:32132 [sender.py:send():235] send: summary | |
| 2022-05-07 18:13:39,930 INFO SenderThread:32132 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 18:13:39,985 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:13:40,589 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:13:40,589 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/wandb-summary.json | |
| 2022-05-07 18:13:41,129 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:13:41,131 DEBUG SenderThread:32132 [sender.py:send():235] send: history | |
| 2022-05-07 18:13:41,131 DEBUG SenderThread:32132 [sender.py:send():235] send: summary | |
| 2022-05-07 18:13:41,132 INFO SenderThread:32132 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 18:13:41,208 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:13:41,589 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/wandb-summary.json | |
| 2022-05-07 18:13:42,305 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:13:42,307 DEBUG SenderThread:32132 [sender.py:send():235] send: history | |
| 2022-05-07 18:13:42,308 DEBUG SenderThread:32132 [sender.py:send():235] send: summary | |
| 2022-05-07 18:13:42,309 INFO SenderThread:32132 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 18:13:42,378 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:13:42,590 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:13:42,590 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/wandb-summary.json | |
| 2022-05-07 18:13:44,591 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:13:53,231 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-05-07 18:13:53,231 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: stop_status | |
| 2022-05-07 18:14:03,884 DEBUG SenderThread:32132 [sender.py:send():235] send: stats | |
| 2022-05-07 18:14:08,287 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-05-07 18:14:08,287 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: stop_status | |
| 2022-05-07 18:14:10,601 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:14:23,368 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-05-07 18:14:23,368 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: stop_status | |
| 2022-05-07 18:14:28,608 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:14:33,965 DEBUG SenderThread:32132 [sender.py:send():235] send: stats | |
| 2022-05-07 18:14:38,815 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-05-07 18:14:38,815 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: stop_status | |
| 2022-05-07 18:14:50,053 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:14:50,104 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: log_artifact | |
| 2022-05-07 18:14:50,104 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: log_artifact | |
| 2022-05-07 18:14:50,213 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:14:50,498 INFO Thread-13 :32132 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/03/1aa3918a194407ac466420b1ea5405 | |
| 2022-05-07 18:14:50,617 INFO Thread-7 :32132 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/media/table/eval/step_5k_5_f901d5b4b0d1263be066.table.json | |
| 2022-05-07 18:14:50,617 INFO Thread-7 :32132 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/media/table | |
| 2022-05-07 18:14:50,617 INFO Thread-7 :32132 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/media/table/eval | |
| 2022-05-07 18:14:50,617 INFO Thread-7 :32132 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/media | |
| 2022-05-07 18:14:50,903 INFO SenderThread:32132 [sender.py:send_request_log_artifact():976] logged artifact run-3jt20u8h-evalstep_5k - {'id': 'QXJ0aWZhY3Q6MTI0MzYzMDQy', 'digest': 'c378d5ba791b4793379472a7d8e8aae6', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5OTAwMjMz', 'latestArtifact': None}, 'version': 'latest'} | |
| 2022-05-07 18:14:50,904 DEBUG SenderThread:32132 [sender.py:send():235] send: files | |
| 2022-05-07 18:14:50,904 INFO SenderThread:32132 [sender.py:_save_file():947] saving file media/table/eval/step_5k_5_f901d5b4b0d1263be066.table.json with policy now | |
| 2022-05-07 18:14:51,150 INFO Thread-14 :32132 [upload_job.py:push():137] Uploaded file /tmp/tmpi1_5w_fvwandb/1ffvxtk5-media/table/eval/step_5k_5_f901d5b4b0d1263be066.table.json | |
| 2022-05-07 18:14:51,322 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:14:51,326 DEBUG SenderThread:32132 [sender.py:send():235] send: history | |
| 2022-05-07 18:14:51,327 DEBUG SenderThread:32132 [sender.py:send():235] send: summary | |
| 2022-05-07 18:14:51,328 INFO SenderThread:32132 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 18:14:51,412 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:14:51,618 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:14:51,619 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/wandb-summary.json | |
| 2022-05-07 18:14:52,550 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:14:52,552 DEBUG SenderThread:32132 [sender.py:send():235] send: history | |
| 2022-05-07 18:14:52,553 DEBUG SenderThread:32132 [sender.py:send():235] send: summary | |
| 2022-05-07 18:14:52,554 INFO SenderThread:32132 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 18:14:52,619 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:14:52,619 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/wandb-summary.json | |
| 2022-05-07 18:14:52,629 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:14:53,619 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:14:53,719 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:14:53,721 DEBUG SenderThread:32132 [sender.py:send():235] send: history | |
| 2022-05-07 18:14:53,722 DEBUG SenderThread:32132 [sender.py:send():235] send: summary | |
| 2022-05-07 18:14:53,722 INFO SenderThread:32132 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 18:14:53,799 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:14:54,003 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-05-07 18:14:54,003 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: stop_status | |
| 2022-05-07 18:14:54,620 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:14:54,620 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/wandb-summary.json | |
| 2022-05-07 18:14:54,983 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:14:54,986 DEBUG SenderThread:32132 [sender.py:send():235] send: history | |
| 2022-05-07 18:14:54,986 DEBUG SenderThread:32132 [sender.py:send():235] send: summary | |
| 2022-05-07 18:14:54,987 INFO SenderThread:32132 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 18:14:55,035 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:14:55,620 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:14:55,621 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/wandb-summary.json | |
| 2022-05-07 18:14:56,226 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:14:56,228 DEBUG SenderThread:32132 [sender.py:send():235] send: history | |
| 2022-05-07 18:14:56,229 DEBUG SenderThread:32132 [sender.py:send():235] send: summary | |
| 2022-05-07 18:14:56,230 INFO SenderThread:32132 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 18:14:56,295 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:14:56,621 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:14:56,621 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/wandb-summary.json | |
| 2022-05-07 18:14:57,621 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:14:58,563 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:14:58,617 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: log_artifact | |
| 2022-05-07 18:14:58,617 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: log_artifact | |
| 2022-05-07 18:14:58,622 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:14:58,622 INFO Thread-7 :32132 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/media/table/eval/step_10k_10_92ae240f2472ae79ad31.table.json | |
| 2022-05-07 18:14:58,622 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/media/table/eval | |
| 2022-05-07 18:14:58,724 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:14:59,622 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:14:59,786 INFO Thread-16 :32132 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/00/66783abbd98b29e02682346c82293a | |
| 2022-05-07 18:14:59,815 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:14:59,888 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:15:00,210 INFO SenderThread:32132 [sender.py:send_request_log_artifact():976] logged artifact run-3jt20u8h-evalstep_10k - {'id': 'QXJ0aWZhY3Q6MTI0MzYzMDkw', 'digest': '15bcd05eec47537f199cd22089cba95f', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5OTAwMjM3', 'latestArtifact': None}, 'version': 'latest'} | |
| 2022-05-07 18:15:00,210 DEBUG SenderThread:32132 [sender.py:send():235] send: files | |
| 2022-05-07 18:15:00,211 INFO SenderThread:32132 [sender.py:_save_file():947] saving file media/table/eval/step_10k_10_92ae240f2472ae79ad31.table.json with policy now | |
| 2022-05-07 18:15:00,211 DEBUG SenderThread:32132 [sender.py:send():235] send: history | |
| 2022-05-07 18:15:00,212 DEBUG SenderThread:32132 [sender.py:send():235] send: summary | |
| 2022-05-07 18:15:00,213 INFO SenderThread:32132 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 18:15:00,466 INFO Thread-17 :32132 [upload_job.py:push():137] Uploaded file /tmp/tmpi1_5w_fvwandb/jmsn8rdp-media/table/eval/step_10k_10_92ae240f2472ae79ad31.table.json | |
| 2022-05-07 18:15:00,623 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:15:00,623 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/wandb-summary.json | |
| 2022-05-07 18:15:00,988 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:15:00,990 DEBUG SenderThread:32132 [sender.py:send():235] send: history | |
| 2022-05-07 18:15:00,990 DEBUG SenderThread:32132 [sender.py:send():235] send: summary | |
| 2022-05-07 18:15:00,991 INFO SenderThread:32132 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 18:15:01,070 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:15:01,623 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:15:01,624 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/wandb-summary.json | |
| 2022-05-07 18:15:02,290 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:15:02,292 DEBUG SenderThread:32132 [sender.py:send():235] send: history | |
| 2022-05-07 18:15:02,293 DEBUG SenderThread:32132 [sender.py:send():235] send: summary | |
| 2022-05-07 18:15:02,294 INFO SenderThread:32132 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 18:15:02,352 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:15:02,624 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:15:02,624 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/wandb-summary.json | |
| 2022-05-07 18:15:03,454 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:15:03,457 DEBUG SenderThread:32132 [sender.py:send():235] send: history | |
| 2022-05-07 18:15:03,457 DEBUG SenderThread:32132 [sender.py:send():235] send: summary | |
| 2022-05-07 18:15:03,458 INFO SenderThread:32132 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 18:15:03,538 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:15:03,625 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/wandb-summary.json | |
| 2022-05-07 18:15:04,045 DEBUG SenderThread:32132 [sender.py:send():235] send: stats | |
| 2022-05-07 18:15:04,625 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:15:04,671 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:15:04,673 DEBUG SenderThread:32132 [sender.py:send():235] send: history | |
| 2022-05-07 18:15:04,674 DEBUG SenderThread:32132 [sender.py:send():235] send: summary | |
| 2022-05-07 18:15:04,675 INFO SenderThread:32132 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 18:15:04,724 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:15:05,625 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/wandb-summary.json | |
| 2022-05-07 18:15:06,626 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:15:07,626 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:15:09,091 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-05-07 18:15:09,091 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: stop_status | |
| 2022-05-07 18:15:20,632 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:15:24,125 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-05-07 18:15:24,126 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: stop_status | |
| 2022-05-07 18:15:32,638 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:15:34,124 DEBUG SenderThread:32132 [sender.py:send():235] send: stats | |
| 2022-05-07 18:15:39,168 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-05-07 18:15:39,168 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: stop_status | |
| 2022-05-07 18:15:43,642 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:15:43,703 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: log_artifact | |
| 2022-05-07 18:15:43,703 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: log_artifact | |
| 2022-05-07 18:15:43,767 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:15:43,826 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: log_artifact | |
| 2022-05-07 18:15:43,921 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:15:44,644 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:15:44,644 INFO Thread-7 :32132 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/media/table/eval/step_15k_15_a7d6e656cdc47be11a4d.table.json | |
| 2022-05-07 18:15:44,644 INFO Thread-7 :32132 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/media/table/eval/final_generation_incorrect_15_27d5678dc49240a7dde2.table.json | |
| 2022-05-07 18:15:44,645 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/media/table/eval | |
| 2022-05-07 18:15:45,644 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:15:45,757 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:15:45,811 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: log_artifact | |
| 2022-05-07 18:15:45,867 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:15:45,921 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: log_artifact | |
| 2022-05-07 18:15:46,114 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:15:46,645 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:15:46,645 INFO Thread-7 :32132 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/media/table/validation[:90%]/final_generation_incorrect_15_932e28f0ba95df4e01c5.table.json | |
| 2022-05-07 18:15:46,645 INFO Thread-7 :32132 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/media/table/validation[:90%]/step_15k_15_b59f86c33aa51dff79ab.table.json | |
| 2022-05-07 18:15:46,645 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/media/table | |
| 2022-05-07 18:15:46,646 INFO Thread-7 :32132 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/media/table/validation[:90%] | |
| 2022-05-07 18:15:47,646 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:15:48,423 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:15:48,463 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: log_artifact | |
| 2022-05-07 18:15:48,552 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:15:48,592 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: log_artifact | |
| 2022-05-07 18:15:48,646 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:15:48,646 INFO Thread-7 :32132 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/media/table/validation[:95%]/step_15k_15_a7d6e656cdc47be11a4d.table.json | |
| 2022-05-07 18:15:48,647 INFO Thread-7 :32132 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/media/table/validation[:95%]/final_generation_incorrect_15_27d5678dc49240a7dde2.table.json | |
| 2022-05-07 18:15:48,647 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/media/table | |
| 2022-05-07 18:15:48,647 INFO Thread-7 :32132 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/media/table/validation[:95%] | |
| 2022-05-07 18:15:48,666 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:15:48,701 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:15:49,647 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:15:50,390 INFO Thread-19 :32132 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/59/70265576a772597adeb57dd33a7a30 | |
| 2022-05-07 18:15:50,648 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:15:50,799 INFO SenderThread:32132 [sender.py:send_request_log_artifact():976] logged artifact run-3jt20u8h-evalstep_15k - {'id': 'QXJ0aWZhY3Q6MTI0MzYzMzM1', 'digest': '1375951cf35bbd2aaa8faa80b86af2b0', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5OTAwMjQ1', 'latestArtifact': None}, 'version': 'latest'} | |
| 2022-05-07 18:15:50,799 DEBUG SenderThread:32132 [sender.py:send():235] send: files | |
| 2022-05-07 18:15:50,799 INFO SenderThread:32132 [sender.py:_save_file():947] saving file media/table/eval/step_15k_15_a7d6e656cdc47be11a4d.table.json with policy now | |
| 2022-05-07 18:15:50,800 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: log_artifact | |
| 2022-05-07 18:15:51,034 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-05-07 18:15:51,050 INFO Thread-20 :32132 [upload_job.py:push():137] Uploaded file /tmp/tmpi1_5w_fvwandb/8fmkeroj-media/table/eval/step_15k_15_a7d6e656cdc47be11a4d.table.json | |
| 2022-05-07 18:15:51,174 INFO Thread-22 :32132 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/6b/3b8e91d9e2c32726db1e53b2d21b7b | |
| 2022-05-07 18:15:51,623 INFO SenderThread:32132 [sender.py:send_request_log_artifact():976] logged artifact run-3jt20u8h-evalfinal_generation_incorrect - {'id': 'QXJ0aWZhY3Q6MTI0MzYzMzUx', 'digest': '2806f32b4eb7c9d877466561e424c0e1', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5OTAwMjQ5', 'latestArtifact': None}, 'version': 'latest'} | |
| 2022-05-07 18:15:51,623 DEBUG SenderThread:32132 [sender.py:send():235] send: files | |
| 2022-05-07 18:15:51,623 INFO SenderThread:32132 [sender.py:_save_file():947] saving file media/table/eval/final_generation_incorrect_15_27d5678dc49240a7dde2.table.json with policy now | |
| 2022-05-07 18:15:51,624 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: log_artifact | |
| 2022-05-07 18:15:51,649 INFO Thread-7 :32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:15:53,292 INFO Thread-23 :32132 [upload_job.py:push():137] Uploaded file /tmp/tmpi1_5w_fvwandb/1xen2c74-media/table/eval/final_generation_incorrect_15_27d5678dc49240a7dde2.table.json | |
| 2022-05-07 18:15:53,872 INFO Thread-25 :32132 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/af/47f5eac929f7d4f6e6c7d604ae396b | |
| 2022-05-07 18:15:55,697 INFO SenderThread:32132 [sender.py:send_request_log_artifact():976] logged artifact run-3jt20u8h-validation90step_15k - {'id': 'QXJ0aWZhY3Q6MTI0MzYzMzU4', 'digest': 'd4593fb16cf602068d6ff40b8ad1567f', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5OTAwMjUw', 'latestArtifact': None}, 'version': 'latest'} | |
| 2022-05-07 18:15:55,697 DEBUG SenderThread:32132 [sender.py:send():235] send: files | |
| 2022-05-07 18:15:55,697 INFO SenderThread:32132 [sender.py:_save_file():947] saving file media/table/validation[[]:90%]/step_15k_15_b59f86c33aa51dff79ab.table.json with policy now | |
| 2022-05-07 18:15:55,699 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: log_artifact | |
| 2022-05-07 18:15:56,108 INFO Thread-28 :32132 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/43/50e41ae72b85005650b1e30f30f93e | |
| 2022-05-07 18:15:56,135 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-05-07 18:15:56,373 INFO Thread-26 :32132 [upload_job.py:push():137] Uploaded file /tmp/tmpi1_5w_fvwandb/1bec4wn7-media/table/validation[:90%]/step_15k_15_b59f86c33aa51dff79ab.table.json | |
| 2022-05-07 18:15:57,081 INFO SenderThread:32132 [sender.py:send_request_log_artifact():976] logged artifact run-3jt20u8h-validation90final_generation_incorrect - {'id': 'QXJ0aWZhY3Q6MTI0MzYzMzc3', 'digest': 'a59be299c2bb989e5e8a7153d0ce9dd8', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5OTAwMjUx', 'latestArtifact': None}, 'version': 'latest'} | |
| 2022-05-07 18:15:57,081 DEBUG SenderThread:32132 [sender.py:send():235] send: files | |
| 2022-05-07 18:15:57,081 INFO SenderThread:32132 [sender.py:_save_file():947] saving file media/table/validation[[]:90%]/final_generation_incorrect_15_932e28f0ba95df4e01c5.table.json with policy now | |
| 2022-05-07 18:15:57,082 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: log_artifact | |
| 2022-05-07 18:15:57,328 INFO Thread-29 :32132 [upload_job.py:push():137] Uploaded file /tmp/tmpi1_5w_fvwandb/a6211j05-media/table/validation[:90%]/final_generation_incorrect_15_932e28f0ba95df4e01c5.table.json | |
| 2022-05-07 18:15:57,900 INFO Thread-31 :32132 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/59/70265576a772597adeb57dd33a7a30 | |
| 2022-05-07 18:15:58,517 INFO SenderThread:32132 [sender.py:send_request_log_artifact():976] logged artifact run-3jt20u8h-validation95step_15k - {'id': 'QXJ0aWZhY3Q6MTI0MzYzMzg4', 'digest': 'ed389500ca9807856176333e266aa986', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5OTAwMjUy', 'latestArtifact': None}, 'version': 'latest'} | |
| 2022-05-07 18:15:58,518 DEBUG SenderThread:32132 [sender.py:send():235] send: files | |
| 2022-05-07 18:15:58,518 INFO SenderThread:32132 [sender.py:_save_file():947] saving file media/table/validation[[]:95%]/step_15k_15_a7d6e656cdc47be11a4d.table.json with policy now | |
| 2022-05-07 18:15:58,519 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: log_artifact | |
| 2022-05-07 18:15:58,769 INFO Thread-32 :32132 [upload_job.py:push():137] Uploaded file /tmp/tmpi1_5w_fvwandb/1w94ueu7-media/table/validation[:95%]/step_15k_15_a7d6e656cdc47be11a4d.table.json | |
| 2022-05-07 18:15:59,023 INFO Thread-34 :32132 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/6b/3b8e91d9e2c32726db1e53b2d21b7b | |
| 2022-05-07 18:15:59,424 INFO SenderThread:32132 [sender.py:send_request_log_artifact():976] logged artifact run-3jt20u8h-validation95final_generation_incorrect - {'id': 'QXJ0aWZhY3Q6MTI0MzYzMzk1', 'digest': '184a1d714543b40a2138ad00e0cd8774', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5OTAwMjUz', 'latestArtifact': None}, 'version': 'latest'} | |
| 2022-05-07 18:15:59,424 DEBUG SenderThread:32132 [sender.py:send():235] send: files | |
| 2022-05-07 18:15:59,425 INFO SenderThread:32132 [sender.py:_save_file():947] saving file media/table/validation[[]:95%]/final_generation_incorrect_15_27d5678dc49240a7dde2.table.json with policy now | |
| 2022-05-07 18:15:59,425 DEBUG SenderThread:32132 [sender.py:send():235] send: telemetry | |
| 2022-05-07 18:15:59,425 DEBUG SenderThread:32132 [sender.py:send():235] send: history | |
| 2022-05-07 18:15:59,426 DEBUG SenderThread:32132 [sender.py:send():235] send: summary | |
| 2022-05-07 18:15:59,427 INFO SenderThread:32132 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 18:15:59,427 DEBUG SenderThread:32132 [sender.py:send():235] send: exit | |
| 2022-05-07 18:15:59,427 INFO SenderThread:32132 [sender.py:send_exit():371] handling exit code: 0 | |
| 2022-05-07 18:15:59,429 INFO SenderThread:32132 [sender.py:send_exit():373] handling runtime: 197 | |
| 2022-05-07 18:15:59,430 INFO SenderThread:32132 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 18:15:59,430 INFO SenderThread:32132 [sender.py:send_exit():379] send defer | |
| 2022-05-07 18:15:59,430 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-05-07 18:15:59,431 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-05-07 18:15:59,432 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: defer | |
| 2022-05-07 18:15:59,432 INFO HandlerThread:32132 [handler.py:handle_request_defer():164] handle defer: 0 | |
| 2022-05-07 18:15:59,434 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: defer | |
| 2022-05-07 18:15:59,434 INFO SenderThread:32132 [sender.py:send_request_defer():388] handle sender defer: 0 | |
| 2022-05-07 18:15:59,434 INFO SenderThread:32132 [sender.py:transition_state():392] send defer: 1 | |
| 2022-05-07 18:15:59,435 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: defer | |
| 2022-05-07 18:15:59,435 INFO HandlerThread:32132 [handler.py:handle_request_defer():164] handle defer: 1 | |
| 2022-05-07 18:15:59,494 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: defer | |
| 2022-05-07 18:15:59,494 INFO SenderThread:32132 [sender.py:send_request_defer():388] handle sender defer: 1 | |
| 2022-05-07 18:15:59,494 INFO SenderThread:32132 [sender.py:transition_state():392] send defer: 2 | |
| 2022-05-07 18:15:59,494 DEBUG SenderThread:32132 [sender.py:send():235] send: stats | |
| 2022-05-07 18:15:59,495 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: defer | |
| 2022-05-07 18:15:59,495 INFO HandlerThread:32132 [handler.py:handle_request_defer():164] handle defer: 2 | |
| 2022-05-07 18:15:59,495 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: defer | |
| 2022-05-07 18:15:59,495 INFO SenderThread:32132 [sender.py:send_request_defer():388] handle sender defer: 2 | |
| 2022-05-07 18:15:59,495 INFO SenderThread:32132 [sender.py:transition_state():392] send defer: 3 | |
| 2022-05-07 18:15:59,495 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: defer | |
| 2022-05-07 18:15:59,495 INFO HandlerThread:32132 [handler.py:handle_request_defer():164] handle defer: 3 | |
| 2022-05-07 18:15:59,495 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: defer | |
| 2022-05-07 18:15:59,495 INFO SenderThread:32132 [sender.py:send_request_defer():388] handle sender defer: 3 | |
| 2022-05-07 18:15:59,495 INFO SenderThread:32132 [sender.py:transition_state():392] send defer: 4 | |
| 2022-05-07 18:15:59,496 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: defer | |
| 2022-05-07 18:15:59,496 INFO HandlerThread:32132 [handler.py:handle_request_defer():164] handle defer: 4 | |
| 2022-05-07 18:15:59,496 DEBUG SenderThread:32132 [sender.py:send():235] send: summary | |
| 2022-05-07 18:15:59,498 INFO SenderThread:32132 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 18:15:59,498 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: defer | |
| 2022-05-07 18:15:59,498 INFO SenderThread:32132 [sender.py:send_request_defer():388] handle sender defer: 4 | |
| 2022-05-07 18:15:59,498 INFO SenderThread:32132 [sender.py:transition_state():392] send defer: 5 | |
| 2022-05-07 18:15:59,498 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: defer | |
| 2022-05-07 18:15:59,498 INFO HandlerThread:32132 [handler.py:handle_request_defer():164] handle defer: 5 | |
| 2022-05-07 18:15:59,498 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: defer | |
| 2022-05-07 18:15:59,498 INFO SenderThread:32132 [sender.py:send_request_defer():388] handle sender defer: 5 | |
| 2022-05-07 18:15:59,532 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-05-07 18:15:59,556 INFO SenderThread:32132 [sender.py:transition_state():392] send defer: 6 | |
| 2022-05-07 18:15:59,556 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-05-07 18:15:59,557 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: defer | |
| 2022-05-07 18:15:59,557 INFO HandlerThread:32132 [handler.py:handle_request_defer():164] handle defer: 6 | |
| 2022-05-07 18:15:59,557 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: defer | |
| 2022-05-07 18:15:59,557 INFO SenderThread:32132 [sender.py:send_request_defer():388] handle sender defer: 6 | |
| 2022-05-07 18:15:59,557 INFO SenderThread:32132 [dir_watcher.py:finish():283] shutting down directory watcher | |
| 2022-05-07 18:15:59,651 INFO SenderThread:32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/config.yaml | |
| 2022-05-07 18:15:59,651 INFO SenderThread:32132 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/wandb-summary.json | |
| 2022-05-07 18:15:59,651 INFO SenderThread:32132 [dir_watcher.py:finish():313] scan: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files | |
| 2022-05-07 18:15:59,651 INFO SenderThread:32132 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/requirements.txt requirements.txt | |
| 2022-05-07 18:15:59,651 INFO SenderThread:32132 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log output.log | |
| 2022-05-07 18:15:59,652 INFO SenderThread:32132 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/wandb-metadata.json wandb-metadata.json | |
| 2022-05-07 18:15:59,652 INFO SenderThread:32132 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/config.yaml config.yaml | |
| 2022-05-07 18:15:59,652 INFO SenderThread:32132 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/wandb-summary.json wandb-summary.json | |
| 2022-05-07 18:15:59,663 INFO SenderThread:32132 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/media/table/validation[:95%]/final_generation_incorrect_15_27d5678dc49240a7dde2.table.json media/table/validation[:95%]/final_generation_incorrect_15_27d5678dc49240a7dde2.table.json | |
| 2022-05-07 18:15:59,663 INFO SenderThread:32132 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/media/table/validation[:95%]/step_15k_15_a7d6e656cdc47be11a4d.table.json media/table/validation[:95%]/step_15k_15_a7d6e656cdc47be11a4d.table.json | |
| 2022-05-07 18:15:59,663 INFO SenderThread:32132 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/media/table/eval/final_generation_incorrect_15_27d5678dc49240a7dde2.table.json media/table/eval/final_generation_incorrect_15_27d5678dc49240a7dde2.table.json | |
| 2022-05-07 18:15:59,663 INFO SenderThread:32132 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/media/table/eval/step_15k_15_a7d6e656cdc47be11a4d.table.json media/table/eval/step_15k_15_a7d6e656cdc47be11a4d.table.json | |
| 2022-05-07 18:15:59,664 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-05-07 18:15:59,664 INFO SenderThread:32132 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/media/table/eval/step_10k_10_92ae240f2472ae79ad31.table.json media/table/eval/step_10k_10_92ae240f2472ae79ad31.table.json | |
| 2022-05-07 18:15:59,665 INFO Thread-35 :32132 [upload_job.py:push():137] Uploaded file /tmp/tmpi1_5w_fvwandb/unjycc4e-media/table/validation[:95%]/final_generation_incorrect_15_27d5678dc49240a7dde2.table.json | |
| 2022-05-07 18:15:59,665 INFO SenderThread:32132 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/media/table/eval/step_5k_5_f901d5b4b0d1263be066.table.json media/table/eval/step_5k_5_f901d5b4b0d1263be066.table.json | |
| 2022-05-07 18:15:59,666 INFO SenderThread:32132 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/media/table/validation[:90%]/final_generation_incorrect_15_932e28f0ba95df4e01c5.table.json media/table/validation[:90%]/final_generation_incorrect_15_932e28f0ba95df4e01c5.table.json | |
| 2022-05-07 18:15:59,666 INFO SenderThread:32132 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/media/table/validation[:90%]/step_15k_15_b59f86c33aa51dff79ab.table.json media/table/validation[:90%]/step_15k_15_b59f86c33aa51dff79ab.table.json | |
| 2022-05-07 18:15:59,666 INFO SenderThread:32132 [sender.py:transition_state():392] send defer: 7 | |
| 2022-05-07 18:15:59,666 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-05-07 18:15:59,667 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: defer | |
| 2022-05-07 18:15:59,667 INFO HandlerThread:32132 [handler.py:handle_request_defer():164] handle defer: 7 | |
| 2022-05-07 18:15:59,667 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: defer | |
| 2022-05-07 18:15:59,667 INFO SenderThread:32132 [sender.py:send_request_defer():388] handle sender defer: 7 | |
| 2022-05-07 18:15:59,668 INFO SenderThread:32132 [file_pusher.py:finish():145] shutting down file pusher | |
| 2022-05-07 18:15:59,768 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-05-07 18:15:59,768 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-05-07 18:15:59,870 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-05-07 18:15:59,870 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-05-07 18:15:59,893 INFO Thread-36 :32132 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/requirements.txt | |
| 2022-05-07 18:15:59,908 INFO Thread-37 :32132 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/output.log | |
| 2022-05-07 18:15:59,915 INFO Thread-38 :32132 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/config.yaml | |
| 2022-05-07 18:15:59,923 INFO Thread-39 :32132 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/files/wandb-summary.json | |
| 2022-05-07 18:15:59,971 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-05-07 18:15:59,971 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-05-07 18:16:00,073 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-05-07 18:16:00,073 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-05-07 18:16:00,124 INFO Thread-6 :32132 [sender.py:transition_state():392] send defer: 8 | |
| 2022-05-07 18:16:00,124 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: defer | |
| 2022-05-07 18:16:00,124 INFO HandlerThread:32132 [handler.py:handle_request_defer():164] handle defer: 8 | |
| 2022-05-07 18:16:00,124 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: defer | |
| 2022-05-07 18:16:00,125 INFO SenderThread:32132 [sender.py:send_request_defer():388] handle sender defer: 8 | |
| 2022-05-07 18:16:00,175 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-05-07 18:16:00,287 INFO SenderThread:32132 [sender.py:transition_state():392] send defer: 9 | |
| 2022-05-07 18:16:00,287 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-05-07 18:16:00,288 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: defer | |
| 2022-05-07 18:16:00,288 INFO HandlerThread:32132 [handler.py:handle_request_defer():164] handle defer: 9 | |
| 2022-05-07 18:16:00,288 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: defer | |
| 2022-05-07 18:16:00,288 INFO SenderThread:32132 [sender.py:send_request_defer():388] handle sender defer: 9 | |
| 2022-05-07 18:16:00,288 INFO SenderThread:32132 [sender.py:transition_state():392] send defer: 10 | |
| 2022-05-07 18:16:00,289 DEBUG SenderThread:32132 [sender.py:send():235] send: final | |
| 2022-05-07 18:16:00,289 DEBUG SenderThread:32132 [sender.py:send():235] send: footer | |
| 2022-05-07 18:16:00,289 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: defer | |
| 2022-05-07 18:16:00,289 INFO HandlerThread:32132 [handler.py:handle_request_defer():164] handle defer: 10 | |
| 2022-05-07 18:16:00,289 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: defer | |
| 2022-05-07 18:16:00,289 INFO SenderThread:32132 [sender.py:send_request_defer():388] handle sender defer: 10 | |
| 2022-05-07 18:16:00,388 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-05-07 18:16:00,389 DEBUG SenderThread:32132 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-05-07 18:16:00,389 INFO SenderThread:32132 [file_pusher.py:join():150] waiting for file pusher | |
| 2022-05-07 18:16:00,515 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: sampled_history | |
| 2022-05-07 18:16:00,517 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: get_summary | |
| 2022-05-07 18:16:00,518 DEBUG HandlerThread:32132 [handler.py:handle_request():141] handle_request: shutdown | |
| 2022-05-07 18:16:00,518 INFO HandlerThread:32132 [handler.py:finish():778] shutting down handler | |
| 2022-05-07 18:16:01,289 INFO WriterThread:32132 [datastore.py:close():281] close: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_181232-3jt20u8h/run-3jt20u8h.wandb | |
| 2022-05-07 18:16:01,414 INFO SenderThread:32132 [sender.py:finish():1078] shutting down sender | |
| 2022-05-07 18:16:01,414 INFO SenderThread:32132 [file_pusher.py:finish():145] shutting down file pusher | |
| 2022-05-07 18:16:01,414 INFO SenderThread:32132 [file_pusher.py:join():150] waiting for file pusher | |
| 2022-05-07 18:16:01,415 INFO MainThread:32132 [internal.py:handle_exit():82] Internal process exited | |