| 2022-05-07 17:58:14,107 INFO MainThread:23847 [internal.py:wandb_internal():92] W&B internal server running at pid: 23847, started at: 2022-05-07 17:58:14.106970 | |
| 2022-05-07 17:58:14,109 INFO WriterThread:23847 [datastore.py:open_for_write():77] open: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/run-3hqzwtal.wandb | |
| 2022-05-07 17:58:14,110 DEBUG SenderThread:23847 [sender.py:send():235] send: header | |
| 2022-05-07 17:58:14,110 DEBUG SenderThread:23847 [sender.py:send():235] send: run | |
| 2022-05-07 17:58:14,188 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: check_version | |
| 2022-05-07 17:58:14,188 INFO SenderThread:23847 [dir_watcher.py:__init__():169] watching files in: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files | |
| 2022-05-07 17:58:14,188 INFO SenderThread:23847 [sender.py:_start_run_threads():812] run started: 3hqzwtal with start time 1651946293 | |
| 2022-05-07 17:58:14,188 DEBUG SenderThread:23847 [sender.py:send():235] send: summary | |
| 2022-05-07 17:58:14,189 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 17:58:14,189 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: check_version | |
| 2022-05-07 17:58:14,248 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: run_start | |
| 2022-05-07 17:58:15,191 INFO Thread-7 :23847 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-summary.json | |
| 2022-05-07 17:58:16,539 DEBUG HandlerThread:23847 [meta.py:__init__():37] meta init | |
| 2022-05-07 17:58:16,539 DEBUG HandlerThread:23847 [meta.py:__init__():51] meta init done | |
| 2022-05-07 17:58:16,539 DEBUG HandlerThread:23847 [meta.py:probe():211] probe | |
| 2022-05-07 17:58:16,541 DEBUG HandlerThread:23847 [meta.py:_setup_git():201] setup git | |
| 2022-05-07 17:58:16,575 DEBUG HandlerThread:23847 [meta.py:_setup_git():208] setup git done | |
| 2022-05-07 17:58:16,575 DEBUG HandlerThread:23847 [meta.py:_save_pip():55] save pip | |
| 2022-05-07 17:58:16,575 DEBUG HandlerThread:23847 [meta.py:_save_pip():69] save pip done | |
| 2022-05-07 17:58:16,575 DEBUG HandlerThread:23847 [meta.py:probe():249] probe done | |
| 2022-05-07 17:58:16,579 DEBUG SenderThread:23847 [sender.py:send():235] send: files | |
| 2022-05-07 17:58:16,579 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-metadata.json with policy now | |
| 2022-05-07 17:58:16,585 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-05-07 17:58:16,586 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: stop_status | |
| 2022-05-07 17:58:16,615 DEBUG SenderThread:23847 [sender.py:send():235] send: telemetry | |
| 2022-05-07 17:58:16,860 INFO Thread-11 :23847 [upload_job.py:push():137] Uploaded file /tmp/tmp6ko7nbgiwandb/2qs680he-wandb-metadata.json | |
| 2022-05-07 17:58:17,195 INFO Thread-7 :23847 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log | |
| 2022-05-07 17:58:17,195 INFO Thread-7 :23847 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/requirements.txt | |
| 2022-05-07 17:58:17,196 INFO Thread-7 :23847 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-metadata.json | |
| 2022-05-07 17:58:19,196 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log | |
| 2022-05-07 17:58:21,197 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log | |
| 2022-05-07 17:58:31,201 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log | |
| 2022-05-07 17:58:31,677 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-05-07 17:58:31,678 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: stop_status | |
| 2022-05-07 17:58:33,202 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log | |
| 2022-05-07 17:58:44,616 DEBUG SenderThread:23847 [sender.py:send():235] send: stats | |
| 2022-05-07 17:58:45,207 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/config.yaml | |
| 2022-05-07 17:58:46,930 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-05-07 17:58:46,930 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: stop_status | |
| 2022-05-07 17:58:57,211 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log | |
| 2022-05-07 17:59:01,985 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-05-07 17:59:01,986 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: stop_status | |
| 2022-05-07 17:59:14,691 DEBUG SenderThread:23847 [sender.py:send():235] send: stats | |
| 2022-05-07 17:59:17,476 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-05-07 17:59:17,476 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: stop_status | |
| 2022-05-07 17:59:19,288 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 17:59:19,378 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 17:59:20,682 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 17:59:20,684 DEBUG SenderThread:23847 [sender.py:send():235] send: history | |
| 2022-05-07 17:59:20,685 DEBUG SenderThread:23847 [sender.py:send():235] send: summary | |
| 2022-05-07 17:59:20,686 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 17:59:20,770 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 17:59:21,221 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log | |
| 2022-05-07 17:59:21,222 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-summary.json | |
| 2022-05-07 17:59:21,849 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 17:59:21,851 DEBUG SenderThread:23847 [sender.py:send():235] send: history | |
| 2022-05-07 17:59:21,852 DEBUG SenderThread:23847 [sender.py:send():235] send: summary | |
| 2022-05-07 17:59:21,853 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 17:59:21,931 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 17:59:22,222 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-summary.json | |
| 2022-05-07 17:59:23,127 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 17:59:23,129 DEBUG SenderThread:23847 [sender.py:send():235] send: history | |
| 2022-05-07 17:59:23,130 DEBUG SenderThread:23847 [sender.py:send():235] send: summary | |
| 2022-05-07 17:59:23,130 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 17:59:23,194 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 17:59:23,222 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log | |
| 2022-05-07 17:59:23,222 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-summary.json | |
| 2022-05-07 17:59:24,322 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 17:59:24,324 DEBUG SenderThread:23847 [sender.py:send():235] send: history | |
| 2022-05-07 17:59:24,325 DEBUG SenderThread:23847 [sender.py:send():235] send: summary | |
| 2022-05-07 17:59:24,325 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 17:59:24,411 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 17:59:25,223 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log | |
| 2022-05-07 17:59:25,223 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-summary.json | |
| 2022-05-07 17:59:32,530 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-05-07 17:59:32,531 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: stop_status | |
| 2022-05-07 17:59:44,766 DEBUG SenderThread:23847 [sender.py:send():235] send: stats | |
| 2022-05-07 17:59:47,568 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-05-07 17:59:47,569 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: stop_status | |
| 2022-05-07 17:59:51,233 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log | |
| 2022-05-07 18:00:02,624 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-05-07 18:00:02,624 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: stop_status | |
| 2022-05-07 18:00:12,240 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log | |
| 2022-05-07 18:00:14,847 DEBUG SenderThread:23847 [sender.py:send():235] send: stats | |
| 2022-05-07 18:00:17,677 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-05-07 18:00:17,677 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: stop_status | |
| 2022-05-07 18:00:32,722 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-05-07 18:00:32,723 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: stop_status | |
| 2022-05-07 18:00:34,050 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:00:34,114 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: log_artifact | |
| 2022-05-07 18:00:34,115 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: log_artifact | |
| 2022-05-07 18:00:34,218 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:00:34,249 INFO Thread-7 :23847 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/eval/epoch_3_5_f901d5b4b0d1263be066.table.json | |
| 2022-05-07 18:00:34,249 INFO Thread-7 :23847 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table | |
| 2022-05-07 18:00:34,250 INFO Thread-7 :23847 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media | |
| 2022-05-07 18:00:34,250 INFO Thread-7 :23847 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/eval | |
| 2022-05-07 18:00:34,574 INFO Thread-13 :23847 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/03/1aa3918a194407ac466420b1ea5405 | |
| 2022-05-07 18:00:35,249 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log | |
| 2022-05-07 18:00:35,435 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:00:35,535 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:00:36,028 INFO SenderThread:23847 [sender.py:send_request_log_artifact():976] logged artifact run-3hqzwtal-evalepoch_3 - {'id': 'QXJ0aWZhY3Q6MTI0MzU4ODAw', 'digest': 'b5c3ea14223fca2fc4c6b3245e57e304', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5ODk5ODky', 'latestArtifact': None}, 'version': 'latest'} | |
| 2022-05-07 18:00:36,028 DEBUG SenderThread:23847 [sender.py:send():235] send: files | |
| 2022-05-07 18:00:36,028 INFO SenderThread:23847 [sender.py:_save_file():947] saving file media/table/eval/epoch_3_5_f901d5b4b0d1263be066.table.json with policy now | |
| 2022-05-07 18:00:36,028 DEBUG SenderThread:23847 [sender.py:send():235] send: history | |
| 2022-05-07 18:00:36,029 DEBUG SenderThread:23847 [sender.py:send():235] send: summary | |
| 2022-05-07 18:00:36,032 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 18:00:36,250 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log | |
| 2022-05-07 18:00:36,250 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-summary.json | |
| 2022-05-07 18:00:36,279 INFO Thread-14 :23847 [upload_job.py:push():137] Uploaded file /tmp/tmp6ko7nbgiwandb/1go5pz4e-media/table/eval/epoch_3_5_f901d5b4b0d1263be066.table.json | |
| 2022-05-07 18:00:36,748 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:00:36,751 DEBUG SenderThread:23847 [sender.py:send():235] send: history | |
| 2022-05-07 18:00:36,751 DEBUG SenderThread:23847 [sender.py:send():235] send: summary | |
| 2022-05-07 18:00:36,752 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 18:00:36,812 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:00:37,250 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-summary.json | |
| 2022-05-07 18:00:37,931 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:00:37,932 DEBUG SenderThread:23847 [sender.py:send():235] send: history | |
| 2022-05-07 18:00:37,933 DEBUG SenderThread:23847 [sender.py:send():235] send: summary | |
| 2022-05-07 18:00:37,934 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 18:00:38,012 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:00:38,251 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log | |
| 2022-05-07 18:00:38,251 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-summary.json | |
| 2022-05-07 18:00:39,225 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:00:39,227 DEBUG SenderThread:23847 [sender.py:send():235] send: history | |
| 2022-05-07 18:00:39,228 DEBUG SenderThread:23847 [sender.py:send():235] send: summary | |
| 2022-05-07 18:00:39,229 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 18:00:39,251 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-summary.json | |
| 2022-05-07 18:00:39,285 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:00:40,252 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log | |
| 2022-05-07 18:00:40,483 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:00:40,485 DEBUG SenderThread:23847 [sender.py:send():235] send: history | |
| 2022-05-07 18:00:40,486 DEBUG SenderThread:23847 [sender.py:send():235] send: summary | |
| 2022-05-07 18:00:40,487 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 18:00:40,564 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:00:41,252 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-summary.json | |
| 2022-05-07 18:00:42,253 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log | |
| 2022-05-07 18:00:42,888 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:00:42,945 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: log_artifact | |
| 2022-05-07 18:00:42,946 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: log_artifact | |
| 2022-05-07 18:00:43,253 INFO Thread-7 :23847 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/eval/epoch_6_10_f901d5b4b0d1263be066.table.json | |
| 2022-05-07 18:00:43,254 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/eval | |
| 2022-05-07 18:00:43,363 INFO Thread-16 :23847 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/03/1aa3918a194407ac466420b1ea5405 | |
| 2022-05-07 18:00:43,590 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:00:43,908 INFO SenderThread:23847 [sender.py:send_request_log_artifact():976] logged artifact run-3hqzwtal-evalepoch_6 - {'id': 'QXJ0aWZhY3Q6MTI0MzU4ODI3', 'digest': 'c7eac83a9da52d82139235864a083759', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5ODk5ODk0', 'latestArtifact': None}, 'version': 'latest'} | |
| 2022-05-07 18:00:43,908 DEBUG SenderThread:23847 [sender.py:send():235] send: files | |
| 2022-05-07 18:00:43,908 INFO SenderThread:23847 [sender.py:_save_file():947] saving file media/table/eval/epoch_6_10_f901d5b4b0d1263be066.table.json with policy now | |
| 2022-05-07 18:00:44,166 INFO Thread-17 :23847 [upload_job.py:push():137] Uploaded file /tmp/tmp6ko7nbgiwandb/230mzitx-media/table/eval/epoch_6_10_f901d5b4b0d1263be066.table.json | |
| 2022-05-07 18:00:44,254 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log | |
| 2022-05-07 18:00:44,709 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:00:44,712 DEBUG SenderThread:23847 [sender.py:send():235] send: history | |
| 2022-05-07 18:00:44,712 DEBUG SenderThread:23847 [sender.py:send():235] send: summary | |
| 2022-05-07 18:00:44,713 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 18:00:44,821 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:00:44,921 DEBUG SenderThread:23847 [sender.py:send():235] send: stats | |
| 2022-05-07 18:00:45,254 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-summary.json | |
| 2022-05-07 18:00:45,935 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:00:45,937 DEBUG SenderThread:23847 [sender.py:send():235] send: history | |
| 2022-05-07 18:00:45,937 DEBUG SenderThread:23847 [sender.py:send():235] send: summary | |
| 2022-05-07 18:00:45,939 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 18:00:46,017 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:00:46,255 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log | |
| 2022-05-07 18:00:46,255 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-summary.json | |
| 2022-05-07 18:00:47,263 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:00:47,265 DEBUG SenderThread:23847 [sender.py:send():235] send: history | |
| 2022-05-07 18:00:47,266 DEBUG SenderThread:23847 [sender.py:send():235] send: summary | |
| 2022-05-07 18:00:47,269 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 18:00:47,324 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:00:47,853 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-05-07 18:00:47,854 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: stop_status | |
| 2022-05-07 18:00:48,256 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log | |
| 2022-05-07 18:00:48,256 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-summary.json | |
| 2022-05-07 18:00:48,450 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:00:48,452 DEBUG SenderThread:23847 [sender.py:send():235] send: history | |
| 2022-05-07 18:00:48,452 DEBUG SenderThread:23847 [sender.py:send():235] send: summary | |
| 2022-05-07 18:00:48,453 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 18:00:48,528 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:00:49,256 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-summary.json | |
| 2022-05-07 18:00:49,681 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:00:49,683 DEBUG SenderThread:23847 [sender.py:send():235] send: history | |
| 2022-05-07 18:00:49,683 DEBUG SenderThread:23847 [sender.py:send():235] send: summary | |
| 2022-05-07 18:00:49,684 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 18:00:49,743 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:00:50,257 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log | |
| 2022-05-07 18:00:50,257 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-summary.json | |
| 2022-05-07 18:00:52,258 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log | |
| 2022-05-07 18:01:02,960 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-05-07 18:01:02,961 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: stop_status | |
| 2022-05-07 18:01:06,264 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log | |
| 2022-05-07 18:01:14,993 DEBUG SenderThread:23847 [sender.py:send():235] send: stats | |
| 2022-05-07 18:01:18,011 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-05-07 18:01:18,012 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: stop_status | |
| 2022-05-07 18:01:18,270 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log | |
| 2022-05-07 18:01:29,470 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:01:29,531 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: log_artifact | |
| 2022-05-07 18:01:29,531 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: log_artifact | |
| 2022-05-07 18:01:29,595 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:01:29,637 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: log_artifact | |
| 2022-05-07 18:01:29,735 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:01:29,962 INFO Thread-19 :23847 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/59/70265576a772597adeb57dd33a7a30 | |
| 2022-05-07 18:01:30,276 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log | |
| 2022-05-07 18:01:30,276 INFO Thread-7 :23847 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/eval/epoch_8_incorrect_15_623faa4883d23d9d4194.table.json | |
| 2022-05-07 18:01:30,276 INFO Thread-7 :23847 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/eval/epoch_8_15_a7d6e656cdc47be11a4d.table.json | |
| 2022-05-07 18:01:30,276 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/eval | |
| 2022-05-07 18:01:30,398 INFO SenderThread:23847 [sender.py:send_request_log_artifact():976] logged artifact run-3hqzwtal-evalepoch_8 - {'id': 'QXJ0aWZhY3Q6MTI0MzU5MDgw', 'digest': 'f438f940059d7667deb4a50791726d2c', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5ODk5OTE3', 'latestArtifact': None}, 'version': 'latest'} | |
| 2022-05-07 18:01:30,398 DEBUG SenderThread:23847 [sender.py:send():235] send: files | |
| 2022-05-07 18:01:30,398 INFO SenderThread:23847 [sender.py:_save_file():947] saving file media/table/eval/epoch_8_15_a7d6e656cdc47be11a4d.table.json with policy now | |
| 2022-05-07 18:01:30,399 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: log_artifact | |
| 2022-05-07 18:01:30,670 INFO Thread-20 :23847 [upload_job.py:push():137] Uploaded file /tmp/tmp6ko7nbgiwandb/sae1wnru-media/table/eval/epoch_8_15_a7d6e656cdc47be11a4d.table.json | |
| 2022-05-07 18:01:30,872 INFO Thread-22 :23847 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/25/29be3032c57f98a337547e6fe497da | |
| 2022-05-07 18:01:31,332 INFO SenderThread:23847 [sender.py:send_request_log_artifact():976] logged artifact run-3hqzwtal-evalepoch_8_incorrect - {'id': 'QXJ0aWZhY3Q6MTI0MzU5MDgy', 'digest': '1083b22586f095b594cda26f5e13869f', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5ODk5OTE4', 'latestArtifact': None}, 'version': 'latest'} | |
| 2022-05-07 18:01:31,332 DEBUG SenderThread:23847 [sender.py:send():235] send: files | |
| 2022-05-07 18:01:31,332 INFO SenderThread:23847 [sender.py:_save_file():947] saving file media/table/eval/epoch_8_incorrect_15_623faa4883d23d9d4194.table.json with policy now | |
| 2022-05-07 18:01:31,613 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:01:31,636 INFO Thread-23 :23847 [upload_job.py:push():137] Uploaded file /tmp/tmp6ko7nbgiwandb/3ku2cp9q-media/table/eval/epoch_8_incorrect_15_623faa4883d23d9d4194.table.json | |
| 2022-05-07 18:01:31,665 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: log_artifact | |
| 2022-05-07 18:01:31,665 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: log_artifact | |
| 2022-05-07 18:01:32,277 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log | |
| 2022-05-07 18:01:32,277 INFO Thread-7 :23847 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/validation[:90%]/epoch_8_15_b59f86c33aa51dff79ab.table.json | |
| 2022-05-07 18:01:32,277 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table | |
| 2022-05-07 18:01:32,277 INFO Thread-7 :23847 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/validation[:90%] | |
| 2022-05-07 18:01:32,643 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:01:32,679 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: log_artifact | |
| 2022-05-07 18:01:32,743 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:01:33,087 INFO Thread-25 :23847 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/af/47f5eac929f7d4f6e6c7d604ae396b | |
| 2022-05-07 18:01:33,277 INFO Thread-7 :23847 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/validation[:90%]/epoch_8_incorrect_15_b59f86c33aa51dff79ab.table.json | |
| 2022-05-07 18:01:33,278 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/validation[:90%] | |
| 2022-05-07 18:01:33,472 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-05-07 18:01:33,627 INFO SenderThread:23847 [sender.py:send_request_log_artifact():976] logged artifact run-3hqzwtal-validation90epoch_8 - {'id': 'QXJ0aWZhY3Q6MTI0MzU5MDky', 'digest': '4a0bf6423c6c046a9c499c17c09d3972', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5ODk5OTIw', 'latestArtifact': None}, 'version': 'latest'} | |
| 2022-05-07 18:01:33,627 DEBUG SenderThread:23847 [sender.py:send():235] send: files | |
| 2022-05-07 18:01:33,627 INFO SenderThread:23847 [sender.py:_save_file():947] saving file media/table/validation[[]:90%]/epoch_8_15_b59f86c33aa51dff79ab.table.json with policy now | |
| 2022-05-07 18:01:33,629 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: log_artifact | |
| 2022-05-07 18:01:33,924 INFO Thread-26 :23847 [upload_job.py:push():137] Uploaded file /tmp/tmp6ko7nbgiwandb/1xj8yudg-media/table/validation[:90%]/epoch_8_15_b59f86c33aa51dff79ab.table.json | |
| 2022-05-07 18:01:34,278 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log | |
| 2022-05-07 18:01:34,323 INFO Thread-28 :23847 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/1d/36af48785046dc58b4f025a3f8aea1 | |
| 2022-05-07 18:01:35,034 INFO SenderThread:23847 [sender.py:send_request_log_artifact():976] logged artifact run-3hqzwtal-validation90epoch_8_incorrect - {'id': 'QXJ0aWZhY3Q6MTI0MzU5MTA0', 'digest': 'b891e308e194fc2dc7887b835c0c4ff7', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5ODk5OTIy', 'latestArtifact': None}, 'version': 'latest'} | |
| 2022-05-07 18:01:35,034 DEBUG SenderThread:23847 [sender.py:send():235] send: files | |
| 2022-05-07 18:01:35,034 INFO SenderThread:23847 [sender.py:_save_file():947] saving file media/table/validation[[]:90%]/epoch_8_incorrect_15_b59f86c33aa51dff79ab.table.json with policy now | |
| 2022-05-07 18:01:35,035 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: stop_status | |
| 2022-05-07 18:01:35,167 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:01:35,261 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: log_artifact | |
| 2022-05-07 18:01:35,262 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: log_artifact | |
| 2022-05-07 18:01:35,278 INFO Thread-7 :23847 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/validation[:95%]/epoch_8_15_a7d6e656cdc47be11a4d.table.json | |
| 2022-05-07 18:01:35,279 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table | |
| 2022-05-07 18:01:35,279 INFO Thread-7 :23847 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/validation[:95%] | |
| 2022-05-07 18:01:35,307 INFO Thread-29 :23847 [upload_job.py:push():137] Uploaded file /tmp/tmp6ko7nbgiwandb/1mnfpbtf-media/table/validation[:90%]/epoch_8_incorrect_15_b59f86c33aa51dff79ab.table.json | |
| 2022-05-07 18:01:35,456 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:01:35,513 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: log_artifact | |
| 2022-05-07 18:01:35,961 INFO Thread-31 :23847 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/59/70265576a772597adeb57dd33a7a30 | |
| 2022-05-07 18:01:36,121 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:01:36,149 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:01:36,279 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log | |
| 2022-05-07 18:01:36,279 INFO Thread-7 :23847 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/validation[:95%]/epoch_8_incorrect_15_623faa4883d23d9d4194.table.json | |
| 2022-05-07 18:01:36,279 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/validation[:95%] | |
| 2022-05-07 18:01:36,772 INFO SenderThread:23847 [sender.py:send_request_log_artifact():976] logged artifact run-3hqzwtal-validation95epoch_8 - {'id': 'QXJ0aWZhY3Q6MTI0MzU5MTEx', 'digest': 'b7f474680722e0d2994717ec03a92b01', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5ODk5OTIz', 'latestArtifact': None}, 'version': 'latest'} | |
| 2022-05-07 18:01:36,773 DEBUG SenderThread:23847 [sender.py:send():235] send: files | |
| 2022-05-07 18:01:36,773 INFO SenderThread:23847 [sender.py:_save_file():947] saving file media/table/validation[[]:95%]/epoch_8_15_a7d6e656cdc47be11a4d.table.json with policy now | |
| 2022-05-07 18:01:36,774 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: log_artifact | |
| 2022-05-07 18:01:37,060 INFO Thread-32 :23847 [upload_job.py:push():137] Uploaded file /tmp/tmp6ko7nbgiwandb/3j6qd40g-media/table/validation[:95%]/epoch_8_15_a7d6e656cdc47be11a4d.table.json | |
| 2022-05-07 18:01:38,905 INFO Thread-34 :23847 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/25/29be3032c57f98a337547e6fe497da | |
| 2022-05-07 18:01:39,144 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-05-07 18:01:39,281 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log | |
| 2022-05-07 18:01:40,840 INFO SenderThread:23847 [sender.py:send_request_log_artifact():976] logged artifact run-3hqzwtal-validation95epoch_8_incorrect - {'id': 'QXJ0aWZhY3Q6MTI0MzU5MTIz', 'digest': '009449b4f57a6cf87bce6279ce908e6c', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5ODk5OTI1', 'latestArtifact': None}, 'version': 'latest'} | |
| 2022-05-07 18:01:40,840 DEBUG SenderThread:23847 [sender.py:send():235] send: files | |
| 2022-05-07 18:01:40,840 INFO SenderThread:23847 [sender.py:_save_file():947] saving file media/table/validation[[]:95%]/epoch_8_incorrect_15_623faa4883d23d9d4194.table.json with policy now | |
| 2022-05-07 18:01:40,841 DEBUG SenderThread:23847 [sender.py:send():235] send: telemetry | |
| 2022-05-07 18:01:40,841 DEBUG SenderThread:23847 [sender.py:send():235] send: history | |
| 2022-05-07 18:01:40,842 DEBUG SenderThread:23847 [sender.py:send():235] send: summary | |
| 2022-05-07 18:01:40,843 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 18:01:40,843 DEBUG SenderThread:23847 [sender.py:send():235] send: exit | |
| 2022-05-07 18:01:40,843 INFO SenderThread:23847 [sender.py:send_exit():371] handling exit code: 0 | |
| 2022-05-07 18:01:40,845 INFO SenderThread:23847 [sender.py:send_exit():373] handling runtime: 204 | |
| 2022-05-07 18:01:40,845 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 18:01:40,846 INFO SenderThread:23847 [sender.py:send_exit():379] send defer | |
| 2022-05-07 18:01:40,846 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-05-07 18:01:40,847 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: defer | |
| 2022-05-07 18:01:40,847 INFO HandlerThread:23847 [handler.py:handle_request_defer():164] handle defer: 0 | |
| 2022-05-07 18:01:40,847 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: defer | |
| 2022-05-07 18:01:40,847 INFO SenderThread:23847 [sender.py:send_request_defer():388] handle sender defer: 0 | |
| 2022-05-07 18:01:40,847 INFO SenderThread:23847 [sender.py:transition_state():392] send defer: 1 | |
| 2022-05-07 18:01:40,850 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: defer | |
| 2022-05-07 18:01:40,850 INFO HandlerThread:23847 [handler.py:handle_request_defer():164] handle defer: 1 | |
| 2022-05-07 18:01:40,857 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: defer | |
| 2022-05-07 18:01:40,857 INFO SenderThread:23847 [sender.py:send_request_defer():388] handle sender defer: 1 | |
| 2022-05-07 18:01:40,857 INFO SenderThread:23847 [sender.py:transition_state():392] send defer: 2 | |
| 2022-05-07 18:01:40,857 DEBUG SenderThread:23847 [sender.py:send():235] send: stats | |
| 2022-05-07 18:01:40,857 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: defer | |
| 2022-05-07 18:01:40,857 INFO HandlerThread:23847 [handler.py:handle_request_defer():164] handle defer: 2 | |
| 2022-05-07 18:01:40,858 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: defer | |
| 2022-05-07 18:01:40,858 INFO SenderThread:23847 [sender.py:send_request_defer():388] handle sender defer: 2 | |
| 2022-05-07 18:01:40,858 INFO SenderThread:23847 [sender.py:transition_state():392] send defer: 3 | |
| 2022-05-07 18:01:40,858 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: defer | |
| 2022-05-07 18:01:40,858 INFO HandlerThread:23847 [handler.py:handle_request_defer():164] handle defer: 3 | |
| 2022-05-07 18:01:40,858 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: defer | |
| 2022-05-07 18:01:40,858 INFO SenderThread:23847 [sender.py:send_request_defer():388] handle sender defer: 3 | |
| 2022-05-07 18:01:40,858 INFO SenderThread:23847 [sender.py:transition_state():392] send defer: 4 | |
| 2022-05-07 18:01:40,859 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: defer | |
| 2022-05-07 18:01:40,859 INFO HandlerThread:23847 [handler.py:handle_request_defer():164] handle defer: 4 | |
| 2022-05-07 18:01:40,859 DEBUG SenderThread:23847 [sender.py:send():235] send: summary | |
| 2022-05-07 18:01:40,860 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 18:01:40,860 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: defer | |
| 2022-05-07 18:01:40,861 INFO SenderThread:23847 [sender.py:send_request_defer():388] handle sender defer: 4 | |
| 2022-05-07 18:01:40,861 INFO SenderThread:23847 [sender.py:transition_state():392] send defer: 5 | |
| 2022-05-07 18:01:40,861 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: defer | |
| 2022-05-07 18:01:40,861 INFO HandlerThread:23847 [handler.py:handle_request_defer():164] handle defer: 5 | |
| 2022-05-07 18:01:40,861 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: defer | |
| 2022-05-07 18:01:40,861 INFO SenderThread:23847 [sender.py:send_request_defer():388] handle sender defer: 5 | |
| 2022-05-07 18:01:40,913 INFO SenderThread:23847 [sender.py:transition_state():392] send defer: 6 | |
| 2022-05-07 18:01:40,914 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: defer | |
| 2022-05-07 18:01:40,914 INFO HandlerThread:23847 [handler.py:handle_request_defer():164] handle defer: 6 | |
| 2022-05-07 18:01:40,914 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: defer | |
| 2022-05-07 18:01:40,914 INFO SenderThread:23847 [sender.py:send_request_defer():388] handle sender defer: 6 | |
| 2022-05-07 18:01:40,914 INFO SenderThread:23847 [dir_watcher.py:finish():283] shutting down directory watcher | |
| 2022-05-07 18:01:40,948 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-05-07 18:01:41,132 INFO Thread-35 :23847 [upload_job.py:push():137] Uploaded file /tmp/tmp6ko7nbgiwandb/2zz6fan7-media/table/validation[:95%]/epoch_8_incorrect_15_623faa4883d23d9d4194.table.json | |
| 2022-05-07 18:01:41,282 INFO SenderThread:23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-summary.json | |
| 2022-05-07 18:01:41,282 INFO SenderThread:23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/config.yaml | |
| 2022-05-07 18:01:41,282 INFO SenderThread:23847 [dir_watcher.py:finish():313] scan: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files | |
| 2022-05-07 18:01:41,283 INFO SenderThread:23847 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/requirements.txt requirements.txt | |
| 2022-05-07 18:01:41,283 INFO SenderThread:23847 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log output.log | |
| 2022-05-07 18:01:41,283 INFO SenderThread:23847 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-metadata.json wandb-metadata.json | |
| 2022-05-07 18:01:41,283 INFO SenderThread:23847 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/config.yaml config.yaml | |
| 2022-05-07 18:01:41,283 INFO SenderThread:23847 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-summary.json wandb-summary.json | |
| 2022-05-07 18:01:41,284 INFO SenderThread:23847 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/validation[:95%]/epoch_8_incorrect_15_623faa4883d23d9d4194.table.json media/table/validation[:95%]/epoch_8_incorrect_15_623faa4883d23d9d4194.table.json | |
| 2022-05-07 18:01:41,284 INFO SenderThread:23847 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/validation[:95%]/epoch_8_15_a7d6e656cdc47be11a4d.table.json media/table/validation[:95%]/epoch_8_15_a7d6e656cdc47be11a4d.table.json | |
| 2022-05-07 18:01:41,286 INFO SenderThread:23847 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/eval/epoch_6_10_f901d5b4b0d1263be066.table.json media/table/eval/epoch_6_10_f901d5b4b0d1263be066.table.json | |
| 2022-05-07 18:01:41,286 INFO SenderThread:23847 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/eval/epoch_8_incorrect_15_623faa4883d23d9d4194.table.json media/table/eval/epoch_8_incorrect_15_623faa4883d23d9d4194.table.json | |
| 2022-05-07 18:01:41,287 INFO SenderThread:23847 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/eval/epoch_8_15_a7d6e656cdc47be11a4d.table.json media/table/eval/epoch_8_15_a7d6e656cdc47be11a4d.table.json | |
| 2022-05-07 18:01:41,287 INFO SenderThread:23847 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/eval/epoch_3_5_f901d5b4b0d1263be066.table.json media/table/eval/epoch_3_5_f901d5b4b0d1263be066.table.json | |
| 2022-05-07 18:01:41,291 INFO SenderThread:23847 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/validation[:90%]/epoch_8_incorrect_15_b59f86c33aa51dff79ab.table.json media/table/validation[:90%]/epoch_8_incorrect_15_b59f86c33aa51dff79ab.table.json | |
| 2022-05-07 18:01:41,291 INFO SenderThread:23847 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/validation[:90%]/epoch_8_15_b59f86c33aa51dff79ab.table.json media/table/validation[:90%]/epoch_8_15_b59f86c33aa51dff79ab.table.json | |
| 2022-05-07 18:01:41,292 INFO SenderThread:23847 [sender.py:transition_state():392] send defer: 7 | |
| 2022-05-07 18:01:41,292 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-05-07 18:01:41,295 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: defer | |
| 2022-05-07 18:01:41,295 INFO HandlerThread:23847 [handler.py:handle_request_defer():164] handle defer: 7 | |
| 2022-05-07 18:01:41,296 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: defer | |
| 2022-05-07 18:01:41,296 INFO SenderThread:23847 [sender.py:send_request_defer():388] handle sender defer: 7 | |
| 2022-05-07 18:01:41,296 INFO SenderThread:23847 [file_pusher.py:finish():145] shutting down file pusher | |
| 2022-05-07 18:01:41,396 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-05-07 18:01:41,396 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-05-07 18:01:41,498 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-05-07 18:01:41,498 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-05-07 18:01:41,546 INFO Thread-36 :23847 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/requirements.txt | |
| 2022-05-07 18:01:41,568 INFO Thread-38 :23847 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/config.yaml | |
| 2022-05-07 18:01:41,600 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-05-07 18:01:41,600 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-05-07 18:01:41,701 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-05-07 18:01:41,702 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-05-07 18:01:41,803 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-05-07 18:01:41,803 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-05-07 18:01:41,842 INFO Thread-37 :23847 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log | |
| 2022-05-07 18:01:41,848 INFO Thread-39 :23847 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-summary.json | |
| 2022-05-07 18:01:41,905 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-05-07 18:01:41,905 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-05-07 18:01:42,006 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-05-07 18:01:42,007 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-05-07 18:01:42,049 INFO Thread-6 :23847 [sender.py:transition_state():392] send defer: 8 | |
| 2022-05-07 18:01:42,049 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: defer | |
| 2022-05-07 18:01:42,049 INFO HandlerThread:23847 [handler.py:handle_request_defer():164] handle defer: 8 | |
| 2022-05-07 18:01:42,050 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: defer | |
| 2022-05-07 18:01:42,050 INFO SenderThread:23847 [sender.py:send_request_defer():388] handle sender defer: 8 | |
| 2022-05-07 18:01:42,108 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-05-07 18:01:42,194 INFO SenderThread:23847 [sender.py:transition_state():392] send defer: 9 | |
| 2022-05-07 18:01:42,194 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-05-07 18:01:42,195 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: defer | |
| 2022-05-07 18:01:42,195 INFO HandlerThread:23847 [handler.py:handle_request_defer():164] handle defer: 9 | |
| 2022-05-07 18:01:42,195 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: defer | |
| 2022-05-07 18:01:42,195 INFO SenderThread:23847 [sender.py:send_request_defer():388] handle sender defer: 9 | |
| 2022-05-07 18:01:42,195 INFO SenderThread:23847 [sender.py:transition_state():392] send defer: 10 | |
| 2022-05-07 18:01:42,196 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: defer | |
| 2022-05-07 18:01:42,196 INFO HandlerThread:23847 [handler.py:handle_request_defer():164] handle defer: 10 | |
| 2022-05-07 18:01:42,196 DEBUG SenderThread:23847 [sender.py:send():235] send: final | |
| 2022-05-07 18:01:42,196 DEBUG SenderThread:23847 [sender.py:send():235] send: footer | |
| 2022-05-07 18:01:42,196 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: defer | |
| 2022-05-07 18:01:42,196 INFO SenderThread:23847 [sender.py:send_request_defer():388] handle sender defer: 10 | |
| 2022-05-07 18:01:42,296 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-05-07 18:01:42,296 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-05-07 18:01:42,296 INFO SenderThread:23847 [file_pusher.py:join():150] waiting for file pusher | |
| 2022-05-07 18:01:42,606 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: sampled_history | |
| 2022-05-07 18:01:42,608 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: get_summary | |
| 2022-05-07 18:01:42,610 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: shutdown | |
| 2022-05-07 18:01:42,610 INFO HandlerThread:23847 [handler.py:finish():778] shutting down handler | |
| 2022-05-07 18:01:43,197 INFO WriterThread:23847 [datastore.py:close():281] close: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/run-3hqzwtal.wandb | |
| 2022-05-07 18:01:43,505 INFO SenderThread:23847 [sender.py:finish():1078] shutting down sender | |
| 2022-05-07 18:01:43,505 INFO SenderThread:23847 [file_pusher.py:finish():145] shutting down file pusher | |
| 2022-05-07 18:01:43,505 INFO SenderThread:23847 [file_pusher.py:join():150] waiting for file pusher | |
| 2022-05-07 18:01:43,507 INFO MainThread:23847 [internal.py:handle_exit():82] Internal process exited | |