2022-05-07 18:20:42,008 INFO MainThread:37252 [internal.py:wandb_internal():92] W&B internal server running at pid: 37252, started at: 2022-05-07 18:20:42.008533 2022-05-07 18:20:42,011 DEBUG SenderThread:37252 [sender.py:send():235] send: header 2022-05-07 18:20:42,011 DEBUG SenderThread:37252 [sender.py:send():235] send: run 2022-05-07 18:20:42,013 INFO WriterThread:37252 [datastore.py:open_for_write():77] open: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/run-10w2xueo.wandb 2022-05-07 18:20:42,121 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: check_version 2022-05-07 18:20:42,121 INFO SenderThread:37252 [dir_watcher.py:__init__():169] watching files in: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files 2022-05-07 18:20:42,122 INFO SenderThread:37252 [sender.py:_start_run_threads():812] run started: 10w2xueo with start time 1651947641 2022-05-07 18:20:42,122 DEBUG SenderThread:37252 [sender.py:send():235] send: summary 2022-05-07 18:20:42,122 INFO SenderThread:37252 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 18:20:42,122 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: check_version 2022-05-07 18:20:42,182 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: run_start 2022-05-07 18:20:43,155 INFO Thread-7 :37252 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/wandb-summary.json 2022-05-07 18:20:44,274 DEBUG HandlerThread:37252 [meta.py:__init__():37] meta init 2022-05-07 18:20:44,274 DEBUG HandlerThread:37252 [meta.py:__init__():51] meta init done 2022-05-07 18:20:44,274 DEBUG HandlerThread:37252 [meta.py:probe():211] probe 2022-05-07 18:20:44,275 DEBUG HandlerThread:37252 [meta.py:_setup_git():201] setup git 2022-05-07 18:20:44,306 DEBUG HandlerThread:37252 [meta.py:_setup_git():208] setup git done 2022-05-07 18:20:44,306 DEBUG HandlerThread:37252 [meta.py:_save_pip():55] save pip 2022-05-07 18:20:44,307 DEBUG HandlerThread:37252 [meta.py:_save_pip():69] save pip done 2022-05-07 18:20:44,307 DEBUG HandlerThread:37252 [meta.py:probe():249] probe done 2022-05-07 18:20:44,309 DEBUG SenderThread:37252 [sender.py:send():235] send: files 2022-05-07 18:20:44,310 INFO SenderThread:37252 [sender.py:_save_file():947] saving file wandb-metadata.json with policy now 2022-05-07 18:20:44,316 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 18:20:44,316 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: stop_status 2022-05-07 18:20:44,591 DEBUG SenderThread:37252 [sender.py:send():235] send: telemetry 2022-05-07 18:20:44,612 INFO Thread-11 :37252 [upload_job.py:push():137] Uploaded file /tmp/tmpakbzv8qrwandb/3pjkdsgl-wandb-metadata.json 2022-05-07 18:20:45,185 INFO Thread-7 :37252 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/output.log 2022-05-07 18:20:45,185 INFO Thread-7 :37252 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/wandb-metadata.json 2022-05-07 18:20:45,185 INFO Thread-7 :37252 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/requirements.txt 2022-05-07 18:20:47,186 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/output.log 2022-05-07 18:20:49,186 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/output.log 2022-05-07 18:20:59,191 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/output.log 2022-05-07 18:20:59,593 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 18:20:59,594 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: stop_status 2022-05-07 18:21:01,192 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/output.log 2022-05-07 18:21:12,356 DEBUG SenderThread:37252 [sender.py:send():235] send: stats 2022-05-07 18:21:13,197 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/config.yaml 2022-05-07 18:21:14,646 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 18:21:14,647 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: stop_status 2022-05-07 18:21:25,203 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/output.log 2022-05-07 18:21:29,690 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 18:21:29,690 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: stop_status 2022-05-07 18:21:42,433 DEBUG SenderThread:37252 [sender.py:send():235] send: stats 2022-05-07 18:21:43,117 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:21:43,184 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:21:44,743 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 18:21:44,743 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: stop_status 2022-05-07 18:21:45,211 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/output.log 2022-05-07 18:21:59,805 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 18:21:59,806 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: stop_status 2022-05-07 18:22:06,490 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:22:06,492 DEBUG SenderThread:37252 [sender.py:send():235] send: history 2022-05-07 18:22:06,493 DEBUG SenderThread:37252 [sender.py:send():235] send: summary 2022-05-07 18:22:06,495 INFO SenderThread:37252 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 18:22:06,576 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:22:07,185 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:22:07,187 DEBUG SenderThread:37252 [sender.py:send():235] send: history 2022-05-07 18:22:07,187 DEBUG SenderThread:37252 [sender.py:send():235] send: summary 2022-05-07 18:22:07,188 INFO SenderThread:37252 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 18:22:07,221 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/wandb-summary.json 2022-05-07 18:22:07,238 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:22:07,879 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:22:07,880 DEBUG SenderThread:37252 [sender.py:send():235] send: history 2022-05-07 18:22:07,881 DEBUG SenderThread:37252 [sender.py:send():235] send: summary 2022-05-07 18:22:07,882 INFO SenderThread:37252 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 18:22:07,941 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:22:08,221 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/wandb-summary.json 2022-05-07 18:22:08,516 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:22:08,518 DEBUG SenderThread:37252 [sender.py:send():235] send: history 2022-05-07 18:22:08,518 DEBUG SenderThread:37252 [sender.py:send():235] send: summary 2022-05-07 18:22:08,519 INFO SenderThread:37252 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 18:22:08,568 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:22:09,222 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/output.log 2022-05-07 18:22:09,222 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/wandb-summary.json 2022-05-07 18:22:11,222 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/output.log 2022-05-07 18:22:12,514 DEBUG SenderThread:37252 [sender.py:send():235] send: stats 2022-05-07 18:22:14,841 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 18:22:14,841 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: stop_status 2022-05-07 18:22:29,878 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 18:22:29,878 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: stop_status 2022-05-07 18:22:35,233 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/output.log 2022-05-07 18:22:42,593 DEBUG SenderThread:37252 [sender.py:send():235] send: stats 2022-05-07 18:22:44,931 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 18:22:44,931 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: stop_status 2022-05-07 18:22:55,241 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/output.log 2022-05-07 18:22:59,978 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 18:22:59,979 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: stop_status 2022-05-07 18:23:12,672 DEBUG SenderThread:37252 [sender.py:send():235] send: stats 2022-05-07 18:23:15,024 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 18:23:15,024 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: stop_status 2022-05-07 18:23:16,963 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:23:17,024 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: log_artifact 2022-05-07 18:23:17,025 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: log_artifact 2022-05-07 18:23:17,150 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:23:17,249 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/output.log 2022-05-07 18:23:17,250 INFO Thread-7 :37252 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/media/table/eval/step_0k_5_92ae240f2472ae79ad31.table.json 2022-05-07 18:23:17,250 INFO Thread-7 :37252 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/media/table/eval 2022-05-07 18:23:17,250 INFO Thread-7 :37252 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/media 2022-05-07 18:23:17,250 INFO Thread-7 :37252 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/media/table 2022-05-07 18:23:17,490 INFO Thread-13 :37252 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/00/66783abbd98b29e02682346c82293a 2022-05-07 18:23:18,035 INFO SenderThread:37252 [sender.py:send_request_log_artifact():976] logged artifact run-10w2xueo-evalstep_0k - {'id': 'QXJ0aWZhY3Q6MTI0MzY1NTU4', 'digest': '1fc9411295ac28c0df16aa1f68611033', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5OTAwMzU2', 'latestArtifact': None}, 'version': 'latest'} 2022-05-07 18:23:18,035 DEBUG SenderThread:37252 [sender.py:send():235] send: files 2022-05-07 18:23:18,035 INFO SenderThread:37252 [sender.py:_save_file():947] saving file media/table/eval/step_0k_5_92ae240f2472ae79ad31.table.json with policy now 2022-05-07 18:23:18,283 INFO Thread-14 :37252 [upload_job.py:push():137] Uploaded file /tmp/tmpakbzv8qrwandb/6qzmd17e-media/table/eval/step_0k_5_92ae240f2472ae79ad31.table.json 2022-05-07 18:23:19,250 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/output.log 2022-05-07 18:23:21,086 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:23:21,089 DEBUG SenderThread:37252 [sender.py:send():235] send: history 2022-05-07 18:23:21,090 DEBUG SenderThread:37252 [sender.py:send():235] send: summary 2022-05-07 18:23:21,091 INFO SenderThread:37252 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 18:23:21,125 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:23:21,251 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/output.log 2022-05-07 18:23:21,252 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/wandb-summary.json 2022-05-07 18:23:21,729 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:23:21,731 DEBUG SenderThread:37252 [sender.py:send():235] send: history 2022-05-07 18:23:21,732 DEBUG SenderThread:37252 [sender.py:send():235] send: summary 2022-05-07 18:23:21,733 INFO SenderThread:37252 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 18:23:21,769 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:23:22,252 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/wandb-summary.json 2022-05-07 18:23:22,363 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:23:22,365 DEBUG SenderThread:37252 [sender.py:send():235] send: history 2022-05-07 18:23:22,366 DEBUG SenderThread:37252 [sender.py:send():235] send: summary 2022-05-07 18:23:22,367 INFO SenderThread:37252 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 18:23:22,412 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:23:23,043 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:23:23,045 DEBUG SenderThread:37252 [sender.py:send():235] send: history 2022-05-07 18:23:23,046 DEBUG SenderThread:37252 [sender.py:send():235] send: summary 2022-05-07 18:23:23,047 INFO SenderThread:37252 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 18:23:23,078 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:23:23,253 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/output.log 2022-05-07 18:23:23,253 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/wandb-summary.json 2022-05-07 18:23:23,586 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:23:23,588 DEBUG SenderThread:37252 [sender.py:send():235] send: history 2022-05-07 18:23:23,589 DEBUG SenderThread:37252 [sender.py:send():235] send: summary 2022-05-07 18:23:23,590 INFO SenderThread:37252 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 18:23:23,626 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:23:24,253 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/wandb-summary.json 2022-05-07 18:23:25,254 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/output.log 2022-05-07 18:23:25,950 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:23:25,996 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: log_artifact 2022-05-07 18:23:25,996 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: log_artifact 2022-05-07 18:23:26,087 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:23:26,111 INFO SenderThread:37252 [sender.py:send_request_log_artifact():976] logged artifact run-10w2xueo-evalstep_0k - {'id': 'QXJ0aWZhY3Q6MTI0MzY1NTU4', 'digest': '1fc9411295ac28c0df16aa1f68611033', 'state': 'COMMITTED', 'aliases': [{'artifactCollectionName': 'run-10w2xueo-evalstep_0k', 'alias': 'latest'}, {'artifactCollectionName': 'run-10w2xueo-evalstep_0k', 'alias': 'v0'}], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5OTAwMzU2', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTI0MzY1NTU4', 'versionIndex': 0}}, 'version': 'v0'} 2022-05-07 18:23:26,111 DEBUG SenderThread:37252 [sender.py:send():235] send: files 2022-05-07 18:23:26,111 INFO SenderThread:37252 [sender.py:_save_file():947] saving file media/table/eval/step_0k_10_92ae240f2472ae79ad31.table.json with policy now 2022-05-07 18:23:26,254 INFO Thread-7 :37252 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/media/table/eval/step_0k_10_92ae240f2472ae79ad31.table.json 2022-05-07 18:23:26,254 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/media/table/eval 2022-05-07 18:23:26,368 INFO Thread-15 :37252 [upload_job.py:push():137] Uploaded file /tmp/tmpakbzv8qrwandb/3uqcqr2v-media/table/eval/step_0k_10_92ae240f2472ae79ad31.table.json 2022-05-07 18:23:27,255 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/output.log 2022-05-07 18:23:28,255 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/output.log 2022-05-07 18:23:30,024 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:23:30,026 DEBUG SenderThread:37252 [sender.py:send():235] send: history 2022-05-07 18:23:30,027 DEBUG SenderThread:37252 [sender.py:send():235] send: summary 2022-05-07 18:23:30,030 INFO SenderThread:37252 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 18:23:30,061 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 18:23:30,061 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: stop_status 2022-05-07 18:23:30,075 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:23:30,256 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/wandb-summary.json 2022-05-07 18:23:30,709 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:23:30,711 DEBUG SenderThread:37252 [sender.py:send():235] send: history 2022-05-07 18:23:30,712 DEBUG SenderThread:37252 [sender.py:send():235] send: summary 2022-05-07 18:23:30,713 INFO SenderThread:37252 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 18:23:30,773 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:23:31,257 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/output.log 2022-05-07 18:23:31,257 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/wandb-summary.json 2022-05-07 18:23:31,432 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:23:31,434 DEBUG SenderThread:37252 [sender.py:send():235] send: history 2022-05-07 18:23:31,435 DEBUG SenderThread:37252 [sender.py:send():235] send: summary 2022-05-07 18:23:31,435 INFO SenderThread:37252 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 18:23:31,468 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:23:32,005 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:23:32,007 DEBUG SenderThread:37252 [sender.py:send():235] send: history 2022-05-07 18:23:32,008 DEBUG SenderThread:37252 [sender.py:send():235] send: summary 2022-05-07 18:23:32,009 INFO SenderThread:37252 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 18:23:32,038 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:23:32,257 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/output.log 2022-05-07 18:23:32,258 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/wandb-summary.json 2022-05-07 18:23:32,549 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:23:32,551 DEBUG SenderThread:37252 [sender.py:send():235] send: history 2022-05-07 18:23:32,552 DEBUG SenderThread:37252 [sender.py:send():235] send: summary 2022-05-07 18:23:32,553 INFO SenderThread:37252 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 18:23:32,581 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:23:33,258 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/output.log 2022-05-07 18:23:33,258 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/wandb-summary.json 2022-05-07 18:23:34,259 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/output.log 2022-05-07 18:23:42,749 DEBUG SenderThread:37252 [sender.py:send():235] send: stats 2022-05-07 18:23:45,265 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 18:23:45,266 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: stop_status 2022-05-07 18:23:50,267 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/output.log 2022-05-07 18:24:00,318 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 18:24:00,318 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: stop_status 2022-05-07 18:24:02,273 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/output.log 2022-05-07 18:24:12,588 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:24:12,658 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: log_artifact 2022-05-07 18:24:12,659 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: log_artifact 2022-05-07 18:24:12,732 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:24:12,788 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: log_artifact 2022-05-07 18:24:12,874 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:24:13,127 INFO Thread-17 :37252 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/59/70265576a772597adeb57dd33a7a30 2022-05-07 18:24:13,278 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/output.log 2022-05-07 18:24:13,278 INFO Thread-7 :37252 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/media/table/eval/step_0k_15_a7d6e656cdc47be11a4d.table.json 2022-05-07 18:24:13,279 INFO Thread-7 :37252 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/media/table/eval/step_0k_incorrect_15_623faa4883d23d9d4194.table.json 2022-05-07 18:24:13,279 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/media/table/eval 2022-05-07 18:24:13,568 INFO SenderThread:37252 [sender.py:send_request_log_artifact():976] logged artifact run-10w2xueo-evalstep_0k - {'id': 'QXJ0aWZhY3Q6MTI0MzY1ODU1', 'digest': '429ae9ca9b8150388808ce6998b8c788', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5OTAwMzU2', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTI0MzY1NTU4', 'versionIndex': 0}}, 'version': 'latest'} 2022-05-07 18:24:13,568 DEBUG SenderThread:37252 [sender.py:send():235] send: files 2022-05-07 18:24:13,568 INFO SenderThread:37252 [sender.py:_save_file():947] saving file media/table/eval/step_0k_15_a7d6e656cdc47be11a4d.table.json with policy now 2022-05-07 18:24:13,568 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: log_artifact 2022-05-07 18:24:13,868 INFO Thread-18 :37252 [upload_job.py:push():137] Uploaded file /tmp/tmpakbzv8qrwandb/156zh9a8-media/table/eval/step_0k_15_a7d6e656cdc47be11a4d.table.json 2022-05-07 18:24:14,021 INFO Thread-20 :37252 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/25/29be3032c57f98a337547e6fe497da 2022-05-07 18:24:14,279 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/output.log 2022-05-07 18:24:14,541 INFO SenderThread:37252 [sender.py:send_request_log_artifact():976] logged artifact run-10w2xueo-evalstep_0k_incorrect - {'id': 'QXJ0aWZhY3Q6MTI0MzY1ODY0', 'digest': '918368a588a4bc961092f64abe63233e', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5OTAwMzY2', 'latestArtifact': None}, 'version': 'latest'} 2022-05-07 18:24:14,541 DEBUG SenderThread:37252 [sender.py:send():235] send: files 2022-05-07 18:24:14,541 INFO SenderThread:37252 [sender.py:_save_file():947] saving file media/table/eval/step_0k_incorrect_15_623faa4883d23d9d4194.table.json with policy now 2022-05-07 18:24:14,542 DEBUG SenderThread:37252 [sender.py:send():235] send: stats 2022-05-07 18:24:14,814 INFO Thread-21 :37252 [upload_job.py:push():137] Uploaded file /tmp/tmpakbzv8qrwandb/3q45pn67-media/table/eval/step_0k_incorrect_15_623faa4883d23d9d4194.table.json 2022-05-07 18:24:15,464 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 18:24:15,465 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: stop_status 2022-05-07 18:24:18,022 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:24:18,067 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: log_artifact 2022-05-07 18:24:18,067 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: log_artifact 2022-05-07 18:24:18,146 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:24:18,186 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: log_artifact 2022-05-07 18:24:18,281 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/output.log 2022-05-07 18:24:18,281 INFO Thread-7 :37252 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/media/table/validation[:90%]/step_0k_incorrect_15_b59f86c33aa51dff79ab.table.json 2022-05-07 18:24:18,281 INFO Thread-7 :37252 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/media/table/validation[:90%]/step_0k_15_b59f86c33aa51dff79ab.table.json 2022-05-07 18:24:18,281 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/media/table 2022-05-07 18:24:18,281 INFO Thread-7 :37252 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/media/table/validation[:90%] 2022-05-07 18:24:18,409 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:24:19,229 INFO Thread-23 :37252 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/af/47f5eac929f7d4f6e6c7d604ae396b 2022-05-07 18:24:19,774 INFO SenderThread:37252 [sender.py:send_request_log_artifact():976] logged artifact run-10w2xueo-validation90step_0k - {'id': 'QXJ0aWZhY3Q6MTI0MzY1ODk0', 'digest': '7d26eb8be387a9f303d800b41f9d0d37', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5OTAwMzY5', 'latestArtifact': None}, 'version': 'latest'} 2022-05-07 18:24:19,774 DEBUG SenderThread:37252 [sender.py:send():235] send: files 2022-05-07 18:24:19,774 INFO SenderThread:37252 [sender.py:_save_file():947] saving file media/table/validation[[]:90%]/step_0k_15_b59f86c33aa51dff79ab.table.json with policy now 2022-05-07 18:24:19,775 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: log_artifact 2022-05-07 18:24:20,075 INFO Thread-24 :37252 [upload_job.py:push():137] Uploaded file /tmp/tmpakbzv8qrwandb/2zjodtcw-media/table/validation[:90%]/step_0k_15_b59f86c33aa51dff79ab.table.json 2022-05-07 18:24:20,202 INFO Thread-26 :37252 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/1d/36af48785046dc58b4f025a3f8aea1 2022-05-07 18:24:20,282 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/output.log 2022-05-07 18:24:20,766 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:24:20,806 INFO SenderThread:37252 [sender.py:send_request_log_artifact():976] logged artifact run-10w2xueo-validation90step_0k_incorrect - {'id': 'QXJ0aWZhY3Q6MTI0MzY1OTA2', 'digest': 'e77745b97875dbbe70731f21e89c0fc2', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5OTAwMzcw', 'latestArtifact': None}, 'version': 'latest'} 2022-05-07 18:24:20,807 DEBUG SenderThread:37252 [sender.py:send():235] send: files 2022-05-07 18:24:20,807 INFO SenderThread:37252 [sender.py:_save_file():947] saving file media/table/validation[[]:90%]/step_0k_incorrect_15_b59f86c33aa51dff79ab.table.json with policy now 2022-05-07 18:24:20,839 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: log_artifact 2022-05-07 18:24:20,839 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: log_artifact 2022-05-07 18:24:20,919 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:24:20,973 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: log_artifact 2022-05-07 18:24:21,050 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:24:21,051 INFO Thread-27 :37252 [upload_job.py:push():137] Uploaded file /tmp/tmpakbzv8qrwandb/pnsv39c2-media/table/validation[:90%]/step_0k_incorrect_15_b59f86c33aa51dff79ab.table.json 2022-05-07 18:24:21,087 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:24:21,282 INFO Thread-7 :37252 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/media/table/validation[:95%]/step_0k_incorrect_15_623faa4883d23d9d4194.table.json 2022-05-07 18:24:21,283 INFO Thread-7 :37252 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/media/table/validation[:95%]/step_0k_15_a7d6e656cdc47be11a4d.table.json 2022-05-07 18:24:21,283 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/media/table 2022-05-07 18:24:21,283 INFO Thread-7 :37252 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/media/table/validation[:95%] 2022-05-07 18:24:21,431 INFO Thread-29 :37252 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/59/70265576a772597adeb57dd33a7a30 2022-05-07 18:24:21,835 INFO SenderThread:37252 [sender.py:send_request_log_artifact():976] logged artifact run-10w2xueo-validation95step_0k - {'id': 'QXJ0aWZhY3Q6MTI0MzY1OTEy', 'digest': '13ab5988ac0fb6d295daa0d3b5b2ad2a', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5OTAwMzcy', 'latestArtifact': None}, 'version': 'latest'} 2022-05-07 18:24:21,835 DEBUG SenderThread:37252 [sender.py:send():235] send: files 2022-05-07 18:24:21,835 INFO SenderThread:37252 [sender.py:_save_file():947] saving file media/table/validation[[]:95%]/step_0k_15_a7d6e656cdc47be11a4d.table.json with policy now 2022-05-07 18:24:21,836 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: log_artifact 2022-05-07 18:24:22,097 INFO Thread-30 :37252 [upload_job.py:push():137] Uploaded file /tmp/tmpakbzv8qrwandb/1s975bqg-media/table/validation[:95%]/step_0k_15_a7d6e656cdc47be11a4d.table.json 2022-05-07 18:24:22,283 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/output.log 2022-05-07 18:24:22,290 INFO Thread-32 :37252 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/25/29be3032c57f98a337547e6fe497da 2022-05-07 18:24:22,718 INFO SenderThread:37252 [sender.py:send_request_log_artifact():976] logged artifact run-10w2xueo-validation95step_0k_incorrect - {'id': 'QXJ0aWZhY3Q6MTI0MzY1OTE3', 'digest': 'be3804996b4e259473afc4fb14457b79', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5OTAwMzcz', 'latestArtifact': None}, 'version': 'latest'} 2022-05-07 18:24:22,718 DEBUG SenderThread:37252 [sender.py:send():235] send: files 2022-05-07 18:24:22,719 INFO SenderThread:37252 [sender.py:_save_file():947] saving file media/table/validation[[]:95%]/step_0k_incorrect_15_623faa4883d23d9d4194.table.json with policy now 2022-05-07 18:24:22,719 DEBUG SenderThread:37252 [sender.py:send():235] send: telemetry 2022-05-07 18:24:22,720 DEBUG SenderThread:37252 [sender.py:send():235] send: history 2022-05-07 18:24:22,721 DEBUG SenderThread:37252 [sender.py:send():235] send: summary 2022-05-07 18:24:22,722 INFO SenderThread:37252 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 18:24:22,974 INFO Thread-33 :37252 [upload_job.py:push():137] Uploaded file /tmp/tmpakbzv8qrwandb/2a6vsuei-media/table/validation[:95%]/step_0k_incorrect_15_623faa4883d23d9d4194.table.json 2022-05-07 18:24:23,284 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/wandb-summary.json 2022-05-07 18:24:23,485 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: poll_exit 2022-05-07 18:24:23,485 DEBUG SenderThread:37252 [sender.py:send():235] send: exit 2022-05-07 18:24:23,485 INFO SenderThread:37252 [sender.py:send_exit():371] handling exit code: 0 2022-05-07 18:24:23,487 INFO SenderThread:37252 [sender.py:send_exit():373] handling runtime: 221 2022-05-07 18:24:23,488 INFO SenderThread:37252 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 18:24:23,488 INFO SenderThread:37252 [sender.py:send_exit():379] send defer 2022-05-07 18:24:23,488 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: poll_exit 2022-05-07 18:24:23,489 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: defer 2022-05-07 18:24:23,489 INFO HandlerThread:37252 [handler.py:handle_request_defer():164] handle defer: 0 2022-05-07 18:24:23,489 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: defer 2022-05-07 18:24:23,489 INFO SenderThread:37252 [sender.py:send_request_defer():388] handle sender defer: 0 2022-05-07 18:24:23,489 INFO SenderThread:37252 [sender.py:transition_state():392] send defer: 1 2022-05-07 18:24:23,490 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: defer 2022-05-07 18:24:23,490 INFO HandlerThread:37252 [handler.py:handle_request_defer():164] handle defer: 1 2022-05-07 18:24:23,552 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: defer 2022-05-07 18:24:23,552 INFO SenderThread:37252 [sender.py:send_request_defer():388] handle sender defer: 1 2022-05-07 18:24:23,552 INFO SenderThread:37252 [sender.py:transition_state():392] send defer: 2 2022-05-07 18:24:23,553 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: defer 2022-05-07 18:24:23,553 INFO HandlerThread:37252 [handler.py:handle_request_defer():164] handle defer: 2 2022-05-07 18:24:23,553 DEBUG SenderThread:37252 [sender.py:send():235] send: stats 2022-05-07 18:24:23,553 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: defer 2022-05-07 18:24:23,553 INFO SenderThread:37252 [sender.py:send_request_defer():388] handle sender defer: 2 2022-05-07 18:24:23,554 INFO SenderThread:37252 [sender.py:transition_state():392] send defer: 3 2022-05-07 18:24:23,554 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: defer 2022-05-07 18:24:23,554 INFO HandlerThread:37252 [handler.py:handle_request_defer():164] handle defer: 3 2022-05-07 18:24:23,554 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: defer 2022-05-07 18:24:23,554 INFO SenderThread:37252 [sender.py:send_request_defer():388] handle sender defer: 3 2022-05-07 18:24:23,554 INFO SenderThread:37252 [sender.py:transition_state():392] send defer: 4 2022-05-07 18:24:23,554 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: defer 2022-05-07 18:24:23,554 INFO HandlerThread:37252 [handler.py:handle_request_defer():164] handle defer: 4 2022-05-07 18:24:23,555 DEBUG SenderThread:37252 [sender.py:send():235] send: summary 2022-05-07 18:24:23,556 INFO SenderThread:37252 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 18:24:23,556 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: defer 2022-05-07 18:24:23,556 INFO SenderThread:37252 [sender.py:send_request_defer():388] handle sender defer: 4 2022-05-07 18:24:23,557 INFO SenderThread:37252 [sender.py:transition_state():392] send defer: 5 2022-05-07 18:24:23,557 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: defer 2022-05-07 18:24:23,557 INFO HandlerThread:37252 [handler.py:handle_request_defer():164] handle defer: 5 2022-05-07 18:24:23,557 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: defer 2022-05-07 18:24:23,557 INFO SenderThread:37252 [sender.py:send_request_defer():388] handle sender defer: 5 2022-05-07 18:24:23,590 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: poll_exit 2022-05-07 18:24:23,623 INFO SenderThread:37252 [sender.py:transition_state():392] send defer: 6 2022-05-07 18:24:23,623 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: poll_exit 2022-05-07 18:24:23,624 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: defer 2022-05-07 18:24:23,624 INFO HandlerThread:37252 [handler.py:handle_request_defer():164] handle defer: 6 2022-05-07 18:24:23,624 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: defer 2022-05-07 18:24:23,624 INFO SenderThread:37252 [sender.py:send_request_defer():388] handle sender defer: 6 2022-05-07 18:24:23,624 INFO SenderThread:37252 [dir_watcher.py:finish():283] shutting down directory watcher 2022-05-07 18:24:23,725 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: poll_exit 2022-05-07 18:24:24,284 INFO Thread-7 :37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/output.log 2022-05-07 18:24:24,285 INFO SenderThread:37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/wandb-summary.json 2022-05-07 18:24:24,285 INFO SenderThread:37252 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/config.yaml 2022-05-07 18:24:24,286 INFO SenderThread:37252 [dir_watcher.py:finish():313] scan: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files 2022-05-07 18:24:24,286 INFO SenderThread:37252 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/requirements.txt requirements.txt 2022-05-07 18:24:24,286 INFO SenderThread:37252 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/output.log output.log 2022-05-07 18:24:24,286 INFO SenderThread:37252 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/wandb-metadata.json wandb-metadata.json 2022-05-07 18:24:24,286 INFO SenderThread:37252 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/config.yaml config.yaml 2022-05-07 18:24:24,286 INFO SenderThread:37252 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/wandb-summary.json wandb-summary.json 2022-05-07 18:24:24,289 INFO SenderThread:37252 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/media/table/validation[:95%]/step_0k_incorrect_15_623faa4883d23d9d4194.table.json media/table/validation[:95%]/step_0k_incorrect_15_623faa4883d23d9d4194.table.json 2022-05-07 18:24:24,289 INFO SenderThread:37252 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/media/table/validation[:95%]/step_0k_15_a7d6e656cdc47be11a4d.table.json media/table/validation[:95%]/step_0k_15_a7d6e656cdc47be11a4d.table.json 2022-05-07 18:24:24,294 INFO SenderThread:37252 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/media/table/eval/step_0k_5_92ae240f2472ae79ad31.table.json media/table/eval/step_0k_5_92ae240f2472ae79ad31.table.json 2022-05-07 18:24:24,294 INFO SenderThread:37252 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/media/table/eval/step_0k_incorrect_15_623faa4883d23d9d4194.table.json media/table/eval/step_0k_incorrect_15_623faa4883d23d9d4194.table.json 2022-05-07 18:24:24,294 INFO SenderThread:37252 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/media/table/eval/step_0k_10_92ae240f2472ae79ad31.table.json media/table/eval/step_0k_10_92ae240f2472ae79ad31.table.json 2022-05-07 18:24:24,294 INFO SenderThread:37252 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/media/table/eval/step_0k_15_a7d6e656cdc47be11a4d.table.json media/table/eval/step_0k_15_a7d6e656cdc47be11a4d.table.json 2022-05-07 18:24:24,295 INFO SenderThread:37252 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/media/table/validation[:90%]/step_0k_15_b59f86c33aa51dff79ab.table.json media/table/validation[:90%]/step_0k_15_b59f86c33aa51dff79ab.table.json 2022-05-07 18:24:24,295 INFO SenderThread:37252 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/media/table/validation[:90%]/step_0k_incorrect_15_b59f86c33aa51dff79ab.table.json media/table/validation[:90%]/step_0k_incorrect_15_b59f86c33aa51dff79ab.table.json 2022-05-07 18:24:24,295 INFO SenderThread:37252 [sender.py:transition_state():392] send defer: 7 2022-05-07 18:24:24,295 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: poll_exit 2022-05-07 18:24:24,296 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: defer 2022-05-07 18:24:24,296 INFO HandlerThread:37252 [handler.py:handle_request_defer():164] handle defer: 7 2022-05-07 18:24:24,296 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: defer 2022-05-07 18:24:24,296 INFO SenderThread:37252 [sender.py:send_request_defer():388] handle sender defer: 7 2022-05-07 18:24:24,296 INFO SenderThread:37252 [file_pusher.py:finish():145] shutting down file pusher 2022-05-07 18:24:24,398 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: poll_exit 2022-05-07 18:24:24,398 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: poll_exit 2022-05-07 18:24:24,500 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: poll_exit 2022-05-07 18:24:24,500 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: poll_exit 2022-05-07 18:24:24,547 INFO Thread-37 :37252 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/wandb-summary.json 2022-05-07 18:24:24,555 INFO Thread-34 :37252 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/requirements.txt 2022-05-07 18:24:24,562 INFO Thread-36 :37252 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/config.yaml 2022-05-07 18:24:24,602 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: poll_exit 2022-05-07 18:24:24,602 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: poll_exit 2022-05-07 18:24:24,704 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: poll_exit 2022-05-07 18:24:24,704 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: poll_exit 2022-05-07 18:24:24,805 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: poll_exit 2022-05-07 18:24:24,806 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: poll_exit 2022-05-07 18:24:24,833 INFO Thread-35 :37252 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/files/output.log 2022-05-07 18:24:24,907 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: poll_exit 2022-05-07 18:24:24,907 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: poll_exit 2022-05-07 18:24:25,009 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: poll_exit 2022-05-07 18:24:25,009 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: poll_exit 2022-05-07 18:24:25,033 INFO Thread-6 :37252 [sender.py:transition_state():392] send defer: 8 2022-05-07 18:24:25,033 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: defer 2022-05-07 18:24:25,033 INFO HandlerThread:37252 [handler.py:handle_request_defer():164] handle defer: 8 2022-05-07 18:24:25,034 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: defer 2022-05-07 18:24:25,034 INFO SenderThread:37252 [sender.py:send_request_defer():388] handle sender defer: 8 2022-05-07 18:24:25,111 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: poll_exit 2022-05-07 18:24:25,201 INFO SenderThread:37252 [sender.py:transition_state():392] send defer: 9 2022-05-07 18:24:25,201 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: poll_exit 2022-05-07 18:24:25,201 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: defer 2022-05-07 18:24:25,202 INFO HandlerThread:37252 [handler.py:handle_request_defer():164] handle defer: 9 2022-05-07 18:24:25,202 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: defer 2022-05-07 18:24:25,202 INFO SenderThread:37252 [sender.py:send_request_defer():388] handle sender defer: 9 2022-05-07 18:24:25,202 INFO SenderThread:37252 [sender.py:transition_state():392] send defer: 10 2022-05-07 18:24:25,203 DEBUG SenderThread:37252 [sender.py:send():235] send: final 2022-05-07 18:24:25,203 DEBUG SenderThread:37252 [sender.py:send():235] send: footer 2022-05-07 18:24:25,203 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: defer 2022-05-07 18:24:25,203 INFO HandlerThread:37252 [handler.py:handle_request_defer():164] handle defer: 10 2022-05-07 18:24:25,203 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: defer 2022-05-07 18:24:25,203 INFO SenderThread:37252 [sender.py:send_request_defer():388] handle sender defer: 10 2022-05-07 18:24:25,303 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: poll_exit 2022-05-07 18:24:25,303 DEBUG SenderThread:37252 [sender.py:send_request():249] send_request: poll_exit 2022-05-07 18:24:25,303 INFO SenderThread:37252 [file_pusher.py:join():150] waiting for file pusher 2022-05-07 18:24:25,522 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: sampled_history 2022-05-07 18:24:25,523 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: get_summary 2022-05-07 18:24:25,525 DEBUG HandlerThread:37252 [handler.py:handle_request():141] handle_request: shutdown 2022-05-07 18:24:25,525 INFO HandlerThread:37252 [handler.py:finish():778] shutting down handler 2022-05-07 18:24:26,203 INFO WriterThread:37252 [datastore.py:close():281] close: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_182041-10w2xueo/run-10w2xueo.wandb 2022-05-07 18:24:26,420 INFO SenderThread:37252 [sender.py:finish():1078] shutting down sender 2022-05-07 18:24:26,420 INFO SenderThread:37252 [file_pusher.py:finish():145] shutting down file pusher 2022-05-07 18:24:26,421 INFO SenderThread:37252 [file_pusher.py:join():150] waiting for file pusher 2022-05-07 18:24:26,421 INFO MainThread:37252 [internal.py:handle_exit():82] Internal process exited