sanchit-gandhi's picture
qf2iwkac: saving weights and logs of step 0k
a869d63
2022-05-29 12:04:59,324 INFO MainThread:1721989 [internal.py:wandb_internal():92] W&B internal server running at pid: 1721989, started at: 2022-05-29 12:04:59.324307
2022-05-29 12:04:59,326 INFO WriterThread:1721989 [datastore.py:open_for_write():77] open: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/run-qf2iwkac.wandb
2022-05-29 12:04:59,327 DEBUG SenderThread:1721989 [sender.py:send():235] send: header
2022-05-29 12:04:59,327 DEBUG SenderThread:1721989 [sender.py:send():235] send: run
2022-05-29 12:04:59,384 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: check_version
2022-05-29 12:04:59,385 INFO SenderThread:1721989 [dir_watcher.py:__init__():169] watching files in: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files
2022-05-29 12:04:59,385 INFO SenderThread:1721989 [sender.py:_start_run_threads():812] run started: qf2iwkac with start time 1653825898
2022-05-29 12:04:59,385 DEBUG SenderThread:1721989 [sender.py:send():235] send: summary
2022-05-29 12:04:59,385 INFO SenderThread:1721989 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-29 12:04:59,385 DEBUG SenderThread:1721989 [sender.py:send_request():249] send_request: check_version
2022-05-29 12:04:59,447 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: run_start
2022-05-29 12:05:00,436 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/wandb-summary.json
2022-05-29 12:05:01,566 DEBUG HandlerThread:1721989 [meta.py:__init__():37] meta init
2022-05-29 12:05:01,566 DEBUG HandlerThread:1721989 [meta.py:__init__():51] meta init done
2022-05-29 12:05:01,566 DEBUG HandlerThread:1721989 [meta.py:probe():211] probe
2022-05-29 12:05:01,568 DEBUG HandlerThread:1721989 [meta.py:_setup_git():201] setup git
2022-05-29 12:05:01,597 DEBUG HandlerThread:1721989 [meta.py:_setup_git():208] setup git done
2022-05-29 12:05:01,597 DEBUG HandlerThread:1721989 [meta.py:_save_pip():55] save pip
2022-05-29 12:05:01,598 DEBUG HandlerThread:1721989 [meta.py:_save_pip():69] save pip done
2022-05-29 12:05:01,598 DEBUG HandlerThread:1721989 [meta.py:probe():249] probe done
2022-05-29 12:05:01,601 DEBUG SenderThread:1721989 [sender.py:send():235] send: files
2022-05-29 12:05:01,601 INFO SenderThread:1721989 [sender.py:_save_file():947] saving file wandb-metadata.json with policy now
2022-05-29 12:05:01,608 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: stop_status
2022-05-29 12:05:01,608 DEBUG SenderThread:1721989 [sender.py:send_request():249] send_request: stop_status
2022-05-29 12:05:01,639 DEBUG SenderThread:1721989 [sender.py:send():235] send: telemetry
2022-05-29 12:05:01,849 INFO Thread-11 :1721989 [upload_job.py:push():137] Uploaded file /tmp/tmpl7u6xxw8wandb/12iwbaqf-wandb-metadata.json
2022-05-29 12:05:02,475 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/wandb-metadata.json
2022-05-29 12:05:02,475 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/requirements.txt
2022-05-29 12:05:02,475 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/output.log
2022-05-29 12:05:04,476 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/output.log
2022-05-29 12:05:06,476 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/output.log
2022-05-29 12:05:16,479 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/output.log
2022-05-29 12:05:16,641 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: stop_status
2022-05-29 12:05:16,641 DEBUG SenderThread:1721989 [sender.py:send_request():249] send_request: stop_status
2022-05-29 12:05:18,480 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/output.log
2022-05-29 12:05:20,481 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/output.log
2022-05-29 12:05:29,648 DEBUG SenderThread:1721989 [sender.py:send():235] send: stats
2022-05-29 12:05:30,485 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/config.yaml
2022-05-29 12:05:31,669 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: stop_status
2022-05-29 12:05:31,669 DEBUG SenderThread:1721989 [sender.py:send_request():249] send_request: stop_status
2022-05-29 12:05:42,516 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:05:42,573 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:05:44,489 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/output.log
2022-05-29 12:05:46,714 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: stop_status
2022-05-29 12:05:46,715 DEBUG SenderThread:1721989 [sender.py:send_request():249] send_request: stop_status
2022-05-29 12:05:59,724 DEBUG SenderThread:1721989 [sender.py:send():235] send: stats
2022-05-29 12:06:01,767 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: stop_status
2022-05-29 12:06:01,767 DEBUG SenderThread:1721989 [sender.py:send_request():249] send_request: stop_status
2022-05-29 12:06:03,840 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:06:03,845 DEBUG SenderThread:1721989 [sender.py:send():235] send: history
2022-05-29 12:06:03,846 DEBUG SenderThread:1721989 [sender.py:send():235] send: summary
2022-05-29 12:06:03,847 INFO SenderThread:1721989 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-29 12:06:03,899 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:06:04,497 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/wandb-summary.json
2022-05-29 12:06:06,497 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/output.log
2022-05-29 12:06:16,816 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: stop_status
2022-05-29 12:06:16,816 DEBUG SenderThread:1721989 [sender.py:send_request():249] send_request: stop_status
2022-05-29 12:06:26,186 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:06:26,188 DEBUG SenderThread:1721989 [sender.py:send():235] send: history
2022-05-29 12:06:26,189 DEBUG SenderThread:1721989 [sender.py:send():235] send: summary
2022-05-29 12:06:26,191 INFO SenderThread:1721989 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-29 12:06:26,256 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:06:26,505 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/wandb-summary.json
2022-05-29 12:06:26,834 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:06:26,836 DEBUG SenderThread:1721989 [sender.py:send():235] send: history
2022-05-29 12:06:26,837 DEBUG SenderThread:1721989 [sender.py:send():235] send: summary
2022-05-29 12:06:26,838 INFO SenderThread:1721989 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-29 12:06:26,884 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:06:27,505 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/wandb-summary.json
2022-05-29 12:06:27,513 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:06:27,515 DEBUG SenderThread:1721989 [sender.py:send():235] send: history
2022-05-29 12:06:27,515 DEBUG SenderThread:1721989 [sender.py:send():235] send: summary
2022-05-29 12:06:27,516 INFO SenderThread:1721989 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-29 12:06:27,570 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:06:28,506 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/wandb-summary.json
2022-05-29 12:06:28,506 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/output.log
2022-05-29 12:06:29,794 DEBUG SenderThread:1721989 [sender.py:send():235] send: stats
2022-05-29 12:06:31,871 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: stop_status
2022-05-29 12:06:31,871 DEBUG SenderThread:1721989 [sender.py:send_request():249] send_request: stop_status
2022-05-29 12:06:46,902 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: stop_status
2022-05-29 12:06:46,903 DEBUG SenderThread:1721989 [sender.py:send_request():249] send_request: stop_status
2022-05-29 12:06:54,515 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/output.log
2022-05-29 12:06:59,870 DEBUG SenderThread:1721989 [sender.py:send():235] send: stats
2022-05-29 12:07:01,952 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: stop_status
2022-05-29 12:07:01,952 DEBUG SenderThread:1721989 [sender.py:send_request():249] send_request: stop_status
2022-05-29 12:07:12,521 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/output.log
2022-05-29 12:07:17,016 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: stop_status
2022-05-29 12:07:17,016 DEBUG SenderThread:1721989 [sender.py:send_request():249] send_request: stop_status
2022-05-29 12:07:29,946 DEBUG SenderThread:1721989 [sender.py:send():235] send: stats
2022-05-29 12:07:32,044 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: stop_status
2022-05-29 12:07:32,044 DEBUG SenderThread:1721989 [sender.py:send_request():249] send_request: stop_status
2022-05-29 12:07:35,982 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:07:36,081 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: log_artifact
2022-05-29 12:07:36,081 DEBUG SenderThread:1721989 [sender.py:send_request():249] send_request: log_artifact
2022-05-29 12:07:36,280 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:07:36,354 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: log_artifact
2022-05-29 12:07:36,453 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:07:36,530 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/output.log
2022-05-29 12:07:36,530 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/media/table/eval/step_0k_5_92ae240f2472ae79ad31.table.json
2022-05-29 12:07:36,530 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/media/table/eval/step_0k_incorrect_5_b30e6671cbaaf5c82b6d.table.json
2022-05-29 12:07:36,530 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/media
2022-05-29 12:07:36,530 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/media/table
2022-05-29 12:07:36,530 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/media/table/eval
2022-05-29 12:07:36,589 INFO Thread-13 :1721989 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/00/66783abbd98b29e02682346c82293a
2022-05-29 12:07:36,995 INFO SenderThread:1721989 [sender.py:send_request_log_artifact():976] logged artifact run-qf2iwkac-evalstep_0k - {'id': 'QXJ0aWZhY3Q6MTM1NjQyNTgy', 'digest': '1fc9411295ac28c0df16aa1f68611033', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjIxNTAwMTY3', 'latestArtifact': None}, 'version': 'latest'}
2022-05-29 12:07:36,995 DEBUG SenderThread:1721989 [sender.py:send():235] send: files
2022-05-29 12:07:36,995 INFO SenderThread:1721989 [sender.py:_save_file():947] saving file media/table/eval/step_0k_5_92ae240f2472ae79ad31.table.json with policy now
2022-05-29 12:07:36,996 DEBUG SenderThread:1721989 [sender.py:send_request():249] send_request: log_artifact
2022-05-29 12:07:37,260 INFO Thread-14 :1721989 [upload_job.py:push():137] Uploaded file /tmp/tmpl7u6xxw8wandb/1pl7w3az-media/table/eval/step_0k_5_92ae240f2472ae79ad31.table.json
2022-05-29 12:07:37,399 INFO Thread-16 :1721989 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/61/78965daa44518dd50bc0e765d67f24
2022-05-29 12:07:37,903 INFO SenderThread:1721989 [sender.py:send_request_log_artifact():976] logged artifact run-qf2iwkac-evalstep_0k_incorrect - {'id': 'QXJ0aWZhY3Q6MTM1NjQyNTkx', 'digest': 'cd720fb526739060a418c512f0e14c5f', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjIxNTAwMTY4', 'latestArtifact': None}, 'version': 'latest'}
2022-05-29 12:07:37,903 DEBUG SenderThread:1721989 [sender.py:send():235] send: files
2022-05-29 12:07:37,903 INFO SenderThread:1721989 [sender.py:_save_file():947] saving file media/table/eval/step_0k_incorrect_5_b30e6671cbaaf5c82b6d.table.json with policy now
2022-05-29 12:07:38,134 INFO Thread-17 :1721989 [upload_job.py:push():137] Uploaded file /tmp/tmpl7u6xxw8wandb/26g70uwz-media/table/eval/step_0k_incorrect_5_b30e6671cbaaf5c82b6d.table.json
2022-05-29 12:07:38,530 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/output.log
2022-05-29 12:07:40,346 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:07:40,349 DEBUG SenderThread:1721989 [sender.py:send():235] send: history
2022-05-29 12:07:40,349 DEBUG SenderThread:1721989 [sender.py:send():235] send: summary
2022-05-29 12:07:40,350 INFO SenderThread:1721989 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-29 12:07:40,392 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:07:40,531 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/wandb-summary.json
2022-05-29 12:07:40,531 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/output.log
2022-05-29 12:07:40,896 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:07:40,899 DEBUG SenderThread:1721989 [sender.py:send():235] send: history
2022-05-29 12:07:40,899 DEBUG SenderThread:1721989 [sender.py:send():235] send: summary
2022-05-29 12:07:40,900 INFO SenderThread:1721989 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-29 12:07:40,946 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:07:41,532 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/wandb-summary.json
2022-05-29 12:07:41,543 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:07:41,545 DEBUG SenderThread:1721989 [sender.py:send():235] send: history
2022-05-29 12:07:41,546 DEBUG SenderThread:1721989 [sender.py:send():235] send: summary
2022-05-29 12:07:41,547 INFO SenderThread:1721989 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-29 12:07:41,579 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:07:42,158 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:07:42,160 DEBUG SenderThread:1721989 [sender.py:send():235] send: history
2022-05-29 12:07:42,161 DEBUG SenderThread:1721989 [sender.py:send():235] send: summary
2022-05-29 12:07:42,162 INFO SenderThread:1721989 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-29 12:07:42,200 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:07:42,532 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/wandb-summary.json
2022-05-29 12:07:42,532 INFO Thread-7 :1721989 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220529_120458-qf2iwkac/files/output.log
2022-05-29 12:07:42,826 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history
2022-05-29 12:07:42,828 DEBUG SenderThread:1721989 [sender.py:send():235] send: history
2022-05-29 12:07:42,828 DEBUG SenderThread:1721989 [sender.py:send():235] send: summary
2022-05-29 12:07:42,829 INFO SenderThread:1721989 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-29 12:07:42,857 DEBUG HandlerThread:1721989 [handler.py:handle_request():141] handle_request: partial_history