| 2022-05-07 18:18:54,038 INFO MainThread:35428 [internal.py:wandb_internal():92] W&B internal server running at pid: 35428, started at: 2022-05-07 18:18:54.037902 | |
| 2022-05-07 18:18:54,040 INFO WriterThread:35428 [datastore.py:open_for_write():77] open: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_181853-3nproo3b/run-3nproo3b.wandb | |
| 2022-05-07 18:18:54,041 DEBUG SenderThread:35428 [sender.py:send():235] send: header | |
| 2022-05-07 18:18:54,041 DEBUG SenderThread:35428 [sender.py:send():235] send: run | |
| 2022-05-07 18:18:54,122 INFO SenderThread:35428 [dir_watcher.py:__init__():169] watching files in: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_181853-3nproo3b/files | |
| 2022-05-07 18:18:54,122 INFO SenderThread:35428 [sender.py:_start_run_threads():812] run started: 3nproo3b with start time 1651947533 | |
| 2022-05-07 18:18:54,122 DEBUG SenderThread:35428 [sender.py:send():235] send: summary | |
| 2022-05-07 18:18:54,123 DEBUG HandlerThread:35428 [handler.py:handle_request():141] handle_request: check_version | |
| 2022-05-07 18:18:54,123 INFO SenderThread:35428 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 18:18:54,123 DEBUG SenderThread:35428 [sender.py:send_request():249] send_request: check_version | |
| 2022-05-07 18:18:54,185 DEBUG HandlerThread:35428 [handler.py:handle_request():141] handle_request: run_start | |
| 2022-05-07 18:18:55,124 INFO Thread-7 :35428 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_181853-3nproo3b/files/wandb-summary.json | |
| 2022-05-07 18:18:56,563 DEBUG HandlerThread:35428 [meta.py:__init__():37] meta init | |
| 2022-05-07 18:18:56,563 DEBUG HandlerThread:35428 [meta.py:__init__():51] meta init done | |
| 2022-05-07 18:18:56,563 DEBUG HandlerThread:35428 [meta.py:probe():211] probe | |
| 2022-05-07 18:18:56,564 DEBUG HandlerThread:35428 [meta.py:_setup_git():201] setup git | |
| 2022-05-07 18:18:56,597 DEBUG HandlerThread:35428 [meta.py:_setup_git():208] setup git done | |
| 2022-05-07 18:18:56,598 DEBUG HandlerThread:35428 [meta.py:_save_pip():55] save pip | |
| 2022-05-07 18:18:56,598 DEBUG HandlerThread:35428 [meta.py:_save_pip():69] save pip done | |
| 2022-05-07 18:18:56,598 DEBUG HandlerThread:35428 [meta.py:probe():249] probe done | |
| 2022-05-07 18:18:56,601 DEBUG SenderThread:35428 [sender.py:send():235] send: files | |
| 2022-05-07 18:18:56,601 INFO SenderThread:35428 [sender.py:_save_file():947] saving file wandb-metadata.json with policy now | |
| 2022-05-07 18:18:56,606 DEBUG HandlerThread:35428 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-05-07 18:18:56,607 DEBUG SenderThread:35428 [sender.py:send_request():249] send_request: stop_status | |
| 2022-05-07 18:18:56,637 DEBUG SenderThread:35428 [sender.py:send():235] send: telemetry | |
| 2022-05-07 18:18:56,881 INFO Thread-11 :35428 [upload_job.py:push():137] Uploaded file /tmp/tmpb1i3o9_4wandb/3h4lcq8i-wandb-metadata.json | |
| 2022-05-07 18:18:57,128 INFO Thread-7 :35428 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_181853-3nproo3b/files/wandb-metadata.json | |
| 2022-05-07 18:18:57,128 INFO Thread-7 :35428 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_181853-3nproo3b/files/requirements.txt | |
| 2022-05-07 18:18:57,128 INFO Thread-7 :35428 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_181853-3nproo3b/files/output.log | |
| 2022-05-07 18:18:59,128 INFO Thread-7 :35428 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_181853-3nproo3b/files/output.log | |
| 2022-05-07 18:19:01,129 INFO Thread-7 :35428 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_181853-3nproo3b/files/output.log | |
| 2022-05-07 18:19:11,133 INFO Thread-7 :35428 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_181853-3nproo3b/files/output.log | |
| 2022-05-07 18:19:11,639 DEBUG HandlerThread:35428 [handler.py:handle_request():141] handle_request: stop_status | |
| 2022-05-07 18:19:11,639 DEBUG SenderThread:35428 [sender.py:send_request():249] send_request: stop_status | |
| 2022-05-07 18:19:13,133 INFO Thread-7 :35428 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_181853-3nproo3b/files/output.log | |
| 2022-05-07 18:19:13,813 DEBUG SenderThread:35428 [sender.py:send():235] send: telemetry | |
| 2022-05-07 18:19:13,813 DEBUG HandlerThread:35428 [handler.py:handle_request():141] handle_request: partial_history | |
| 2022-05-07 18:19:15,134 INFO Thread-7 :35428 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_181853-3nproo3b/files/output.log | |
| 2022-05-07 18:19:16,561 DEBUG SenderThread:35428 [sender.py:send():235] send: exit | |
| 2022-05-07 18:19:16,561 DEBUG HandlerThread:35428 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-05-07 18:19:16,561 INFO SenderThread:35428 [sender.py:send_exit():371] handling exit code: 1 | |
| 2022-05-07 18:19:16,561 INFO SenderThread:35428 [sender.py:send_exit():373] handling runtime: 22 | |
| 2022-05-07 18:19:16,562 INFO SenderThread:35428 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 18:19:16,562 INFO SenderThread:35428 [sender.py:send_exit():379] send defer | |
| 2022-05-07 18:19:16,562 DEBUG SenderThread:35428 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-05-07 18:19:16,563 DEBUG HandlerThread:35428 [handler.py:handle_request():141] handle_request: defer | |
| 2022-05-07 18:19:16,563 INFO HandlerThread:35428 [handler.py:handle_request_defer():164] handle defer: 0 | |
| 2022-05-07 18:19:16,563 DEBUG SenderThread:35428 [sender.py:send_request():249] send_request: defer | |
| 2022-05-07 18:19:16,563 INFO SenderThread:35428 [sender.py:send_request_defer():388] handle sender defer: 0 | |
| 2022-05-07 18:19:16,563 INFO SenderThread:35428 [sender.py:transition_state():392] send defer: 1 | |
| 2022-05-07 18:19:16,563 DEBUG HandlerThread:35428 [handler.py:handle_request():141] handle_request: defer | |
| 2022-05-07 18:19:16,563 INFO HandlerThread:35428 [handler.py:handle_request_defer():164] handle defer: 1 | |
| 2022-05-07 18:19:16,620 DEBUG SenderThread:35428 [sender.py:send_request():249] send_request: defer | |
| 2022-05-07 18:19:16,620 INFO SenderThread:35428 [sender.py:send_request_defer():388] handle sender defer: 1 | |
| 2022-05-07 18:19:16,620 INFO SenderThread:35428 [sender.py:transition_state():392] send defer: 2 | |
| 2022-05-07 18:19:16,620 DEBUG SenderThread:35428 [sender.py:send():235] send: stats | |
| 2022-05-07 18:19:16,621 DEBUG HandlerThread:35428 [handler.py:handle_request():141] handle_request: defer | |
| 2022-05-07 18:19:16,621 INFO HandlerThread:35428 [handler.py:handle_request_defer():164] handle defer: 2 | |
| 2022-05-07 18:19:16,621 DEBUG SenderThread:35428 [sender.py:send_request():249] send_request: defer | |
| 2022-05-07 18:19:16,621 INFO SenderThread:35428 [sender.py:send_request_defer():388] handle sender defer: 2 | |
| 2022-05-07 18:19:16,621 INFO SenderThread:35428 [sender.py:transition_state():392] send defer: 3 | |
| 2022-05-07 18:19:16,621 DEBUG HandlerThread:35428 [handler.py:handle_request():141] handle_request: defer | |
| 2022-05-07 18:19:16,621 INFO HandlerThread:35428 [handler.py:handle_request_defer():164] handle defer: 3 | |
| 2022-05-07 18:19:16,621 DEBUG SenderThread:35428 [sender.py:send_request():249] send_request: defer | |
| 2022-05-07 18:19:16,621 INFO SenderThread:35428 [sender.py:send_request_defer():388] handle sender defer: 3 | |
| 2022-05-07 18:19:16,621 INFO SenderThread:35428 [sender.py:transition_state():392] send defer: 4 | |
| 2022-05-07 18:19:16,622 DEBUG HandlerThread:35428 [handler.py:handle_request():141] handle_request: defer | |
| 2022-05-07 18:19:16,622 INFO HandlerThread:35428 [handler.py:handle_request_defer():164] handle defer: 4 | |
| 2022-05-07 18:19:16,622 DEBUG SenderThread:35428 [sender.py:send():235] send: summary | |
| 2022-05-07 18:19:16,622 INFO SenderThread:35428 [sender.py:_save_file():947] saving file wandb-summary.json with policy end | |
| 2022-05-07 18:19:16,622 DEBUG SenderThread:35428 [sender.py:send_request():249] send_request: defer | |
| 2022-05-07 18:19:16,622 INFO SenderThread:35428 [sender.py:send_request_defer():388] handle sender defer: 4 | |
| 2022-05-07 18:19:16,622 INFO SenderThread:35428 [sender.py:transition_state():392] send defer: 5 | |
| 2022-05-07 18:19:16,622 DEBUG HandlerThread:35428 [handler.py:handle_request():141] handle_request: defer | |
| 2022-05-07 18:19:16,622 INFO HandlerThread:35428 [handler.py:handle_request_defer():164] handle defer: 5 | |
| 2022-05-07 18:19:16,622 DEBUG SenderThread:35428 [sender.py:send_request():249] send_request: defer | |
| 2022-05-07 18:19:16,623 INFO SenderThread:35428 [sender.py:send_request_defer():388] handle sender defer: 5 | |
| 2022-05-07 18:19:16,664 DEBUG HandlerThread:35428 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-05-07 18:19:16,689 INFO SenderThread:35428 [sender.py:transition_state():392] send defer: 6 | |
| 2022-05-07 18:19:16,690 DEBUG SenderThread:35428 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-05-07 18:19:16,690 DEBUG HandlerThread:35428 [handler.py:handle_request():141] handle_request: defer | |
| 2022-05-07 18:19:16,690 INFO HandlerThread:35428 [handler.py:handle_request_defer():164] handle defer: 6 | |
| 2022-05-07 18:19:16,690 DEBUG SenderThread:35428 [sender.py:send_request():249] send_request: defer | |
| 2022-05-07 18:19:16,690 INFO SenderThread:35428 [sender.py:send_request_defer():388] handle sender defer: 6 | |
| 2022-05-07 18:19:16,691 INFO SenderThread:35428 [dir_watcher.py:finish():283] shutting down directory watcher | |
| 2022-05-07 18:19:16,791 DEBUG HandlerThread:35428 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-05-07 18:19:17,135 INFO Thread-7 :35428 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_181853-3nproo3b/files/config.yaml | |
| 2022-05-07 18:19:17,135 INFO SenderThread:35428 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_181853-3nproo3b/files/wandb-summary.json | |
| 2022-05-07 18:19:17,136 INFO SenderThread:35428 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_181853-3nproo3b/files/output.log | |
| 2022-05-07 18:19:17,136 INFO SenderThread:35428 [dir_watcher.py:finish():313] scan: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_181853-3nproo3b/files | |
| 2022-05-07 18:19:17,136 INFO SenderThread:35428 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_181853-3nproo3b/files/requirements.txt requirements.txt | |
| 2022-05-07 18:19:17,136 INFO SenderThread:35428 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_181853-3nproo3b/files/output.log output.log | |
| 2022-05-07 18:19:17,136 INFO SenderThread:35428 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_181853-3nproo3b/files/wandb-metadata.json wandb-metadata.json | |
| 2022-05-07 18:19:17,137 INFO SenderThread:35428 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_181853-3nproo3b/files/config.yaml config.yaml | |
| 2022-05-07 18:19:17,137 INFO SenderThread:35428 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_181853-3nproo3b/files/wandb-summary.json wandb-summary.json | |
| 2022-05-07 18:19:17,139 INFO SenderThread:35428 [sender.py:transition_state():392] send defer: 7 | |
| 2022-05-07 18:19:17,140 DEBUG SenderThread:35428 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-05-07 18:19:17,140 DEBUG HandlerThread:35428 [handler.py:handle_request():141] handle_request: defer | |
| 2022-05-07 18:19:17,141 INFO HandlerThread:35428 [handler.py:handle_request_defer():164] handle defer: 7 | |
| 2022-05-07 18:19:17,143 DEBUG SenderThread:35428 [sender.py:send_request():249] send_request: defer | |
| 2022-05-07 18:19:17,146 INFO SenderThread:35428 [sender.py:send_request_defer():388] handle sender defer: 7 | |
| 2022-05-07 18:19:17,148 INFO SenderThread:35428 [file_pusher.py:finish():145] shutting down file pusher | |
| 2022-05-07 18:19:17,241 DEBUG HandlerThread:35428 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-05-07 18:19:17,242 DEBUG SenderThread:35428 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-05-07 18:19:17,343 DEBUG HandlerThread:35428 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-05-07 18:19:17,343 DEBUG SenderThread:35428 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-05-07 18:19:17,366 INFO Thread-12 :35428 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-dummy/wandb/run-20220507_181853-3nproo3b/files/requirements.txt | |
| 2022-05-07 18:19:17,375 INFO Thread-15 :35428 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-dummy/wandb/run-20220507_181853-3nproo3b/files/wandb-summary.json | |
| 2022-05-07 18:19:17,395 INFO Thread-14 :35428 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-dummy/wandb/run-20220507_181853-3nproo3b/files/config.yaml | |
| 2022-05-07 18:19:17,445 DEBUG HandlerThread:35428 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-05-07 18:19:17,445 DEBUG SenderThread:35428 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-05-07 18:19:17,483 INFO Thread-13 :35428 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-dummy/wandb/run-20220507_181853-3nproo3b/files/output.log | |
| 2022-05-07 18:19:17,547 DEBUG HandlerThread:35428 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-05-07 18:19:17,547 DEBUG SenderThread:35428 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-05-07 18:19:17,648 DEBUG HandlerThread:35428 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-05-07 18:19:17,649 DEBUG SenderThread:35428 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-05-07 18:19:17,684 INFO Thread-6 :35428 [sender.py:transition_state():392] send defer: 8 | |
| 2022-05-07 18:19:17,684 DEBUG HandlerThread:35428 [handler.py:handle_request():141] handle_request: defer | |
| 2022-05-07 18:19:17,684 INFO HandlerThread:35428 [handler.py:handle_request_defer():164] handle defer: 8 | |
| 2022-05-07 18:19:17,684 DEBUG SenderThread:35428 [sender.py:send_request():249] send_request: defer | |
| 2022-05-07 18:19:17,685 INFO SenderThread:35428 [sender.py:send_request_defer():388] handle sender defer: 8 | |
| 2022-05-07 18:19:17,750 DEBUG HandlerThread:35428 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-05-07 18:19:22,092 INFO SenderThread:35428 [sender.py:transition_state():392] send defer: 9 | |
| 2022-05-07 18:19:22,092 DEBUG SenderThread:35428 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-05-07 18:19:22,093 DEBUG HandlerThread:35428 [handler.py:handle_request():141] handle_request: defer | |
| 2022-05-07 18:19:22,093 INFO HandlerThread:35428 [handler.py:handle_request_defer():164] handle defer: 9 | |
| 2022-05-07 18:19:22,093 DEBUG SenderThread:35428 [sender.py:send_request():249] send_request: defer | |
| 2022-05-07 18:19:22,093 INFO SenderThread:35428 [sender.py:send_request_defer():388] handle sender defer: 9 | |
| 2022-05-07 18:19:22,093 INFO SenderThread:35428 [sender.py:transition_state():392] send defer: 10 | |
| 2022-05-07 18:19:22,093 DEBUG SenderThread:35428 [sender.py:send():235] send: final | |
| 2022-05-07 18:19:22,094 DEBUG HandlerThread:35428 [handler.py:handle_request():141] handle_request: defer | |
| 2022-05-07 18:19:22,094 INFO HandlerThread:35428 [handler.py:handle_request_defer():164] handle defer: 10 | |
| 2022-05-07 18:19:22,094 DEBUG SenderThread:35428 [sender.py:send():235] send: footer | |
| 2022-05-07 18:19:22,094 DEBUG SenderThread:35428 [sender.py:send_request():249] send_request: defer | |
| 2022-05-07 18:19:22,094 INFO SenderThread:35428 [sender.py:send_request_defer():388] handle sender defer: 10 | |
| 2022-05-07 18:19:22,194 DEBUG HandlerThread:35428 [handler.py:handle_request():141] handle_request: poll_exit | |
| 2022-05-07 18:19:22,195 DEBUG SenderThread:35428 [sender.py:send_request():249] send_request: poll_exit | |
| 2022-05-07 18:19:22,195 INFO SenderThread:35428 [file_pusher.py:join():150] waiting for file pusher | |
| 2022-05-07 18:19:24,366 DEBUG HandlerThread:35428 [handler.py:handle_request():141] handle_request: sampled_history | |
| 2022-05-07 18:19:24,367 DEBUG HandlerThread:35428 [handler.py:handle_request():141] handle_request: get_summary | |
| 2022-05-07 18:19:24,368 DEBUG HandlerThread:35428 [handler.py:handle_request():141] handle_request: shutdown | |
| 2022-05-07 18:19:24,368 INFO HandlerThread:35428 [handler.py:finish():778] shutting down handler | |
| 2022-05-07 18:19:25,094 INFO WriterThread:35428 [datastore.py:close():281] close: /home/sanchitgandhi/flax-dummy/wandb/run-20220507_181853-3nproo3b/run-3nproo3b.wandb | |
| 2022-05-07 18:19:25,265 INFO SenderThread:35428 [sender.py:finish():1078] shutting down sender | |
| 2022-05-07 18:19:25,265 INFO SenderThread:35428 [file_pusher.py:finish():145] shutting down file pusher | |
| 2022-05-07 18:19:25,265 INFO SenderThread:35428 [file_pusher.py:join():150] waiting for file pusher | |
| 2022-05-07 18:19:25,267 INFO MainThread:35428 [internal.py:handle_exit():82] Internal process exited | |