2022-01-14 21:28:56,265 INFO MainThread:8253 [internal.py:wandb_internal():87] W&B internal server running at pid: 8253, started at: 2022-01-14 21:28:56.265129 2022-01-14 21:28:56,268 DEBUG SenderThread:8253 [sender.py:send():234] send: header 2022-01-14 21:28:56,268 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: check_version 2022-01-14 21:28:56,268 INFO WriterThread:8253 [datastore.py:open_for_write():77] open: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/run-32qdb4k5.wandb 2022-01-14 21:28:56,268 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: check_version 2022-01-14 21:28:56,352 DEBUG SenderThread:8253 [sender.py:send():234] send: run 2022-01-14 21:28:56,515 INFO SenderThread:8253 [dir_watcher.py:__init__():169] watching files in: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files 2022-01-14 21:28:56,515 INFO SenderThread:8253 [sender.py:_start_run_threads():804] run started: 32qdb4k5 with start time 1642195735 2022-01-14 21:28:56,515 DEBUG SenderThread:8253 [sender.py:send():234] send: summary 2022-01-14 21:28:56,515 INFO SenderThread:8253 [sender.py:_save_file():939] saving file wandb-summary.json with policy end 2022-01-14 21:28:56,515 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: run_start 2022-01-14 21:28:57,561 INFO Thread-8 :8253 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/wandb-summary.json 2022-01-14 21:28:58,974 DEBUG HandlerThread:8253 [meta.py:__init__():40] meta init 2022-01-14 21:28:58,974 DEBUG HandlerThread:8253 [meta.py:__init__():54] meta init done 2022-01-14 21:28:58,974 DEBUG HandlerThread:8253 [meta.py:probe():214] probe 2022-01-14 21:28:58,975 DEBUG HandlerThread:8253 [meta.py:_setup_git():204] setup git 2022-01-14 21:28:59,006 DEBUG HandlerThread:8253 [meta.py:_setup_git():211] setup git done 2022-01-14 21:28:59,006 DEBUG HandlerThread:8253 [meta.py:_save_code():92] save code 2022-01-14 21:28:59,018 DEBUG HandlerThread:8253 [meta.py:_save_code():113] save code done 2022-01-14 21:28:59,018 DEBUG HandlerThread:8253 [meta.py:_save_patches():130] save patches 2022-01-14 21:28:59,561 INFO Thread-8 :8253 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/code/run_mlm_flax.py 2022-01-14 21:28:59,562 INFO Thread-8 :8253 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/code 2022-01-14 21:29:01,562 INFO Thread-8 :8253 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/output.log 2022-01-14 21:29:03,563 INFO Thread-8 :8253 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/output.log 2022-01-14 21:29:03,563 INFO Thread-8 :8253 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/diff.patch 2022-01-14 21:29:08,274 ERROR HandlerThread:8253 [meta.py:_save_patches():171] Error generating diff: Command '['git', 'diff', '--submodule=diff', 'HEAD']' timed out after 5 seconds 2022-01-14 21:29:08,274 DEBUG HandlerThread:8253 [meta.py:_save_patches():172] save patches done 2022-01-14 21:29:08,274 DEBUG HandlerThread:8253 [meta.py:_save_pip():58] save pip 2022-01-14 21:29:08,275 DEBUG HandlerThread:8253 [meta.py:_save_pip():72] save pip done 2022-01-14 21:29:08,275 DEBUG HandlerThread:8253 [meta.py:probe():252] probe done 2022-01-14 21:29:08,283 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status 2022-01-14 21:29:08,284 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status 2022-01-14 21:29:08,566 DEBUG SenderThread:8253 [sender.py:send():234] send: config 2022-01-14 21:29:08,566 DEBUG SenderThread:8253 [sender.py:send():234] send: config 2022-01-14 21:29:08,566 DEBUG SenderThread:8253 [sender.py:send():234] send: config 2022-01-14 21:29:08,567 DEBUG SenderThread:8253 [sender.py:send():234] send: files 2022-01-14 21:29:08,567 INFO SenderThread:8253 [sender.py:_save_file():939] saving file wandb-metadata.json with policy now 2022-01-14 21:29:08,567 INFO SenderThread:8253 [sender.py:_save_file():939] saving file code/run_mlm_flax.py with policy now 2022-01-14 21:29:08,571 INFO Thread-8 :8253 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/requirements.txt 2022-01-14 21:29:08,571 INFO Thread-8 :8253 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/wandb-metadata.json 2022-01-14 21:29:09,032 INFO Thread-12 :8253 [upload_job.py:push():137] Uploaded file /tmp/tmpdg54qv_0wandb/w1tibuxq-code/run_mlm_flax.py 2022-01-14 21:29:09,069 INFO Thread-11 :8253 [upload_job.py:push():137] Uploaded file /tmp/tmpdg54qv_0wandb/35h4ryp5-wandb-metadata.json 2022-01-14 21:29:09,571 INFO Thread-8 :8253 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/output.log 2022-01-14 21:29:21,520 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status 2022-01-14 21:29:21,521 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status 2022-01-14 21:29:27,049 DEBUG SenderThread:8253 [sender.py:send():234] send: stats 2022-01-14 21:29:27,579 INFO Thread-8 :8253 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/config.yaml 2022-01-14 21:29:36,656 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status 2022-01-14 21:29:36,657 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status 2022-01-14 21:29:51,794 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status 2022-01-14 21:29:51,795 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status 2022-01-14 21:29:57,119 DEBUG SenderThread:8253 [sender.py:send():234] send: stats 2022-01-14 21:29:58,591 INFO Thread-8 :8253 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/output.log 2022-01-14 21:30:06,945 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status 2022-01-14 21:30:06,945 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status 2022-01-14 21:30:22,126 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status 2022-01-14 21:30:22,127 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status 2022-01-14 21:30:27,189 DEBUG SenderThread:8253 [sender.py:send():234] send: stats 2022-01-14 21:30:37,330 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status 2022-01-14 21:30:37,330 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status 2022-01-14 21:30:52,532 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status 2022-01-14 21:30:52,532 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status 2022-01-14 21:30:57,257 DEBUG SenderThread:8253 [sender.py:send():234] send: stats 2022-01-14 21:31:07,691 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status 2022-01-14 21:31:07,692 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status 2022-01-14 21:31:22,944 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status 2022-01-14 21:31:22,945 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status 2022-01-14 21:31:27,323 DEBUG SenderThread:8253 [sender.py:send():234] send: stats 2022-01-14 21:31:38,085 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status 2022-01-14 21:31:38,086 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status 2022-01-14 21:31:53,231 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status 2022-01-14 21:31:53,231 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status 2022-01-14 21:31:57,395 DEBUG SenderThread:8253 [sender.py:send():234] send: stats 2022-01-14 21:32:08,366 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status 2022-01-14 21:32:08,367 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status 2022-01-14 21:32:16,649 INFO Thread-8 :8253 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/output.log 2022-01-14 21:32:16,893 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit 2022-01-14 21:32:16,893 DEBUG SenderThread:8253 [sender.py:send():234] send: telemetry 2022-01-14 21:32:16,893 DEBUG SenderThread:8253 [sender.py:send():234] send: exit 2022-01-14 21:32:16,893 INFO SenderThread:8253 [sender.py:send_exit():366] handling exit code: 1 2022-01-14 21:32:16,894 INFO SenderThread:8253 [sender.py:send_exit():368] handling runtime: 200 2022-01-14 21:32:16,894 INFO SenderThread:8253 [sender.py:_save_file():939] saving file wandb-summary.json with policy end 2022-01-14 21:32:16,894 INFO SenderThread:8253 [sender.py:send_exit():374] send defer 2022-01-14 21:32:16,894 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit 2022-01-14 21:32:16,895 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer 2022-01-14 21:32:16,895 INFO HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 0 2022-01-14 21:32:16,895 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: defer 2022-01-14 21:32:16,895 INFO SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 0 2022-01-14 21:32:16,895 INFO SenderThread:8253 [sender.py:transition_state():387] send defer: 1 2022-01-14 21:32:16,896 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer 2022-01-14 21:32:16,896 INFO HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 1 2022-01-14 21:32:16,941 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: defer 2022-01-14 21:32:16,941 INFO SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 1 2022-01-14 21:32:16,941 INFO SenderThread:8253 [sender.py:transition_state():387] send defer: 2 2022-01-14 21:32:16,941 DEBUG SenderThread:8253 [sender.py:send():234] send: stats 2022-01-14 21:32:16,942 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer 2022-01-14 21:32:16,942 INFO HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 2 2022-01-14 21:32:16,942 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: defer 2022-01-14 21:32:16,942 INFO SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 2 2022-01-14 21:32:16,942 INFO SenderThread:8253 [sender.py:transition_state():387] send defer: 3 2022-01-14 21:32:16,942 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer 2022-01-14 21:32:16,942 INFO HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 3 2022-01-14 21:32:16,943 DEBUG SenderThread:8253 [sender.py:send():234] send: summary 2022-01-14 21:32:16,943 INFO SenderThread:8253 [sender.py:_save_file():939] saving file wandb-summary.json with policy end 2022-01-14 21:32:16,943 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: defer 2022-01-14 21:32:16,943 INFO SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 3 2022-01-14 21:32:16,943 INFO SenderThread:8253 [sender.py:transition_state():387] send defer: 4 2022-01-14 21:32:16,943 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer 2022-01-14 21:32:16,943 INFO HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 4 2022-01-14 21:32:16,944 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: defer 2022-01-14 21:32:16,944 INFO SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 4 2022-01-14 21:32:16,997 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit 2022-01-14 21:32:17,650 INFO Thread-8 :8253 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/output.log 2022-01-14 21:32:17,650 INFO Thread-8 :8253 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/wandb-summary.json 2022-01-14 21:32:17,685 INFO SenderThread:8253 [sender.py:transition_state():387] send defer: 5 2022-01-14 21:32:17,686 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit 2022-01-14 21:32:17,686 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer 2022-01-14 21:32:17,686 INFO HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 5 2022-01-14 21:32:17,686 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: defer 2022-01-14 21:32:17,686 INFO SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 5 2022-01-14 21:32:17,687 INFO SenderThread:8253 [dir_watcher.py:finish():283] shutting down directory watcher 2022-01-14 21:32:17,787 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit 2022-01-14 21:32:18,650 INFO Thread-8 :8253 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/config.yaml 2022-01-14 21:32:18,651 INFO SenderThread:8253 [dir_watcher.py:finish():313] scan: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files 2022-01-14 21:32:18,651 INFO SenderThread:8253 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/config.yaml config.yaml 2022-01-14 21:32:18,651 INFO SenderThread:8253 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/diff.patch diff.patch 2022-01-14 21:32:18,651 INFO SenderThread:8253 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/requirements.txt requirements.txt 2022-01-14 21:32:18,652 INFO SenderThread:8253 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/output.log output.log 2022-01-14 21:32:18,652 INFO SenderThread:8253 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/wandb-summary.json wandb-summary.json 2022-01-14 21:32:18,652 INFO SenderThread:8253 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/wandb-metadata.json wandb-metadata.json 2022-01-14 21:32:18,656 INFO SenderThread:8253 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/code/run_mlm_flax.py code/run_mlm_flax.py 2022-01-14 21:32:18,656 INFO SenderThread:8253 [sender.py:transition_state():387] send defer: 6 2022-01-14 21:32:18,656 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit 2022-01-14 21:32:18,657 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer 2022-01-14 21:32:18,657 INFO HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 6 2022-01-14 21:32:18,662 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: defer 2022-01-14 21:32:18,665 INFO SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 6 2022-01-14 21:32:18,665 INFO SenderThread:8253 [file_pusher.py:finish():177] shutting down file pusher 2022-01-14 21:32:18,757 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit 2022-01-14 21:32:18,758 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit 2022-01-14 21:32:18,859 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit 2022-01-14 21:32:18,860 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit 2022-01-14 21:32:18,961 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit 2022-01-14 21:32:18,962 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit 2022-01-14 21:32:19,063 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit 2022-01-14 21:32:19,064 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit 2022-01-14 21:32:19,139 INFO Thread-15 :8253 [upload_job.py:push():137] Uploaded file /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/output.log 2022-01-14 21:32:19,148 INFO Thread-14 :8253 [upload_job.py:push():137] Uploaded file /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/requirements.txt 2022-01-14 21:32:19,165 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit 2022-01-14 21:32:19,165 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit 2022-01-14 21:32:19,171 INFO Thread-13 :8253 [upload_job.py:push():137] Uploaded file /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/config.yaml 2022-01-14 21:32:19,267 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit 2022-01-14 21:32:19,267 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit 2022-01-14 21:32:19,288 INFO Thread-16 :8253 [upload_job.py:push():137] Uploaded file /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/wandb-summary.json 2022-01-14 21:32:19,370 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit 2022-01-14 21:32:19,370 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit 2022-01-14 21:32:19,472 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit 2022-01-14 21:32:19,472 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit 2022-01-14 21:32:19,489 INFO Thread-7 :8253 [sender.py:transition_state():387] send defer: 7 2022-01-14 21:32:19,489 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer 2022-01-14 21:32:19,490 INFO HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 7 2022-01-14 21:32:19,490 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: defer 2022-01-14 21:32:19,490 INFO SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 7 2022-01-14 21:32:19,573 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit 2022-01-14 21:32:19,915 INFO SenderThread:8253 [sender.py:transition_state():387] send defer: 8 2022-01-14 21:32:19,916 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit 2022-01-14 21:32:19,916 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer 2022-01-14 21:32:19,916 INFO HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 8 2022-01-14 21:32:19,917 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: defer 2022-01-14 21:32:19,917 INFO SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 8 2022-01-14 21:32:19,917 INFO SenderThread:8253 [sender.py:transition_state():387] send defer: 9 2022-01-14 21:32:19,917 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer 2022-01-14 21:32:19,917 INFO HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 9 2022-01-14 21:32:19,918 DEBUG SenderThread:8253 [sender.py:send():234] send: final 2022-01-14 21:32:19,918 DEBUG SenderThread:8253 [sender.py:send():234] send: footer 2022-01-14 21:32:19,918 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: defer 2022-01-14 21:32:19,918 INFO SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 9 2022-01-14 21:32:20,017 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit 2022-01-14 21:32:20,018 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit 2022-01-14 21:32:20,018 INFO SenderThread:8253 [file_pusher.py:join():182] waiting for file pusher 2022-01-14 21:32:20,278 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: get_summary 2022-01-14 21:32:20,278 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: sampled_history 2022-01-14 21:32:20,279 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: shutdown 2022-01-14 21:32:20,279 INFO HandlerThread:8253 [handler.py:finish():731] shutting down handler 2022-01-14 21:32:20,918 INFO WriterThread:8253 [datastore.py:close():281] close: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/run-32qdb4k5.wandb 2022-01-14 21:32:21,277 INFO SenderThread:8253 [sender.py:finish():1070] shutting down sender 2022-01-14 21:32:21,277 INFO SenderThread:8253 [file_pusher.py:finish():177] shutting down file pusher 2022-01-14 21:32:21,277 INFO SenderThread:8253 [file_pusher.py:join():182] waiting for file pusher 2022-01-14 21:32:21,279 INFO MainThread:8253 [internal.py:handle_exit():77] Internal process exited