|
2022-01-14 21:28:56,265 INFO MainThread:8253 [internal.py:wandb_internal():87] W&B internal server running at pid: 8253, started at: 2022-01-14 21:28:56.265129 |
|
2022-01-14 21:28:56,268 DEBUG SenderThread:8253 [sender.py:send():234] send: header |
|
2022-01-14 21:28:56,268 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: check_version |
|
2022-01-14 21:28:56,268 INFO WriterThread:8253 [datastore.py:open_for_write():77] open: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/run-32qdb4k5.wandb |
|
2022-01-14 21:28:56,268 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: check_version |
|
2022-01-14 21:28:56,352 DEBUG SenderThread:8253 [sender.py:send():234] send: run |
|
2022-01-14 21:28:56,515 INFO SenderThread:8253 [dir_watcher.py:__init__():169] watching files in: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files |
|
2022-01-14 21:28:56,515 INFO SenderThread:8253 [sender.py:_start_run_threads():804] run started: 32qdb4k5 with start time 1642195735 |
|
2022-01-14 21:28:56,515 DEBUG SenderThread:8253 [sender.py:send():234] send: summary |
|
2022-01-14 21:28:56,515 INFO SenderThread:8253 [sender.py:_save_file():939] saving file wandb-summary.json with policy end |
|
2022-01-14 21:28:56,515 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: run_start |
|
2022-01-14 21:28:57,561 INFO Thread-8 :8253 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/wandb-summary.json |
|
2022-01-14 21:28:58,974 DEBUG HandlerThread:8253 [meta.py:__init__():40] meta init |
|
2022-01-14 21:28:58,974 DEBUG HandlerThread:8253 [meta.py:__init__():54] meta init done |
|
2022-01-14 21:28:58,974 DEBUG HandlerThread:8253 [meta.py:probe():214] probe |
|
2022-01-14 21:28:58,975 DEBUG HandlerThread:8253 [meta.py:_setup_git():204] setup git |
|
2022-01-14 21:28:59,006 DEBUG HandlerThread:8253 [meta.py:_setup_git():211] setup git done |
|
2022-01-14 21:28:59,006 DEBUG HandlerThread:8253 [meta.py:_save_code():92] save code |
|
2022-01-14 21:28:59,018 DEBUG HandlerThread:8253 [meta.py:_save_code():113] save code done |
|
2022-01-14 21:28:59,018 DEBUG HandlerThread:8253 [meta.py:_save_patches():130] save patches |
|
2022-01-14 21:28:59,561 INFO Thread-8 :8253 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/code/run_mlm_flax.py |
|
2022-01-14 21:28:59,562 INFO Thread-8 :8253 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/code |
|
2022-01-14 21:29:01,562 INFO Thread-8 :8253 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/output.log |
|
2022-01-14 21:29:03,563 INFO Thread-8 :8253 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/output.log |
|
2022-01-14 21:29:03,563 INFO Thread-8 :8253 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/diff.patch |
|
2022-01-14 21:29:08,274 ERROR HandlerThread:8253 [meta.py:_save_patches():171] Error generating diff: Command '['git', 'diff', '--submodule=diff', 'HEAD']' timed out after 5 seconds |
|
2022-01-14 21:29:08,274 DEBUG HandlerThread:8253 [meta.py:_save_patches():172] save patches done |
|
2022-01-14 21:29:08,274 DEBUG HandlerThread:8253 [meta.py:_save_pip():58] save pip |
|
2022-01-14 21:29:08,275 DEBUG HandlerThread:8253 [meta.py:_save_pip():72] save pip done |
|
2022-01-14 21:29:08,275 DEBUG HandlerThread:8253 [meta.py:probe():252] probe done |
|
2022-01-14 21:29:08,283 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status |
|
2022-01-14 21:29:08,284 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status |
|
2022-01-14 21:29:08,566 DEBUG SenderThread:8253 [sender.py:send():234] send: config |
|
2022-01-14 21:29:08,566 DEBUG SenderThread:8253 [sender.py:send():234] send: config |
|
2022-01-14 21:29:08,566 DEBUG SenderThread:8253 [sender.py:send():234] send: config |
|
2022-01-14 21:29:08,567 DEBUG SenderThread:8253 [sender.py:send():234] send: files |
|
2022-01-14 21:29:08,567 INFO SenderThread:8253 [sender.py:_save_file():939] saving file wandb-metadata.json with policy now |
|
2022-01-14 21:29:08,567 INFO SenderThread:8253 [sender.py:_save_file():939] saving file code/run_mlm_flax.py with policy now |
|
2022-01-14 21:29:08,571 INFO Thread-8 :8253 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/requirements.txt |
|
2022-01-14 21:29:08,571 INFO Thread-8 :8253 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/wandb-metadata.json |
|
2022-01-14 21:29:09,032 INFO Thread-12 :8253 [upload_job.py:push():137] Uploaded file /tmp/tmpdg54qv_0wandb/w1tibuxq-code/run_mlm_flax.py |
|
2022-01-14 21:29:09,069 INFO Thread-11 :8253 [upload_job.py:push():137] Uploaded file /tmp/tmpdg54qv_0wandb/35h4ryp5-wandb-metadata.json |
|
2022-01-14 21:29:09,571 INFO Thread-8 :8253 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/output.log |
|
2022-01-14 21:29:21,520 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status |
|
2022-01-14 21:29:21,521 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status |
|
2022-01-14 21:29:27,049 DEBUG SenderThread:8253 [sender.py:send():234] send: stats |
|
2022-01-14 21:29:27,579 INFO Thread-8 :8253 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/config.yaml |
|
2022-01-14 21:29:36,656 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status |
|
2022-01-14 21:29:36,657 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status |
|
2022-01-14 21:29:51,794 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status |
|
2022-01-14 21:29:51,795 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status |
|
2022-01-14 21:29:57,119 DEBUG SenderThread:8253 [sender.py:send():234] send: stats |
|
2022-01-14 21:29:58,591 INFO Thread-8 :8253 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/output.log |
|
2022-01-14 21:30:06,945 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status |
|
2022-01-14 21:30:06,945 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status |
|
2022-01-14 21:30:22,126 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status |
|
2022-01-14 21:30:22,127 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status |
|
2022-01-14 21:30:27,189 DEBUG SenderThread:8253 [sender.py:send():234] send: stats |
|
2022-01-14 21:30:37,330 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status |
|
2022-01-14 21:30:37,330 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status |
|
2022-01-14 21:30:52,532 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status |
|
2022-01-14 21:30:52,532 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status |
|
2022-01-14 21:30:57,257 DEBUG SenderThread:8253 [sender.py:send():234] send: stats |
|
2022-01-14 21:31:07,691 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status |
|
2022-01-14 21:31:07,692 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status |
|
2022-01-14 21:31:22,944 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status |
|
2022-01-14 21:31:22,945 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status |
|
2022-01-14 21:31:27,323 DEBUG SenderThread:8253 [sender.py:send():234] send: stats |
|
2022-01-14 21:31:38,085 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status |
|
2022-01-14 21:31:38,086 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status |
|
2022-01-14 21:31:53,231 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status |
|
2022-01-14 21:31:53,231 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status |
|
2022-01-14 21:31:57,395 DEBUG SenderThread:8253 [sender.py:send():234] send: stats |
|
2022-01-14 21:32:08,366 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status |
|
2022-01-14 21:32:08,367 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status |
|
2022-01-14 21:32:16,649 INFO Thread-8 :8253 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/output.log |
|
2022-01-14 21:32:16,893 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit |
|
2022-01-14 21:32:16,893 DEBUG SenderThread:8253 [sender.py:send():234] send: telemetry |
|
2022-01-14 21:32:16,893 DEBUG SenderThread:8253 [sender.py:send():234] send: exit |
|
2022-01-14 21:32:16,893 INFO SenderThread:8253 [sender.py:send_exit():366] handling exit code: 1 |
|
2022-01-14 21:32:16,894 INFO SenderThread:8253 [sender.py:send_exit():368] handling runtime: 200 |
|
2022-01-14 21:32:16,894 INFO SenderThread:8253 [sender.py:_save_file():939] saving file wandb-summary.json with policy end |
|
2022-01-14 21:32:16,894 INFO SenderThread:8253 [sender.py:send_exit():374] send defer |
|
2022-01-14 21:32:16,894 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit |
|
2022-01-14 21:32:16,895 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer |
|
2022-01-14 21:32:16,895 INFO HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 0 |
|
2022-01-14 21:32:16,895 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: defer |
|
2022-01-14 21:32:16,895 INFO SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 0 |
|
2022-01-14 21:32:16,895 INFO SenderThread:8253 [sender.py:transition_state():387] send defer: 1 |
|
2022-01-14 21:32:16,896 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer |
|
2022-01-14 21:32:16,896 INFO HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 1 |
|
2022-01-14 21:32:16,941 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: defer |
|
2022-01-14 21:32:16,941 INFO SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 1 |
|
2022-01-14 21:32:16,941 INFO SenderThread:8253 [sender.py:transition_state():387] send defer: 2 |
|
2022-01-14 21:32:16,941 DEBUG SenderThread:8253 [sender.py:send():234] send: stats |
|
2022-01-14 21:32:16,942 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer |
|
2022-01-14 21:32:16,942 INFO HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 2 |
|
2022-01-14 21:32:16,942 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: defer |
|
2022-01-14 21:32:16,942 INFO SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 2 |
|
2022-01-14 21:32:16,942 INFO SenderThread:8253 [sender.py:transition_state():387] send defer: 3 |
|
2022-01-14 21:32:16,942 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer |
|
2022-01-14 21:32:16,942 INFO HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 3 |
|
2022-01-14 21:32:16,943 DEBUG SenderThread:8253 [sender.py:send():234] send: summary |
|
2022-01-14 21:32:16,943 INFO SenderThread:8253 [sender.py:_save_file():939] saving file wandb-summary.json with policy end |
|
2022-01-14 21:32:16,943 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: defer |
|
2022-01-14 21:32:16,943 INFO SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 3 |
|
2022-01-14 21:32:16,943 INFO SenderThread:8253 [sender.py:transition_state():387] send defer: 4 |
|
2022-01-14 21:32:16,943 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer |
|
2022-01-14 21:32:16,943 INFO HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 4 |
|
2022-01-14 21:32:16,944 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: defer |
|
2022-01-14 21:32:16,944 INFO SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 4 |
|
2022-01-14 21:32:16,997 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit |
|
2022-01-14 21:32:17,650 INFO Thread-8 :8253 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/output.log |
|
2022-01-14 21:32:17,650 INFO Thread-8 :8253 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/wandb-summary.json |
|
2022-01-14 21:32:17,685 INFO SenderThread:8253 [sender.py:transition_state():387] send defer: 5 |
|
2022-01-14 21:32:17,686 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit |
|
2022-01-14 21:32:17,686 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer |
|
2022-01-14 21:32:17,686 INFO HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 5 |
|
2022-01-14 21:32:17,686 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: defer |
|
2022-01-14 21:32:17,686 INFO SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 5 |
|
2022-01-14 21:32:17,687 INFO SenderThread:8253 [dir_watcher.py:finish():283] shutting down directory watcher |
|
2022-01-14 21:32:17,787 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit |
|
2022-01-14 21:32:18,650 INFO Thread-8 :8253 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/config.yaml |
|
2022-01-14 21:32:18,651 INFO SenderThread:8253 [dir_watcher.py:finish():313] scan: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files |
|
2022-01-14 21:32:18,651 INFO SenderThread:8253 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/config.yaml config.yaml |
|
2022-01-14 21:32:18,651 INFO SenderThread:8253 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/diff.patch diff.patch |
|
2022-01-14 21:32:18,651 INFO SenderThread:8253 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/requirements.txt requirements.txt |
|
2022-01-14 21:32:18,652 INFO SenderThread:8253 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/output.log output.log |
|
2022-01-14 21:32:18,652 INFO SenderThread:8253 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/wandb-summary.json wandb-summary.json |
|
2022-01-14 21:32:18,652 INFO SenderThread:8253 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/wandb-metadata.json wandb-metadata.json |
|
2022-01-14 21:32:18,656 INFO SenderThread:8253 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/code/run_mlm_flax.py code/run_mlm_flax.py |
|
2022-01-14 21:32:18,656 INFO SenderThread:8253 [sender.py:transition_state():387] send defer: 6 |
|
2022-01-14 21:32:18,656 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit |
|
2022-01-14 21:32:18,657 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer |
|
2022-01-14 21:32:18,657 INFO HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 6 |
|
2022-01-14 21:32:18,662 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: defer |
|
2022-01-14 21:32:18,665 INFO SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 6 |
|
2022-01-14 21:32:18,665 INFO SenderThread:8253 [file_pusher.py:finish():177] shutting down file pusher |
|
2022-01-14 21:32:18,757 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit |
|
2022-01-14 21:32:18,758 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit |
|
2022-01-14 21:32:18,859 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit |
|
2022-01-14 21:32:18,860 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit |
|
2022-01-14 21:32:18,961 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit |
|
2022-01-14 21:32:18,962 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit |
|
2022-01-14 21:32:19,063 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit |
|
2022-01-14 21:32:19,064 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit |
|
2022-01-14 21:32:19,139 INFO Thread-15 :8253 [upload_job.py:push():137] Uploaded file /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/output.log |
|
2022-01-14 21:32:19,148 INFO Thread-14 :8253 [upload_job.py:push():137] Uploaded file /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/requirements.txt |
|
2022-01-14 21:32:19,165 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit |
|
2022-01-14 21:32:19,165 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit |
|
2022-01-14 21:32:19,171 INFO Thread-13 :8253 [upload_job.py:push():137] Uploaded file /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/config.yaml |
|
2022-01-14 21:32:19,267 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit |
|
2022-01-14 21:32:19,267 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit |
|
2022-01-14 21:32:19,288 INFO Thread-16 :8253 [upload_job.py:push():137] Uploaded file /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/wandb-summary.json |
|
2022-01-14 21:32:19,370 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit |
|
2022-01-14 21:32:19,370 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit |
|
2022-01-14 21:32:19,472 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit |
|
2022-01-14 21:32:19,472 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit |
|
2022-01-14 21:32:19,489 INFO Thread-7 :8253 [sender.py:transition_state():387] send defer: 7 |
|
2022-01-14 21:32:19,489 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer |
|
2022-01-14 21:32:19,490 INFO HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 7 |
|
2022-01-14 21:32:19,490 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: defer |
|
2022-01-14 21:32:19,490 INFO SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 7 |
|
2022-01-14 21:32:19,573 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit |
|
2022-01-14 21:32:19,915 INFO SenderThread:8253 [sender.py:transition_state():387] send defer: 8 |
|
2022-01-14 21:32:19,916 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit |
|
2022-01-14 21:32:19,916 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer |
|
2022-01-14 21:32:19,916 INFO HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 8 |
|
2022-01-14 21:32:19,917 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: defer |
|
2022-01-14 21:32:19,917 INFO SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 8 |
|
2022-01-14 21:32:19,917 INFO SenderThread:8253 [sender.py:transition_state():387] send defer: 9 |
|
2022-01-14 21:32:19,917 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer |
|
2022-01-14 21:32:19,917 INFO HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 9 |
|
2022-01-14 21:32:19,918 DEBUG SenderThread:8253 [sender.py:send():234] send: final |
|
2022-01-14 21:32:19,918 DEBUG SenderThread:8253 [sender.py:send():234] send: footer |
|
2022-01-14 21:32:19,918 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: defer |
|
2022-01-14 21:32:19,918 INFO SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 9 |
|
2022-01-14 21:32:20,017 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit |
|
2022-01-14 21:32:20,018 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit |
|
2022-01-14 21:32:20,018 INFO SenderThread:8253 [file_pusher.py:join():182] waiting for file pusher |
|
2022-01-14 21:32:20,278 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: get_summary |
|
2022-01-14 21:32:20,278 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: sampled_history |
|
2022-01-14 21:32:20,279 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: shutdown |
|
2022-01-14 21:32:20,279 INFO HandlerThread:8253 [handler.py:finish():731] shutting down handler |
|
2022-01-14 21:32:20,918 INFO WriterThread:8253 [datastore.py:close():281] close: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/run-32qdb4k5.wandb |
|
2022-01-14 21:32:21,277 INFO SenderThread:8253 [sender.py:finish():1070] shutting down sender |
|
2022-01-14 21:32:21,277 INFO SenderThread:8253 [file_pusher.py:finish():177] shutting down file pusher |
|
2022-01-14 21:32:21,277 INFO SenderThread:8253 [file_pusher.py:join():182] waiting for file pusher |
|
2022-01-14 21:32:21,279 INFO MainThread:8253 [internal.py:handle_exit():77] Internal process exited |
|
|