dat
Saving weights and logs at step 1252
f291f93
raw
history blame
32.9 kB
2021-07-14 19:25:15,501 INFO MainThread:558812 [internal.py:wandb_internal():88] W&B internal server running at pid: 558812, started at: 2021-07-14 19:25:15.501475
2021-07-14 19:25:15,503 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: check_version
2021-07-14 19:25:15,504 INFO WriterThread:558812 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/run-fdn39gsq.wandb
2021-07-14 19:25:15,505 DEBUG SenderThread:558812 [sender.py:send():179] send: header
2021-07-14 19:25:15,505 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: check_version
2021-07-14 19:25:15,550 DEBUG SenderThread:558812 [sender.py:send():179] send: run
2021-07-14 19:25:15,726 INFO SenderThread:558812 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files
2021-07-14 19:25:15,726 INFO SenderThread:558812 [sender.py:_start_run_threads():716] run started: fdn39gsq with start time 1626290714
2021-07-14 19:25:15,726 DEBUG SenderThread:558812 [sender.py:send():179] send: summary
2021-07-14 19:25:15,726 INFO SenderThread:558812 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 19:25:15,727 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: run_start
2021-07-14 19:25:16,728 INFO Thread-8 :558812 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/wandb-summary.json
2021-07-14 19:25:16,843 DEBUG HandlerThread:558812 [meta.py:__init__():39] meta init
2021-07-14 19:25:16,843 DEBUG HandlerThread:558812 [meta.py:__init__():53] meta init done
2021-07-14 19:25:16,843 DEBUG HandlerThread:558812 [meta.py:probe():210] probe
2021-07-14 19:25:16,844 DEBUG HandlerThread:558812 [meta.py:_setup_git():200] setup git
2021-07-14 19:25:16,874 DEBUG HandlerThread:558812 [meta.py:_setup_git():207] setup git done
2021-07-14 19:25:16,874 DEBUG HandlerThread:558812 [meta.py:_save_pip():57] save pip
2021-07-14 19:25:16,875 DEBUG HandlerThread:558812 [meta.py:_save_pip():71] save pip done
2021-07-14 19:25:16,875 DEBUG HandlerThread:558812 [meta.py:probe():252] probe done
2021-07-14 19:25:16,878 DEBUG SenderThread:558812 [sender.py:send():179] send: files
2021-07-14 19:25:16,878 INFO SenderThread:558812 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-14 19:25:16,883 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:25:16,884 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:25:17,012 DEBUG SenderThread:558812 [sender.py:send():179] send: config
2021-07-14 19:25:17,013 DEBUG SenderThread:558812 [sender.py:send():179] send: config
2021-07-14 19:25:17,013 DEBUG SenderThread:558812 [sender.py:send():179] send: config
2021-07-14 19:25:17,401 INFO Thread-11 :558812 [upload_job.py:push():137] Uploaded file /tmp/tmp2t3dxezvwandb/1xrc7l9y-wandb-metadata.json
2021-07-14 19:25:17,727 INFO Thread-8 :558812 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/requirements.txt
2021-07-14 19:25:17,727 INFO Thread-8 :558812 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/wandb-metadata.json
2021-07-14 19:25:17,727 INFO Thread-8 :558812 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/output.log
2021-07-14 19:25:32,014 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:25:32,014 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:25:35,734 INFO Thread-8 :558812 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/output.log
2021-07-14 19:25:44,927 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:25:46,738 INFO Thread-8 :558812 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/config.yaml
2021-07-14 19:25:47,145 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:25:47,146 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:26:02,275 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:26:02,276 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:26:14,992 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:26:17,407 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:26:17,407 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:26:32,541 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:26:32,541 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:26:45,071 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:26:47,675 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:26:47,676 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:27:02,817 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:27:02,817 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:27:15,150 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:27:17,954 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:27:17,955 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:27:33,130 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:27:33,130 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:27:45,227 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:27:48,264 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:27:48,264 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:28:03,396 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:28:03,397 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:28:15,303 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:28:18,531 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:28:18,532 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:28:33,662 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:28:33,662 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:28:45,377 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:28:48,794 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:28:48,795 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:29:03,929 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:29:03,929 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:29:15,441 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:29:19,068 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:29:19,068 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:29:34,207 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:29:34,207 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:29:45,507 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:29:49,342 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:29:49,342 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:30:04,484 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:30:04,485 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:30:15,573 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:30:19,618 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:30:19,619 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:30:34,752 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:30:34,753 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:30:45,637 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:30:49,884 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:30:49,885 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:31:05,016 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:31:05,017 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:31:15,702 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:31:20,150 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:31:20,150 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:31:35,285 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:31:35,285 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:31:45,768 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:31:50,417 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:31:50,418 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:32:05,556 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:32:05,557 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:32:15,834 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:32:20,694 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:32:20,695 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:32:35,831 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:32:35,831 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:32:45,903 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:32:50,966 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:32:50,967 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:33:06,100 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:33:06,100 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:33:15,972 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:33:21,242 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:33:21,242 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:33:36,374 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:33:36,374 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:33:46,047 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:33:51,520 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:33:51,520 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:34:06,655 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:34:06,655 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:34:16,121 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:34:21,789 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:34:21,790 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:34:36,924 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:34:36,925 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:34:46,200 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:34:52,186 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:34:52,187 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:35:07,323 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:35:07,323 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:35:16,277 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:35:22,455 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:35:22,455 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:35:37,590 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:35:37,590 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:35:46,355 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:35:52,727 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:35:52,727 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:36:07,858 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:36:07,858 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:36:16,432 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:36:22,989 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:36:22,989 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:36:38,123 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:36:38,124 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:36:46,506 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:36:53,255 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:36:53,255 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:37:08,418 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:37:08,419 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:37:16,581 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:37:23,567 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:37:23,567 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:37:38,702 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:37:38,702 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:37:46,660 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:37:53,835 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:37:53,835 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:38:08,972 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:38:08,972 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:38:16,734 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:38:24,103 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:38:24,103 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:38:39,236 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:38:39,236 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:38:46,811 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:38:54,369 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:38:54,369 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:39:09,502 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:39:09,503 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:39:16,888 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:39:24,635 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:39:24,635 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:39:39,766 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:39:39,766 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:39:46,967 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:39:54,898 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:39:54,899 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:40:10,033 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:40:10,033 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:40:17,045 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:40:25,168 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:40:25,168 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:40:40,300 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 19:40:40,300 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: stop_status
2021-07-14 19:40:47,122 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:40:49,097 INFO Thread-8 :558812 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/output.log
2021-07-14 19:40:50,137 DEBUG SenderThread:558812 [sender.py:send():179] send: telemetry
2021-07-14 19:40:50,137 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:40:50,137 DEBUG SenderThread:558812 [sender.py:send():179] send: exit
2021-07-14 19:40:50,137 INFO SenderThread:558812 [sender.py:send_exit():287] handling exit code: 1
2021-07-14 19:40:50,138 INFO SenderThread:558812 [sender.py:send_exit():295] send defer
2021-07-14 19:40:50,138 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:40:50,138 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:40:50,139 INFO HandlerThread:558812 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-14 19:40:50,139 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: defer
2021-07-14 19:40:50,139 INFO SenderThread:558812 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-14 19:40:50,139 INFO SenderThread:558812 [sender.py:transition_state():308] send defer: 1
2021-07-14 19:40:50,139 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:40:50,139 INFO HandlerThread:558812 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-14 19:40:50,231 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: defer
2021-07-14 19:40:50,231 INFO SenderThread:558812 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-14 19:40:50,231 INFO SenderThread:558812 [sender.py:transition_state():308] send defer: 2
2021-07-14 19:40:50,231 DEBUG SenderThread:558812 [sender.py:send():179] send: stats
2021-07-14 19:40:50,232 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:40:50,232 INFO HandlerThread:558812 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-14 19:40:50,232 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: defer
2021-07-14 19:40:50,233 INFO SenderThread:558812 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-14 19:40:50,233 INFO SenderThread:558812 [sender.py:transition_state():308] send defer: 3
2021-07-14 19:40:50,233 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:40:50,233 INFO HandlerThread:558812 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-14 19:40:50,233 DEBUG SenderThread:558812 [sender.py:send():179] send: summary
2021-07-14 19:40:50,234 INFO SenderThread:558812 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 19:40:50,234 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: defer
2021-07-14 19:40:50,234 INFO SenderThread:558812 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-14 19:40:50,234 INFO SenderThread:558812 [sender.py:transition_state():308] send defer: 4
2021-07-14 19:40:50,234 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:40:50,234 INFO HandlerThread:558812 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-14 19:40:50,234 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: defer
2021-07-14 19:40:50,235 INFO SenderThread:558812 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-14 19:40:50,240 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:40:50,432 INFO SenderThread:558812 [sender.py:transition_state():308] send defer: 5
2021-07-14 19:40:50,432 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:40:50,433 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:40:50,433 INFO HandlerThread:558812 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-14 19:40:50,433 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: defer
2021-07-14 19:40:50,433 INFO SenderThread:558812 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-14 19:40:50,433 INFO SenderThread:558812 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-14 19:40:50,534 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:40:51,098 INFO Thread-8 :558812 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/wandb-summary.json
2021-07-14 19:40:51,099 INFO SenderThread:558812 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/output.log
2021-07-14 19:40:51,099 INFO SenderThread:558812 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/config.yaml
2021-07-14 19:40:51,099 INFO SenderThread:558812 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files
2021-07-14 19:40:51,099 INFO SenderThread:558812 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/requirements.txt requirements.txt
2021-07-14 19:40:51,100 INFO SenderThread:558812 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/output.log output.log
2021-07-14 19:40:51,100 INFO SenderThread:558812 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/wandb-metadata.json wandb-metadata.json
2021-07-14 19:40:51,100 INFO SenderThread:558812 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/config.yaml config.yaml
2021-07-14 19:40:51,101 INFO SenderThread:558812 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/wandb-summary.json wandb-summary.json
2021-07-14 19:40:51,101 INFO SenderThread:558812 [sender.py:transition_state():308] send defer: 6
2021-07-14 19:40:51,104 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:40:51,110 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:40:51,110 INFO HandlerThread:558812 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-14 19:40:51,113 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: defer
2021-07-14 19:40:51,114 INFO SenderThread:558812 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-14 19:40:51,114 INFO SenderThread:558812 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 19:40:51,206 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:40:51,206 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:40:51,309 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:40:51,309 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:40:51,411 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:40:51,411 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:40:51,512 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:40:51,513 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:40:51,533 INFO Thread-14 :558812 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/config.yaml
2021-07-14 19:40:51,534 INFO Thread-12 :558812 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/requirements.txt
2021-07-14 19:40:51,535 INFO Thread-15 :558812 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/wandb-summary.json
2021-07-14 19:40:51,537 INFO Thread-13 :558812 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/files/output.log
2021-07-14 19:40:51,614 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:40:51,615 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:40:51,716 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:40:51,717 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:40:51,737 INFO Thread-7 :558812 [sender.py:transition_state():308] send defer: 7
2021-07-14 19:40:51,738 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:40:51,738 INFO HandlerThread:558812 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-14 19:40:51,738 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: defer
2021-07-14 19:40:51,738 INFO SenderThread:558812 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-14 19:40:51,818 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:40:52,030 INFO SenderThread:558812 [sender.py:transition_state():308] send defer: 8
2021-07-14 19:40:52,030 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:40:52,031 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:40:52,031 INFO HandlerThread:558812 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-14 19:40:52,031 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: defer
2021-07-14 19:40:52,031 INFO SenderThread:558812 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-14 19:40:52,031 INFO SenderThread:558812 [sender.py:transition_state():308] send defer: 9
2021-07-14 19:40:52,032 DEBUG SenderThread:558812 [sender.py:send():179] send: final
2021-07-14 19:40:52,032 DEBUG SenderThread:558812 [sender.py:send():179] send: footer
2021-07-14 19:40:52,032 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: defer
2021-07-14 19:40:52,032 INFO HandlerThread:558812 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-14 19:40:52,032 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: defer
2021-07-14 19:40:52,032 INFO SenderThread:558812 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-14 19:40:52,132 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 19:40:52,133 DEBUG SenderThread:558812 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 19:40:52,133 INFO SenderThread:558812 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 19:40:52,134 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: get_summary
2021-07-14 19:40:52,135 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-14 19:40:52,135 DEBUG HandlerThread:558812 [handler.py:handle_request():124] handle_request: shutdown
2021-07-14 19:40:52,135 INFO HandlerThread:558812 [handler.py:finish():638] shutting down handler
2021-07-14 19:40:53,033 INFO WriterThread:558812 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_192514-fdn39gsq/run-fdn39gsq.wandb
2021-07-14 19:40:53,133 INFO SenderThread:558812 [sender.py:finish():945] shutting down sender
2021-07-14 19:40:53,133 INFO SenderThread:558812 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 19:40:53,133 INFO SenderThread:558812 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 19:40:53,136 INFO MainThread:558812 [internal.py:handle_exit():78] Internal process exited