| 2021-07-14 23:11:47,821 INFO MainThread:604064 [internal.py:wandb_internal():88] W&B internal server running at pid: 604064, started at: 2021-07-14 23:11:47.821366 | |
| 2021-07-14 23:11:47,823 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: check_version | |
| 2021-07-14 23:11:47,824 INFO WriterThread:604064 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/run-gkn68kcy.wandb | |
| 2021-07-14 23:11:47,824 DEBUG SenderThread:604064 [sender.py:send():179] send: header | |
| 2021-07-14 23:11:47,825 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: check_version | |
| 2021-07-14 23:11:47,862 DEBUG SenderThread:604064 [sender.py:send():179] send: run | |
| 2021-07-14 23:11:48,044 INFO SenderThread:604064 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files | |
| 2021-07-14 23:11:48,044 INFO SenderThread:604064 [sender.py:_start_run_threads():716] run started: gkn68kcy with start time 1626304307 | |
| 2021-07-14 23:11:48,044 DEBUG SenderThread:604064 [sender.py:send():179] send: summary | |
| 2021-07-14 23:11:48,044 INFO SenderThread:604064 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-14 23:11:48,045 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: run_start | |
| 2021-07-14 23:11:49,047 INFO Thread-8 :604064 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/wandb-summary.json | |
| 2021-07-14 23:11:49,155 DEBUG HandlerThread:604064 [meta.py:__init__():39] meta init | |
| 2021-07-14 23:11:49,155 DEBUG HandlerThread:604064 [meta.py:__init__():53] meta init done | |
| 2021-07-14 23:11:49,155 DEBUG HandlerThread:604064 [meta.py:probe():210] probe | |
| 2021-07-14 23:11:49,156 DEBUG HandlerThread:604064 [meta.py:_setup_git():200] setup git | |
| 2021-07-14 23:11:49,187 DEBUG HandlerThread:604064 [meta.py:_setup_git():207] setup git done | |
| 2021-07-14 23:11:49,188 DEBUG HandlerThread:604064 [meta.py:_save_pip():57] save pip | |
| 2021-07-14 23:11:49,188 DEBUG HandlerThread:604064 [meta.py:_save_pip():71] save pip done | |
| 2021-07-14 23:11:49,188 DEBUG HandlerThread:604064 [meta.py:probe():252] probe done | |
| 2021-07-14 23:11:49,192 DEBUG SenderThread:604064 [sender.py:send():179] send: files | |
| 2021-07-14 23:11:49,192 INFO SenderThread:604064 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now | |
| 2021-07-14 23:11:49,199 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:11:49,200 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:11:49,336 DEBUG SenderThread:604064 [sender.py:send():179] send: config | |
| 2021-07-14 23:11:49,336 DEBUG SenderThread:604064 [sender.py:send():179] send: config | |
| 2021-07-14 23:11:49,337 DEBUG SenderThread:604064 [sender.py:send():179] send: config | |
| 2021-07-14 23:11:49,685 INFO Thread-11 :604064 [upload_job.py:push():137] Uploaded file /tmp/tmpnh56hsgtwandb/3jt8aozu-wandb-metadata.json | |
| 2021-07-14 23:11:50,046 INFO Thread-8 :604064 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/output.log | |
| 2021-07-14 23:11:50,046 INFO Thread-8 :604064 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/requirements.txt | |
| 2021-07-14 23:11:50,046 INFO Thread-8 :604064 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/wandb-metadata.json | |
| 2021-07-14 23:12:04,051 INFO Thread-8 :604064 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/output.log | |
| 2021-07-14 23:12:04,338 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:12:04,339 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:12:17,233 DEBUG SenderThread:604064 [sender.py:send():179] send: stats | |
| 2021-07-14 23:12:19,058 INFO Thread-8 :604064 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/config.yaml | |
| 2021-07-14 23:12:19,470 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:12:19,471 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:12:34,607 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:12:34,607 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:12:47,304 DEBUG SenderThread:604064 [sender.py:send():179] send: stats | |
| 2021-07-14 23:12:49,759 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:12:49,760 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:13:04,899 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:13:04,899 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:13:17,375 DEBUG SenderThread:604064 [sender.py:send():179] send: stats | |
| 2021-07-14 23:13:20,032 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:13:20,032 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:13:35,167 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:13:35,168 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:13:47,450 DEBUG SenderThread:604064 [sender.py:send():179] send: stats | |
| 2021-07-14 23:13:50,301 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:13:50,301 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:14:05,433 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:14:05,434 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:14:17,527 DEBUG SenderThread:604064 [sender.py:send():179] send: stats | |
| 2021-07-14 23:14:20,564 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:14:20,564 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:14:35,694 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:14:35,695 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:14:47,605 DEBUG SenderThread:604064 [sender.py:send():179] send: stats | |
| 2021-07-14 23:14:50,827 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:14:50,827 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:15:05,965 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:15:05,965 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:15:17,682 DEBUG SenderThread:604064 [sender.py:send():179] send: stats | |
| 2021-07-14 23:15:21,099 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:15:21,100 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:15:36,236 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:15:36,237 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:15:47,752 DEBUG SenderThread:604064 [sender.py:send():179] send: stats | |
| 2021-07-14 23:15:51,383 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:15:51,384 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:16:06,514 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:16:06,515 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:16:17,832 DEBUG SenderThread:604064 [sender.py:send():179] send: stats | |
| 2021-07-14 23:16:21,647 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:16:21,647 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:16:36,777 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:16:36,778 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:16:47,906 DEBUG SenderThread:604064 [sender.py:send():179] send: stats | |
| 2021-07-14 23:16:51,911 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:16:51,911 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:17:07,045 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:17:07,045 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:17:17,984 DEBUG SenderThread:604064 [sender.py:send():179] send: stats | |
| 2021-07-14 23:17:22,180 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:17:22,180 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:17:37,312 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:17:37,313 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:17:48,061 DEBUG SenderThread:604064 [sender.py:send():179] send: stats | |
| 2021-07-14 23:17:52,447 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:17:52,447 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:18:07,584 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:18:07,584 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:18:18,134 DEBUG SenderThread:604064 [sender.py:send():179] send: stats | |
| 2021-07-14 23:18:22,718 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:18:22,718 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:18:37,852 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:18:37,852 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:18:48,202 DEBUG SenderThread:604064 [sender.py:send():179] send: stats | |
| 2021-07-14 23:18:52,983 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:18:52,984 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:19:08,118 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:19:08,118 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:19:18,280 DEBUG SenderThread:604064 [sender.py:send():179] send: stats | |
| 2021-07-14 23:19:23,249 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:19:23,249 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:19:38,406 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:19:38,407 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:19:48,357 DEBUG SenderThread:604064 [sender.py:send():179] send: stats | |
| 2021-07-14 23:19:53,538 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:19:53,539 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:20:08,668 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:20:08,668 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:20:18,431 DEBUG SenderThread:604064 [sender.py:send():179] send: stats | |
| 2021-07-14 23:20:23,808 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:20:23,809 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:20:38,943 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:20:38,943 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:20:48,507 DEBUG SenderThread:604064 [sender.py:send():179] send: stats | |
| 2021-07-14 23:20:54,079 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:20:54,079 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:21:09,222 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:21:09,223 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:21:18,586 DEBUG SenderThread:604064 [sender.py:send():179] send: stats | |
| 2021-07-14 23:21:24,353 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:21:24,354 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:21:39,483 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:21:39,484 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:21:48,663 DEBUG SenderThread:604064 [sender.py:send():179] send: stats | |
| 2021-07-14 23:21:55,394 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:21:55,394 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:22:10,527 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:22:10,528 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:22:18,741 DEBUG SenderThread:604064 [sender.py:send():179] send: stats | |
| 2021-07-14 23:22:25,659 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:22:25,659 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:22:40,790 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:22:40,790 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:22:48,820 DEBUG SenderThread:604064 [sender.py:send():179] send: stats | |
| 2021-07-14 23:22:55,922 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:22:55,923 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:23:11,058 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:23:11,059 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:23:18,897 DEBUG SenderThread:604064 [sender.py:send():179] send: stats | |
| 2021-07-14 23:23:26,197 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:23:26,198 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:23:41,329 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:23:41,329 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:23:48,974 DEBUG SenderThread:604064 [sender.py:send():179] send: stats | |
| 2021-07-14 23:23:56,463 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:23:56,463 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:24:11,593 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:24:11,593 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:24:19,051 DEBUG SenderThread:604064 [sender.py:send():179] send: stats | |
| 2021-07-14 23:24:26,724 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:24:26,724 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:24:41,858 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:24:41,858 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:24:49,130 DEBUG SenderThread:604064 [sender.py:send():179] send: stats | |
| 2021-07-14 23:24:56,991 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:24:56,991 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:25:12,121 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:25:12,122 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:25:19,207 DEBUG SenderThread:604064 [sender.py:send():179] send: stats | |
| 2021-07-14 23:25:27,253 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:25:27,254 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:25:42,385 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:25:42,386 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:25:49,284 DEBUG SenderThread:604064 [sender.py:send():179] send: stats | |
| 2021-07-14 23:25:57,527 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 23:25:57,527 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 23:26:06,351 INFO Thread-8 :604064 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/output.log | |
| 2021-07-14 23:26:07,081 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 23:26:07,082 DEBUG SenderThread:604064 [sender.py:send():179] send: telemetry | |
| 2021-07-14 23:26:07,082 DEBUG SenderThread:604064 [sender.py:send():179] send: exit | |
| 2021-07-14 23:26:07,082 INFO SenderThread:604064 [sender.py:send_exit():287] handling exit code: 1 | |
| 2021-07-14 23:26:07,083 INFO SenderThread:604064 [sender.py:send_exit():295] send defer | |
| 2021-07-14 23:26:07,084 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 23:26:07,084 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 23:26:07,085 INFO HandlerThread:604064 [handler.py:handle_request_defer():141] handle defer: 0 | |
| 2021-07-14 23:26:07,085 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 23:26:07,085 INFO SenderThread:604064 [sender.py:send_request_defer():304] handle sender defer: 0 | |
| 2021-07-14 23:26:07,085 INFO SenderThread:604064 [sender.py:transition_state():308] send defer: 1 | |
| 2021-07-14 23:26:07,085 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 23:26:07,085 INFO HandlerThread:604064 [handler.py:handle_request_defer():141] handle defer: 1 | |
| 2021-07-14 23:26:07,130 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 23:26:07,130 INFO SenderThread:604064 [sender.py:send_request_defer():304] handle sender defer: 1 | |
| 2021-07-14 23:26:07,130 INFO SenderThread:604064 [sender.py:transition_state():308] send defer: 2 | |
| 2021-07-14 23:26:07,131 DEBUG SenderThread:604064 [sender.py:send():179] send: stats | |
| 2021-07-14 23:26:07,131 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 23:26:07,131 INFO HandlerThread:604064 [handler.py:handle_request_defer():141] handle defer: 2 | |
| 2021-07-14 23:26:07,132 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 23:26:07,132 INFO SenderThread:604064 [sender.py:send_request_defer():304] handle sender defer: 2 | |
| 2021-07-14 23:26:07,132 INFO SenderThread:604064 [sender.py:transition_state():308] send defer: 3 | |
| 2021-07-14 23:26:07,132 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 23:26:07,132 INFO HandlerThread:604064 [handler.py:handle_request_defer():141] handle defer: 3 | |
| 2021-07-14 23:26:07,132 DEBUG SenderThread:604064 [sender.py:send():179] send: summary | |
| 2021-07-14 23:26:07,133 INFO SenderThread:604064 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-14 23:26:07,133 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 23:26:07,133 INFO SenderThread:604064 [sender.py:send_request_defer():304] handle sender defer: 3 | |
| 2021-07-14 23:26:07,133 INFO SenderThread:604064 [sender.py:transition_state():308] send defer: 4 | |
| 2021-07-14 23:26:07,134 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 23:26:07,134 INFO HandlerThread:604064 [handler.py:handle_request_defer():141] handle defer: 4 | |
| 2021-07-14 23:26:07,134 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 23:26:07,134 INFO SenderThread:604064 [sender.py:send_request_defer():304] handle sender defer: 4 | |
| 2021-07-14 23:26:07,228 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 23:26:07,305 INFO SenderThread:604064 [sender.py:transition_state():308] send defer: 5 | |
| 2021-07-14 23:26:07,305 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 23:26:07,306 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 23:26:07,306 INFO HandlerThread:604064 [handler.py:handle_request_defer():141] handle defer: 5 | |
| 2021-07-14 23:26:07,306 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 23:26:07,306 INFO SenderThread:604064 [sender.py:send_request_defer():304] handle sender defer: 5 | |
| 2021-07-14 23:26:07,306 INFO SenderThread:604064 [dir_watcher.py:finish():282] shutting down directory watcher | |
| 2021-07-14 23:26:07,351 INFO Thread-8 :604064 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/config.yaml | |
| 2021-07-14 23:26:07,351 INFO SenderThread:604064 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/output.log | |
| 2021-07-14 23:26:07,352 INFO SenderThread:604064 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/wandb-summary.json | |
| 2021-07-14 23:26:07,352 INFO SenderThread:604064 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files | |
| 2021-07-14 23:26:07,352 INFO SenderThread:604064 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/requirements.txt requirements.txt | |
| 2021-07-14 23:26:07,352 INFO SenderThread:604064 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/output.log output.log | |
| 2021-07-14 23:26:07,352 INFO SenderThread:604064 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/wandb-metadata.json wandb-metadata.json | |
| 2021-07-14 23:26:07,352 INFO SenderThread:604064 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/config.yaml config.yaml | |
| 2021-07-14 23:26:07,356 INFO SenderThread:604064 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/wandb-summary.json wandb-summary.json | |
| 2021-07-14 23:26:07,359 INFO SenderThread:604064 [sender.py:transition_state():308] send defer: 6 | |
| 2021-07-14 23:26:07,360 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 23:26:07,360 INFO HandlerThread:604064 [handler.py:handle_request_defer():141] handle defer: 6 | |
| 2021-07-14 23:26:07,360 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 23:26:07,361 INFO SenderThread:604064 [sender.py:send_request_defer():304] handle sender defer: 6 | |
| 2021-07-14 23:26:07,361 INFO SenderThread:604064 [file_pusher.py:finish():177] shutting down file pusher | |
| 2021-07-14 23:26:07,407 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 23:26:07,408 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 23:26:07,513 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 23:26:07,514 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 23:26:07,616 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 23:26:07,616 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 23:26:07,718 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 23:26:07,718 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 23:26:07,820 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 23:26:07,820 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 23:26:07,840 INFO Thread-14 :604064 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/config.yaml | |
| 2021-07-14 23:26:07,841 INFO Thread-13 :604064 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/output.log | |
| 2021-07-14 23:26:07,874 INFO Thread-12 :604064 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/requirements.txt | |
| 2021-07-14 23:26:07,875 INFO Thread-15 :604064 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/wandb-summary.json | |
| 2021-07-14 23:26:07,922 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 23:26:07,922 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 23:26:08,024 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 23:26:08,024 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 23:26:08,076 INFO Thread-7 :604064 [sender.py:transition_state():308] send defer: 7 | |
| 2021-07-14 23:26:08,077 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 23:26:08,077 INFO HandlerThread:604064 [handler.py:handle_request_defer():141] handle defer: 7 | |
| 2021-07-14 23:26:08,077 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 23:26:08,077 INFO SenderThread:604064 [sender.py:send_request_defer():304] handle sender defer: 7 | |
| 2021-07-14 23:26:08,126 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 23:26:08,360 INFO SenderThread:604064 [sender.py:transition_state():308] send defer: 8 | |
| 2021-07-14 23:26:08,360 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 23:26:08,361 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 23:26:08,361 INFO HandlerThread:604064 [handler.py:handle_request_defer():141] handle defer: 8 | |
| 2021-07-14 23:26:08,361 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 23:26:08,361 INFO SenderThread:604064 [sender.py:send_request_defer():304] handle sender defer: 8 | |
| 2021-07-14 23:26:08,361 INFO SenderThread:604064 [sender.py:transition_state():308] send defer: 9 | |
| 2021-07-14 23:26:08,362 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 23:26:08,362 INFO HandlerThread:604064 [handler.py:handle_request_defer():141] handle defer: 9 | |
| 2021-07-14 23:26:08,362 DEBUG SenderThread:604064 [sender.py:send():179] send: final | |
| 2021-07-14 23:26:08,362 DEBUG SenderThread:604064 [sender.py:send():179] send: footer | |
| 2021-07-14 23:26:08,362 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 23:26:08,362 INFO SenderThread:604064 [sender.py:send_request_defer():304] handle sender defer: 9 | |
| 2021-07-14 23:26:08,462 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 23:26:08,462 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 23:26:08,463 INFO SenderThread:604064 [file_pusher.py:join():182] waiting for file pusher | |
| 2021-07-14 23:26:08,464 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: get_summary | |
| 2021-07-14 23:26:08,465 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: sampled_history | |
| 2021-07-14 23:26:08,465 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: shutdown | |
| 2021-07-14 23:26:08,465 INFO HandlerThread:604064 [handler.py:finish():638] shutting down handler | |
| 2021-07-14 23:26:09,363 INFO WriterThread:604064 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/run-gkn68kcy.wandb | |
| 2021-07-14 23:26:09,463 INFO SenderThread:604064 [sender.py:finish():945] shutting down sender | |
| 2021-07-14 23:26:09,463 INFO SenderThread:604064 [file_pusher.py:finish():177] shutting down file pusher | |
| 2021-07-14 23:26:09,463 INFO SenderThread:604064 [file_pusher.py:join():182] waiting for file pusher | |
| 2021-07-14 23:26:09,466 INFO MainThread:604064 [internal.py:handle_exit():78] Internal process exited | |