| 2021-07-15 02:15:59,792 INFO MainThread:644701 [internal.py:wandb_internal():88] W&B internal server running at pid: 644701, started at: 2021-07-15 02:15:59.792106 | |
| 2021-07-15 02:15:59,795 DEBUG SenderThread:644701 [sender.py:send():179] send: header | |
| 2021-07-15 02:15:59,795 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: check_version | |
| 2021-07-15 02:15:59,795 INFO WriterThread:644701 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210715_021559-38yj0n5v/run-38yj0n5v.wandb | |
| 2021-07-15 02:15:59,796 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: check_version | |
| 2021-07-15 02:15:59,833 DEBUG SenderThread:644701 [sender.py:send():179] send: run | |
| 2021-07-15 02:16:00,007 INFO SenderThread:644701 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210715_021559-38yj0n5v/files | |
| 2021-07-15 02:16:00,007 INFO SenderThread:644701 [sender.py:_start_run_threads():716] run started: 38yj0n5v with start time 1626315359 | |
| 2021-07-15 02:16:00,007 DEBUG SenderThread:644701 [sender.py:send():179] send: summary | |
| 2021-07-15 02:16:00,008 INFO SenderThread:644701 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-15 02:16:00,008 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: run_start | |
| 2021-07-15 02:16:01,010 INFO Thread-8 :644701 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_021559-38yj0n5v/files/wandb-summary.json | |
| 2021-07-15 02:16:01,117 DEBUG HandlerThread:644701 [meta.py:__init__():39] meta init | |
| 2021-07-15 02:16:01,117 DEBUG HandlerThread:644701 [meta.py:__init__():53] meta init done | |
| 2021-07-15 02:16:01,117 DEBUG HandlerThread:644701 [meta.py:probe():210] probe | |
| 2021-07-15 02:16:01,118 DEBUG HandlerThread:644701 [meta.py:_setup_git():200] setup git | |
| 2021-07-15 02:16:01,149 DEBUG HandlerThread:644701 [meta.py:_setup_git():207] setup git done | |
| 2021-07-15 02:16:01,150 DEBUG HandlerThread:644701 [meta.py:_save_pip():57] save pip | |
| 2021-07-15 02:16:01,150 DEBUG HandlerThread:644701 [meta.py:_save_pip():71] save pip done | |
| 2021-07-15 02:16:01,150 DEBUG HandlerThread:644701 [meta.py:probe():252] probe done | |
| 2021-07-15 02:16:01,154 DEBUG SenderThread:644701 [sender.py:send():179] send: files | |
| 2021-07-15 02:16:01,154 INFO SenderThread:644701 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now | |
| 2021-07-15 02:16:01,160 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:16:01,161 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:16:01,290 DEBUG SenderThread:644701 [sender.py:send():179] send: config | |
| 2021-07-15 02:16:01,291 DEBUG SenderThread:644701 [sender.py:send():179] send: config | |
| 2021-07-15 02:16:01,291 DEBUG SenderThread:644701 [sender.py:send():179] send: config | |
| 2021-07-15 02:16:01,718 INFO Thread-11 :644701 [upload_job.py:push():137] Uploaded file /tmp/tmp__ipqk3vwandb/1qcixa2k-wandb-metadata.json | |
| 2021-07-15 02:16:02,009 INFO Thread-8 :644701 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_021559-38yj0n5v/files/output.log | |
| 2021-07-15 02:16:02,009 INFO Thread-8 :644701 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_021559-38yj0n5v/files/requirements.txt | |
| 2021-07-15 02:16:02,009 INFO Thread-8 :644701 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_021559-38yj0n5v/files/wandb-metadata.json | |
| 2021-07-15 02:16:16,015 INFO Thread-8 :644701 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_021559-38yj0n5v/files/output.log | |
| 2021-07-15 02:16:16,292 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:16:16,293 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:16:29,202 DEBUG SenderThread:644701 [sender.py:send():179] send: stats | |
| 2021-07-15 02:16:31,021 INFO Thread-8 :644701 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_021559-38yj0n5v/files/config.yaml | |
| 2021-07-15 02:16:31,425 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:16:31,425 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:16:46,555 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:16:46,555 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:16:59,284 DEBUG SenderThread:644701 [sender.py:send():179] send: stats | |
| 2021-07-15 02:17:01,687 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:17:01,687 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:17:16,819 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:17:16,820 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:17:29,359 DEBUG SenderThread:644701 [sender.py:send():179] send: stats | |
| 2021-07-15 02:17:31,951 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:17:31,951 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:17:47,083 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:17:47,083 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:17:59,439 DEBUG SenderThread:644701 [sender.py:send():179] send: stats | |
| 2021-07-15 02:18:02,215 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:18:02,215 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:18:17,355 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:18:17,356 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:18:29,519 DEBUG SenderThread:644701 [sender.py:send():179] send: stats | |
| 2021-07-15 02:18:32,491 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:18:32,492 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:18:47,624 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:18:47,624 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:18:59,595 DEBUG SenderThread:644701 [sender.py:send():179] send: stats | |
| 2021-07-15 02:19:02,759 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:19:02,759 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:19:17,890 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:19:17,890 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:19:29,672 DEBUG SenderThread:644701 [sender.py:send():179] send: stats | |
| 2021-07-15 02:19:33,021 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:19:33,022 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:19:48,153 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:19:48,154 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:19:59,751 DEBUG SenderThread:644701 [sender.py:send():179] send: stats | |
| 2021-07-15 02:20:03,293 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:20:03,294 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:20:18,425 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:20:18,426 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:20:29,828 DEBUG SenderThread:644701 [sender.py:send():179] send: stats | |
| 2021-07-15 02:20:33,560 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:20:33,560 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:20:48,726 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:20:48,726 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:20:59,906 DEBUG SenderThread:644701 [sender.py:send():179] send: stats | |
| 2021-07-15 02:21:03,857 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:21:03,858 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:21:18,990 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:21:18,991 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:21:29,980 DEBUG SenderThread:644701 [sender.py:send():179] send: stats | |
| 2021-07-15 02:21:34,126 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:21:34,126 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:21:49,258 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:21:49,258 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:22:00,053 DEBUG SenderThread:644701 [sender.py:send():179] send: stats | |
| 2021-07-15 02:22:04,390 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:22:04,391 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:22:19,527 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:22:19,527 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:22:30,130 DEBUG SenderThread:644701 [sender.py:send():179] send: stats | |
| 2021-07-15 02:22:34,658 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:22:34,658 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:22:49,790 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:22:49,790 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:23:00,206 DEBUG SenderThread:644701 [sender.py:send():179] send: stats | |
| 2021-07-15 02:23:04,919 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:23:04,920 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:23:20,062 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:23:20,063 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:23:30,267 DEBUG SenderThread:644701 [sender.py:send():179] send: stats | |
| 2021-07-15 02:23:35,199 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:23:35,199 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:23:50,332 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:23:50,332 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:24:00,346 DEBUG SenderThread:644701 [sender.py:send():179] send: stats | |
| 2021-07-15 02:24:05,465 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:24:05,466 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:24:20,598 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:24:20,598 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:24:30,424 DEBUG SenderThread:644701 [sender.py:send():179] send: stats | |
| 2021-07-15 02:24:35,751 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:24:35,751 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:24:50,888 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:24:50,888 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:25:00,500 DEBUG SenderThread:644701 [sender.py:send():179] send: stats | |
| 2021-07-15 02:25:06,021 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:25:06,022 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:25:21,156 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:25:21,157 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:25:30,575 DEBUG SenderThread:644701 [sender.py:send():179] send: stats | |
| 2021-07-15 02:25:36,290 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:25:36,291 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:25:51,426 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:25:51,426 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:26:00,654 DEBUG SenderThread:644701 [sender.py:send():179] send: stats | |
| 2021-07-15 02:26:06,562 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:26:06,562 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:26:21,692 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:26:21,693 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:26:30,729 DEBUG SenderThread:644701 [sender.py:send():179] send: stats | |
| 2021-07-15 02:26:36,825 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:26:36,825 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:26:51,959 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:26:51,959 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:27:00,798 DEBUG SenderThread:644701 [sender.py:send():179] send: stats | |
| 2021-07-15 02:27:07,091 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:27:07,091 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:27:22,224 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:27:22,224 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:27:30,870 DEBUG SenderThread:644701 [sender.py:send():179] send: stats | |
| 2021-07-15 02:27:37,360 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:27:37,360 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:27:52,491 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:27:52,491 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:28:00,938 DEBUG SenderThread:644701 [sender.py:send():179] send: stats | |
| 2021-07-15 02:28:07,622 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:28:07,622 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:28:22,754 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:28:22,755 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:28:31,010 DEBUG SenderThread:644701 [sender.py:send():179] send: stats | |
| 2021-07-15 02:28:37,888 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:28:37,888 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:28:53,020 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:28:53,021 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:29:01,085 DEBUG SenderThread:644701 [sender.py:send():179] send: stats | |
| 2021-07-15 02:29:08,157 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:29:08,157 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:29:23,289 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:29:23,289 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:29:31,158 DEBUG SenderThread:644701 [sender.py:send():179] send: stats | |
| 2021-07-15 02:29:38,420 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:29:38,420 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:29:53,553 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:29:53,553 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:29:58,389 INFO Thread-8 :644701 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_021559-38yj0n5v/files/output.log | |
| 2021-07-15 02:30:01,235 DEBUG SenderThread:644701 [sender.py:send():179] send: stats | |
| 2021-07-15 02:30:08,702 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:30:08,702 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:30:23,843 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:30:23,843 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:30:31,315 DEBUG SenderThread:644701 [sender.py:send():179] send: stats | |
| 2021-07-15 02:30:38,973 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:30:38,973 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:30:54,105 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:30:54,106 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:31:01,399 DEBUG SenderThread:644701 [sender.py:send():179] send: stats | |
| 2021-07-15 02:31:09,240 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:31:09,240 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:31:24,379 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:31:24,379 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:31:31,480 DEBUG SenderThread:644701 [sender.py:send():179] send: stats | |
| 2021-07-15 02:31:39,512 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:31:39,512 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:31:54,644 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-15 02:31:54,644 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-15 02:32:01,553 DEBUG SenderThread:644701 [sender.py:send():179] send: stats | |
| 2021-07-15 02:32:04,443 INFO Thread-8 :644701 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_021559-38yj0n5v/files/output.log | |
| 2021-07-15 02:32:04,474 DEBUG SenderThread:644701 [sender.py:send():179] send: telemetry | |
| 2021-07-15 02:32:04,474 DEBUG SenderThread:644701 [sender.py:send():179] send: exit | |
| 2021-07-15 02:32:04,474 INFO SenderThread:644701 [sender.py:send_exit():287] handling exit code: 1 | |
| 2021-07-15 02:32:04,476 INFO SenderThread:644701 [sender.py:send_exit():295] send defer | |
| 2021-07-15 02:32:04,476 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-15 02:32:04,477 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-15 02:32:04,477 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-15 02:32:04,477 INFO HandlerThread:644701 [handler.py:handle_request_defer():141] handle defer: 0 | |
| 2021-07-15 02:32:04,478 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: defer | |
| 2021-07-15 02:32:04,478 INFO SenderThread:644701 [sender.py:send_request_defer():304] handle sender defer: 0 | |
| 2021-07-15 02:32:04,478 INFO SenderThread:644701 [sender.py:transition_state():308] send defer: 1 | |
| 2021-07-15 02:32:04,478 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-15 02:32:04,478 INFO HandlerThread:644701 [handler.py:handle_request_defer():141] handle defer: 1 | |
| 2021-07-15 02:32:04,561 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: defer | |
| 2021-07-15 02:32:04,561 INFO SenderThread:644701 [sender.py:send_request_defer():304] handle sender defer: 1 | |
| 2021-07-15 02:32:04,562 INFO SenderThread:644701 [sender.py:transition_state():308] send defer: 2 | |
| 2021-07-15 02:32:04,562 DEBUG SenderThread:644701 [sender.py:send():179] send: stats | |
| 2021-07-15 02:32:04,562 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-15 02:32:04,562 INFO HandlerThread:644701 [handler.py:handle_request_defer():141] handle defer: 2 | |
| 2021-07-15 02:32:04,562 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: defer | |
| 2021-07-15 02:32:04,563 INFO SenderThread:644701 [sender.py:send_request_defer():304] handle sender defer: 2 | |
| 2021-07-15 02:32:04,563 INFO SenderThread:644701 [sender.py:transition_state():308] send defer: 3 | |
| 2021-07-15 02:32:04,563 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-15 02:32:04,563 INFO HandlerThread:644701 [handler.py:handle_request_defer():141] handle defer: 3 | |
| 2021-07-15 02:32:04,563 DEBUG SenderThread:644701 [sender.py:send():179] send: summary | |
| 2021-07-15 02:32:04,564 INFO SenderThread:644701 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-15 02:32:04,564 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: defer | |
| 2021-07-15 02:32:04,564 INFO SenderThread:644701 [sender.py:send_request_defer():304] handle sender defer: 3 | |
| 2021-07-15 02:32:04,564 INFO SenderThread:644701 [sender.py:transition_state():308] send defer: 4 | |
| 2021-07-15 02:32:04,565 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-15 02:32:04,565 INFO HandlerThread:644701 [handler.py:handle_request_defer():141] handle defer: 4 | |
| 2021-07-15 02:32:04,565 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: defer | |
| 2021-07-15 02:32:04,565 INFO SenderThread:644701 [sender.py:send_request_defer():304] handle sender defer: 4 | |
| 2021-07-15 02:32:04,580 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-15 02:32:04,749 INFO SenderThread:644701 [sender.py:transition_state():308] send defer: 5 | |
| 2021-07-15 02:32:04,749 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-15 02:32:04,749 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-15 02:32:04,750 INFO HandlerThread:644701 [handler.py:handle_request_defer():141] handle defer: 5 | |
| 2021-07-15 02:32:04,750 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: defer | |
| 2021-07-15 02:32:04,750 INFO SenderThread:644701 [sender.py:send_request_defer():304] handle sender defer: 5 | |
| 2021-07-15 02:32:04,750 INFO SenderThread:644701 [dir_watcher.py:finish():282] shutting down directory watcher | |
| 2021-07-15 02:32:04,851 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-15 02:32:05,444 INFO Thread-8 :644701 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_021559-38yj0n5v/files/output.log | |
| 2021-07-15 02:32:05,445 INFO SenderThread:644701 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_021559-38yj0n5v/files/wandb-summary.json | |
| 2021-07-15 02:32:05,445 INFO SenderThread:644701 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_021559-38yj0n5v/files/config.yaml | |
| 2021-07-15 02:32:05,445 INFO SenderThread:644701 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210715_021559-38yj0n5v/files | |
| 2021-07-15 02:32:05,445 INFO SenderThread:644701 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_021559-38yj0n5v/files/requirements.txt requirements.txt | |
| 2021-07-15 02:32:05,446 INFO SenderThread:644701 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_021559-38yj0n5v/files/output.log output.log | |
| 2021-07-15 02:32:05,446 INFO SenderThread:644701 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_021559-38yj0n5v/files/wandb-metadata.json wandb-metadata.json | |
| 2021-07-15 02:32:05,446 INFO SenderThread:644701 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_021559-38yj0n5v/files/config.yaml config.yaml | |
| 2021-07-15 02:32:05,450 INFO SenderThread:644701 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_021559-38yj0n5v/files/wandb-summary.json wandb-summary.json | |
| 2021-07-15 02:32:05,453 INFO SenderThread:644701 [sender.py:transition_state():308] send defer: 6 | |
| 2021-07-15 02:32:05,453 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-15 02:32:05,455 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-15 02:32:05,457 INFO HandlerThread:644701 [handler.py:handle_request_defer():141] handle defer: 6 | |
| 2021-07-15 02:32:05,458 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: defer | |
| 2021-07-15 02:32:05,461 INFO SenderThread:644701 [sender.py:send_request_defer():304] handle sender defer: 6 | |
| 2021-07-15 02:32:05,461 INFO SenderThread:644701 [file_pusher.py:finish():177] shutting down file pusher | |
| 2021-07-15 02:32:05,556 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-15 02:32:05,556 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-15 02:32:05,658 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-15 02:32:05,659 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-15 02:32:05,761 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-15 02:32:05,761 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-15 02:32:05,863 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-15 02:32:05,863 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-15 02:32:05,888 INFO Thread-15 :644701 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_021559-38yj0n5v/files/wandb-summary.json | |
| 2021-07-15 02:32:05,892 INFO Thread-13 :644701 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_021559-38yj0n5v/files/output.log | |
| 2021-07-15 02:32:05,894 INFO Thread-14 :644701 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_021559-38yj0n5v/files/config.yaml | |
| 2021-07-15 02:32:05,895 INFO Thread-12 :644701 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_021559-38yj0n5v/files/requirements.txt | |
| 2021-07-15 02:32:05,965 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-15 02:32:05,965 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-15 02:32:06,067 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-15 02:32:06,067 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-15 02:32:06,096 INFO Thread-7 :644701 [sender.py:transition_state():308] send defer: 7 | |
| 2021-07-15 02:32:06,096 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-15 02:32:06,096 INFO HandlerThread:644701 [handler.py:handle_request_defer():141] handle defer: 7 | |
| 2021-07-15 02:32:06,097 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: defer | |
| 2021-07-15 02:32:06,097 INFO SenderThread:644701 [sender.py:send_request_defer():304] handle sender defer: 7 | |
| 2021-07-15 02:32:06,169 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-15 02:32:06,370 INFO SenderThread:644701 [sender.py:transition_state():308] send defer: 8 | |
| 2021-07-15 02:32:06,370 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-15 02:32:06,371 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-15 02:32:06,371 INFO HandlerThread:644701 [handler.py:handle_request_defer():141] handle defer: 8 | |
| 2021-07-15 02:32:06,371 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: defer | |
| 2021-07-15 02:32:06,371 INFO SenderThread:644701 [sender.py:send_request_defer():304] handle sender defer: 8 | |
| 2021-07-15 02:32:06,371 INFO SenderThread:644701 [sender.py:transition_state():308] send defer: 9 | |
| 2021-07-15 02:32:06,372 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-15 02:32:06,372 INFO HandlerThread:644701 [handler.py:handle_request_defer():141] handle defer: 9 | |
| 2021-07-15 02:32:06,372 DEBUG SenderThread:644701 [sender.py:send():179] send: final | |
| 2021-07-15 02:32:06,372 DEBUG SenderThread:644701 [sender.py:send():179] send: footer | |
| 2021-07-15 02:32:06,372 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: defer | |
| 2021-07-15 02:32:06,372 INFO SenderThread:644701 [sender.py:send_request_defer():304] handle sender defer: 9 | |
| 2021-07-15 02:32:06,472 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-15 02:32:06,472 DEBUG SenderThread:644701 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-15 02:32:06,473 INFO SenderThread:644701 [file_pusher.py:join():182] waiting for file pusher | |
| 2021-07-15 02:32:06,474 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: get_summary | |
| 2021-07-15 02:32:06,475 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: sampled_history | |
| 2021-07-15 02:32:06,475 DEBUG HandlerThread:644701 [handler.py:handle_request():124] handle_request: shutdown | |
| 2021-07-15 02:32:06,475 INFO HandlerThread:644701 [handler.py:finish():638] shutting down handler | |
| 2021-07-15 02:32:07,372 INFO WriterThread:644701 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210715_021559-38yj0n5v/run-38yj0n5v.wandb | |
| 2021-07-15 02:32:07,473 INFO SenderThread:644701 [sender.py:finish():945] shutting down sender | |
| 2021-07-15 02:32:07,473 INFO SenderThread:644701 [file_pusher.py:finish():177] shutting down file pusher | |
| 2021-07-15 02:32:07,473 INFO SenderThread:644701 [file_pusher.py:join():182] waiting for file pusher | |
| 2021-07-15 02:32:07,477 INFO MainThread:644701 [internal.py:handle_exit():78] Internal process exited | |