| 2021-07-14 22:19:21,314 INFO MainThread:593294 [internal.py:wandb_internal():88] W&B internal server running at pid: 593294, started at: 2021-07-14 22:19:21.314432 | |
| 2021-07-14 22:19:21,317 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: check_version | |
| 2021-07-14 22:19:21,317 INFO WriterThread:593294 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/run-s091gfok.wandb | |
| 2021-07-14 22:19:21,318 DEBUG SenderThread:593294 [sender.py:send():179] send: header | |
| 2021-07-14 22:19:21,318 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: check_version | |
| 2021-07-14 22:19:21,357 DEBUG SenderThread:593294 [sender.py:send():179] send: run | |
| 2021-07-14 22:19:21,536 INFO SenderThread:593294 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files | |
| 2021-07-14 22:19:21,536 INFO SenderThread:593294 [sender.py:_start_run_threads():716] run started: s091gfok with start time 1626301160 | |
| 2021-07-14 22:19:21,536 DEBUG SenderThread:593294 [sender.py:send():179] send: summary | |
| 2021-07-14 22:19:21,537 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: run_start | |
| 2021-07-14 22:19:21,537 INFO SenderThread:593294 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-14 22:19:22,539 INFO Thread-8 :593294 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/wandb-summary.json | |
| 2021-07-14 22:19:22,632 DEBUG HandlerThread:593294 [meta.py:__init__():39] meta init | |
| 2021-07-14 22:19:22,632 DEBUG HandlerThread:593294 [meta.py:__init__():53] meta init done | |
| 2021-07-14 22:19:22,632 DEBUG HandlerThread:593294 [meta.py:probe():210] probe | |
| 2021-07-14 22:19:22,634 DEBUG HandlerThread:593294 [meta.py:_setup_git():200] setup git | |
| 2021-07-14 22:19:22,663 DEBUG HandlerThread:593294 [meta.py:_setup_git():207] setup git done | |
| 2021-07-14 22:19:22,663 DEBUG HandlerThread:593294 [meta.py:_save_pip():57] save pip | |
| 2021-07-14 22:19:22,664 DEBUG HandlerThread:593294 [meta.py:_save_pip():71] save pip done | |
| 2021-07-14 22:19:22,664 DEBUG HandlerThread:593294 [meta.py:probe():252] probe done | |
| 2021-07-14 22:19:22,667 DEBUG SenderThread:593294 [sender.py:send():179] send: files | |
| 2021-07-14 22:19:22,667 INFO SenderThread:593294 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now | |
| 2021-07-14 22:19:22,674 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:19:22,674 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:19:22,802 DEBUG SenderThread:593294 [sender.py:send():179] send: config | |
| 2021-07-14 22:19:22,803 DEBUG SenderThread:593294 [sender.py:send():179] send: config | |
| 2021-07-14 22:19:22,803 DEBUG SenderThread:593294 [sender.py:send():179] send: config | |
| 2021-07-14 22:19:23,119 INFO Thread-11 :593294 [upload_job.py:push():137] Uploaded file /tmp/tmpn0n6xzzmwandb/2vhpic31-wandb-metadata.json | |
| 2021-07-14 22:19:23,537 INFO Thread-8 :593294 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/wandb-metadata.json | |
| 2021-07-14 22:19:23,537 INFO Thread-8 :593294 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/requirements.txt | |
| 2021-07-14 22:19:23,538 INFO Thread-8 :593294 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/output.log | |
| 2021-07-14 22:19:37,543 INFO Thread-8 :593294 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/output.log | |
| 2021-07-14 22:19:37,804 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:19:37,804 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:19:39,545 INFO Thread-8 :593294 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/output.log | |
| 2021-07-14 22:19:50,715 DEBUG SenderThread:593294 [sender.py:send():179] send: stats | |
| 2021-07-14 22:19:52,550 INFO Thread-8 :593294 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/config.yaml | |
| 2021-07-14 22:19:52,936 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:19:52,936 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:20:08,079 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:20:08,080 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:20:20,789 DEBUG SenderThread:593294 [sender.py:send():179] send: stats | |
| 2021-07-14 22:20:23,215 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:20:23,215 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:20:38,362 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:20:38,363 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:20:50,861 DEBUG SenderThread:593294 [sender.py:send():179] send: stats | |
| 2021-07-14 22:20:53,496 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:20:53,496 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:21:08,625 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:21:08,625 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:21:20,932 DEBUG SenderThread:593294 [sender.py:send():179] send: stats | |
| 2021-07-14 22:21:23,756 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:21:23,757 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:21:38,885 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:21:38,886 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:21:50,997 DEBUG SenderThread:593294 [sender.py:send():179] send: stats | |
| 2021-07-14 22:21:54,016 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:21:54,016 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:22:09,146 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:22:09,147 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:22:21,114 DEBUG SenderThread:593294 [sender.py:send():179] send: stats | |
| 2021-07-14 22:22:24,279 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:22:24,279 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:22:39,412 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:22:39,413 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:22:51,192 DEBUG SenderThread:593294 [sender.py:send():179] send: stats | |
| 2021-07-14 22:22:54,548 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:22:54,548 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:23:09,678 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:23:09,678 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:23:21,267 DEBUG SenderThread:593294 [sender.py:send():179] send: stats | |
| 2021-07-14 22:23:24,814 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:23:24,814 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:23:39,949 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:23:39,949 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:23:51,337 DEBUG SenderThread:593294 [sender.py:send():179] send: stats | |
| 2021-07-14 22:23:55,081 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:23:55,082 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:24:10,212 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:24:10,212 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:24:21,405 DEBUG SenderThread:593294 [sender.py:send():179] send: stats | |
| 2021-07-14 22:24:25,345 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:24:25,346 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:24:40,483 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:24:40,483 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:24:51,475 DEBUG SenderThread:593294 [sender.py:send():179] send: stats | |
| 2021-07-14 22:24:55,615 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:24:55,615 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:25:10,746 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:25:10,746 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:25:21,548 DEBUG SenderThread:593294 [sender.py:send():179] send: stats | |
| 2021-07-14 22:25:25,876 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:25:25,876 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:25:41,015 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:25:41,016 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:25:51,619 DEBUG SenderThread:593294 [sender.py:send():179] send: stats | |
| 2021-07-14 22:25:56,148 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:25:56,148 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:26:11,280 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:26:11,280 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:26:21,695 DEBUG SenderThread:593294 [sender.py:send():179] send: stats | |
| 2021-07-14 22:26:26,412 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:26:26,413 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:26:41,546 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:26:41,547 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:26:51,772 DEBUG SenderThread:593294 [sender.py:send():179] send: stats | |
| 2021-07-14 22:26:56,683 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:26:56,683 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:27:11,816 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:27:11,816 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:27:21,849 DEBUG SenderThread:593294 [sender.py:send():179] send: stats | |
| 2021-07-14 22:27:26,950 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:27:26,950 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:27:29,710 INFO Thread-8 :593294 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/output.log | |
| 2021-07-14 22:27:42,097 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:27:42,097 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:27:51,925 DEBUG SenderThread:593294 [sender.py:send():179] send: stats | |
| 2021-07-14 22:27:57,249 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:27:57,250 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:28:12,383 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:28:12,384 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:28:22,007 DEBUG SenderThread:593294 [sender.py:send():179] send: stats | |
| 2021-07-14 22:28:27,521 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:28:27,522 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:28:42,658 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 22:28:42,658 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 22:28:49,741 INFO Thread-8 :593294 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/output.log | |
| 2021-07-14 22:28:51,116 DEBUG SenderThread:593294 [sender.py:send():179] send: telemetry | |
| 2021-07-14 22:28:51,116 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 22:28:51,116 DEBUG SenderThread:593294 [sender.py:send():179] send: exit | |
| 2021-07-14 22:28:51,116 INFO SenderThread:593294 [sender.py:send_exit():287] handling exit code: 1 | |
| 2021-07-14 22:28:51,117 INFO SenderThread:593294 [sender.py:send_exit():295] send defer | |
| 2021-07-14 22:28:51,117 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 22:28:51,118 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 22:28:51,118 INFO HandlerThread:593294 [handler.py:handle_request_defer():141] handle defer: 0 | |
| 2021-07-14 22:28:51,118 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 22:28:51,118 INFO SenderThread:593294 [sender.py:send_request_defer():304] handle sender defer: 0 | |
| 2021-07-14 22:28:51,118 INFO SenderThread:593294 [sender.py:transition_state():308] send defer: 1 | |
| 2021-07-14 22:28:51,118 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 22:28:51,118 INFO HandlerThread:593294 [handler.py:handle_request_defer():141] handle defer: 1 | |
| 2021-07-14 22:28:51,182 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 22:28:51,182 INFO SenderThread:593294 [sender.py:send_request_defer():304] handle sender defer: 1 | |
| 2021-07-14 22:28:51,183 INFO SenderThread:593294 [sender.py:transition_state():308] send defer: 2 | |
| 2021-07-14 22:28:51,183 DEBUG SenderThread:593294 [sender.py:send():179] send: stats | |
| 2021-07-14 22:28:51,183 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 22:28:51,183 INFO HandlerThread:593294 [handler.py:handle_request_defer():141] handle defer: 2 | |
| 2021-07-14 22:28:51,184 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 22:28:51,184 INFO SenderThread:593294 [sender.py:send_request_defer():304] handle sender defer: 2 | |
| 2021-07-14 22:28:51,184 INFO SenderThread:593294 [sender.py:transition_state():308] send defer: 3 | |
| 2021-07-14 22:28:51,184 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 22:28:51,184 INFO HandlerThread:593294 [handler.py:handle_request_defer():141] handle defer: 3 | |
| 2021-07-14 22:28:51,184 DEBUG SenderThread:593294 [sender.py:send():179] send: summary | |
| 2021-07-14 22:28:51,185 INFO SenderThread:593294 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-14 22:28:51,185 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 22:28:51,185 INFO SenderThread:593294 [sender.py:send_request_defer():304] handle sender defer: 3 | |
| 2021-07-14 22:28:51,185 INFO SenderThread:593294 [sender.py:transition_state():308] send defer: 4 | |
| 2021-07-14 22:28:51,185 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 22:28:51,185 INFO HandlerThread:593294 [handler.py:handle_request_defer():141] handle defer: 4 | |
| 2021-07-14 22:28:51,185 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 22:28:51,185 INFO SenderThread:593294 [sender.py:send_request_defer():304] handle sender defer: 4 | |
| 2021-07-14 22:28:51,220 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 22:28:51,361 INFO SenderThread:593294 [sender.py:transition_state():308] send defer: 5 | |
| 2021-07-14 22:28:51,361 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 22:28:51,361 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 22:28:51,362 INFO HandlerThread:593294 [handler.py:handle_request_defer():141] handle defer: 5 | |
| 2021-07-14 22:28:51,362 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 22:28:51,362 INFO SenderThread:593294 [sender.py:send_request_defer():304] handle sender defer: 5 | |
| 2021-07-14 22:28:51,362 INFO SenderThread:593294 [dir_watcher.py:finish():282] shutting down directory watcher | |
| 2021-07-14 22:28:51,463 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 22:28:51,742 INFO Thread-8 :593294 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/wandb-summary.json | |
| 2021-07-14 22:28:51,743 INFO SenderThread:593294 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/config.yaml | |
| 2021-07-14 22:28:51,743 INFO SenderThread:593294 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/output.log | |
| 2021-07-14 22:28:51,743 INFO SenderThread:593294 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files | |
| 2021-07-14 22:28:51,743 INFO SenderThread:593294 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/requirements.txt requirements.txt | |
| 2021-07-14 22:28:51,744 INFO SenderThread:593294 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/output.log output.log | |
| 2021-07-14 22:28:51,744 INFO SenderThread:593294 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/wandb-metadata.json wandb-metadata.json | |
| 2021-07-14 22:28:51,744 INFO SenderThread:593294 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/config.yaml config.yaml | |
| 2021-07-14 22:28:51,744 INFO SenderThread:593294 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/wandb-summary.json wandb-summary.json | |
| 2021-07-14 22:28:51,750 INFO SenderThread:593294 [sender.py:transition_state():308] send defer: 6 | |
| 2021-07-14 22:28:51,750 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 22:28:51,751 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 22:28:51,751 INFO HandlerThread:593294 [handler.py:handle_request_defer():141] handle defer: 6 | |
| 2021-07-14 22:28:51,754 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 22:28:51,754 INFO SenderThread:593294 [sender.py:send_request_defer():304] handle sender defer: 6 | |
| 2021-07-14 22:28:51,754 INFO SenderThread:593294 [file_pusher.py:finish():177] shutting down file pusher | |
| 2021-07-14 22:28:51,856 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 22:28:51,856 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 22:28:51,958 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 22:28:51,958 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 22:28:52,060 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 22:28:52,061 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 22:28:52,162 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 22:28:52,163 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 22:28:52,191 INFO Thread-13 :593294 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/output.log | |
| 2021-07-14 22:28:52,198 INFO Thread-12 :593294 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/requirements.txt | |
| 2021-07-14 22:28:52,200 INFO Thread-14 :593294 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/config.yaml | |
| 2021-07-14 22:28:52,212 INFO Thread-15 :593294 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/wandb-summary.json | |
| 2021-07-14 22:28:52,264 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 22:28:52,265 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 22:28:52,366 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 22:28:52,366 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 22:28:52,413 INFO Thread-7 :593294 [sender.py:transition_state():308] send defer: 7 | |
| 2021-07-14 22:28:52,413 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 22:28:52,413 INFO HandlerThread:593294 [handler.py:handle_request_defer():141] handle defer: 7 | |
| 2021-07-14 22:28:52,413 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 22:28:52,414 INFO SenderThread:593294 [sender.py:send_request_defer():304] handle sender defer: 7 | |
| 2021-07-14 22:28:52,468 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 22:28:52,536 INFO SenderThread:593294 [sender.py:transition_state():308] send defer: 8 | |
| 2021-07-14 22:28:52,536 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 22:28:52,537 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 22:28:52,537 INFO HandlerThread:593294 [handler.py:handle_request_defer():141] handle defer: 8 | |
| 2021-07-14 22:28:52,537 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 22:28:52,537 INFO SenderThread:593294 [sender.py:send_request_defer():304] handle sender defer: 8 | |
| 2021-07-14 22:28:52,537 INFO SenderThread:593294 [sender.py:transition_state():308] send defer: 9 | |
| 2021-07-14 22:28:52,538 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 22:28:52,538 INFO HandlerThread:593294 [handler.py:handle_request_defer():141] handle defer: 9 | |
| 2021-07-14 22:28:52,538 DEBUG SenderThread:593294 [sender.py:send():179] send: final | |
| 2021-07-14 22:28:52,538 DEBUG SenderThread:593294 [sender.py:send():179] send: footer | |
| 2021-07-14 22:28:52,538 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 22:28:52,538 INFO SenderThread:593294 [sender.py:send_request_defer():304] handle sender defer: 9 | |
| 2021-07-14 22:28:52,638 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 22:28:52,638 DEBUG SenderThread:593294 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 22:28:52,638 INFO SenderThread:593294 [file_pusher.py:join():182] waiting for file pusher | |
| 2021-07-14 22:28:52,640 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: get_summary | |
| 2021-07-14 22:28:52,640 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: sampled_history | |
| 2021-07-14 22:28:52,641 DEBUG HandlerThread:593294 [handler.py:handle_request():124] handle_request: shutdown | |
| 2021-07-14 22:28:52,641 INFO HandlerThread:593294 [handler.py:finish():638] shutting down handler | |
| 2021-07-14 22:28:53,538 INFO WriterThread:593294 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/run-s091gfok.wandb | |
| 2021-07-14 22:28:53,639 INFO SenderThread:593294 [sender.py:finish():945] shutting down sender | |
| 2021-07-14 22:28:53,639 INFO SenderThread:593294 [file_pusher.py:finish():177] shutting down file pusher | |
| 2021-07-14 22:28:53,639 INFO SenderThread:593294 [file_pusher.py:join():182] waiting for file pusher | |
| 2021-07-14 22:28:53,641 INFO MainThread:593294 [internal.py:handle_exit():78] Internal process exited | |