| 2021-07-14 21:23:51,351 INFO MainThread:580021 [internal.py:wandb_internal():88] W&B internal server running at pid: 580021, started at: 2021-07-14 21:23:51.351484 | |
| 2021-07-14 21:23:51,353 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: check_version | |
| 2021-07-14 21:23:51,354 INFO WriterThread:580021 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/run-6qjir2uu.wandb | |
| 2021-07-14 21:23:51,355 DEBUG SenderThread:580021 [sender.py:send():179] send: header | |
| 2021-07-14 21:23:51,355 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: check_version | |
| 2021-07-14 21:23:51,392 DEBUG SenderThread:580021 [sender.py:send():179] send: run | |
| 2021-07-14 21:23:51,563 INFO SenderThread:580021 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files | |
| 2021-07-14 21:23:51,563 INFO SenderThread:580021 [sender.py:_start_run_threads():716] run started: 6qjir2uu with start time 1626297830 | |
| 2021-07-14 21:23:51,563 DEBUG SenderThread:580021 [sender.py:send():179] send: summary | |
| 2021-07-14 21:23:51,563 INFO SenderThread:580021 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-14 21:23:51,563 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: run_start | |
| 2021-07-14 21:23:52,572 INFO Thread-8 :580021 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/wandb-summary.json | |
| 2021-07-14 21:23:52,677 DEBUG HandlerThread:580021 [meta.py:__init__():39] meta init | |
| 2021-07-14 21:23:52,677 DEBUG HandlerThread:580021 [meta.py:__init__():53] meta init done | |
| 2021-07-14 21:23:52,677 DEBUG HandlerThread:580021 [meta.py:probe():210] probe | |
| 2021-07-14 21:23:52,678 DEBUG HandlerThread:580021 [meta.py:_setup_git():200] setup git | |
| 2021-07-14 21:23:52,708 DEBUG HandlerThread:580021 [meta.py:_setup_git():207] setup git done | |
| 2021-07-14 21:23:52,708 DEBUG HandlerThread:580021 [meta.py:_save_pip():57] save pip | |
| 2021-07-14 21:23:52,709 DEBUG HandlerThread:580021 [meta.py:_save_pip():71] save pip done | |
| 2021-07-14 21:23:52,709 DEBUG HandlerThread:580021 [meta.py:probe():252] probe done | |
| 2021-07-14 21:23:52,712 DEBUG SenderThread:580021 [sender.py:send():179] send: files | |
| 2021-07-14 21:23:52,712 INFO SenderThread:580021 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now | |
| 2021-07-14 21:23:52,718 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:23:52,718 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:23:52,863 DEBUG SenderThread:580021 [sender.py:send():179] send: config | |
| 2021-07-14 21:23:52,864 DEBUG SenderThread:580021 [sender.py:send():179] send: config | |
| 2021-07-14 21:23:52,864 DEBUG SenderThread:580021 [sender.py:send():179] send: config | |
| 2021-07-14 21:23:53,163 INFO Thread-11 :580021 [upload_job.py:push():137] Uploaded file /tmp/tmpzpobzd3rwandb/1kqe8dfg-wandb-metadata.json | |
| 2021-07-14 21:23:53,571 INFO Thread-8 :580021 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/wandb-metadata.json | |
| 2021-07-14 21:23:53,571 INFO Thread-8 :580021 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:23:53,571 INFO Thread-8 :580021 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/requirements.txt | |
| 2021-07-14 21:24:07,576 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:24:07,865 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:24:07,865 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:24:13,578 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:24:20,750 DEBUG SenderThread:580021 [sender.py:send():179] send: stats | |
| 2021-07-14 21:24:22,581 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/config.yaml | |
| 2021-07-14 21:24:23,011 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:24:23,011 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:24:38,163 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:24:38,163 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:24:50,833 DEBUG SenderThread:580021 [sender.py:send():179] send: stats | |
| 2021-07-14 21:24:53,294 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:24:53,294 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:25:08,436 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:25:08,437 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:25:20,911 DEBUG SenderThread:580021 [sender.py:send():179] send: stats | |
| 2021-07-14 21:25:23,576 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:25:23,576 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:25:37,608 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:25:38,709 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:25:38,709 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:25:39,609 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:25:41,610 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:25:43,610 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:25:45,611 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:25:47,612 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:25:49,613 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:25:50,992 DEBUG SenderThread:580021 [sender.py:send():179] send: stats | |
| 2021-07-14 21:25:51,614 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:25:53,614 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:25:53,875 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:25:53,875 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:25:55,615 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:25:57,616 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:25:59,617 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:26:01,617 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:26:03,618 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:26:05,619 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:26:07,620 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:26:09,007 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:26:09,007 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:26:09,621 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:26:11,622 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:26:13,623 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:26:15,624 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:26:17,625 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:26:19,625 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:26:21,071 DEBUG SenderThread:580021 [sender.py:send():179] send: stats | |
| 2021-07-14 21:26:21,626 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:26:23,627 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:26:24,137 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:26:24,137 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:26:25,628 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:26:27,629 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:26:29,629 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:26:31,630 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:26:33,631 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:26:35,632 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:26:37,633 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:26:39,266 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:26:39,266 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:26:39,633 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:26:41,634 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:26:43,635 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:26:45,636 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:26:47,637 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:26:49,637 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:26:51,148 DEBUG SenderThread:580021 [sender.py:send():179] send: stats | |
| 2021-07-14 21:26:51,638 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:26:53,639 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:26:54,396 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:26:54,396 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:26:55,640 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:26:57,641 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:26:59,641 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:27:01,642 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:27:03,643 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:27:05,644 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:27:07,644 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:27:09,528 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:27:09,528 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:27:09,645 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:27:11,646 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:27:13,647 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:27:15,648 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:27:17,649 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:27:19,649 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:27:21,220 DEBUG SenderThread:580021 [sender.py:send():179] send: stats | |
| 2021-07-14 21:27:21,650 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:27:23,651 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:27:24,659 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:27:24,659 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:27:25,652 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:27:27,653 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:27:29,653 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:27:31,654 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:27:33,655 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:27:35,656 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:27:37,656 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:27:39,657 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:27:39,790 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:27:39,790 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:27:41,658 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:27:43,659 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:27:46,660 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:27:48,661 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:27:50,662 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:27:51,295 DEBUG SenderThread:580021 [sender.py:send():179] send: stats | |
| 2021-07-14 21:27:52,662 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:27:54,663 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:27:54,922 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:27:54,923 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:27:56,664 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:27:58,665 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:28:00,666 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:28:02,667 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:28:04,668 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:28:06,668 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:28:08,669 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:28:10,056 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:28:10,057 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:28:10,670 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:28:12,671 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:28:14,671 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:28:21,367 DEBUG SenderThread:580021 [sender.py:send():179] send: stats | |
| 2021-07-14 21:28:25,189 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:28:25,189 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:28:37,228 DEBUG SenderThread:580021 [sender.py:send():179] send: history | |
| 2021-07-14 21:28:37,229 DEBUG SenderThread:580021 [sender.py:send():179] send: summary | |
| 2021-07-14 21:28:37,231 INFO SenderThread:580021 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-14 21:28:37,680 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/wandb-summary.json | |
| 2021-07-14 21:28:38,681 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:28:39,681 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:28:40,320 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:28:40,320 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:28:40,681 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:28:42,682 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:28:44,683 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:28:46,684 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:28:48,685 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:28:50,686 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:28:51,442 DEBUG SenderThread:580021 [sender.py:send():179] send: stats | |
| 2021-07-14 21:28:52,687 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:28:54,687 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:28:55,463 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:28:55,463 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:28:56,688 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:28:58,689 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:29:00,690 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:29:02,691 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:29:04,691 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:29:06,692 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:29:08,693 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:29:10,610 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:29:10,611 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:29:10,694 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:29:12,695 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:29:14,695 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:29:16,696 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:29:18,697 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:29:20,698 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:29:21,521 DEBUG SenderThread:580021 [sender.py:send():179] send: stats | |
| 2021-07-14 21:29:22,699 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:29:24,700 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:29:25,747 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:29:25,748 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:29:26,701 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:29:28,702 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:29:30,703 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:29:32,704 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:29:34,705 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:29:36,706 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:29:38,707 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:29:40,707 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:29:40,883 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:29:40,883 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:29:42,708 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:29:44,709 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:29:46,710 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:29:48,711 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:29:50,712 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:29:51,599 DEBUG SenderThread:580021 [sender.py:send():179] send: stats | |
| 2021-07-14 21:29:52,713 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:29:54,714 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:29:56,025 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:29:56,025 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:29:56,715 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:29:58,715 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:30:00,716 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:30:02,717 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:30:04,718 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:30:06,719 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:30:08,720 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:30:10,721 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:30:11,164 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:30:11,165 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:30:12,722 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:30:14,723 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:30:16,724 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:30:18,724 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:30:20,725 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:30:21,667 DEBUG SenderThread:580021 [sender.py:send():179] send: stats | |
| 2021-07-14 21:30:22,726 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:30:24,727 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:30:26,298 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:30:26,298 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:30:26,728 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:30:28,729 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:30:30,730 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:30:33,731 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:30:35,732 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:30:37,733 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:30:39,734 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:30:41,436 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:30:41,437 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:30:41,735 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:30:43,736 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:30:45,737 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:30:47,738 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:30:49,739 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:30:51,734 DEBUG SenderThread:580021 [sender.py:send():179] send: stats | |
| 2021-07-14 21:30:51,740 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:30:53,741 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:30:55,742 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:30:56,575 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:30:56,576 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:30:57,742 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:30:59,743 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:31:01,744 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:31:03,745 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:31:05,746 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:31:07,747 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:31:09,748 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:31:11,717 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:31:11,717 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:31:11,749 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:31:13,749 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:31:15,750 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:31:21,803 DEBUG SenderThread:580021 [sender.py:send():179] send: stats | |
| 2021-07-14 21:31:26,853 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:31:26,853 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:31:37,846 DEBUG SenderThread:580021 [sender.py:send():179] send: history | |
| 2021-07-14 21:31:37,847 DEBUG SenderThread:580021 [sender.py:send():179] send: summary | |
| 2021-07-14 21:31:37,847 INFO SenderThread:580021 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-14 21:31:38,759 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/wandb-summary.json | |
| 2021-07-14 21:31:39,760 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:31:41,981 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:31:41,982 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:31:51,878 DEBUG SenderThread:580021 [sender.py:send():179] send: stats | |
| 2021-07-14 21:31:57,123 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:31:57,123 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:31:59,768 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:32:01,769 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:32:04,447 DEBUG SenderThread:580021 [sender.py:send():179] send: history | |
| 2021-07-14 21:32:04,448 DEBUG SenderThread:580021 [sender.py:send():179] send: summary | |
| 2021-07-14 21:32:04,448 INFO SenderThread:580021 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-14 21:32:04,770 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/wandb-summary.json | |
| 2021-07-14 21:32:07,771 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:32:12,254 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:32:12,255 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:32:21,959 DEBUG SenderThread:580021 [sender.py:send():179] send: stats | |
| 2021-07-14 21:32:27,390 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:32:27,391 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:32:37,785 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:32:41,786 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:32:42,870 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 21:32:42,871 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 21:32:43,788 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:32:52,027 DEBUG SenderThread:580021 [sender.py:send():179] send: stats | |
| 2021-07-14 21:32:59,795 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:32:59,895 DEBUG SenderThread:580021 [sender.py:send():179] send: telemetry | |
| 2021-07-14 21:32:59,896 DEBUG SenderThread:580021 [sender.py:send():179] send: exit | |
| 2021-07-14 21:32:59,896 INFO SenderThread:580021 [sender.py:send_exit():287] handling exit code: 1 | |
| 2021-07-14 21:32:59,896 INFO SenderThread:580021 [sender.py:send_exit():295] send defer | |
| 2021-07-14 21:32:59,896 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 21:32:59,897 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 21:32:59,897 INFO HandlerThread:580021 [handler.py:handle_request_defer():141] handle defer: 0 | |
| 2021-07-14 21:32:59,897 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 21:32:59,897 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 21:32:59,897 INFO SenderThread:580021 [sender.py:send_request_defer():304] handle sender defer: 0 | |
| 2021-07-14 21:32:59,897 INFO SenderThread:580021 [sender.py:transition_state():308] send defer: 1 | |
| 2021-07-14 21:32:59,897 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 21:32:59,898 INFO HandlerThread:580021 [handler.py:handle_request_defer():141] handle defer: 1 | |
| 2021-07-14 21:32:59,944 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 21:32:59,944 INFO SenderThread:580021 [sender.py:send_request_defer():304] handle sender defer: 1 | |
| 2021-07-14 21:32:59,944 INFO SenderThread:580021 [sender.py:transition_state():308] send defer: 2 | |
| 2021-07-14 21:32:59,944 DEBUG SenderThread:580021 [sender.py:send():179] send: stats | |
| 2021-07-14 21:32:59,945 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 21:32:59,945 INFO HandlerThread:580021 [handler.py:handle_request_defer():141] handle defer: 2 | |
| 2021-07-14 21:32:59,945 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 21:32:59,945 INFO SenderThread:580021 [sender.py:send_request_defer():304] handle sender defer: 2 | |
| 2021-07-14 21:32:59,945 INFO SenderThread:580021 [sender.py:transition_state():308] send defer: 3 | |
| 2021-07-14 21:32:59,945 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 21:32:59,946 INFO HandlerThread:580021 [handler.py:handle_request_defer():141] handle defer: 3 | |
| 2021-07-14 21:32:59,946 DEBUG SenderThread:580021 [sender.py:send():179] send: summary | |
| 2021-07-14 21:32:59,946 INFO SenderThread:580021 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-14 21:32:59,946 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 21:32:59,946 INFO SenderThread:580021 [sender.py:send_request_defer():304] handle sender defer: 3 | |
| 2021-07-14 21:32:59,946 INFO SenderThread:580021 [sender.py:transition_state():308] send defer: 4 | |
| 2021-07-14 21:32:59,947 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 21:32:59,947 INFO HandlerThread:580021 [handler.py:handle_request_defer():141] handle defer: 4 | |
| 2021-07-14 21:32:59,947 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 21:32:59,947 INFO SenderThread:580021 [sender.py:send_request_defer():304] handle sender defer: 4 | |
| 2021-07-14 21:32:59,999 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 21:33:00,122 INFO SenderThread:580021 [sender.py:transition_state():308] send defer: 5 | |
| 2021-07-14 21:33:00,122 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 21:33:00,122 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 21:33:00,122 INFO HandlerThread:580021 [handler.py:handle_request_defer():141] handle defer: 5 | |
| 2021-07-14 21:33:00,123 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 21:33:00,123 INFO SenderThread:580021 [sender.py:send_request_defer():304] handle sender defer: 5 | |
| 2021-07-14 21:33:00,123 INFO SenderThread:580021 [dir_watcher.py:finish():282] shutting down directory watcher | |
| 2021-07-14 21:33:00,224 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 21:33:00,795 INFO Thread-8 :580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/config.yaml | |
| 2021-07-14 21:33:00,796 INFO SenderThread:580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:33:00,796 INFO SenderThread:580021 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/wandb-summary.json | |
| 2021-07-14 21:33:00,796 INFO SenderThread:580021 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files | |
| 2021-07-14 21:33:00,796 INFO SenderThread:580021 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/requirements.txt requirements.txt | |
| 2021-07-14 21:33:00,796 INFO SenderThread:580021 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log output.log | |
| 2021-07-14 21:33:00,797 INFO SenderThread:580021 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/wandb-metadata.json wandb-metadata.json | |
| 2021-07-14 21:33:00,797 INFO SenderThread:580021 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/config.yaml config.yaml | |
| 2021-07-14 21:33:00,797 INFO SenderThread:580021 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/wandb-summary.json wandb-summary.json | |
| 2021-07-14 21:33:00,797 INFO SenderThread:580021 [sender.py:transition_state():308] send defer: 6 | |
| 2021-07-14 21:33:00,797 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 21:33:00,801 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 21:33:00,801 INFO HandlerThread:580021 [handler.py:handle_request_defer():141] handle defer: 6 | |
| 2021-07-14 21:33:00,804 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 21:33:00,805 INFO SenderThread:580021 [sender.py:send_request_defer():304] handle sender defer: 6 | |
| 2021-07-14 21:33:00,805 INFO SenderThread:580021 [file_pusher.py:finish():177] shutting down file pusher | |
| 2021-07-14 21:33:00,903 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 21:33:00,903 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 21:33:01,005 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 21:33:01,006 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 21:33:01,108 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 21:33:01,108 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 21:33:01,210 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 21:33:01,210 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 21:33:01,236 INFO Thread-14 :580021 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/config.yaml | |
| 2021-07-14 21:33:01,247 INFO Thread-12 :580021 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/requirements.txt | |
| 2021-07-14 21:33:01,250 INFO Thread-15 :580021 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/wandb-summary.json | |
| 2021-07-14 21:33:01,286 INFO Thread-13 :580021 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/files/output.log | |
| 2021-07-14 21:33:01,312 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 21:33:01,312 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 21:33:01,414 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 21:33:01,414 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 21:33:01,486 INFO Thread-7 :580021 [sender.py:transition_state():308] send defer: 7 | |
| 2021-07-14 21:33:01,487 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 21:33:01,487 INFO HandlerThread:580021 [handler.py:handle_request_defer():141] handle defer: 7 | |
| 2021-07-14 21:33:01,487 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 21:33:01,487 INFO SenderThread:580021 [sender.py:send_request_defer():304] handle sender defer: 7 | |
| 2021-07-14 21:33:01,516 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 21:33:01,759 INFO SenderThread:580021 [sender.py:transition_state():308] send defer: 8 | |
| 2021-07-14 21:33:01,759 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 21:33:01,759 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 21:33:01,760 INFO HandlerThread:580021 [handler.py:handle_request_defer():141] handle defer: 8 | |
| 2021-07-14 21:33:01,760 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 21:33:01,760 INFO SenderThread:580021 [sender.py:send_request_defer():304] handle sender defer: 8 | |
| 2021-07-14 21:33:01,760 INFO SenderThread:580021 [sender.py:transition_state():308] send defer: 9 | |
| 2021-07-14 21:33:01,760 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: defer | |
| 2021-07-14 21:33:01,760 DEBUG SenderThread:580021 [sender.py:send():179] send: final | |
| 2021-07-14 21:33:01,761 INFO HandlerThread:580021 [handler.py:handle_request_defer():141] handle defer: 9 | |
| 2021-07-14 21:33:01,761 DEBUG SenderThread:580021 [sender.py:send():179] send: footer | |
| 2021-07-14 21:33:01,761 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: defer | |
| 2021-07-14 21:33:01,761 INFO SenderThread:580021 [sender.py:send_request_defer():304] handle sender defer: 9 | |
| 2021-07-14 21:33:01,861 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: poll_exit | |
| 2021-07-14 21:33:01,861 DEBUG SenderThread:580021 [sender.py:send_request():193] send_request: poll_exit | |
| 2021-07-14 21:33:01,861 INFO SenderThread:580021 [file_pusher.py:join():182] waiting for file pusher | |
| 2021-07-14 21:33:01,863 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: get_summary | |
| 2021-07-14 21:33:01,864 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: sampled_history | |
| 2021-07-14 21:33:01,865 DEBUG HandlerThread:580021 [handler.py:handle_request():124] handle_request: shutdown | |
| 2021-07-14 21:33:01,866 INFO HandlerThread:580021 [handler.py:finish():638] shutting down handler | |
| 2021-07-14 21:33:02,761 INFO WriterThread:580021 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_212350-6qjir2uu/run-6qjir2uu.wandb | |
| 2021-07-14 21:33:02,862 INFO SenderThread:580021 [sender.py:finish():945] shutting down sender | |
| 2021-07-14 21:33:02,862 INFO SenderThread:580021 [file_pusher.py:finish():177] shutting down file pusher | |
| 2021-07-14 21:33:02,862 INFO SenderThread:580021 [file_pusher.py:join():182] waiting for file pusher | |
| 2021-07-14 21:33:02,864 INFO MainThread:580021 [internal.py:handle_exit():78] Internal process exited | |