2021-07-13 11:34:36,371 INFO MainThread:353216 [internal.py:wandb_internal():88] W&B internal server running at pid: 353216, started at: 2021-07-13 11:34:36.371423 2021-07-13 11:34:36,374 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: check_version 2021-07-13 11:34:36,374 DEBUG SenderThread:353216 [sender.py:send():179] send: header 2021-07-13 11:34:36,374 INFO WriterThread:353216 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/run-1vxvoloi.wandb 2021-07-13 11:34:36,375 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: check_version 2021-07-13 11:34:36,412 DEBUG SenderThread:353216 [sender.py:send():179] send: run 2021-07-13 11:34:36,584 INFO SenderThread:353216 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files 2021-07-13 11:34:36,585 INFO SenderThread:353216 [sender.py:_start_run_threads():716] run started: 1vxvoloi with start time 1626176075 2021-07-13 11:34:36,585 DEBUG SenderThread:353216 [sender.py:send():179] send: summary 2021-07-13 11:34:36,585 INFO SenderThread:353216 [sender.py:_save_file():841] saving file wandb-summary.json with policy end 2021-07-13 11:34:36,585 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: run_start 2021-07-13 11:34:37,598 INFO Thread-8 :353216 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/wandb-summary.json 2021-07-13 11:34:37,732 DEBUG HandlerThread:353216 [meta.py:__init__():39] meta init 2021-07-13 11:34:37,732 DEBUG HandlerThread:353216 [meta.py:__init__():53] meta init done 2021-07-13 11:34:37,732 DEBUG HandlerThread:353216 [meta.py:probe():210] probe 2021-07-13 11:34:37,733 DEBUG HandlerThread:353216 [meta.py:_setup_git():200] setup git 2021-07-13 11:34:37,764 DEBUG HandlerThread:353216 [meta.py:_setup_git():207] setup git done 2021-07-13 11:34:37,764 DEBUG HandlerThread:353216 [meta.py:_save_pip():57] save pip 2021-07-13 11:34:37,765 DEBUG HandlerThread:353216 [meta.py:_save_pip():71] save pip done 2021-07-13 11:34:37,765 DEBUG HandlerThread:353216 [meta.py:probe():252] probe done 2021-07-13 11:34:37,769 DEBUG SenderThread:353216 [sender.py:send():179] send: files 2021-07-13 11:34:37,769 INFO SenderThread:353216 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now 2021-07-13 11:34:37,775 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:34:37,776 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:34:37,906 DEBUG SenderThread:353216 [sender.py:send():179] send: config 2021-07-13 11:34:37,906 DEBUG SenderThread:353216 [sender.py:send():179] send: config 2021-07-13 11:34:37,906 DEBUG SenderThread:353216 [sender.py:send():179] send: config 2021-07-13 11:34:38,222 INFO Thread-11 :353216 [upload_job.py:push():137] Uploaded file /tmp/tmpokklr1agwandb/32ipfyng-wandb-metadata.json 2021-07-13 11:34:38,597 INFO Thread-8 :353216 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/requirements.txt 2021-07-13 11:34:38,597 INFO Thread-8 :353216 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/wandb-metadata.json 2021-07-13 11:34:38,598 INFO Thread-8 :353216 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:34:52,602 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:34:52,909 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:34:52,909 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:34:54,603 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:35:05,813 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:35:07,608 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/config.yaml 2021-07-13 11:35:08,039 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:35:08,039 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:35:23,172 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:35:23,173 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:35:35,884 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:35:38,310 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:35:38,310 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:35:53,444 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:35:53,444 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:36:05,948 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:36:08,575 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:36:08,575 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:36:22,634 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:36:23,746 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:36:23,747 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:36:36,019 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:36:38,888 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:36:38,889 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:36:54,028 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:36:54,029 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:37:06,096 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:37:09,466 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:37:09,467 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:37:24,600 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:37:24,600 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:37:36,175 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:37:39,736 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:37:39,736 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:37:54,868 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:37:54,868 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:37:56,667 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:37:58,668 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:38:00,669 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:38:02,669 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:38:04,670 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:38:06,251 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:38:06,671 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:38:08,672 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:38:10,000 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:38:10,000 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:38:10,672 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:38:12,673 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:38:14,674 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:38:16,675 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:38:18,676 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:38:20,676 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:38:22,677 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:38:24,678 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:38:25,132 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:38:25,132 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:38:26,679 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:38:28,679 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:38:30,680 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:38:32,681 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:38:34,682 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:38:36,327 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:38:36,682 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:38:38,683 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:38:40,286 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:38:40,286 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:38:40,684 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:38:42,684 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:38:44,685 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:38:46,686 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:38:48,687 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:38:50,687 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:38:52,688 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:38:54,689 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:38:55,418 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:38:55,418 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:38:56,690 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:38:58,691 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:39:00,691 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:39:02,692 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:39:04,693 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:39:06,402 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:39:06,694 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:39:08,694 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:39:10,549 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:39:10,549 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:39:10,695 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:39:12,696 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:39:14,696 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:39:16,698 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:39:18,698 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:39:20,699 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:39:22,700 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:39:24,701 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:39:25,682 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:39:25,682 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:39:26,701 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:39:28,702 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:39:30,702 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:39:32,703 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:39:34,704 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:39:36,479 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:39:36,705 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:39:38,706 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:39:40,706 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:39:40,814 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:39:40,814 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:39:42,707 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:39:44,708 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:39:46,708 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:39:48,709 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:39:50,710 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:39:53,711 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:39:55,712 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:39:55,957 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:39:55,957 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:39:57,713 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:39:59,714 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:40:01,714 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:40:03,715 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:40:05,716 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:40:06,556 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:40:07,717 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:40:09,718 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:40:11,098 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:40:11,099 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:40:11,718 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:40:13,719 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:40:15,720 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:40:17,721 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:40:19,722 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:40:21,723 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:40:23,723 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:40:25,724 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:40:26,232 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:40:26,233 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:40:27,725 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:40:29,726 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:40:31,727 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:40:33,728 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:40:35,728 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:40:36,634 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:40:37,729 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:40:39,730 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:40:41,362 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:40:41,362 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:40:41,731 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:40:43,732 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:40:45,732 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:40:47,733 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:40:49,734 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:40:51,735 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:40:53,736 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:40:55,737 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:40:56,493 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:40:56,494 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:40:57,738 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:40:59,739 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:41:01,739 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:41:03,740 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:41:05,741 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:41:06,712 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:41:07,742 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:41:09,743 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:41:11,648 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:41:11,648 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:41:11,744 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:41:13,745 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:41:15,746 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:41:17,747 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:41:19,748 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:41:21,749 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:41:23,750 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:41:25,750 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:41:26,781 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:41:26,781 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:41:27,751 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:41:29,752 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:41:31,753 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:41:33,754 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:41:35,754 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:41:36,789 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:41:37,755 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:41:39,756 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:41:41,757 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:41:41,912 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:41:41,912 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:41:43,757 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:41:45,758 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:41:47,759 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:41:49,760 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:41:51,760 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:41:53,761 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:41:55,762 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:41:57,041 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:41:57,042 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:41:57,762 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:41:59,763 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:42:01,764 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:42:03,765 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:42:05,765 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:42:06,883 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:42:07,766 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:42:09,767 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:42:11,767 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:42:12,179 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:42:12,180 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:42:13,768 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:42:15,769 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:42:17,770 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:42:19,770 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:42:21,771 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:42:23,771 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:42:25,772 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:42:27,320 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:42:27,320 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:42:27,773 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:42:29,773 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:42:31,774 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:42:33,775 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:42:35,776 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:42:36,960 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:42:37,776 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:42:39,777 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:42:41,778 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:42:42,460 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:42:42,460 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:42:43,779 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:42:45,779 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:42:47,780 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:42:49,781 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:42:51,781 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:42:53,782 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:42:55,783 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:42:57,600 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:42:57,601 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:42:57,783 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:42:59,784 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:43:01,785 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:43:03,785 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:43:05,786 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:43:07,038 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:43:07,787 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:43:09,787 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:43:11,788 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:43:12,730 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:43:12,730 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:43:13,789 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:43:15,790 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:43:17,790 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:43:19,791 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:43:21,792 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:43:23,793 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:43:25,793 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:43:27,860 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:43:27,860 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:43:37,114 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:43:42,995 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:43:42,995 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:43:47,346 DEBUG SenderThread:353216 [sender.py:send():179] send: history 2021-07-13 11:43:47,346 DEBUG SenderThread:353216 [sender.py:send():179] send: summary 2021-07-13 11:43:47,347 INFO SenderThread:353216 [sender.py:_save_file():841] saving file wandb-summary.json with policy end 2021-07-13 11:43:47,801 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/wandb-summary.json 2021-07-13 11:43:47,801 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:43:49,802 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:43:50,802 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:43:52,803 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:43:54,803 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:43:56,804 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:43:58,127 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:43:58,127 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:43:58,805 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:44:00,805 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:44:02,806 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:44:04,807 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:44:06,808 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:44:07,193 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:44:08,808 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:44:10,809 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:44:12,810 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:44:13,273 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:44:13,273 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:44:14,811 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:44:16,811 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:44:18,812 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:44:20,813 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:44:22,813 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:44:24,814 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:44:26,815 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:44:28,406 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:44:28,406 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:44:28,816 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:44:30,816 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:44:32,817 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:44:34,818 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:44:36,819 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:44:37,272 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:44:38,820 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:44:40,820 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:44:42,821 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:44:43,539 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:44:43,539 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:44:44,822 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:44:46,823 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:44:48,824 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:44:50,824 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:44:52,825 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:44:54,826 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:44:56,827 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:44:58,671 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:44:58,671 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:44:58,827 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:45:00,828 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:45:02,829 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:45:04,830 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:45:06,831 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:45:07,349 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:45:08,832 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:45:10,833 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:45:12,833 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:45:13,803 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:45:13,803 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:45:14,834 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:45:16,835 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:45:18,836 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:45:20,837 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:45:22,838 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:45:24,838 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:45:26,839 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:45:28,840 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:45:28,938 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:45:28,938 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:45:30,840 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:45:32,841 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:45:34,842 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:45:36,843 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:45:37,427 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:45:38,844 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:45:40,844 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:45:42,845 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:45:44,070 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:45:44,071 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:45:44,846 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:45:46,847 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:45:48,848 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:45:50,848 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:45:52,849 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:45:54,850 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:45:56,851 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:45:58,852 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:45:59,201 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:45:59,201 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:46:00,852 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:46:02,853 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:46:04,854 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:46:06,855 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:46:07,505 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:46:08,855 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:46:10,856 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:46:12,857 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:46:14,334 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:46:14,334 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:46:14,858 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:46:16,859 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:46:18,860 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:46:20,861 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:46:22,862 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:46:24,862 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:46:26,863 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:46:28,864 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:46:29,468 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:46:29,468 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:46:30,865 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:46:32,865 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:46:34,866 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:46:36,867 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:46:37,583 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:46:38,868 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:46:40,869 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:46:42,870 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:46:44,601 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:46:44,602 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:46:44,870 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:46:46,871 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:46:48,872 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:46:50,873 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:46:52,874 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:46:54,874 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:46:56,875 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:46:58,876 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:46:59,732 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:46:59,733 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:47:00,877 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:47:02,878 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:47:04,879 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:47:06,880 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:47:07,661 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:47:08,881 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:47:10,881 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:47:13,882 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:47:14,866 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:47:14,866 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:47:15,883 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:47:17,884 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:47:19,885 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:47:21,886 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:47:23,887 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:47:25,887 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:47:27,888 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:47:29,889 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:47:30,009 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:47:30,009 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:47:31,889 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:47:33,890 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:47:35,891 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:47:37,741 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:47:37,892 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:47:39,892 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:47:41,893 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:47:43,894 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:47:45,150 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:47:45,150 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:47:45,895 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:47:47,895 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:47:49,896 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:47:51,897 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:47:53,897 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:47:55,898 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:47:57,899 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:47:59,899 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:48:00,283 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:48:00,284 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:48:01,900 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:48:03,901 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:48:05,902 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:48:07,817 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:48:07,902 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:48:09,903 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:48:11,904 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:48:13,904 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:48:15,414 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:48:15,414 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:48:15,905 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:48:17,906 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:48:19,907 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:48:21,907 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:48:23,908 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:48:25,908 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:48:27,909 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:48:29,910 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:48:30,544 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:48:30,545 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:48:31,910 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:48:33,911 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:48:35,912 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:48:37,895 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:48:37,913 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:48:39,914 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:48:41,914 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:48:43,915 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:48:45,687 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:48:45,687 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:48:45,916 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:48:47,916 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:48:49,917 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:48:51,918 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:48:53,918 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:48:55,919 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:48:57,920 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:48:59,920 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:49:00,825 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:49:00,826 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:49:01,921 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:49:03,922 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:49:05,923 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:49:07,924 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:49:07,971 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:49:09,924 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:49:11,925 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:49:13,926 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:49:15,926 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:49:15,957 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:49:15,957 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:49:31,088 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:49:31,089 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:49:38,049 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:49:38,378 DEBUG SenderThread:353216 [sender.py:send():179] send: history 2021-07-13 11:49:38,378 DEBUG SenderThread:353216 [sender.py:send():179] send: summary 2021-07-13 11:49:38,379 INFO SenderThread:353216 [sender.py:_save_file():841] saving file wandb-summary.json with policy end 2021-07-13 11:49:38,935 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/wandb-summary.json 2021-07-13 11:49:39,936 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:49:40,936 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:49:41,937 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:49:43,938 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:49:45,939 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:49:46,219 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:49:46,219 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:49:47,940 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:49:49,940 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:49:51,941 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:49:53,942 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:49:55,943 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:49:57,944 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:49:59,945 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:50:01,352 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:50:01,352 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:50:01,945 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:50:03,946 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:50:05,947 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:50:07,948 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:50:08,127 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:50:09,948 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:50:11,949 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:50:13,950 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:50:15,951 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:50:16,484 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:50:16,484 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:50:17,952 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:50:19,952 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:50:21,953 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:50:23,954 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:50:25,955 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:50:27,956 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:50:29,957 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:50:31,618 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:50:31,618 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:50:31,957 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:50:34,959 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:50:36,960 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:50:38,205 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:50:38,961 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:50:40,962 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:50:42,963 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:50:44,963 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:50:46,747 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:50:46,748 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:50:46,964 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:50:48,965 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:50:50,966 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:50:52,967 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:50:54,968 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:50:56,969 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:50:58,970 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:51:00,970 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:51:01,878 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:51:01,878 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:51:02,971 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:51:04,972 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:51:06,973 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:51:08,281 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:51:08,974 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:51:10,975 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:51:12,976 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:51:14,977 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:51:16,978 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:51:17,021 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:51:17,021 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:51:18,980 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:51:20,980 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:51:22,981 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:51:24,982 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:51:26,983 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:51:28,984 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:51:30,985 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:51:32,159 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:51:32,159 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:51:32,986 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:51:34,987 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:51:36,988 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:51:38,360 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:51:38,989 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:51:40,990 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:51:42,991 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:51:44,992 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:51:46,992 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:51:47,293 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:51:47,293 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:51:48,993 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:51:50,994 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:51:52,995 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:51:54,996 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:51:56,997 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:51:58,997 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:52:00,998 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:52:02,426 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:52:02,427 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:52:02,999 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:52:05,000 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:52:07,001 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:52:08,437 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:52:09,002 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:52:11,003 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:52:13,003 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:52:15,004 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:52:17,005 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:52:17,560 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:52:17,560 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:52:19,006 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:52:21,007 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:52:23,007 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:52:25,008 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:52:27,009 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:52:29,010 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:52:31,010 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:52:32,696 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:52:32,696 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:52:33,011 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:52:35,012 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:52:37,013 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:52:38,515 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:52:39,013 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:52:41,014 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:52:43,015 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:52:45,015 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:52:47,016 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:52:47,827 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:52:47,827 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:52:49,017 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:52:51,017 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:52:53,018 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:52:55,019 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:52:57,019 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:52:59,020 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:53:01,020 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:53:02,961 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:53:02,962 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:53:03,021 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:53:05,022 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:53:07,022 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:53:08,593 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:53:09,023 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:53:11,024 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:53:13,024 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:53:15,025 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:53:17,026 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:53:18,093 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:53:18,093 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:53:19,027 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:53:21,027 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:53:23,028 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:53:25,029 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:53:27,030 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:53:29,030 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:53:31,031 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:53:33,032 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:53:33,226 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:53:33,226 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:53:35,032 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:53:37,033 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:53:38,672 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:53:39,033 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:53:41,034 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:53:43,035 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:53:45,035 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:53:47,036 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:53:48,358 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:53:48,359 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:53:49,037 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:53:51,038 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:53:53,038 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:53:55,039 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:53:57,040 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:53:59,041 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:54:01,041 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:54:03,042 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:54:03,503 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:54:03,503 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:54:05,043 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:54:07,044 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:54:08,748 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:54:09,045 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:54:11,045 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:54:13,046 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:54:15,047 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:54:17,048 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:54:18,648 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:54:18,649 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:54:19,049 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:54:21,049 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:54:23,050 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:54:25,051 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:54:27,052 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:54:29,053 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:54:31,054 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:54:33,055 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:54:33,812 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:54:33,812 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:54:35,056 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:54:37,057 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:54:38,827 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:54:39,058 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:54:41,059 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:54:43,059 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:54:45,060 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:54:47,061 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:54:48,944 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:54:48,945 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:54:49,062 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:54:51,063 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:54:53,064 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:54:55,065 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:54:57,065 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:54:59,066 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:55:01,067 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:55:03,068 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:55:04,077 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:55:04,077 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:55:05,069 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:55:07,070 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:55:08,904 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:55:19,209 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:55:19,209 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:55:29,385 DEBUG SenderThread:353216 [sender.py:send():179] send: history 2021-07-13 11:55:29,385 DEBUG SenderThread:353216 [sender.py:send():179] send: summary 2021-07-13 11:55:29,386 INFO SenderThread:353216 [sender.py:_save_file():841] saving file wandb-summary.json with policy end 2021-07-13 11:55:30,079 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/wandb-summary.json 2021-07-13 11:55:31,079 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:55:33,080 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:55:34,343 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:55:34,344 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:55:35,081 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:55:37,082 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:55:38,981 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:55:39,083 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:55:41,083 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:55:43,084 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:55:45,085 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:55:47,086 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:55:49,087 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:55:49,528 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:55:49,528 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:55:51,088 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:55:53,088 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:55:55,089 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:55:57,090 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:55:59,091 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:56:01,092 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:56:03,093 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:56:04,677 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:56:04,678 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:56:05,094 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:56:07,095 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:56:09,060 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:56:09,095 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:56:12,097 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:56:14,098 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:56:16,098 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:56:18,099 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:56:19,807 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:56:19,807 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:56:20,100 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:56:22,101 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:56:24,102 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:56:26,103 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:56:28,104 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:56:30,104 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:56:32,106 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:56:34,106 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:56:34,941 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:56:34,941 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:56:36,107 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:56:38,108 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:56:39,136 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:56:40,109 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:56:42,110 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:56:44,111 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:56:46,112 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:56:48,113 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:56:50,074 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:56:50,074 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:56:50,114 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:56:52,114 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:56:54,115 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:56:56,116 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:56:58,117 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:57:00,118 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:57:02,119 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:57:04,120 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:57:05,206 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:57:05,206 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:57:06,121 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:57:08,122 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:57:09,215 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:57:10,123 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:57:12,124 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:57:14,125 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:57:16,126 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:57:18,127 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:57:20,127 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:57:20,338 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:57:20,338 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:57:22,128 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:57:24,129 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:57:26,130 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:57:28,131 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:57:30,131 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:57:32,132 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:57:34,133 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:57:35,468 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:57:35,469 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:57:36,134 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:57:38,135 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:57:39,293 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:57:40,136 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:57:42,137 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:57:44,138 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:57:46,139 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:57:48,140 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:57:50,141 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:57:50,600 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:57:50,600 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:57:52,142 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:57:54,143 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:57:56,144 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:57:58,145 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:58:00,146 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:58:02,147 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:58:04,148 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:58:05,736 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:58:05,736 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:58:06,149 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:58:08,149 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:58:09,364 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:58:10,151 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:58:12,151 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:58:14,152 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:58:16,154 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:58:18,154 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:58:20,155 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:58:20,876 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:58:20,876 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:58:22,156 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:58:24,157 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:58:26,158 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:58:28,159 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:58:30,159 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:58:32,160 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:58:34,161 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:58:36,010 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:58:36,011 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:58:36,162 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:58:38,162 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:58:39,435 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:58:40,163 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:58:42,164 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:58:44,165 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:58:46,165 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:58:48,166 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:58:50,167 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:58:51,146 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:58:51,147 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:58:52,168 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:58:54,169 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:58:56,169 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:58:58,170 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:59:00,171 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:59:02,172 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:59:04,173 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:59:06,173 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:59:06,278 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:59:06,278 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:59:08,174 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:59:09,508 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:59:10,175 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:59:12,176 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:59:14,176 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:59:16,177 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:59:18,178 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:59:20,179 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:59:21,407 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:59:21,407 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:59:22,180 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:59:24,180 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:59:26,181 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:59:28,182 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:59:30,183 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:59:32,184 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:59:34,185 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:59:36,185 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:59:36,559 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:59:36,560 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:59:38,186 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:59:39,586 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 11:59:40,187 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:59:42,188 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:59:44,189 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:59:46,190 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:59:48,190 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:59:50,191 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:59:51,712 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 11:59:51,713 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 11:59:52,192 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:59:54,193 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:59:56,193 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 11:59:58,194 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:00:00,195 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:00:03,196 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:00:05,197 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:00:06,846 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:00:06,847 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:00:07,198 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:00:09,199 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:00:09,660 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 12:00:11,200 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:00:13,200 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:00:15,201 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:00:17,202 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:00:19,203 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:00:21,203 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:00:21,978 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:00:21,978 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:00:23,204 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:00:25,205 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:00:27,206 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:00:29,206 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:00:31,207 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:00:33,208 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:00:35,209 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:00:37,111 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:00:37,111 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:00:37,210 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:00:39,211 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:00:39,735 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 12:00:41,211 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:00:43,212 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:00:45,213 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:00:47,214 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:00:49,215 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:00:51,216 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:00:52,245 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:00:52,245 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:00:53,216 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:00:55,217 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:00:57,218 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:00:59,219 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:01:07,378 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:01:07,379 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:01:09,812 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 12:01:20,445 DEBUG SenderThread:353216 [sender.py:send():179] send: history 2021-07-13 12:01:20,446 DEBUG SenderThread:353216 [sender.py:send():179] send: summary 2021-07-13 12:01:20,446 INFO SenderThread:353216 [sender.py:_save_file():841] saving file wandb-summary.json with policy end 2021-07-13 12:01:21,226 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/wandb-summary.json 2021-07-13 12:01:22,226 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:01:22,523 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:01:22,524 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:01:23,227 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:01:25,228 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:01:27,228 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:01:29,229 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:01:31,230 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:01:33,231 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:01:35,232 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:01:37,233 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:01:37,665 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:01:37,665 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:01:39,234 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:01:39,889 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 12:01:41,235 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:01:43,236 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:01:45,237 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:01:47,238 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:01:49,239 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:01:51,240 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:01:52,795 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:01:52,796 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:01:53,241 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:01:55,242 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:01:57,243 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:01:59,244 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:02:01,245 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:02:03,246 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:02:05,247 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:02:07,248 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:02:07,929 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:02:07,930 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:02:09,249 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:02:09,967 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 12:02:11,250 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:02:13,251 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:02:15,252 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:02:17,253 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:02:19,254 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:02:21,255 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:02:23,062 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:02:23,062 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:02:23,256 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:02:25,257 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:02:27,258 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:02:29,259 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:02:31,260 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:02:33,261 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:02:35,262 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:02:37,262 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:02:38,194 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:02:38,194 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:02:39,263 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:02:40,044 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 12:02:41,265 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:02:43,266 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:02:45,267 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:02:47,268 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:02:49,269 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:02:51,270 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:02:53,271 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:02:53,326 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:02:53,327 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:02:55,272 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:02:57,273 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:02:59,274 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:03:01,275 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:03:03,276 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:03:05,277 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:03:07,278 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:03:08,458 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:03:08,458 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:03:09,279 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:03:10,122 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 12:03:11,280 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:03:13,281 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:03:15,282 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:03:17,283 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:03:19,284 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:03:21,285 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:03:23,286 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:03:23,591 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:03:23,591 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:03:25,287 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:03:27,288 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:03:29,289 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:03:31,290 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:03:33,291 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:03:35,292 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:03:37,293 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:03:38,720 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:03:38,720 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:03:39,294 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:03:40,201 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 12:03:41,295 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:03:43,296 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:03:45,297 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:03:47,298 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:03:49,299 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:03:51,300 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:03:53,301 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:03:53,852 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:03:53,852 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:03:55,302 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:03:57,303 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:03:59,304 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:04:01,305 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:04:03,306 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:04:05,307 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:04:07,308 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:04:08,985 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:04:08,986 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:04:09,309 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:04:10,279 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 12:04:11,310 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:04:13,311 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:04:15,312 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:04:17,313 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:04:20,314 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:04:22,315 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:04:24,119 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:04:24,120 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:04:24,316 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:04:26,317 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:04:28,318 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:04:30,319 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:04:32,320 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:04:34,321 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:04:36,322 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:04:38,323 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:04:39,266 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:04:39,266 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:04:40,324 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:04:40,357 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 12:04:42,325 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:04:44,326 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:04:46,327 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:04:48,328 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:04:50,329 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:04:52,330 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:04:54,331 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:04:54,399 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:04:54,400 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:04:56,332 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:04:58,333 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:05:00,334 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:05:02,335 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:05:04,336 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:05:06,337 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:05:08,339 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:05:09,530 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:05:09,530 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:05:10,339 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:05:10,438 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 12:05:12,340 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:05:14,342 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:05:16,343 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:05:18,344 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:05:20,345 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:05:22,346 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:05:24,347 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:05:24,661 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:05:24,661 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:05:26,348 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:05:28,349 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:05:30,350 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:05:32,351 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:05:34,352 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:05:36,353 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:05:38,354 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:05:39,803 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:05:39,804 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:05:40,355 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:05:40,517 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 12:05:42,356 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:05:44,357 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:05:46,358 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:05:48,359 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:05:50,360 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:05:52,361 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:05:54,362 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:05:54,994 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:05:54,995 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:05:56,363 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:05:58,364 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:06:00,365 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:06:02,366 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:06:04,366 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:06:06,368 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:06:08,368 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:06:10,133 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:06:10,133 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:06:10,369 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:06:10,594 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 12:06:12,370 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:06:14,371 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:06:16,372 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:06:18,373 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:06:20,374 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:06:22,374 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:06:24,375 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:06:25,266 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:06:25,267 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:06:26,376 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:06:28,377 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:06:30,377 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:06:32,378 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:06:34,379 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:06:36,380 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:06:38,381 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:06:40,381 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:06:40,397 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:06:40,397 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:06:40,671 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 12:06:42,382 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:06:44,383 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:06:46,384 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:06:48,384 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:06:50,385 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:06:55,527 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:06:55,527 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:07:10,658 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:07:10,658 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:07:10,787 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 12:07:11,468 DEBUG SenderThread:353216 [sender.py:send():179] send: history 2021-07-13 12:07:11,468 DEBUG SenderThread:353216 [sender.py:send():179] send: summary 2021-07-13 12:07:11,469 INFO SenderThread:353216 [sender.py:_save_file():841] saving file wandb-summary.json with policy end 2021-07-13 12:07:12,393 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/wandb-summary.json 2021-07-13 12:07:12,393 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:07:13,394 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:07:14,394 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:07:16,395 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:07:18,396 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:07:20,397 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:07:22,398 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:07:24,398 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:07:25,789 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:07:25,790 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:07:26,399 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:07:28,400 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:07:30,401 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:07:32,402 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:07:34,403 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:07:36,403 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:07:38,404 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:07:40,405 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:07:40,828 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 12:07:40,920 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:07:40,920 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:07:42,406 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:07:44,407 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:07:46,408 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:07:48,408 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:07:50,409 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:07:52,410 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:07:54,411 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:07:56,056 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:07:56,056 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:07:56,412 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:07:58,413 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:08:00,414 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:08:02,415 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:08:04,415 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:08:06,416 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:08:09,417 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:08:10,904 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 12:08:11,259 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:08:11,259 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:08:11,418 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:08:13,419 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:08:15,420 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:08:17,421 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:08:19,422 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:08:21,423 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:08:23,423 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:08:25,424 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:08:26,390 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:08:26,390 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:08:27,425 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:08:29,426 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:08:31,427 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:08:33,427 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:08:35,428 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:08:37,429 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:08:39,430 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:08:40,967 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 12:08:41,431 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:08:41,523 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:08:41,523 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:08:43,431 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:08:45,432 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:08:47,433 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:08:49,434 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:08:51,435 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:08:53,436 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:08:55,436 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:08:56,655 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:08:56,655 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:08:57,437 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:08:59,438 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:09:01,439 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:09:03,440 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:09:05,441 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:09:07,442 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:09:09,443 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:09:11,033 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 12:09:11,444 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:09:11,806 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:09:11,806 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:09:13,445 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:09:15,446 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:09:17,447 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:09:19,448 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:09:21,448 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:09:23,449 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:09:25,450 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:09:26,945 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:09:26,945 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:09:27,451 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:09:29,452 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:09:31,453 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:09:33,454 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:09:35,455 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:09:37,456 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:09:39,456 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:09:41,102 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 12:09:41,457 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:09:42,142 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:09:42,142 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:09:43,458 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:09:45,459 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:09:47,460 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:09:49,461 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:09:51,462 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:09:53,463 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:09:55,464 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:09:57,274 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:09:57,274 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:09:57,464 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:09:59,465 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:10:01,466 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:10:03,467 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:10:05,468 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:10:07,469 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:10:09,470 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:10:11,173 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 12:10:11,471 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:10:12,413 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:10:12,414 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:10:13,472 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:10:15,473 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:10:17,474 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:10:19,475 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:10:21,476 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:10:23,476 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:10:25,477 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:10:27,478 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:10:27,548 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:10:27,548 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:10:29,479 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:10:31,480 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:10:33,481 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:10:35,482 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:10:37,483 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:10:39,484 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:10:41,245 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 12:10:41,485 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:10:42,679 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:10:42,680 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:10:43,486 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:10:45,487 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:10:47,487 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:10:49,488 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:10:51,489 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:10:53,490 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:10:55,491 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:10:57,492 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:10:57,812 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:10:57,812 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:10:59,493 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:11:01,494 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:11:03,495 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:11:05,496 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:11:07,497 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:11:09,497 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:11:11,317 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 12:11:11,498 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:11:12,942 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:11:12,942 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:11:13,499 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:11:15,500 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:11:17,501 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:11:19,501 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:11:21,502 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:11:23,503 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:11:25,504 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:11:27,505 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:11:28,163 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:11:28,164 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:11:29,506 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:11:31,507 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:11:33,508 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:11:35,509 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:11:37,510 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:11:39,510 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:11:41,392 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 12:11:41,511 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:11:43,299 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:11:43,299 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:11:43,512 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:11:45,513 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:11:47,514 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:11:49,515 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:11:51,516 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:11:53,517 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:11:55,517 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:11:57,518 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:11:58,429 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:11:58,430 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:11:59,519 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:12:01,520 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:12:03,521 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:12:05,522 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:12:07,523 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:12:09,524 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:12:11,468 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 12:12:11,525 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:12:13,526 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:12:13,840 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:12:13,840 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:12:15,527 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:12:17,527 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:12:20,529 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:12:22,530 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:12:24,531 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:12:26,532 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:12:28,533 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:12:28,970 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:12:28,971 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:12:30,534 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:12:32,535 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:12:34,535 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:12:36,536 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:12:38,537 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:12:40,538 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:12:41,538 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 12:12:44,115 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:12:44,116 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:12:59,247 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:12:59,248 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:13:02,477 DEBUG SenderThread:353216 [sender.py:send():179] send: history 2021-07-13 12:13:02,477 DEBUG SenderThread:353216 [sender.py:send():179] send: summary 2021-07-13 12:13:02,478 INFO SenderThread:353216 [sender.py:_save_file():841] saving file wandb-summary.json with policy end 2021-07-13 12:13:02,549 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/wandb-summary.json 2021-07-13 12:13:03,549 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:13:04,550 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:13:06,551 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:13:08,552 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:13:10,552 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:13:11,616 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 12:13:12,553 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:13:14,473 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:13:14,474 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:13:14,554 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:13:16,555 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:13:18,556 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:13:20,557 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:13:22,558 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:13:24,559 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:13:26,560 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:13:28,561 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:13:29,632 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:13:29,632 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:13:30,561 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:13:32,562 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:13:34,563 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:13:36,564 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:13:38,565 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:13:40,565 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:13:41,684 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 12:13:42,566 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:13:44,567 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:13:44,764 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:13:44,764 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:13:46,568 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:13:48,569 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:13:50,570 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:13:52,571 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:13:54,572 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:13:56,573 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:13:58,574 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:13:59,895 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:13:59,896 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:14:00,575 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:14:02,576 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:14:04,577 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:14:06,577 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:14:08,578 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:14:10,579 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:14:11,756 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 12:14:12,580 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:14:14,581 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:14:15,058 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:14:15,058 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:14:16,581 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:14:18,582 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:14:20,583 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:14:22,584 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:14:24,585 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:14:26,585 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:14:28,586 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:14:30,189 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:14:30,190 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:14:30,587 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:14:32,588 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:14:34,589 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:14:36,589 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:14:38,590 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:14:40,591 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:14:41,831 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 12:14:42,592 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:14:44,593 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:14:45,328 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:14:45,328 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:14:46,593 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:14:48,594 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:14:50,595 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:14:52,596 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:14:54,597 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:14:56,598 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:14:58,598 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:15:00,465 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:15:00,465 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:15:00,599 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:15:02,600 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:15:04,601 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:15:06,601 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:15:08,602 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:15:10,603 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:15:11,905 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 12:15:12,604 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:15:14,605 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:15:15,598 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:15:15,598 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:15:16,606 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:15:18,606 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:15:20,607 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:15:22,608 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:15:24,609 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:15:26,610 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:15:28,611 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:15:30,611 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:15:30,729 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:15:30,730 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:15:32,612 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:15:34,613 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:15:36,613 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:15:38,614 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:15:40,615 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:15:41,980 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 12:15:42,616 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:15:44,616 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:15:45,872 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:15:45,872 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:15:46,617 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:15:48,618 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:15:50,619 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:15:52,620 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:15:54,620 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:15:56,621 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:15:58,622 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:16:00,623 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:16:01,008 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:16:01,008 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:16:02,623 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:16:04,624 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:16:06,625 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:16:08,626 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:16:10,626 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:16:12,058 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 12:16:12,627 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:16:14,628 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:16:16,148 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:16:16,148 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:16:16,629 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:16:19,630 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:16:21,631 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:16:23,632 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:16:25,633 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:16:27,633 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:16:29,634 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:16:31,287 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:16:31,287 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:16:31,635 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:16:33,636 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:16:35,637 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:16:37,637 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:16:39,638 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:16:41,639 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:16:42,135 DEBUG SenderThread:353216 [sender.py:send():179] send: stats 2021-07-13 12:16:43,640 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:16:45,640 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:16:46,418 DEBUG HandlerThread:353216 [handler.py:handle_request():124] handle_request: stop_status 2021-07-13 12:16:46,419 DEBUG SenderThread:353216 [sender.py:send_request():193] send_request: stop_status 2021-07-13 12:16:47,641 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:16:49,642 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:16:51,643 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:16:53,644 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:16:55,488 WARNING MainThread:353216 [internal.py:wandb_internal():147] Internal process interrupt: 1 2021-07-13 12:16:55,645 INFO Thread-8 :353216 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/files/output.log 2021-07-13 12:16:55,695 WARNING MainThread:353216 [internal.py:wandb_internal():147] Internal process interrupt: 2 2021-07-13 12:16:55,695 ERROR MainThread:353216 [internal.py:wandb_internal():150] Internal process interrupted. 2021-07-13 12:16:55,767 INFO SenderThread:353216 [sender.py:finish():945] shutting down sender 2021-07-13 12:16:55,767 INFO SenderThread:353216 [dir_watcher.py:finish():282] shutting down directory watcher 2021-07-13 12:16:55,768 INFO WriterThread:353216 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210713_113435-1vxvoloi/run-1vxvoloi.wandb 2021-07-13 12:16:55,768 INFO HandlerThread:353216 [handler.py:finish():638] shutting down handler 2021-07-13 12:16:55,990 INFO MainThread:353216 [internal.py:handle_exit():78] Internal process exited