Hugging Face's logo

flax-community
/
pino-bigbird-roberta-base

Fill-Mask
Transformers
PyTorch
JAX
TensorBoard
Dutch
big_bird
Inference Endpoints
Model card Files Metrics Community
2
pino-bigbird-roberta-base / wandb /run-20210714_204013-1ss5wuki /logs /debug-internal.log
dat
Saving weights and logs at step 1252
f291f93
raw history blame
No virus
122 kB
2021-07-14 20:40:13,861 INFO MainThread:571895 [internal.py:wandb_internal():88] W&B internal server running at pid: 571895, started at: 2021-07-14 20:40:13.861224
2021-07-14 20:40:13,863 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: check_version
2021-07-14 20:40:13,863 INFO WriterThread:571895 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/run-1ss5wuki.wandb
2021-07-14 20:40:13,864 DEBUG SenderThread:571895 [sender.py:send():179] send: header
2021-07-14 20:40:13,865 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: check_version
2021-07-14 20:40:13,901 DEBUG SenderThread:571895 [sender.py:send():179] send: run
2021-07-14 20:40:14,075 INFO SenderThread:571895 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files
2021-07-14 20:40:14,075 INFO SenderThread:571895 [sender.py:_start_run_threads():716] run started: 1ss5wuki with start time 1626295213
2021-07-14 20:40:14,075 DEBUG SenderThread:571895 [sender.py:send():179] send: summary
2021-07-14 20:40:14,075 INFO SenderThread:571895 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 20:40:14,076 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: run_start
2021-07-14 20:40:15,077 INFO Thread-8 :571895 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/wandb-summary.json
2021-07-14 20:40:15,202 DEBUG HandlerThread:571895 [meta.py:__init__():39] meta init
2021-07-14 20:40:15,202 DEBUG HandlerThread:571895 [meta.py:__init__():53] meta init done
2021-07-14 20:40:15,202 DEBUG HandlerThread:571895 [meta.py:probe():210] probe
2021-07-14 20:40:15,204 DEBUG HandlerThread:571895 [meta.py:_setup_git():200] setup git
2021-07-14 20:40:15,233 DEBUG HandlerThread:571895 [meta.py:_setup_git():207] setup git done
2021-07-14 20:40:15,233 DEBUG HandlerThread:571895 [meta.py:_save_pip():57] save pip
2021-07-14 20:40:15,234 DEBUG HandlerThread:571895 [meta.py:_save_pip():71] save pip done
2021-07-14 20:40:15,234 DEBUG HandlerThread:571895 [meta.py:probe():252] probe done
2021-07-14 20:40:15,237 DEBUG SenderThread:571895 [sender.py:send():179] send: files
2021-07-14 20:40:15,237 INFO SenderThread:571895 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-14 20:40:15,242 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:40:15,243 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:40:15,378 DEBUG SenderThread:571895 [sender.py:send():179] send: config
2021-07-14 20:40:15,378 DEBUG SenderThread:571895 [sender.py:send():179] send: config
2021-07-14 20:40:15,378 DEBUG SenderThread:571895 [sender.py:send():179] send: config
2021-07-14 20:40:15,680 INFO Thread-11 :571895 [upload_job.py:push():137] Uploaded file /tmp/tmpltn3tpoawandb/xaf39rs3-wandb-metadata.json
2021-07-14 20:40:16,076 INFO Thread-8 :571895 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:40:16,076 INFO Thread-8 :571895 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/requirements.txt
2021-07-14 20:40:16,076 INFO Thread-8 :571895 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/wandb-metadata.json
2021-07-14 20:40:30,082 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:40:30,380 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:40:30,380 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:40:36,084 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:40:43,279 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:40:45,088 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/config.yaml
2021-07-14 20:40:45,527 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:40:45,527 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:41:00,677 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:41:00,678 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:41:13,357 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:41:15,809 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:41:15,810 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:41:30,943 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:41:30,943 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:41:43,433 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:41:46,076 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:41:46,076 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:42:01,291 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:42:01,291 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:42:04,119 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:42:06,120 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:42:08,121 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:42:10,121 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:42:12,122 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:42:13,508 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:42:14,123 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:42:16,124 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:42:16,427 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:42:16,428 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:42:18,125 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:42:20,126 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:42:22,126 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:42:24,127 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:42:26,128 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:42:28,129 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:42:30,130 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:42:31,560 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:42:31,560 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:42:32,131 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:42:34,132 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:42:36,132 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:42:38,133 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:42:40,134 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:42:42,135 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:42:43,582 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:42:44,136 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:42:46,137 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:42:46,690 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:42:46,691 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:42:48,138 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:42:50,138 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:42:52,139 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:42:54,140 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:42:56,141 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:42:58,142 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:43:00,143 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:43:01,821 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:43:01,822 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:43:02,144 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:43:04,144 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:43:06,145 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:43:08,146 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:43:10,147 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:43:12,148 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:43:13,656 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:43:14,149 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:43:16,149 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:43:16,955 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:43:16,955 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:43:18,150 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:43:20,151 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:43:22,152 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:43:24,152 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:43:26,153 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:43:28,154 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:43:30,155 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:43:32,085 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:43:32,085 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:43:32,156 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:43:34,156 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:43:36,157 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:43:38,158 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:43:40,159 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:43:43,160 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:43:43,732 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:43:45,161 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:43:47,162 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:43:47,216 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:43:47,216 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:43:49,163 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:43:51,164 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:43:53,165 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:43:55,166 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:43:57,167 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:43:59,167 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:44:01,168 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:44:02,347 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:44:02,347 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:44:03,169 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:44:05,170 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:44:07,171 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:44:09,172 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:44:11,173 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:44:13,174 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:44:13,809 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:44:15,175 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:44:17,175 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:44:17,486 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:44:17,486 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:44:19,176 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:44:21,177 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:44:23,178 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:44:25,179 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:44:27,180 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:44:29,181 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:44:31,182 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:44:32,616 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:44:32,617 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:44:33,183 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:44:35,183 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:44:37,184 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:44:39,185 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:44:41,186 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:44:43,885 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:44:47,747 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:44:47,748 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:45:02,772 DEBUG SenderThread:571895 [sender.py:send():179] send: history
2021-07-14 20:45:02,773 DEBUG SenderThread:571895 [sender.py:send():179] send: summary
2021-07-14 20:45:02,773 INFO SenderThread:571895 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 20:45:02,892 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:45:02,892 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:45:03,195 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/wandb-summary.json
2021-07-14 20:45:05,196 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:45:07,197 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:45:09,198 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:45:11,199 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:45:13,200 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:45:13,963 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:45:15,201 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:45:17,202 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:45:18,023 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:45:18,023 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:45:19,203 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:45:21,204 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:45:23,205 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:45:25,205 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:45:27,206 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:45:29,207 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:45:31,208 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:45:33,158 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:45:33,158 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:45:33,209 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:45:35,210 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:45:37,211 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:45:39,212 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:45:41,213 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:45:43,214 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:45:44,039 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:45:45,215 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:45:47,215 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:45:48,289 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:45:48,290 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:45:49,216 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:45:51,217 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:45:53,218 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:45:55,219 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:45:57,220 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:45:59,221 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:46:01,222 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:46:03,223 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:46:03,423 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:46:03,423 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:46:05,224 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:46:07,225 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:46:09,225 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:46:11,226 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:46:13,227 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:46:14,113 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:46:15,228 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:46:17,229 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:46:18,562 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:46:18,562 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:46:19,230 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:46:21,231 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:46:23,232 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:46:25,232 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:46:27,233 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:46:29,234 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:46:31,235 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:46:33,236 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:46:33,700 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:46:33,701 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:46:35,237 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:46:37,238 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:46:39,238 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:46:41,239 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:46:43,240 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:46:44,182 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:46:45,241 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:46:47,242 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:46:49,017 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:46:49,018 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:46:49,243 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:46:51,244 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:46:53,245 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:46:55,246 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:46:57,246 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:46:59,247 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:47:01,248 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:47:03,249 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:47:04,147 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:47:04,148 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:47:05,250 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:47:07,251 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:47:09,251 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:47:11,252 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:47:13,253 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:47:14,252 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:47:15,254 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:47:17,255 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:47:19,256 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:47:19,281 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:47:19,282 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:47:21,256 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:47:23,257 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:47:25,258 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:47:27,259 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:47:29,260 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:47:31,261 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:47:33,261 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:47:34,416 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:47:34,417 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:47:35,262 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:47:37,263 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:47:39,264 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:47:41,264 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:47:44,324 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:47:49,548 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:47:49,548 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:48:03,384 DEBUG SenderThread:571895 [sender.py:send():179] send: history
2021-07-14 20:48:03,384 DEBUG SenderThread:571895 [sender.py:send():179] send: summary
2021-07-14 20:48:03,384 INFO SenderThread:571895 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 20:48:04,274 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/wandb-summary.json
2021-07-14 20:48:04,709 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:48:04,709 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:48:05,274 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:48:07,275 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:48:09,276 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:48:11,276 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:48:13,277 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:48:14,393 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:48:15,278 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:48:17,279 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:48:19,853 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:48:19,853 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:48:20,280 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:48:22,281 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:48:24,282 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:48:26,283 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:48:28,284 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:48:30,285 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:48:32,286 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:48:34,287 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:48:34,982 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:48:34,982 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:48:36,289 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:48:38,289 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:48:40,290 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:48:42,291 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:48:44,292 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:48:44,472 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:48:46,293 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:48:48,294 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:48:50,123 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:48:50,123 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:48:50,295 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:48:52,296 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:48:54,297 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:48:56,298 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:48:58,299 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:49:00,300 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:49:02,301 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:49:04,302 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:49:05,257 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:49:05,257 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:49:06,303 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:49:08,304 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:49:10,305 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:49:12,306 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:49:14,306 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:49:14,547 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:49:16,307 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:49:18,308 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:49:20,309 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:49:20,388 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:49:20,388 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:49:22,310 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:49:24,311 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:49:26,312 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:49:28,313 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:49:30,314 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:49:32,315 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:49:34,316 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:49:35,518 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:49:35,519 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:49:36,317 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:49:38,318 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:49:40,319 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:49:42,320 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:49:44,321 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:49:44,617 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:49:46,322 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:49:48,323 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:49:50,323 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:49:50,651 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:49:50,651 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:49:52,324 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:49:54,325 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:49:56,326 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:49:58,327 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:50:00,328 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:50:02,329 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:50:04,330 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:50:05,778 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:50:05,778 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:50:06,331 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:50:08,332 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:50:10,333 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:50:12,334 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:50:14,335 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:50:14,687 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:50:16,336 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:50:18,337 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:50:20,338 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:50:20,908 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:50:20,909 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:50:22,338 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:50:24,339 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:50:26,340 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:50:28,341 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:50:30,342 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:50:32,343 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:50:34,344 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:50:36,046 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:50:36,046 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:50:36,345 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:50:38,346 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:50:40,347 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:50:42,349 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:50:44,762 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:50:51,178 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:50:51,179 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:51:03,989 DEBUG SenderThread:571895 [sender.py:send():179] send: history
2021-07-14 20:51:03,989 DEBUG SenderThread:571895 [sender.py:send():179] send: summary
2021-07-14 20:51:03,989 INFO SenderThread:571895 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 20:51:04,358 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/wandb-summary.json
2021-07-14 20:51:05,358 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:51:06,319 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:51:06,320 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:51:06,359 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:51:08,359 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:51:10,360 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:51:12,361 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:51:14,362 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:51:14,836 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:51:16,363 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:51:18,364 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:51:20,365 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:51:21,506 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:51:21,506 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:51:22,366 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:51:24,367 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:51:26,368 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:51:28,368 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:51:30,369 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:51:32,370 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:51:34,371 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:51:36,372 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:51:36,646 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:51:36,646 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:51:38,373 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:51:40,374 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:51:42,374 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:51:44,375 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:51:44,913 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:51:46,376 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:51:48,377 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:51:50,378 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:51:51,781 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:51:51,782 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:51:52,379 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:51:54,380 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:51:56,381 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:51:58,382 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:52:00,382 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:52:02,383 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:52:04,384 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:52:06,385 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:52:07,036 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:52:07,037 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:52:08,386 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:52:10,387 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:52:12,388 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:52:14,389 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:52:14,990 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:52:16,390 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:52:18,390 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:52:20,391 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:52:22,168 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:52:22,168 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:52:22,392 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:52:24,393 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:52:26,394 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:52:28,394 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:52:30,395 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:52:32,396 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:52:34,397 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:52:36,398 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:52:37,299 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:52:37,299 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:52:38,399 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:52:40,400 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:52:42,401 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:52:44,402 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:52:45,064 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:52:46,403 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:52:48,403 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:52:50,404 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:52:52,405 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:52:52,429 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:52:52,430 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:52:54,406 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:52:56,407 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:52:58,407 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:53:00,408 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:53:02,409 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:53:04,410 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:53:06,411 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:53:07,562 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:53:07,562 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:53:08,412 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:53:10,413 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:53:12,413 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:53:14,414 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:53:15,139 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:53:16,415 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:53:18,416 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:53:20,417 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:53:22,418 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:53:22,691 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:53:22,691 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:53:24,418 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:53:26,419 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:53:28,420 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:53:30,421 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:53:32,422 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:53:34,422 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:53:36,423 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:53:37,827 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:53:37,828 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:53:38,424 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:53:40,425 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:53:42,426 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:53:45,218 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:53:52,960 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:53:52,961 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:54:04,598 DEBUG SenderThread:571895 [sender.py:send():179] send: history
2021-07-14 20:54:04,598 DEBUG SenderThread:571895 [sender.py:send():179] send: summary
2021-07-14 20:54:04,599 INFO SenderThread:571895 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 20:54:05,435 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:54:05,435 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/wandb-summary.json
2021-07-14 20:54:07,436 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:54:08,101 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:54:08,101 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:54:09,437 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:54:11,438 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:54:13,439 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:54:15,328 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:54:15,439 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:54:17,440 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:54:19,441 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:54:21,442 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:54:23,260 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:54:23,260 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:54:23,442 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:54:25,443 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:54:27,444 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:54:29,445 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:54:31,446 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:54:33,446 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:54:35,447 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:54:37,448 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:54:38,393 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:54:38,394 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:54:39,449 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:54:41,450 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:54:43,450 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:54:45,402 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:54:45,451 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:54:47,452 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:54:49,453 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:54:51,453 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:54:53,454 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:54:53,524 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:54:53,525 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:54:55,455 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:54:57,456 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:54:59,457 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:55:01,457 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:55:03,458 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:55:05,459 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:55:07,460 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:55:08,656 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:55:08,656 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:55:09,461 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:55:11,462 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:55:13,463 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:55:15,464 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:55:15,475 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:55:17,465 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:55:19,466 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:55:21,466 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:55:23,467 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:55:23,787 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:55:23,787 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:55:25,468 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:55:27,469 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:55:29,470 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:55:31,471 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:55:33,472 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:55:35,473 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:55:37,474 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:55:38,916 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:55:38,916 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:55:39,474 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:55:41,475 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:55:43,476 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:55:45,477 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:55:45,549 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:55:47,478 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:55:49,479 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:55:51,480 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:55:53,481 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:55:54,045 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:55:54,045 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:55:55,482 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:55:57,483 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:55:59,483 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:56:01,484 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:56:03,485 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:56:05,486 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:56:07,487 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:56:09,184 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:56:09,184 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:56:09,488 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:56:11,489 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:56:13,490 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:56:15,491 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:56:15,622 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:56:17,491 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:56:19,492 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:56:21,493 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:56:23,494 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:56:24,318 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:56:24,318 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:56:25,495 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:56:27,496 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:56:29,497 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:56:31,498 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:56:33,498 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:56:35,499 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:56:37,500 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:56:39,450 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:56:39,450 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:56:39,501 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:56:41,502 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:56:43,503 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:56:45,697 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:56:54,581 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:56:54,581 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:57:07,513 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:57:07,638 DEBUG SenderThread:571895 [sender.py:send():179] send: history
2021-07-14 20:57:07,639 DEBUG SenderThread:571895 [sender.py:send():179] send: summary
2021-07-14 20:57:07,639 INFO SenderThread:571895 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 20:57:08,514 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/wandb-summary.json
2021-07-14 20:57:09,514 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:57:09,723 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:57:09,723 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:57:10,514 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:57:11,515 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:57:13,515 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:57:15,516 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:57:15,770 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:57:17,517 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:57:19,518 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:57:21,519 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:57:23,520 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:57:24,917 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:57:24,918 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:57:25,521 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:57:27,521 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:57:29,522 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:57:31,523 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:57:33,524 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:57:35,525 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:57:37,526 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:57:39,527 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:57:40,062 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:57:40,062 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:57:41,527 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:57:43,528 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:57:45,529 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:57:45,840 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:57:47,530 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:57:49,531 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:57:51,532 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:57:53,532 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:57:55,199 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:57:55,199 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:57:55,534 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:57:57,534 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:57:59,535 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:58:01,536 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:58:03,537 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:58:05,538 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:58:07,539 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:58:09,539 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:58:10,399 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:58:10,400 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:58:11,540 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:58:13,541 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:58:15,542 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:58:15,914 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:58:17,542 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:58:19,543 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:58:21,544 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:58:23,545 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:58:25,546 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:58:25,548 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:58:25,549 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:58:27,547 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:58:29,548 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:58:31,549 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:58:34,550 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:58:36,551 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:58:38,552 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:58:40,553 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:58:40,686 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:58:40,687 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:58:42,554 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:58:44,555 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:58:45,989 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:58:46,556 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:58:48,557 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:58:50,558 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:58:52,558 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:58:54,559 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:58:55,822 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:58:55,823 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:58:56,560 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:58:58,561 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:59:00,562 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:59:02,563 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:59:04,564 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:59:06,565 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:59:08,566 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:59:10,566 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:59:10,961 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:59:10,961 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:59:12,567 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:59:14,568 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:59:16,065 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:59:16,569 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:59:18,570 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:59:20,571 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:59:22,571 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:59:24,572 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:59:26,105 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:59:26,106 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:59:26,573 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:59:28,574 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:59:30,575 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:59:32,576 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:59:34,577 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:59:36,578 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:59:38,579 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:59:40,580 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:59:41,239 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:59:41,240 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 20:59:42,581 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:59:44,582 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:59:46,141 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 20:59:46,583 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 20:59:56,377 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 20:59:56,378 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:00:08,249 DEBUG SenderThread:571895 [sender.py:send():179] send: history
2021-07-14 21:00:08,249 DEBUG SenderThread:571895 [sender.py:send():179] send: summary
2021-07-14 21:00:08,252 INFO SenderThread:571895 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 21:00:08,591 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/wandb-summary.json
2021-07-14 21:00:10,592 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 21:00:11,516 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:00:11,516 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:00:12,593 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 21:00:14,594 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 21:00:16,217 DEBUG SenderThread:571895 [sender.py:send():179] send: stats
2021-07-14 21:00:16,595 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 21:00:18,596 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 21:00:20,597 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 21:00:22,598 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 21:00:24,599 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 21:00:26,600 INFO Thread-8 :571895 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/files/output.log
2021-07-14 21:00:26,720 DEBUG HandlerThread:571895 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:00:26,720 DEBUG SenderThread:571895 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:00:27,548 WARNING MainThread:571895 [internal.py:wandb_internal():147] Internal process interrupt: 1
2021-07-14 21:00:27,730 WARNING MainThread:571895 [internal.py:wandb_internal():147] Internal process interrupt: 2
2021-07-14 21:00:27,730 ERROR MainThread:571895 [internal.py:wandb_internal():150] Internal process interrupted.
2021-07-14 21:00:27,856 INFO SenderThread:571895 [sender.py:finish():945] shutting down sender
2021-07-14 21:00:27,856 INFO SenderThread:571895 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-14 21:00:27,877 INFO WriterThread:571895 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_204013-1ss5wuki/run-1ss5wuki.wandb
2021-07-14 21:00:27,879 INFO HandlerThread:571895 [handler.py:finish():638] shutting down handler
2021-07-14 21:00:28,464 INFO MainThread:571895 [internal.py:handle_exit():78] Internal process exited