| 2021-07-14 19:48:02,284 INFO MainThread:562187 [internal.py:wandb_internal():88] W&B internal server running at pid: 562187, started at: 2021-07-14 19:48:02.284377 | |
| 2021-07-14 19:48:02,286 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: check_version | |
| 2021-07-14 19:48:02,287 INFO WriterThread:562187 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/run-35ol5oes.wandb | |
| 2021-07-14 19:48:02,287 DEBUG SenderThread:562187 [sender.py:send():179] send: header | |
| 2021-07-14 19:48:02,288 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: check_version | |
| 2021-07-14 19:48:02,326 DEBUG SenderThread:562187 [sender.py:send():179] send: run | |
| 2021-07-14 19:48:02,511 INFO SenderThread:562187 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files | |
| 2021-07-14 19:48:02,511 INFO SenderThread:562187 [sender.py:_start_run_threads():716] run started: 35ol5oes with start time 1626292081 | |
| 2021-07-14 19:48:02,511 DEBUG SenderThread:562187 [sender.py:send():179] send: summary | |
| 2021-07-14 19:48:02,511 INFO SenderThread:562187 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-14 19:48:02,512 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: run_start | |
| 2021-07-14 19:48:03,515 INFO Thread-8 :562187 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/wandb-summary.json | |
| 2021-07-14 19:48:03,596 DEBUG HandlerThread:562187 [meta.py:__init__():39] meta init | |
| 2021-07-14 19:48:03,596 DEBUG HandlerThread:562187 [meta.py:__init__():53] meta init done | |
| 2021-07-14 19:48:03,596 DEBUG HandlerThread:562187 [meta.py:probe():210] probe | |
| 2021-07-14 19:48:03,598 DEBUG HandlerThread:562187 [meta.py:_setup_git():200] setup git | |
| 2021-07-14 19:48:03,626 DEBUG HandlerThread:562187 [meta.py:_setup_git():207] setup git done | |
| 2021-07-14 19:48:03,626 DEBUG HandlerThread:562187 [meta.py:_save_pip():57] save pip | |
| 2021-07-14 19:48:03,627 DEBUG HandlerThread:562187 [meta.py:_save_pip():71] save pip done | |
| 2021-07-14 19:48:03,627 DEBUG HandlerThread:562187 [meta.py:probe():252] probe done | |
| 2021-07-14 19:48:03,630 DEBUG SenderThread:562187 [sender.py:send():179] send: files | |
| 2021-07-14 19:48:03,630 INFO SenderThread:562187 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now | |
| 2021-07-14 19:48:03,636 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:48:03,636 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:48:03,766 DEBUG SenderThread:562187 [sender.py:send():179] send: config | |
| 2021-07-14 19:48:03,767 DEBUG SenderThread:562187 [sender.py:send():179] send: config | |
| 2021-07-14 19:48:03,767 DEBUG SenderThread:562187 [sender.py:send():179] send: config | |
| 2021-07-14 19:48:04,089 INFO Thread-11 :562187 [upload_job.py:push():137] Uploaded file /tmp/tmp3z0bl71kwandb/1glaoo69-wandb-metadata.json | |
| 2021-07-14 19:48:04,514 INFO Thread-8 :562187 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/requirements.txt | |
| 2021-07-14 19:48:04,515 INFO Thread-8 :562187 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/wandb-metadata.json | |
| 2021-07-14 19:48:04,515 INFO Thread-8 :562187 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:48:18,768 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:48:18,769 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:48:22,522 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:48:26,524 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:48:31,679 DEBUG SenderThread:562187 [sender.py:send():179] send: stats | |
| 2021-07-14 19:48:33,527 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/config.yaml | |
| 2021-07-14 19:48:33,914 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:48:33,915 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:48:49,071 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:48:49,071 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:49:01,760 DEBUG SenderThread:562187 [sender.py:send():179] send: stats | |
| 2021-07-14 19:49:04,205 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:49:04,205 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:49:19,336 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:49:19,337 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:49:31,840 DEBUG SenderThread:562187 [sender.py:send():179] send: stats | |
| 2021-07-14 19:49:34,471 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:49:34,472 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:49:49,607 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:49:49,607 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:50:01,917 DEBUG SenderThread:562187 [sender.py:send():179] send: stats | |
| 2021-07-14 19:50:04,746 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:50:04,746 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:50:12,565 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:50:14,566 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:50:16,566 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:50:18,567 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:50:19,933 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:50:19,933 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:50:20,568 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:50:22,568 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:50:24,569 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:50:26,570 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:50:28,571 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:50:30,571 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:50:31,989 DEBUG SenderThread:562187 [sender.py:send():179] send: stats | |
| 2021-07-14 19:50:32,572 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:50:34,573 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:50:35,105 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:50:35,105 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:50:36,574 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:50:38,574 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:50:40,575 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:50:42,576 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:50:44,577 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:50:46,577 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:50:49,578 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:50:50,335 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:50:50,335 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:50:51,579 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:50:53,580 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:50:55,581 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:50:57,582 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:50:59,582 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:51:01,583 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:51:02,061 DEBUG SenderThread:562187 [sender.py:send():179] send: stats | |
| 2021-07-14 19:51:03,584 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:51:05,545 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:51:05,545 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:51:05,585 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:51:07,585 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:51:09,586 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:51:11,587 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:51:13,588 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:51:15,588 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:51:17,589 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:51:19,590 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:51:20,681 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:51:20,681 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:51:21,591 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:51:23,592 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:51:25,592 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:51:27,593 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:51:29,594 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:51:31,595 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:51:32,130 DEBUG SenderThread:562187 [sender.py:send():179] send: stats | |
| 2021-07-14 19:51:33,596 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:51:35,597 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:51:35,814 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:51:35,814 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:51:37,598 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:51:39,598 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:51:41,599 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:51:43,600 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:51:45,601 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:51:47,602 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:51:49,603 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:51:51,016 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:51:51,016 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:51:51,604 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:51:53,604 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:51:55,605 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:51:57,606 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:51:59,607 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:52:01,608 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:52:02,201 DEBUG SenderThread:562187 [sender.py:send():179] send: stats | |
| 2021-07-14 19:52:03,609 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:52:05,610 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:52:06,662 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:52:06,662 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:52:07,610 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:52:09,611 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:52:11,612 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:52:13,613 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:52:15,614 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:52:17,615 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:52:19,616 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:52:21,616 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:52:21,794 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:52:21,795 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:52:23,617 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:52:25,618 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:52:27,619 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:52:29,620 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:52:31,621 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:52:32,271 DEBUG SenderThread:562187 [sender.py:send():179] send: stats | |
| 2021-07-14 19:52:33,621 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:52:35,622 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:52:36,983 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:52:36,983 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:52:37,623 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:52:39,624 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:52:41,625 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:52:43,626 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:52:45,626 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:52:47,627 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:52:49,628 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:52:51,629 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:52:52,116 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:52:52,116 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:52:53,630 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:52:55,631 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:52:57,631 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:52:59,632 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:53:01,633 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:53:02,363 DEBUG SenderThread:562187 [sender.py:send():179] send: stats | |
| 2021-07-14 19:53:03,634 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:53:05,635 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:53:07,249 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:53:07,249 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:53:07,635 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:53:09,636 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:53:11,637 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:53:13,638 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:53:15,639 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:53:17,639 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:53:19,640 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:53:21,641 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:53:22,471 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:53:22,471 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:53:23,642 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:53:25,643 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:53:27,643 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:53:29,644 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:53:32,421 DEBUG SenderThread:562187 [sender.py:send():179] send: stats | |
| 2021-07-14 19:53:32,645 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:53:34,646 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:53:36,647 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:53:37,603 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:53:37,603 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:53:38,648 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:53:40,649 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:53:42,650 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:53:44,650 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:53:46,651 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:53:48,652 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:53:50,653 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:53:52,654 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:53:52,734 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:53:52,735 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:53:54,655 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:53:56,656 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:53:58,656 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:54:00,657 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:54:02,498 DEBUG SenderThread:562187 [sender.py:send():179] send: stats | |
| 2021-07-14 19:54:02,658 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:54:04,659 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:54:06,660 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:54:07,935 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:54:07,936 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:54:08,661 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:54:10,662 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:54:12,662 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:54:14,663 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:54:16,664 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:54:18,665 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:54:20,665 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:54:22,666 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:54:23,067 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:54:23,067 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:54:24,667 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:54:26,668 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:54:28,669 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:54:30,669 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:54:32,587 DEBUG SenderThread:562187 [sender.py:send():179] send: stats | |
| 2021-07-14 19:54:32,670 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:54:34,671 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:54:36,672 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:54:38,197 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:54:38,198 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:54:38,672 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:54:40,673 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:54:53,388 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:54:53,388 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:54:58,979 DEBUG SenderThread:562187 [sender.py:send():179] send: history | |
| 2021-07-14 19:54:58,980 DEBUG SenderThread:562187 [sender.py:send():179] send: summary | |
| 2021-07-14 19:54:58,982 INFO SenderThread:562187 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-14 19:54:59,680 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/wandb-summary.json | |
| 2021-07-14 19:55:00,680 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:55:01,681 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:55:02,681 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:55:02,698 DEBUG SenderThread:562187 [sender.py:send():179] send: stats | |
| 2021-07-14 19:55:04,682 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:55:06,683 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:55:08,520 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:55:08,520 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:55:08,684 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:55:10,685 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:55:12,686 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:55:14,686 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:55:16,687 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:55:18,688 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:55:20,689 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:55:22,690 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:55:23,651 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:55:23,652 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:55:24,690 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:55:26,691 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:55:28,692 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:55:30,693 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:55:32,694 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:55:32,807 DEBUG SenderThread:562187 [sender.py:send():179] send: stats | |
| 2021-07-14 19:55:34,694 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:55:36,695 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:55:38,696 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:55:38,783 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:55:38,784 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:55:40,697 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:55:42,698 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:55:44,699 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:55:46,700 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:55:48,701 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:55:50,702 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:55:52,702 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:55:53,947 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:55:53,947 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:55:54,703 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:55:56,704 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:55:58,705 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:56:00,705 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:56:02,706 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:56:02,885 DEBUG SenderThread:562187 [sender.py:send():179] send: stats | |
| 2021-07-14 19:56:04,707 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:56:06,708 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:56:08,709 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:56:09,079 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:56:09,079 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:56:10,710 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:56:12,710 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:56:14,711 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:56:16,712 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:56:18,713 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:56:20,713 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:56:22,714 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:56:24,213 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:56:24,214 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:56:24,715 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:56:26,716 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:56:28,717 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:56:31,717 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:56:32,963 DEBUG SenderThread:562187 [sender.py:send():179] send: stats | |
| 2021-07-14 19:56:33,718 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:56:35,719 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:56:37,720 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:56:39,402 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:56:39,402 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:56:39,721 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:56:41,722 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:56:43,723 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:56:45,723 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:56:47,724 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:56:49,725 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:56:51,726 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:56:53,727 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:56:54,535 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:56:54,535 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:56:55,728 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:56:57,728 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:56:59,729 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:57:01,730 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:57:03,040 DEBUG SenderThread:562187 [sender.py:send():179] send: stats | |
| 2021-07-14 19:57:03,731 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:57:05,732 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:57:07,732 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:57:09,667 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:57:09,668 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:57:09,733 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:57:11,734 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:57:13,735 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:57:15,736 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:57:17,737 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:57:19,738 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:57:21,738 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:57:23,739 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:57:24,868 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:57:24,869 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:57:25,740 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:57:27,741 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:57:29,742 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:57:31,742 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:57:33,119 DEBUG SenderThread:562187 [sender.py:send():179] send: stats | |
| 2021-07-14 19:57:33,743 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:57:35,744 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:57:37,745 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:57:39,746 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:57:40,034 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:57:40,035 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:57:41,747 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:57:43,748 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:57:45,748 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:57:47,749 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:57:49,750 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:57:51,751 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:57:53,752 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:57:55,168 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:57:55,169 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:57:55,753 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:57:57,754 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:57:59,754 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:58:01,755 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:58:03,197 DEBUG SenderThread:562187 [sender.py:send():179] send: stats | |
| 2021-07-14 19:58:03,756 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:58:05,757 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:58:07,758 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:58:09,759 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:58:10,343 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:58:10,343 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:58:11,760 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:58:13,760 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:58:15,761 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:58:17,762 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:58:19,763 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:58:21,764 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:58:23,765 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:58:25,474 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:58:25,474 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:58:25,765 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:58:27,766 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:58:29,767 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:58:31,768 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:58:33,273 DEBUG SenderThread:562187 [sender.py:send():179] send: stats | |
| 2021-07-14 19:58:33,768 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:58:35,769 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:58:37,770 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:58:39,771 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:58:40,607 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:58:40,608 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:58:41,772 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:58:43,773 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:58:45,774 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:58:47,774 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:58:49,775 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:58:51,776 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:58:53,777 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:58:55,778 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:58:55,784 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:58:55,785 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:58:57,779 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:58:59,779 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:59:01,780 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:59:03,352 DEBUG SenderThread:562187 [sender.py:send():179] send: stats | |
| 2021-07-14 19:59:04,781 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:59:06,782 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:59:08,783 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:59:10,784 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:59:10,915 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:59:10,916 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:59:12,785 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:59:14,786 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:59:16,787 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:59:18,788 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:59:20,789 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:59:22,790 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:59:24,791 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:59:26,047 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:59:26,047 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:59:26,791 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:59:28,792 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:59:30,793 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:59:33,430 DEBUG SenderThread:562187 [sender.py:send():179] send: stats | |
| 2021-07-14 19:59:41,212 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:59:41,212 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:59:47,400 DEBUG SenderThread:562187 [sender.py:send():179] send: history | |
| 2021-07-14 19:59:47,400 DEBUG SenderThread:562187 [sender.py:send():179] send: summary | |
| 2021-07-14 19:59:47,401 INFO SenderThread:562187 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-14 19:59:47,800 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/wandb-summary.json | |
| 2021-07-14 19:59:48,800 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:59:49,801 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:59:50,801 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:59:52,802 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:59:54,803 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:59:56,346 DEBUG HandlerThread:562187 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-14 19:59:56,346 DEBUG SenderThread:562187 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-14 19:59:56,804 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:59:58,365 WARNING MainThread:562187 [internal.py:wandb_internal():147] Internal process interrupt: 1 | |
| 2021-07-14 19:59:58,805 INFO Thread-8 :562187 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/files/output.log | |
| 2021-07-14 19:59:59,281 WARNING MainThread:562187 [internal.py:wandb_internal():147] Internal process interrupt: 2 | |
| 2021-07-14 19:59:59,281 ERROR MainThread:562187 [internal.py:wandb_internal():150] Internal process interrupted. | |
| 2021-07-14 20:00:00,010 INFO SenderThread:562187 [sender.py:finish():945] shutting down sender | |
| 2021-07-14 20:00:00,010 INFO WriterThread:562187 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_194801-35ol5oes/run-35ol5oes.wandb | |
| 2021-07-14 20:00:00,010 INFO SenderThread:562187 [dir_watcher.py:finish():282] shutting down directory watcher | |
| 2021-07-14 20:00:00,011 INFO HandlerThread:562187 [handler.py:finish():638] shutting down handler | |
| 2021-07-14 20:00:00,345 INFO MainThread:562187 [internal.py:handle_exit():78] Internal process exited | |