| 2021-07-13 13:54:22,213 INFO MainThread:370605 [internal.py:wandb_internal():88] W&B internal server running at pid: 370605, started at: 2021-07-13 13:54:22.213034 | |
| 2021-07-13 13:54:22,215 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: check_version | |
| 2021-07-13 13:54:22,215 INFO WriterThread:370605 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/run-328w5urh.wandb | |
| 2021-07-13 13:54:22,216 DEBUG SenderThread:370605 [sender.py:send():179] send: header | |
| 2021-07-13 13:54:22,216 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: check_version | |
| 2021-07-13 13:54:22,252 DEBUG SenderThread:370605 [sender.py:send():179] send: run | |
| 2021-07-13 13:54:22,436 INFO SenderThread:370605 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files | |
| 2021-07-13 13:54:22,436 INFO SenderThread:370605 [sender.py:_start_run_threads():716] run started: 328w5urh with start time 1626184461 | |
| 2021-07-13 13:54:22,436 DEBUG SenderThread:370605 [sender.py:send():179] send: summary | |
| 2021-07-13 13:54:22,436 INFO SenderThread:370605 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-13 13:54:22,436 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: run_start | |
| 2021-07-13 13:54:23,439 INFO Thread-8 :370605 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/wandb-summary.json | |
| 2021-07-13 13:54:23,510 DEBUG HandlerThread:370605 [meta.py:__init__():39] meta init | |
| 2021-07-13 13:54:23,510 DEBUG HandlerThread:370605 [meta.py:__init__():53] meta init done | |
| 2021-07-13 13:54:23,510 DEBUG HandlerThread:370605 [meta.py:probe():210] probe | |
| 2021-07-13 13:54:23,511 DEBUG HandlerThread:370605 [meta.py:_setup_git():200] setup git | |
| 2021-07-13 13:54:23,537 DEBUG HandlerThread:370605 [meta.py:_setup_git():207] setup git done | |
| 2021-07-13 13:54:23,537 DEBUG HandlerThread:370605 [meta.py:_save_pip():57] save pip | |
| 2021-07-13 13:54:23,538 DEBUG HandlerThread:370605 [meta.py:_save_pip():71] save pip done | |
| 2021-07-13 13:54:23,538 DEBUG HandlerThread:370605 [meta.py:probe():252] probe done | |
| 2021-07-13 13:54:23,541 DEBUG SenderThread:370605 [sender.py:send():179] send: files | |
| 2021-07-13 13:54:23,541 INFO SenderThread:370605 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now | |
| 2021-07-13 13:54:23,546 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 13:54:23,546 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 13:54:23,675 DEBUG SenderThread:370605 [sender.py:send():179] send: config | |
| 2021-07-13 13:54:23,676 DEBUG SenderThread:370605 [sender.py:send():179] send: config | |
| 2021-07-13 13:54:23,676 DEBUG SenderThread:370605 [sender.py:send():179] send: config | |
| 2021-07-13 13:54:24,012 INFO Thread-11 :370605 [upload_job.py:push():137] Uploaded file /tmp/tmph3r95lhcwandb/2rt0dekr-wandb-metadata.json | |
| 2021-07-13 13:54:24,437 INFO Thread-8 :370605 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/wandb-metadata.json | |
| 2021-07-13 13:54:24,438 INFO Thread-8 :370605 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/requirements.txt | |
| 2021-07-13 13:54:24,438 INFO Thread-8 :370605 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:54:38,678 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 13:54:38,678 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 13:54:40,443 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:54:51,587 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 13:54:53,448 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/config.yaml | |
| 2021-07-13 13:54:53,816 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 13:54:53,816 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 13:55:08,948 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 13:55:08,948 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 13:55:21,654 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 13:55:24,087 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 13:55:24,087 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 13:55:39,220 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 13:55:39,220 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 13:55:51,717 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 13:55:54,361 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 13:55:54,362 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 13:56:09,496 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 13:56:09,497 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 13:56:10,471 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:56:21,781 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 13:56:24,670 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 13:56:24,671 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 13:56:39,808 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 13:56:39,808 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 13:56:51,854 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 13:56:54,939 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 13:56:54,939 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 13:57:10,069 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 13:57:10,070 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 13:57:21,932 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 13:57:25,202 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 13:57:25,202 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 13:57:40,332 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 13:57:40,333 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 13:57:52,011 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 13:57:55,469 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 13:57:55,470 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 13:58:06,518 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:58:08,519 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:58:10,519 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:58:10,612 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 13:58:10,612 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 13:58:12,520 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:58:14,521 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:58:16,521 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:58:18,522 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:58:20,523 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:58:22,083 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 13:58:22,524 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:58:24,524 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:58:25,747 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 13:58:25,747 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 13:58:26,525 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:58:28,526 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:58:30,526 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:58:32,527 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:58:34,528 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:58:37,529 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:58:39,530 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:58:40,877 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 13:58:40,877 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 13:58:41,530 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:58:43,531 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:58:45,532 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:58:47,532 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:58:49,533 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:58:51,534 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:58:52,152 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 13:58:53,535 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:58:55,535 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:58:56,007 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 13:58:56,007 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 13:58:57,536 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:58:59,537 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:59:01,537 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:59:03,538 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:59:05,539 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:59:07,539 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:59:09,540 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:59:11,136 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 13:59:11,137 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 13:59:11,541 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:59:13,541 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:59:15,542 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:59:17,542 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:59:20,543 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:59:22,221 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 13:59:22,544 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:59:24,545 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:59:26,266 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 13:59:26,267 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 13:59:26,546 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:59:28,546 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:59:30,547 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:59:32,548 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:59:34,548 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:59:36,549 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:59:38,550 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:59:40,551 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:59:41,400 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 13:59:41,400 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 13:59:42,551 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:59:44,552 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:59:46,553 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:59:48,553 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:59:50,554 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:59:52,293 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 13:59:52,555 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:59:54,555 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:59:56,532 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 13:59:56,532 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 13:59:56,556 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 13:59:58,557 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:00:01,558 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:00:03,559 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:00:05,559 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:00:07,560 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:00:09,561 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:00:11,561 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:00:11,660 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:00:11,661 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:00:13,562 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:00:15,563 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:00:17,564 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:00:19,564 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:00:21,565 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:00:22,363 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:00:23,566 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:00:25,566 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:00:26,794 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:00:26,794 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:00:27,567 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:00:29,568 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:00:31,569 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:00:33,569 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:00:35,570 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:00:37,571 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:00:39,572 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:00:41,573 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:00:41,928 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:00:41,928 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:00:43,573 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:00:46,574 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:00:48,575 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:00:50,576 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:00:52,434 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:00:52,576 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:00:54,577 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:00:56,578 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:00:57,061 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:00:57,062 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:00:58,579 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:01:00,580 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:01:02,580 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:01:04,581 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:01:06,582 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:01:08,582 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:01:10,583 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:01:12,206 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:01:12,206 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:01:12,584 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:01:14,585 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:01:16,586 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:01:18,586 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:01:20,587 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:01:22,504 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:01:22,588 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:01:24,588 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:01:26,589 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:01:27,339 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:01:27,339 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:01:28,590 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:01:31,591 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:01:33,592 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:01:35,592 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:01:37,593 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:01:39,594 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:01:41,595 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:01:42,468 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:01:42,469 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:01:43,595 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:01:45,596 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:01:47,597 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:01:49,598 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:01:51,598 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:01:52,577 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:01:53,599 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:01:55,600 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:01:57,598 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:01:57,598 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:01:57,601 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:01:59,602 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:02:01,602 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:02:03,603 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:02:05,604 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:02:07,605 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:02:09,605 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:02:11,606 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:02:12,728 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:02:12,729 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:02:14,607 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:02:16,608 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:02:18,609 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:02:20,609 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:02:22,610 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:02:22,651 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:02:24,611 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:02:26,612 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:02:27,858 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:02:27,858 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:02:28,613 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:02:30,613 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:02:32,614 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:02:34,615 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:02:36,616 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:02:38,617 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:02:40,618 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:02:42,618 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:02:43,145 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:02:43,145 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:02:44,619 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:02:46,620 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:02:48,621 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:02:50,622 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:02:52,622 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:02:52,725 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:02:54,623 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:02:57,624 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:02:58,279 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:02:58,280 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:02:59,625 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:03:01,626 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:03:03,627 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:03:05,627 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:03:07,628 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:03:09,629 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:03:11,630 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:03:13,411 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:03:13,411 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:03:13,631 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:03:15,631 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:03:17,632 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:03:19,633 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:03:21,634 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:03:22,801 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:03:23,635 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:03:25,635 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:03:28,543 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:03:28,543 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:03:43,673 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:03:43,673 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:03:46,235 DEBUG SenderThread:370605 [sender.py:send():179] send: history | |
| 2021-07-13 14:03:46,235 DEBUG SenderThread:370605 [sender.py:send():179] send: summary | |
| 2021-07-13 14:03:46,235 INFO SenderThread:370605 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-13 14:03:46,642 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/wandb-summary.json | |
| 2021-07-13 14:03:46,643 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:03:47,643 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:03:49,643 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:03:51,644 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:03:52,875 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:03:53,645 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:03:55,645 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:03:57,646 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:03:58,802 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:03:58,802 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:03:59,647 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:04:01,648 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:04:03,648 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:04:05,649 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:04:08,650 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:04:10,651 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:04:12,652 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:04:13,932 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:04:13,932 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:04:14,652 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:04:16,653 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:04:18,654 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:04:20,655 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:04:22,655 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:04:22,949 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:04:24,656 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:04:26,657 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:04:28,657 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:04:29,063 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:04:29,063 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:04:30,658 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:04:32,659 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:04:34,660 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:04:36,661 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:04:38,661 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:04:40,662 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:04:42,662 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:04:44,196 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:04:44,196 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:04:44,663 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:04:46,664 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:04:48,665 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:04:50,665 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:04:53,025 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:04:53,666 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:04:55,667 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:04:57,668 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:04:59,327 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:04:59,328 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:04:59,669 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:05:01,669 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:05:03,670 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:05:05,671 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:05:07,671 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:05:09,672 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:05:11,673 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:05:13,674 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:05:14,535 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:05:14,535 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:05:15,674 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:05:17,675 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:05:19,676 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:05:21,677 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:05:23,098 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:05:23,678 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:05:25,678 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:05:27,679 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:05:29,665 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:05:29,666 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:05:29,680 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:05:31,680 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:05:33,681 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:05:36,682 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:05:38,683 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:05:40,684 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:05:42,685 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:05:44,685 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:05:44,800 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:05:44,800 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:05:46,686 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:05:48,687 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:05:50,687 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:05:52,688 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:05:53,173 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:05:54,689 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:05:56,690 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:05:58,690 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:05:59,932 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:05:59,932 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:06:00,691 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:06:02,692 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:06:04,692 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:06:06,693 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:06:08,694 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:06:10,695 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:06:12,695 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:06:14,696 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:06:15,064 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:06:15,064 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:06:16,697 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:06:19,698 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:06:21,698 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:06:23,249 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:06:23,699 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:06:25,700 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:06:27,700 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:06:29,701 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:06:30,205 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:06:30,205 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:06:31,702 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:06:33,702 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:06:35,703 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:06:37,704 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:06:39,705 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:06:41,706 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:06:43,706 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:06:45,343 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:06:45,343 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:06:45,707 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:06:47,708 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:06:49,709 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:06:51,709 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:06:53,323 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:06:53,710 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:06:55,711 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:06:57,712 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:06:59,712 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:07:00,473 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:07:00,473 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:07:02,714 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:07:04,715 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:07:06,715 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:07:08,716 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:07:10,717 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:07:12,718 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:07:14,718 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:07:15,603 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:07:15,604 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:07:16,719 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:07:18,720 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:07:20,720 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:07:22,721 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:07:23,401 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:07:24,722 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:07:26,723 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:07:28,724 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:07:30,724 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:07:30,745 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:07:30,746 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:07:32,725 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:07:34,726 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:07:36,726 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:07:38,727 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:07:40,728 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:07:42,729 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:07:45,730 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:07:45,880 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:07:45,880 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:07:47,731 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:07:49,731 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:07:51,732 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:07:53,478 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:07:53,733 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:07:55,734 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:07:57,735 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:07:59,735 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:08:01,026 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:08:01,026 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:08:01,736 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:08:03,737 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:08:05,738 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:08:07,738 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:08:09,739 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:08:11,740 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:08:13,741 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:08:15,742 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:08:16,156 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:08:16,157 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:08:17,742 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:08:19,743 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:08:21,744 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:08:23,554 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:08:23,745 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:08:25,746 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:08:27,746 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:08:30,748 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:08:31,288 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:08:31,288 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:08:32,748 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:08:34,749 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:08:36,750 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:08:38,751 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:08:40,752 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:08:42,753 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:08:44,753 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:08:46,419 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:08:46,420 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:08:46,754 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:08:48,755 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:08:50,756 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:08:52,757 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:08:53,629 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:08:54,757 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:08:56,758 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:08:58,759 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:09:00,760 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:09:01,622 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:09:01,623 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:09:02,761 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:09:04,761 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:09:06,762 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:09:16,754 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:09:16,754 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:09:23,706 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:09:28,234 DEBUG SenderThread:370605 [sender.py:send():179] send: history | |
| 2021-07-13 14:09:28,235 DEBUG SenderThread:370605 [sender.py:send():179] send: summary | |
| 2021-07-13 14:09:28,235 INFO SenderThread:370605 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-13 14:09:28,769 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/wandb-summary.json | |
| 2021-07-13 14:09:29,770 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:09:31,771 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:09:31,939 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:09:31,939 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:09:33,771 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:09:35,772 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:09:37,773 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:09:39,774 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:09:41,774 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:09:43,775 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:09:45,776 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:09:47,080 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:09:47,080 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:09:47,776 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:09:49,777 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:09:51,778 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:09:53,779 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:09:53,783 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:09:55,780 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:09:57,781 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:09:59,781 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:10:01,782 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:10:02,211 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:10:02,212 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:10:03,783 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:10:05,784 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:10:07,784 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:10:09,785 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:10:11,786 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:10:13,787 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:10:15,787 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:10:17,345 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:10:17,346 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:10:17,788 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:10:19,789 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:10:21,789 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:10:23,790 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:10:23,860 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:10:26,791 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:10:28,792 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:10:30,793 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:10:32,477 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:10:32,477 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:10:32,794 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:10:34,795 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:10:36,796 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:10:38,796 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:10:40,797 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:10:42,798 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:10:44,799 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:10:46,799 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:10:47,659 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:10:47,660 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:10:48,800 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:10:50,801 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:10:52,802 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:10:53,934 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:10:54,802 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:10:56,803 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:10:58,804 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:11:00,805 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:11:02,788 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:11:02,788 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:11:02,806 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:11:04,806 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:11:06,807 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:11:09,808 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:11:11,809 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:11:13,810 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:11:15,811 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:11:17,811 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:11:17,931 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:11:17,931 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:11:19,812 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:11:21,813 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:11:23,814 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:11:24,009 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:11:25,815 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:11:27,816 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:11:29,817 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:11:31,818 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:11:33,064 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:11:33,064 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:11:33,818 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:11:35,819 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:11:37,820 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:11:39,821 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:11:41,822 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:11:43,822 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:11:45,823 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:11:47,824 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:11:48,205 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:11:48,205 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:11:49,824 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:11:52,826 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:11:54,085 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:11:54,827 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:11:56,827 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:11:58,828 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:12:00,829 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:12:02,830 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:12:03,345 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:12:03,345 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:12:04,831 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:12:06,831 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:12:08,832 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:12:10,833 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:12:12,834 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:12:14,835 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:12:16,836 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:12:18,488 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:12:18,488 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:12:18,836 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:12:20,837 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:12:22,838 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:12:24,158 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:12:24,839 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:12:26,840 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:12:28,840 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:12:30,841 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:12:32,842 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:12:33,620 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:12:33,620 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:12:34,843 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:12:37,844 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:12:39,845 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:12:41,846 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:12:43,847 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:12:45,848 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:12:47,848 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:12:48,759 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:12:48,759 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:12:49,849 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:12:51,850 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:12:53,851 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:12:54,236 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:12:55,851 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:12:57,852 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:12:59,853 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:13:01,854 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:13:03,855 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:13:03,891 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:13:03,891 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:13:05,856 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:13:07,857 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:13:09,858 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:13:11,858 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:13:13,859 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:13:15,860 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:13:17,861 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:13:19,022 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:13:19,022 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:13:20,862 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:13:22,863 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:13:24,311 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:13:24,863 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:13:26,864 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:13:28,865 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:13:30,866 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:13:32,867 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:13:34,244 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:13:34,245 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:13:34,867 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:13:36,868 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:13:38,869 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:13:40,870 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:13:42,871 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:13:44,872 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:13:46,872 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:13:48,873 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:13:49,378 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:13:49,378 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:13:50,874 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:13:52,875 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:13:54,384 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:13:54,875 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:13:56,876 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:13:58,877 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:14:00,878 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:14:03,879 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:14:04,508 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:14:04,508 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:14:05,880 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:14:07,880 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:14:09,881 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:14:11,882 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:14:13,883 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:14:15,884 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:14:17,885 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:14:19,638 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:14:19,638 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:14:19,886 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:14:21,887 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:14:23,888 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:14:24,460 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:14:25,889 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:14:27,890 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:14:29,891 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:14:31,891 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:14:33,892 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:14:34,777 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:14:34,778 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:14:35,893 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:14:37,894 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:14:39,895 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:14:41,896 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:14:43,896 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:14:46,898 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:14:48,899 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:14:49,913 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:14:49,913 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:14:54,534 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:15:05,045 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:15:05,046 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:15:10,221 DEBUG SenderThread:370605 [sender.py:send():179] send: history | |
| 2021-07-13 14:15:10,221 DEBUG SenderThread:370605 [sender.py:send():179] send: summary | |
| 2021-07-13 14:15:10,222 INFO SenderThread:370605 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-13 14:15:10,906 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/wandb-summary.json | |
| 2021-07-13 14:15:10,906 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:15:12,907 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:15:14,908 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:15:16,909 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:15:18,909 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:15:20,176 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:15:20,177 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:15:20,910 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:15:22,911 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:15:24,610 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:15:24,912 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:15:26,913 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:15:28,914 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:15:30,914 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:15:32,915 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:15:34,916 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:15:35,306 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:15:35,307 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:15:36,917 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:15:38,918 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:15:40,919 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:15:42,919 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:15:44,920 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:15:46,921 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:15:48,922 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:15:50,435 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:15:50,436 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:15:50,923 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:15:52,923 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:15:54,684 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:15:54,924 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:15:56,925 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:15:59,926 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:16:01,927 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:16:03,928 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:16:05,566 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:16:05,566 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:16:05,929 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:16:07,930 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:16:09,930 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:16:11,931 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:16:13,932 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:16:15,933 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:16:17,934 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:16:19,934 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:16:20,776 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:16:20,776 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:16:21,935 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:16:23,936 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:16:24,756 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:16:25,937 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:16:27,938 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:16:29,939 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:16:31,940 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:16:33,941 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:16:35,908 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:16:35,908 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:16:35,942 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:16:37,943 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:16:39,944 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:16:41,945 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:16:44,946 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:16:46,947 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:16:48,948 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:16:50,948 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:16:51,042 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:16:51,042 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:16:52,949 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:16:54,831 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:16:54,950 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:16:56,951 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:16:58,952 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:17:00,953 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:17:02,954 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:17:04,955 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:17:06,176 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:17:06,176 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:17:06,956 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:17:08,956 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:17:10,957 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:17:12,958 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:17:14,959 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:17:16,960 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:17:18,961 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:17:20,962 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:17:21,314 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:17:21,314 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:17:22,963 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:17:24,905 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:17:24,964 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:17:27,965 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:17:29,966 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:17:31,966 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:17:33,967 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:17:35,968 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:17:36,444 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:17:36,445 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:17:37,969 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:17:39,970 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:17:41,971 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:17:43,971 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:17:45,972 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:17:47,973 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:17:49,974 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:17:51,585 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:17:51,585 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:17:51,975 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:17:53,975 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:17:54,978 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:17:55,976 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:17:57,977 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:17:59,978 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:18:01,979 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:18:03,980 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:18:05,980 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:18:06,717 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:18:06,717 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:18:07,981 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:18:10,983 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:18:12,983 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:18:14,984 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:18:16,985 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:18:18,986 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:18:20,987 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:18:21,846 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:18:21,846 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:18:22,988 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:18:24,989 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:18:25,050 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:18:26,989 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:18:28,990 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:18:30,991 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:18:32,992 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:18:34,992 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:18:36,979 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:18:36,979 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:18:36,993 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:18:38,994 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:18:40,995 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:18:42,996 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:18:44,997 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:18:46,998 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:18:48,998 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:18:50,999 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:18:52,115 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:18:52,115 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:18:54,001 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:18:55,126 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:18:56,002 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:18:58,002 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:19:00,003 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:19:02,004 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:19:04,005 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:19:06,006 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:19:07,245 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:19:07,245 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:19:08,007 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:19:10,008 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:19:12,009 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:19:14,010 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:19:16,010 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:19:18,011 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:19:20,012 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:19:22,013 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:19:22,390 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:19:22,391 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:19:24,014 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:19:25,201 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:19:26,015 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:19:28,016 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:19:30,016 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:19:32,017 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:19:34,018 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:19:37,019 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:19:37,523 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:19:37,523 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:19:39,020 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:19:41,021 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:19:43,022 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:19:45,022 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:19:47,023 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:19:49,024 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:19:51,025 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:19:52,656 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:19:52,656 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:19:53,026 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:19:55,027 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:19:55,275 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:19:57,027 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:19:59,028 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:20:01,029 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:20:03,030 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:20:05,030 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:20:07,031 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:20:07,785 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:20:07,785 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:20:09,032 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:20:11,033 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:20:13,034 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:20:15,035 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:20:17,035 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:20:20,037 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:20:22,038 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:20:22,916 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:20:22,916 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:20:24,038 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:20:25,347 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:20:26,039 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:20:28,040 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:20:30,041 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:20:32,041 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:20:38,058 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:20:38,058 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:20:52,218 DEBUG SenderThread:370605 [sender.py:send():179] send: history | |
| 2021-07-13 14:20:52,218 DEBUG SenderThread:370605 [sender.py:send():179] send: summary | |
| 2021-07-13 14:20:52,219 INFO SenderThread:370605 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-13 14:20:53,049 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/wandb-summary.json | |
| 2021-07-13 14:20:53,245 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:20:53,246 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:20:54,049 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:20:55,423 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:20:56,050 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:20:58,051 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:21:00,052 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:21:02,053 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:21:04,054 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:21:06,055 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:21:08,055 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:21:08,390 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:21:08,390 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:21:10,056 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:21:12,057 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:21:14,058 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:21:16,059 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:21:18,060 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:21:20,061 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:21:22,061 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:21:23,520 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:21:23,521 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:21:24,062 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:21:25,497 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:21:26,063 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:21:28,064 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:21:30,065 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:21:32,066 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:21:35,067 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:21:37,068 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:21:38,656 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:21:38,656 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:21:39,069 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:21:41,069 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:21:43,070 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:21:45,071 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:21:47,072 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:21:49,073 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:21:51,074 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:21:53,075 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:21:53,797 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:21:53,797 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:21:55,076 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:21:55,572 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:21:57,077 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:21:59,078 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:22:01,078 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:22:03,079 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:22:05,080 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:22:07,080 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:22:08,940 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:22:08,941 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:22:09,081 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:22:11,082 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:22:13,083 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:22:15,083 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:22:18,085 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:22:20,085 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:22:22,086 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:22:24,075 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:22:24,075 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:22:24,087 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:22:25,647 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:22:26,088 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:22:28,089 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:22:30,089 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:22:32,090 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:22:34,091 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:22:36,092 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:22:38,093 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:22:39,206 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:22:39,207 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:22:40,094 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:22:42,095 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:22:44,095 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:22:46,096 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:22:48,097 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:22:50,098 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:22:52,099 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:22:54,100 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:22:54,350 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:22:54,350 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:22:55,721 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:22:56,101 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:22:58,102 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:23:00,102 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:23:02,103 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:23:04,104 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:23:06,105 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:23:08,106 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:23:09,479 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:23:09,479 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:23:10,106 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:23:12,107 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:23:14,108 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:23:16,109 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:23:18,110 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:23:20,110 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:23:22,111 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:23:24,112 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:23:24,684 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:23:24,685 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:23:25,795 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:23:26,113 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:23:28,114 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:23:31,115 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:23:33,116 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:23:35,117 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:23:37,117 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:23:39,118 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:23:39,814 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:23:39,814 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:23:41,119 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:23:43,120 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:23:45,120 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:23:47,121 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:23:49,122 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:23:51,123 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:23:53,124 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:23:54,947 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:23:54,947 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:23:55,125 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:23:55,869 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:23:57,126 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:23:59,126 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:24:01,127 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:24:03,128 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:24:05,129 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:24:07,130 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:24:09,130 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:24:10,079 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:24:10,079 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:24:11,131 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:24:14,132 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:24:16,133 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:24:18,134 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:24:20,135 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:24:22,136 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:24:24,137 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:24:25,212 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:24:25,212 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:24:25,944 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:24:26,138 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:24:28,138 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:24:30,139 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:24:32,140 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:24:34,141 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:24:36,142 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:24:38,143 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:24:40,144 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:24:40,343 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:24:40,343 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:24:40,345 WARNING SenderThread:370605 [sender.py:send_request_stop_status():237] Failed to check stop requested status: Could not find a suitable TLS CA certificate bundle, invalid path: /home/dat/pino/lib/python3.8/site-packages/certifi/cacert.pem | |
| 2021-07-13 14:24:42,144 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:24:44,145 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:24:46,146 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:24:48,147 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:24:50,147 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:24:52,148 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:24:54,149 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:24:55,347 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:24:55,347 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:24:55,349 WARNING SenderThread:370605 [sender.py:send_request_stop_status():237] Failed to check stop requested status: Could not find a suitable TLS CA certificate bundle, invalid path: /home/dat/pino/lib/python3.8/site-packages/certifi/cacert.pem | |
| 2021-07-13 14:24:56,021 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:24:56,022 ERROR FileStreamThread:370605 [file_stream.py:_thread_except_body():311] generic exception in filestream thread | |
| Traceback (most recent call last): | |
| File "/home/dat/pino/lib/python3.8/site-packages/wandb/sdk/internal/file_stream.py", line 307, in _thread_except_body | |
| File "/home/dat/pino/lib/python3.8/site-packages/wandb/sdk/internal/file_stream.py", line 276, in _thread_body | |
| File "/home/dat/pino/lib/python3.8/site-packages/wandb/sdk/internal/file_stream.py", line 348, in _send | |
| File "/home/dat/pino/lib/python3.8/site-packages/wandb/sdk/internal/file_stream.py", line 418, in request_with_retry | |
| File "/home/dat/pino/lib/python3.8/site-packages/requests/sessions.py", line 590, in post | |
| File "/home/dat/pino/lib/python3.8/site-packages/requests/sessions.py", line 542, in request | |
| File "/home/dat/pino/lib/python3.8/site-packages/requests/sessions.py", line 655, in send | |
| File "/home/dat/pino/lib/python3.8/site-packages/requests/adapters.py", line 416, in send | |
| File "/home/dat/pino/lib/python3.8/site-packages/requests/adapters.py", line 227, in cert_verify | |
| OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /home/dat/pino/lib/python3.8/site-packages/certifi/cacert.pem | |
| 2021-07-13 14:24:57,151 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:24:59,152 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:25:01,152 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:25:03,153 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:25:05,154 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:25:07,155 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:25:09,156 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:25:10,351 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:25:10,351 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:25:10,353 WARNING SenderThread:370605 [sender.py:send_request_stop_status():237] Failed to check stop requested status: Could not find a suitable TLS CA certificate bundle, invalid path: /home/dat/pino/lib/python3.8/site-packages/certifi/cacert.pem | |
| 2021-07-13 14:25:11,157 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:25:13,157 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:25:15,158 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:25:17,159 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:25:19,160 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:25:21,161 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:25:23,162 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:25:25,163 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:25:25,354 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:25:25,355 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:25:25,356 WARNING SenderThread:370605 [sender.py:send_request_stop_status():237] Failed to check stop requested status: Could not find a suitable TLS CA certificate bundle, invalid path: /home/dat/pino/lib/python3.8/site-packages/certifi/cacert.pem | |
| 2021-07-13 14:25:26,095 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:25:27,164 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:25:29,164 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:25:31,165 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:25:33,166 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:25:35,167 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:25:37,168 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:25:40,169 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:25:40,358 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:25:40,358 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:25:40,360 WARNING SenderThread:370605 [sender.py:send_request_stop_status():237] Failed to check stop requested status: Could not find a suitable TLS CA certificate bundle, invalid path: /home/dat/pino/lib/python3.8/site-packages/certifi/cacert.pem | |
| 2021-07-13 14:25:42,170 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:25:44,171 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:25:46,171 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:25:48,172 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:25:50,173 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:25:52,174 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:25:54,174 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:25:55,362 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:25:55,362 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:25:55,364 WARNING SenderThread:370605 [sender.py:send_request_stop_status():237] Failed to check stop requested status: Could not find a suitable TLS CA certificate bundle, invalid path: /home/dat/pino/lib/python3.8/site-packages/certifi/cacert.pem | |
| 2021-07-13 14:25:56,168 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:25:56,175 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:25:58,176 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:26:00,177 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:26:02,178 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:26:04,179 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:26:06,180 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:26:08,181 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:26:10,182 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:26:10,365 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:26:10,366 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:26:10,367 WARNING SenderThread:370605 [sender.py:send_request_stop_status():237] Failed to check stop requested status: Could not find a suitable TLS CA certificate bundle, invalid path: /home/dat/pino/lib/python3.8/site-packages/certifi/cacert.pem | |
| 2021-07-13 14:26:12,182 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:26:25,369 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:26:25,369 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:26:25,371 WARNING SenderThread:370605 [sender.py:send_request_stop_status():237] Failed to check stop requested status: Could not find a suitable TLS CA certificate bundle, invalid path: /home/dat/pino/lib/python3.8/site-packages/certifi/cacert.pem | |
| 2021-07-13 14:26:26,240 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:26:34,200 DEBUG SenderThread:370605 [sender.py:send():179] send: history | |
| 2021-07-13 14:26:34,200 DEBUG SenderThread:370605 [sender.py:send():179] send: summary | |
| 2021-07-13 14:26:34,200 INFO SenderThread:370605 [sender.py:_save_file():841] saving file wandb-summary.json with policy end | |
| 2021-07-13 14:26:35,191 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/wandb-summary.json | |
| 2021-07-13 14:26:36,191 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:26:38,192 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:26:40,193 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:26:40,372 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:26:40,372 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:26:40,374 WARNING SenderThread:370605 [sender.py:send_request_stop_status():237] Failed to check stop requested status: Could not find a suitable TLS CA certificate bundle, invalid path: /home/dat/pino/lib/python3.8/site-packages/certifi/cacert.pem | |
| 2021-07-13 14:26:42,194 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:26:44,194 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:26:46,195 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:26:48,196 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:26:50,197 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:26:52,198 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:26:55,199 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:26:55,375 DEBUG HandlerThread:370605 [handler.py:handle_request():124] handle_request: stop_status | |
| 2021-07-13 14:26:55,375 DEBUG SenderThread:370605 [sender.py:send_request():193] send_request: stop_status | |
| 2021-07-13 14:26:55,377 WARNING SenderThread:370605 [sender.py:send_request_stop_status():237] Failed to check stop requested status: Could not find a suitable TLS CA certificate bundle, invalid path: /home/dat/pino/lib/python3.8/site-packages/certifi/cacert.pem | |
| 2021-07-13 14:26:56,313 DEBUG SenderThread:370605 [sender.py:send():179] send: stats | |
| 2021-07-13 14:26:57,200 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:26:59,201 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:27:01,202 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:27:03,202 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:27:05,203 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:27:07,204 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:27:09,205 INFO Thread-8 :370605 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log | |
| 2021-07-13 14:27:09,826 WARNING MainThread:370605 [internal.py:wandb_internal():147] Internal process interrupt: 1 | |
| 2021-07-13 14:27:09,994 WARNING MainThread:370605 [internal.py:wandb_internal():147] Internal process interrupt: 2 | |
| 2021-07-13 14:27:09,994 ERROR MainThread:370605 [internal.py:wandb_internal():150] Internal process interrupted. | |
| 2021-07-13 14:27:10,569 INFO SenderThread:370605 [sender.py:finish():945] shutting down sender | |
| 2021-07-13 14:27:10,569 INFO SenderThread:370605 [dir_watcher.py:finish():282] shutting down directory watcher | |
| 2021-07-13 14:27:10,569 INFO WriterThread:370605 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/run-328w5urh.wandb | |
| 2021-07-13 14:27:10,571 INFO HandlerThread:370605 [handler.py:finish():638] shutting down handler | |
| 2021-07-13 14:27:11,207 INFO SenderThread:370605 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files | |
| 2021-07-13 14:27:11,207 INFO SenderThread:370605 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/requirements.txt requirements.txt | |
| 2021-07-13 14:27:11,207 INFO SenderThread:370605 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/output.log output.log | |
| 2021-07-13 14:27:11,207 INFO SenderThread:370605 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/wandb-metadata.json wandb-metadata.json | |
| 2021-07-13 14:27:11,207 INFO SenderThread:370605 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/config.yaml config.yaml | |
| 2021-07-13 14:27:11,207 INFO SenderThread:370605 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_135421-328w5urh/files/wandb-summary.json wandb-summary.json | |
| 2021-07-13 14:27:11,208 INFO SenderThread:370605 [file_pusher.py:finish():177] shutting down file pusher | |
| 2021-07-13 14:27:11,208 INFO SenderThread:370605 [file_pusher.py:join():182] waiting for file pusher | |
| 2021-07-13 14:27:11,709 ERROR SenderThread:370605 [file_stream.py:finish():395] FileStream exception | |
| Traceback (most recent call last): | |
| File "/home/dat/pino/lib/python3.8/site-packages/wandb/sdk/internal/file_stream.py", line 307, in _thread_except_body | |
| File "/home/dat/pino/lib/python3.8/site-packages/wandb/sdk/internal/file_stream.py", line 276, in _thread_body | |
| File "/home/dat/pino/lib/python3.8/site-packages/wandb/sdk/internal/file_stream.py", line 348, in _send | |
| File "/home/dat/pino/lib/python3.8/site-packages/wandb/sdk/internal/file_stream.py", line 418, in request_with_retry | |
| File "/home/dat/pino/lib/python3.8/site-packages/requests/sessions.py", line 590, in post | |
| File "/home/dat/pino/lib/python3.8/site-packages/requests/sessions.py", line 542, in request | |
| File "/home/dat/pino/lib/python3.8/site-packages/requests/sessions.py", line 655, in send | |
| File "/home/dat/pino/lib/python3.8/site-packages/requests/adapters.py", line 416, in send | |
| File "/home/dat/pino/lib/python3.8/site-packages/requests/adapters.py", line 227, in cert_verify | |
| OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /home/dat/pino/lib/python3.8/site-packages/certifi/cacert.pem | |
| 2021-07-13 14:27:11,710 ERROR MainThread:370605 [internal.py:wandb_internal():159] Thread SenderThread: | |
| Traceback (most recent call last): | |
| File "/home/dat/pino/lib/python3.8/site-packages/wandb/sdk/internal/internal_util.py", line 55, in run | |
| File "/home/dat/pino/lib/python3.8/site-packages/wandb/sdk/internal/internal_util.py", line 106, in _run | |
| File "/home/dat/pino/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 295, in _finish | |
| File "/home/dat/pino/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 956, in finish | |
| File "/home/dat/pino/lib/python3.8/site-packages/wandb/sdk/internal/file_stream.py", line 397, in finish | |
| File "/home/dat/pino/lib/python3.8/site-packages/six.py", line 702, in reraise | |
| File "/home/dat/pino/lib/python3.8/site-packages/wandb/sdk/internal/file_stream.py", line 307, in _thread_except_body | |
| File "/home/dat/pino/lib/python3.8/site-packages/wandb/sdk/internal/file_stream.py", line 276, in _thread_body | |
| File "/home/dat/pino/lib/python3.8/site-packages/wandb/sdk/internal/file_stream.py", line 348, in _send | |
| File "/home/dat/pino/lib/python3.8/site-packages/wandb/sdk/internal/file_stream.py", line 418, in request_with_retry | |
| File "/home/dat/pino/lib/python3.8/site-packages/requests/sessions.py", line 590, in post | |
| File "/home/dat/pino/lib/python3.8/site-packages/requests/sessions.py", line 542, in request | |
| File "/home/dat/pino/lib/python3.8/site-packages/requests/sessions.py", line 655, in send | |
| File "/home/dat/pino/lib/python3.8/site-packages/requests/adapters.py", line 416, in send | |
| File "/home/dat/pino/lib/python3.8/site-packages/requests/adapters.py", line 227, in cert_verify | |
| OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /home/dat/pino/lib/python3.8/site-packages/certifi/cacert.pem | |
| 2021-07-13 14:27:13,716 INFO MainThread:370605 [internal.py:handle_exit():78] Internal process exited | |