2021-07-15 00:52:29,330 INFO    MainThread:624595 [internal.py:wandb_internal():88] W&B internal server running at pid: 624595, started at: 2021-07-15 00:52:29.329937
2021-07-15 00:52:29,332 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: check_version
2021-07-15 00:52:29,333 INFO    WriterThread:624595 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/run-361cvuow.wandb
2021-07-15 00:52:29,333 DEBUG   SenderThread:624595 [sender.py:send():179] send: header
2021-07-15 00:52:29,333 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: check_version
2021-07-15 00:52:29,368 DEBUG   SenderThread:624595 [sender.py:send():179] send: run
2021-07-15 00:52:29,550 INFO    SenderThread:624595 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files
2021-07-15 00:52:29,550 INFO    SenderThread:624595 [sender.py:_start_run_threads():716] run started: 361cvuow with start time 1626310348
2021-07-15 00:52:29,550 DEBUG   SenderThread:624595 [sender.py:send():179] send: summary
2021-07-15 00:52:29,551 INFO    SenderThread:624595 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-15 00:52:29,551 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: run_start
2021-07-15 00:52:30,553 INFO    Thread-8  :624595 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/wandb-summary.json
2021-07-15 00:52:30,640 DEBUG   HandlerThread:624595 [meta.py:__init__():39] meta init
2021-07-15 00:52:30,640 DEBUG   HandlerThread:624595 [meta.py:__init__():53] meta init done
2021-07-15 00:52:30,640 DEBUG   HandlerThread:624595 [meta.py:probe():210] probe
2021-07-15 00:52:30,641 DEBUG   HandlerThread:624595 [meta.py:_setup_git():200] setup git
2021-07-15 00:52:30,670 DEBUG   HandlerThread:624595 [meta.py:_setup_git():207] setup git done
2021-07-15 00:52:30,670 DEBUG   HandlerThread:624595 [meta.py:_save_pip():57] save pip
2021-07-15 00:52:30,671 DEBUG   HandlerThread:624595 [meta.py:_save_pip():71] save pip done
2021-07-15 00:52:30,671 DEBUG   HandlerThread:624595 [meta.py:probe():252] probe done
2021-07-15 00:52:30,674 DEBUG   SenderThread:624595 [sender.py:send():179] send: files
2021-07-15 00:52:30,674 INFO    SenderThread:624595 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-15 00:52:30,680 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:52:30,680 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:52:30,812 DEBUG   SenderThread:624595 [sender.py:send():179] send: config
2021-07-15 00:52:30,812 DEBUG   SenderThread:624595 [sender.py:send():179] send: config
2021-07-15 00:52:30,813 DEBUG   SenderThread:624595 [sender.py:send():179] send: config
2021-07-15 00:52:31,126 INFO    Thread-11 :624595 [upload_job.py:push():137] Uploaded file /tmp/tmp7sk_pd_0wandb/2r26ralk-wandb-metadata.json
2021-07-15 00:52:31,551 INFO    Thread-8  :624595 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/wandb-metadata.json
2021-07-15 00:52:31,552 INFO    Thread-8  :624595 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/requirements.txt
2021-07-15 00:52:31,552 INFO    Thread-8  :624595 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/output.log
2021-07-15 00:52:45,557 INFO    Thread-8  :624595 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/output.log
2021-07-15 00:52:45,814 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:52:45,814 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:52:58,723 DEBUG   SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:53:00,564 INFO    Thread-8  :624595 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/config.yaml
2021-07-15 00:53:00,947 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:53:00,947 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:53:16,080 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:53:16,080 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:53:28,803 DEBUG   SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:53:31,216 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:53:31,216 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:53:46,347 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:53:46,347 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:53:58,879 DEBUG   SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:54:01,496 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:54:01,496 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:54:16,630 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:54:16,630 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:54:28,953 DEBUG   SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:54:31,760 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:54:31,761 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:54:46,892 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:54:46,893 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:54:59,029 DEBUG   SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:55:02,026 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:55:02,027 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:55:17,159 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:55:17,160 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:55:29,107 DEBUG   SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:55:32,292 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:55:32,292 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:55:47,422 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:55:47,422 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:55:59,186 DEBUG   SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:56:02,555 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:56:02,556 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:56:17,688 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:56:17,688 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:56:29,261 DEBUG   SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:56:32,824 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:56:32,824 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:56:47,955 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:56:47,955 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:56:59,339 DEBUG   SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:57:03,086 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:57:03,086 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:57:18,219 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:57:18,219 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:57:29,416 DEBUG   SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:57:33,350 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:57:33,350 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:57:48,482 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:57:48,483 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:57:59,490 DEBUG   SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:58:03,614 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:58:03,614 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:58:17,674 INFO    Thread-8  :624595 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/output.log
2021-07-15 00:58:18,763 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:58:18,763 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:58:29,567 DEBUG   SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:58:33,909 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:58:33,910 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:58:49,058 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:58:49,059 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:58:59,648 DEBUG   SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:59:04,188 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:59:04,188 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:59:19,321 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:59:19,322 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:59:29,729 DEBUG   SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:59:34,455 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:59:34,455 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:59:41,706 INFO    Thread-8  :624595 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/output.log
2021-07-15 00:59:43,394 DEBUG   SenderThread:624595 [sender.py:send():179] send: telemetry
2021-07-15 00:59:43,394 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:59:43,395 DEBUG   SenderThread:624595 [sender.py:send():179] send: exit
2021-07-15 00:59:43,395 INFO    SenderThread:624595 [sender.py:send_exit():287] handling exit code: 1
2021-07-15 00:59:43,395 INFO    SenderThread:624595 [sender.py:send_exit():295] send defer
2021-07-15 00:59:43,395 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:59:43,396 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:59:43,396 INFO    HandlerThread:624595 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-15 00:59:43,396 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: defer
2021-07-15 00:59:43,396 INFO    SenderThread:624595 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-15 00:59:43,396 INFO    SenderThread:624595 [sender.py:transition_state():308] send defer: 1
2021-07-15 00:59:43,396 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:59:43,396 INFO    HandlerThread:624595 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-15 00:59:43,467 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: defer
2021-07-15 00:59:43,467 INFO    SenderThread:624595 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-15 00:59:43,467 INFO    SenderThread:624595 [sender.py:transition_state():308] send defer: 2
2021-07-15 00:59:43,467 DEBUG   SenderThread:624595 [sender.py:send():179] send: stats
2021-07-15 00:59:43,467 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:59:43,467 INFO    HandlerThread:624595 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-15 00:59:43,468 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: defer
2021-07-15 00:59:43,468 INFO    SenderThread:624595 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-15 00:59:43,468 INFO    SenderThread:624595 [sender.py:transition_state():308] send defer: 3
2021-07-15 00:59:43,468 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:59:43,468 INFO    HandlerThread:624595 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-15 00:59:43,468 DEBUG   SenderThread:624595 [sender.py:send():179] send: summary
2021-07-15 00:59:43,468 INFO    SenderThread:624595 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-15 00:59:43,468 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: defer
2021-07-15 00:59:43,469 INFO    SenderThread:624595 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-15 00:59:43,469 INFO    SenderThread:624595 [sender.py:transition_state():308] send defer: 4
2021-07-15 00:59:43,469 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:59:43,469 INFO    HandlerThread:624595 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-15 00:59:43,469 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: defer
2021-07-15 00:59:43,469 INFO    SenderThread:624595 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-15 00:59:43,498 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:59:43,675 INFO    SenderThread:624595 [sender.py:transition_state():308] send defer: 5
2021-07-15 00:59:43,675 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:59:43,676 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:59:43,676 INFO    HandlerThread:624595 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-15 00:59:43,676 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: defer
2021-07-15 00:59:43,676 INFO    SenderThread:624595 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-15 00:59:43,676 INFO    SenderThread:624595 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-15 00:59:43,707 INFO    SenderThread:624595 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/config.yaml
2021-07-15 00:59:43,708 INFO    SenderThread:624595 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/output.log
2021-07-15 00:59:43,708 INFO    SenderThread:624595 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/wandb-summary.json
2021-07-15 00:59:43,708 INFO    SenderThread:624595 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files
2021-07-15 00:59:43,708 INFO    SenderThread:624595 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/requirements.txt requirements.txt
2021-07-15 00:59:43,708 INFO    SenderThread:624595 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/output.log output.log
2021-07-15 00:59:43,708 INFO    SenderThread:624595 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/wandb-metadata.json wandb-metadata.json
2021-07-15 00:59:43,708 INFO    SenderThread:624595 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/config.yaml config.yaml
2021-07-15 00:59:43,708 INFO    SenderThread:624595 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/wandb-summary.json wandb-summary.json
2021-07-15 00:59:43,709 INFO    SenderThread:624595 [sender.py:transition_state():308] send defer: 6
2021-07-15 00:59:43,709 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:59:43,715 INFO    HandlerThread:624595 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-15 00:59:43,720 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: defer
2021-07-15 00:59:43,720 INFO    SenderThread:624595 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-15 00:59:43,721 INFO    SenderThread:624595 [file_pusher.py:finish():177] shutting down file pusher
2021-07-15 00:59:43,778 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:59:43,778 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:59:43,880 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:59:43,880 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:59:43,982 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:59:43,982 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:59:44,084 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:59:44,084 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:59:44,146 INFO    Thread-13 :624595 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/output.log
2021-07-15 00:59:44,161 INFO    Thread-15 :624595 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/wandb-summary.json
2021-07-15 00:59:44,186 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:59:44,187 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:59:44,194 INFO    Thread-12 :624595 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/requirements.txt
2021-07-15 00:59:44,198 INFO    Thread-14 :624595 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/files/config.yaml
2021-07-15 00:59:44,289 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:59:44,289 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:59:44,391 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:59:44,391 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:59:44,399 INFO    Thread-7  :624595 [sender.py:transition_state():308] send defer: 7
2021-07-15 00:59:44,399 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:59:44,399 INFO    HandlerThread:624595 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-15 00:59:44,399 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: defer
2021-07-15 00:59:44,400 INFO    SenderThread:624595 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-15 00:59:44,493 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:59:44,686 INFO    SenderThread:624595 [sender.py:transition_state():308] send defer: 8
2021-07-15 00:59:44,687 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:59:44,687 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:59:44,687 INFO    HandlerThread:624595 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-15 00:59:44,687 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: defer
2021-07-15 00:59:44,688 INFO    SenderThread:624595 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-15 00:59:44,688 INFO    SenderThread:624595 [sender.py:transition_state():308] send defer: 9
2021-07-15 00:59:44,688 DEBUG   SenderThread:624595 [sender.py:send():179] send: final
2021-07-15 00:59:44,688 DEBUG   SenderThread:624595 [sender.py:send():179] send: footer
2021-07-15 00:59:44,688 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:59:44,689 INFO    HandlerThread:624595 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-15 00:59:44,689 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: defer
2021-07-15 00:59:44,689 INFO    SenderThread:624595 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-15 00:59:44,789 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:59:44,789 DEBUG   SenderThread:624595 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:59:44,789 INFO    SenderThread:624595 [file_pusher.py:join():182] waiting for file pusher
2021-07-15 00:59:44,791 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: get_summary
2021-07-15 00:59:44,791 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-15 00:59:44,792 DEBUG   HandlerThread:624595 [handler.py:handle_request():124] handle_request: shutdown
2021-07-15 00:59:44,792 INFO    HandlerThread:624595 [handler.py:finish():638] shutting down handler
2021-07-15 00:59:45,689 INFO    WriterThread:624595 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210715_005228-361cvuow/run-361cvuow.wandb
2021-07-15 00:59:45,789 INFO    SenderThread:624595 [sender.py:finish():945] shutting down sender
2021-07-15 00:59:45,790 INFO    SenderThread:624595 [file_pusher.py:finish():177] shutting down file pusher
2021-07-15 00:59:45,790 INFO    SenderThread:624595 [file_pusher.py:join():182] waiting for file pusher
2021-07-15 00:59:45,792 INFO    MainThread:624595 [internal.py:handle_exit():78] Internal process exited