2021-07-14 22:19:21,314 INFO    MainThread:593294 [internal.py:wandb_internal():88] W&B internal server running at pid: 593294, started at: 2021-07-14 22:19:21.314432
2021-07-14 22:19:21,317 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: check_version
2021-07-14 22:19:21,317 INFO    WriterThread:593294 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/run-s091gfok.wandb
2021-07-14 22:19:21,318 DEBUG   SenderThread:593294 [sender.py:send():179] send: header
2021-07-14 22:19:21,318 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: check_version
2021-07-14 22:19:21,357 DEBUG   SenderThread:593294 [sender.py:send():179] send: run
2021-07-14 22:19:21,536 INFO    SenderThread:593294 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files
2021-07-14 22:19:21,536 INFO    SenderThread:593294 [sender.py:_start_run_threads():716] run started: s091gfok with start time 1626301160
2021-07-14 22:19:21,536 DEBUG   SenderThread:593294 [sender.py:send():179] send: summary
2021-07-14 22:19:21,537 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: run_start
2021-07-14 22:19:21,537 INFO    SenderThread:593294 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 22:19:22,539 INFO    Thread-8  :593294 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/wandb-summary.json
2021-07-14 22:19:22,632 DEBUG   HandlerThread:593294 [meta.py:__init__():39] meta init
2021-07-14 22:19:22,632 DEBUG   HandlerThread:593294 [meta.py:__init__():53] meta init done
2021-07-14 22:19:22,632 DEBUG   HandlerThread:593294 [meta.py:probe():210] probe
2021-07-14 22:19:22,634 DEBUG   HandlerThread:593294 [meta.py:_setup_git():200] setup git
2021-07-14 22:19:22,663 DEBUG   HandlerThread:593294 [meta.py:_setup_git():207] setup git done
2021-07-14 22:19:22,663 DEBUG   HandlerThread:593294 [meta.py:_save_pip():57] save pip
2021-07-14 22:19:22,664 DEBUG   HandlerThread:593294 [meta.py:_save_pip():71] save pip done
2021-07-14 22:19:22,664 DEBUG   HandlerThread:593294 [meta.py:probe():252] probe done
2021-07-14 22:19:22,667 DEBUG   SenderThread:593294 [sender.py:send():179] send: files
2021-07-14 22:19:22,667 INFO    SenderThread:593294 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-14 22:19:22,674 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:19:22,674 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:19:22,802 DEBUG   SenderThread:593294 [sender.py:send():179] send: config
2021-07-14 22:19:22,803 DEBUG   SenderThread:593294 [sender.py:send():179] send: config
2021-07-14 22:19:22,803 DEBUG   SenderThread:593294 [sender.py:send():179] send: config
2021-07-14 22:19:23,119 INFO    Thread-11 :593294 [upload_job.py:push():137] Uploaded file /tmp/tmpn0n6xzzmwandb/2vhpic31-wandb-metadata.json
2021-07-14 22:19:23,537 INFO    Thread-8  :593294 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/wandb-metadata.json
2021-07-14 22:19:23,537 INFO    Thread-8  :593294 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/requirements.txt
2021-07-14 22:19:23,538 INFO    Thread-8  :593294 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/output.log
2021-07-14 22:19:37,543 INFO    Thread-8  :593294 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/output.log
2021-07-14 22:19:37,804 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:19:37,804 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:19:39,545 INFO    Thread-8  :593294 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/output.log
2021-07-14 22:19:50,715 DEBUG   SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:19:52,550 INFO    Thread-8  :593294 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/config.yaml
2021-07-14 22:19:52,936 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:19:52,936 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:20:08,079 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:20:08,080 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:20:20,789 DEBUG   SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:20:23,215 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:20:23,215 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:20:38,362 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:20:38,363 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:20:50,861 DEBUG   SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:20:53,496 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:20:53,496 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:21:08,625 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:21:08,625 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:21:20,932 DEBUG   SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:21:23,756 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:21:23,757 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:21:38,885 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:21:38,886 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:21:50,997 DEBUG   SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:21:54,016 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:21:54,016 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:22:09,146 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:22:09,147 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:22:21,114 DEBUG   SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:22:24,279 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:22:24,279 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:22:39,412 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:22:39,413 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:22:51,192 DEBUG   SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:22:54,548 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:22:54,548 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:23:09,678 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:23:09,678 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:23:21,267 DEBUG   SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:23:24,814 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:23:24,814 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:23:39,949 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:23:39,949 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:23:51,337 DEBUG   SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:23:55,081 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:23:55,082 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:24:10,212 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:24:10,212 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:24:21,405 DEBUG   SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:24:25,345 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:24:25,346 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:24:40,483 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:24:40,483 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:24:51,475 DEBUG   SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:24:55,615 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:24:55,615 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:25:10,746 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:25:10,746 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:25:21,548 DEBUG   SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:25:25,876 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:25:25,876 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:25:41,015 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:25:41,016 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:25:51,619 DEBUG   SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:25:56,148 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:25:56,148 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:26:11,280 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:26:11,280 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:26:21,695 DEBUG   SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:26:26,412 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:26:26,413 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:26:41,546 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:26:41,547 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:26:51,772 DEBUG   SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:26:56,683 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:26:56,683 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:27:11,816 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:27:11,816 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:27:21,849 DEBUG   SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:27:26,950 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:27:26,950 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:27:29,710 INFO    Thread-8  :593294 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/output.log
2021-07-14 22:27:42,097 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:27:42,097 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:27:51,925 DEBUG   SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:27:57,249 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:27:57,250 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:28:12,383 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:28:12,384 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:28:22,007 DEBUG   SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:28:27,521 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:28:27,522 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:28:42,658 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:28:42,658 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:28:49,741 INFO    Thread-8  :593294 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/output.log
2021-07-14 22:28:51,116 DEBUG   SenderThread:593294 [sender.py:send():179] send: telemetry
2021-07-14 22:28:51,116 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 22:28:51,116 DEBUG   SenderThread:593294 [sender.py:send():179] send: exit
2021-07-14 22:28:51,116 INFO    SenderThread:593294 [sender.py:send_exit():287] handling exit code: 1
2021-07-14 22:28:51,117 INFO    SenderThread:593294 [sender.py:send_exit():295] send defer
2021-07-14 22:28:51,117 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 22:28:51,118 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: defer
2021-07-14 22:28:51,118 INFO    HandlerThread:593294 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-14 22:28:51,118 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: defer
2021-07-14 22:28:51,118 INFO    SenderThread:593294 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-14 22:28:51,118 INFO    SenderThread:593294 [sender.py:transition_state():308] send defer: 1
2021-07-14 22:28:51,118 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: defer
2021-07-14 22:28:51,118 INFO    HandlerThread:593294 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-14 22:28:51,182 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: defer
2021-07-14 22:28:51,182 INFO    SenderThread:593294 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-14 22:28:51,183 INFO    SenderThread:593294 [sender.py:transition_state():308] send defer: 2
2021-07-14 22:28:51,183 DEBUG   SenderThread:593294 [sender.py:send():179] send: stats
2021-07-14 22:28:51,183 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: defer
2021-07-14 22:28:51,183 INFO    HandlerThread:593294 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-14 22:28:51,184 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: defer
2021-07-14 22:28:51,184 INFO    SenderThread:593294 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-14 22:28:51,184 INFO    SenderThread:593294 [sender.py:transition_state():308] send defer: 3
2021-07-14 22:28:51,184 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: defer
2021-07-14 22:28:51,184 INFO    HandlerThread:593294 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-14 22:28:51,184 DEBUG   SenderThread:593294 [sender.py:send():179] send: summary
2021-07-14 22:28:51,185 INFO    SenderThread:593294 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 22:28:51,185 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: defer
2021-07-14 22:28:51,185 INFO    SenderThread:593294 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-14 22:28:51,185 INFO    SenderThread:593294 [sender.py:transition_state():308] send defer: 4
2021-07-14 22:28:51,185 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: defer
2021-07-14 22:28:51,185 INFO    HandlerThread:593294 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-14 22:28:51,185 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: defer
2021-07-14 22:28:51,185 INFO    SenderThread:593294 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-14 22:28:51,220 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 22:28:51,361 INFO    SenderThread:593294 [sender.py:transition_state():308] send defer: 5
2021-07-14 22:28:51,361 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 22:28:51,361 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: defer
2021-07-14 22:28:51,362 INFO    HandlerThread:593294 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-14 22:28:51,362 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: defer
2021-07-14 22:28:51,362 INFO    SenderThread:593294 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-14 22:28:51,362 INFO    SenderThread:593294 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-14 22:28:51,463 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 22:28:51,742 INFO    Thread-8  :593294 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/wandb-summary.json
2021-07-14 22:28:51,743 INFO    SenderThread:593294 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/config.yaml
2021-07-14 22:28:51,743 INFO    SenderThread:593294 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/output.log
2021-07-14 22:28:51,743 INFO    SenderThread:593294 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files
2021-07-14 22:28:51,743 INFO    SenderThread:593294 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/requirements.txt requirements.txt
2021-07-14 22:28:51,744 INFO    SenderThread:593294 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/output.log output.log
2021-07-14 22:28:51,744 INFO    SenderThread:593294 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/wandb-metadata.json wandb-metadata.json
2021-07-14 22:28:51,744 INFO    SenderThread:593294 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/config.yaml config.yaml
2021-07-14 22:28:51,744 INFO    SenderThread:593294 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/wandb-summary.json wandb-summary.json
2021-07-14 22:28:51,750 INFO    SenderThread:593294 [sender.py:transition_state():308] send defer: 6
2021-07-14 22:28:51,750 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 22:28:51,751 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: defer
2021-07-14 22:28:51,751 INFO    HandlerThread:593294 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-14 22:28:51,754 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: defer
2021-07-14 22:28:51,754 INFO    SenderThread:593294 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-14 22:28:51,754 INFO    SenderThread:593294 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 22:28:51,856 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 22:28:51,856 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 22:28:51,958 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 22:28:51,958 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 22:28:52,060 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 22:28:52,061 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 22:28:52,162 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 22:28:52,163 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 22:28:52,191 INFO    Thread-13 :593294 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/output.log
2021-07-14 22:28:52,198 INFO    Thread-12 :593294 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/requirements.txt
2021-07-14 22:28:52,200 INFO    Thread-14 :593294 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/config.yaml
2021-07-14 22:28:52,212 INFO    Thread-15 :593294 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/files/wandb-summary.json
2021-07-14 22:28:52,264 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 22:28:52,265 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 22:28:52,366 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 22:28:52,366 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 22:28:52,413 INFO    Thread-7  :593294 [sender.py:transition_state():308] send defer: 7
2021-07-14 22:28:52,413 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: defer
2021-07-14 22:28:52,413 INFO    HandlerThread:593294 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-14 22:28:52,413 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: defer
2021-07-14 22:28:52,414 INFO    SenderThread:593294 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-14 22:28:52,468 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 22:28:52,536 INFO    SenderThread:593294 [sender.py:transition_state():308] send defer: 8
2021-07-14 22:28:52,536 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 22:28:52,537 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: defer
2021-07-14 22:28:52,537 INFO    HandlerThread:593294 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-14 22:28:52,537 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: defer
2021-07-14 22:28:52,537 INFO    SenderThread:593294 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-14 22:28:52,537 INFO    SenderThread:593294 [sender.py:transition_state():308] send defer: 9
2021-07-14 22:28:52,538 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: defer
2021-07-14 22:28:52,538 INFO    HandlerThread:593294 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-14 22:28:52,538 DEBUG   SenderThread:593294 [sender.py:send():179] send: final
2021-07-14 22:28:52,538 DEBUG   SenderThread:593294 [sender.py:send():179] send: footer
2021-07-14 22:28:52,538 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: defer
2021-07-14 22:28:52,538 INFO    SenderThread:593294 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-14 22:28:52,638 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 22:28:52,638 DEBUG   SenderThread:593294 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 22:28:52,638 INFO    SenderThread:593294 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 22:28:52,640 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: get_summary
2021-07-14 22:28:52,640 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-14 22:28:52,641 DEBUG   HandlerThread:593294 [handler.py:handle_request():124] handle_request: shutdown
2021-07-14 22:28:52,641 INFO    HandlerThread:593294 [handler.py:finish():638] shutting down handler
2021-07-14 22:28:53,538 INFO    WriterThread:593294 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_221920-s091gfok/run-s091gfok.wandb
2021-07-14 22:28:53,639 INFO    SenderThread:593294 [sender.py:finish():945] shutting down sender
2021-07-14 22:28:53,639 INFO    SenderThread:593294 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 22:28:53,639 INFO    SenderThread:593294 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 22:28:53,641 INFO    MainThread:593294 [internal.py:handle_exit():78] Internal process exited