2021-07-14 21:00:58,647 INFO    MainThread:574948 [internal.py:wandb_internal():88] W&B internal server running at pid: 574948, started at: 2021-07-14 21:00:58.647052
2021-07-14 21:00:58,649 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: check_version
2021-07-14 21:00:58,649 INFO    WriterThread:574948 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/run-dn4954s5.wandb
2021-07-14 21:00:58,650 DEBUG   SenderThread:574948 [sender.py:send():179] send: header
2021-07-14 21:00:58,650 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: check_version
2021-07-14 21:00:58,686 DEBUG   SenderThread:574948 [sender.py:send():179] send: run
2021-07-14 21:00:58,856 INFO    SenderThread:574948 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files
2021-07-14 21:00:58,856 INFO    SenderThread:574948 [sender.py:_start_run_threads():716] run started: dn4954s5 with start time 1626296458
2021-07-14 21:00:58,856 DEBUG   SenderThread:574948 [sender.py:send():179] send: summary
2021-07-14 21:00:58,857 INFO    SenderThread:574948 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 21:00:58,857 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: run_start
2021-07-14 21:00:59,858 INFO    Thread-8  :574948 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/wandb-summary.json
2021-07-14 21:00:59,959 DEBUG   HandlerThread:574948 [meta.py:__init__():39] meta init
2021-07-14 21:00:59,959 DEBUG   HandlerThread:574948 [meta.py:__init__():53] meta init done
2021-07-14 21:00:59,959 DEBUG   HandlerThread:574948 [meta.py:probe():210] probe
2021-07-14 21:00:59,960 DEBUG   HandlerThread:574948 [meta.py:_setup_git():200] setup git
2021-07-14 21:00:59,989 DEBUG   HandlerThread:574948 [meta.py:_setup_git():207] setup git done
2021-07-14 21:00:59,989 DEBUG   HandlerThread:574948 [meta.py:_save_pip():57] save pip
2021-07-14 21:00:59,989 DEBUG   HandlerThread:574948 [meta.py:_save_pip():71] save pip done
2021-07-14 21:00:59,990 DEBUG   HandlerThread:574948 [meta.py:probe():252] probe done
2021-07-14 21:00:59,992 DEBUG   SenderThread:574948 [sender.py:send():179] send: files
2021-07-14 21:00:59,993 INFO    SenderThread:574948 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-14 21:00:59,998 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:00:59,999 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:01:00,132 DEBUG   SenderThread:574948 [sender.py:send():179] send: config
2021-07-14 21:01:00,133 DEBUG   SenderThread:574948 [sender.py:send():179] send: config
2021-07-14 21:01:00,133 DEBUG   SenderThread:574948 [sender.py:send():179] send: config
2021-07-14 21:01:00,445 INFO    Thread-11 :574948 [upload_job.py:push():137] Uploaded file /tmp/tmpqq6ketpgwandb/y90ikahm-wandb-metadata.json
2021-07-14 21:01:00,857 INFO    Thread-8  :574948 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/output.log
2021-07-14 21:01:00,857 INFO    Thread-8  :574948 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/requirements.txt
2021-07-14 21:01:00,857 INFO    Thread-8  :574948 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/wandb-metadata.json
2021-07-14 21:01:14,863 INFO    Thread-8  :574948 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/output.log
2021-07-14 21:01:15,134 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:01:15,135 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:01:20,865 INFO    Thread-8  :574948 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/output.log
2021-07-14 21:01:28,042 DEBUG   SenderThread:574948 [sender.py:send():179] send: stats
2021-07-14 21:01:29,869 INFO    Thread-8  :574948 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/config.yaml
2021-07-14 21:01:30,286 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:01:30,287 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:01:45,439 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:01:45,440 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:01:58,125 DEBUG   SenderThread:574948 [sender.py:send():179] send: stats
2021-07-14 21:02:00,570 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:02:00,571 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:02:15,704 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:02:15,704 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:02:28,204 DEBUG   SenderThread:574948 [sender.py:send():179] send: stats
2021-07-14 21:02:30,841 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 21:02:30,841 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: stop_status
2021-07-14 21:02:32,894 INFO    Thread-8  :574948 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/output.log
2021-07-14 21:02:33,493 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:02:33,493 DEBUG   SenderThread:574948 [sender.py:send():179] send: telemetry
2021-07-14 21:02:33,493 DEBUG   SenderThread:574948 [sender.py:send():179] send: exit
2021-07-14 21:02:33,493 INFO    SenderThread:574948 [sender.py:send_exit():287] handling exit code: 1
2021-07-14 21:02:33,494 INFO    SenderThread:574948 [sender.py:send_exit():295] send defer
2021-07-14 21:02:33,494 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:02:33,494 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:02:33,494 INFO    HandlerThread:574948 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-14 21:02:33,495 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: defer
2021-07-14 21:02:33,495 INFO    SenderThread:574948 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-14 21:02:33,495 INFO    SenderThread:574948 [sender.py:transition_state():308] send defer: 1
2021-07-14 21:02:33,495 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:02:33,495 INFO    HandlerThread:574948 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-14 21:02:33,518 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: defer
2021-07-14 21:02:33,518 INFO    SenderThread:574948 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-14 21:02:33,518 INFO    SenderThread:574948 [sender.py:transition_state():308] send defer: 2
2021-07-14 21:02:33,518 DEBUG   SenderThread:574948 [sender.py:send():179] send: stats
2021-07-14 21:02:33,519 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:02:33,519 INFO    HandlerThread:574948 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-14 21:02:33,519 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: defer
2021-07-14 21:02:33,519 INFO    SenderThread:574948 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-14 21:02:33,519 INFO    SenderThread:574948 [sender.py:transition_state():308] send defer: 3
2021-07-14 21:02:33,520 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:02:33,520 INFO    HandlerThread:574948 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-14 21:02:33,520 DEBUG   SenderThread:574948 [sender.py:send():179] send: summary
2021-07-14 21:02:33,520 INFO    SenderThread:574948 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 21:02:33,520 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: defer
2021-07-14 21:02:33,520 INFO    SenderThread:574948 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-14 21:02:33,520 INFO    SenderThread:574948 [sender.py:transition_state():308] send defer: 4
2021-07-14 21:02:33,521 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:02:33,521 INFO    HandlerThread:574948 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-14 21:02:33,521 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: defer
2021-07-14 21:02:33,521 INFO    SenderThread:574948 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-14 21:02:33,596 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:02:33,707 INFO    SenderThread:574948 [sender.py:transition_state():308] send defer: 5
2021-07-14 21:02:33,708 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:02:33,708 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:02:33,708 INFO    HandlerThread:574948 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-14 21:02:33,709 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: defer
2021-07-14 21:02:33,709 INFO    SenderThread:574948 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-14 21:02:33,709 INFO    SenderThread:574948 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-14 21:02:33,810 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:02:33,895 INFO    Thread-8  :574948 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/config.yaml
2021-07-14 21:02:33,895 INFO    SenderThread:574948 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/output.log
2021-07-14 21:02:33,895 INFO    SenderThread:574948 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/wandb-summary.json
2021-07-14 21:02:33,895 INFO    SenderThread:574948 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files
2021-07-14 21:02:33,896 INFO    SenderThread:574948 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/requirements.txt requirements.txt
2021-07-14 21:02:33,896 INFO    SenderThread:574948 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/output.log output.log
2021-07-14 21:02:33,896 INFO    SenderThread:574948 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/wandb-metadata.json wandb-metadata.json
2021-07-14 21:02:33,896 INFO    SenderThread:574948 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/config.yaml config.yaml
2021-07-14 21:02:33,896 INFO    SenderThread:574948 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/wandb-summary.json wandb-summary.json
2021-07-14 21:02:33,896 INFO    SenderThread:574948 [sender.py:transition_state():308] send defer: 6
2021-07-14 21:02:33,897 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:02:33,908 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:02:33,908 INFO    HandlerThread:574948 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-14 21:02:33,909 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: defer
2021-07-14 21:02:33,909 INFO    SenderThread:574948 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-14 21:02:33,909 INFO    SenderThread:574948 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 21:02:34,002 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:02:34,002 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:02:34,104 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 21:02:34,104 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 21:02:34,203 WARNING MainThread:574948 [internal.py:wandb_internal():147] Internal process interrupt: 1
2021-07-14 21:02:34,354 INFO    Thread-14 :574948 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/config.yaml
2021-07-14 21:02:34,366 INFO    Thread-13 :574948 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/output.log
2021-07-14 21:02:34,377 INFO    Thread-12 :574948 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/requirements.txt
2021-07-14 21:02:34,379 INFO    Thread-15 :574948 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/files/wandb-summary.json
2021-07-14 21:02:34,579 INFO    Thread-7  :574948 [sender.py:transition_state():308] send defer: 7
2021-07-14 21:02:34,580 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:02:34,580 INFO    HandlerThread:574948 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-14 21:02:34,580 DEBUG   SenderThread:574948 [sender.py:send_request():193] send_request: defer
2021-07-14 21:02:34,580 INFO    SenderThread:574948 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-14 21:02:34,801 WARNING MainThread:574948 [internal.py:wandb_internal():147] Internal process interrupt: 2
2021-07-14 21:02:34,801 ERROR   MainThread:574948 [internal.py:wandb_internal():150] Internal process interrupted.
2021-07-14 21:02:34,872 INFO    SenderThread:574948 [sender.py:transition_state():308] send defer: 8
2021-07-14 21:02:34,872 INFO    SenderThread:574948 [sender.py:finish():945] shutting down sender
2021-07-14 21:02:34,872 INFO    SenderThread:574948 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 21:02:34,872 INFO    SenderThread:574948 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 21:02:34,873 DEBUG   HandlerThread:574948 [handler.py:handle_request():124] handle_request: defer
2021-07-14 21:02:34,873 INFO    HandlerThread:574948 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-14 21:02:34,875 INFO    HandlerThread:574948 [handler.py:finish():638] shutting down handler
2021-07-14 21:02:35,521 INFO    WriterThread:574948 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_210057-dn4954s5/run-dn4954s5.wandb
2021-07-14 21:02:35,521 INFO    MainThread:574948 [internal.py:handle_exit():78] Internal process exited