2021-07-13 11:11:12,514 INFO    MainThread:348072 [internal.py:wandb_internal():88] W&B internal server running at pid: 348072, started at: 2021-07-13 11:11:12.514575
2021-07-13 11:11:12,516 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: check_version
2021-07-13 11:11:12,517 INFO    WriterThread:348072 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/run-3cwslju0.wandb
2021-07-13 11:11:12,518 DEBUG   SenderThread:348072 [sender.py:send():179] send: header
2021-07-13 11:11:12,518 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: check_version
2021-07-13 11:11:12,553 DEBUG   SenderThread:348072 [sender.py:send():179] send: run
2021-07-13 11:11:12,741 INFO    SenderThread:348072 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files
2021-07-13 11:11:12,741 INFO    SenderThread:348072 [sender.py:_start_run_threads():716] run started: 3cwslju0 with start time 1626174671
2021-07-13 11:11:12,741 DEBUG   SenderThread:348072 [sender.py:send():179] send: summary
2021-07-13 11:11:12,742 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: run_start
2021-07-13 11:11:12,743 INFO    SenderThread:348072 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 11:11:13,745 INFO    Thread-8  :348072 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/wandb-summary.json
2021-07-13 11:11:14,000 DEBUG   HandlerThread:348072 [meta.py:__init__():39] meta init
2021-07-13 11:11:14,000 DEBUG   HandlerThread:348072 [meta.py:__init__():53] meta init done
2021-07-13 11:11:14,000 DEBUG   HandlerThread:348072 [meta.py:probe():210] probe
2021-07-13 11:11:14,001 DEBUG   HandlerThread:348072 [meta.py:_setup_git():200] setup git
2021-07-13 11:11:14,031 DEBUG   HandlerThread:348072 [meta.py:_setup_git():207] setup git done
2021-07-13 11:11:14,031 DEBUG   HandlerThread:348072 [meta.py:_save_pip():57] save pip
2021-07-13 11:11:14,031 DEBUG   HandlerThread:348072 [meta.py:_save_pip():71] save pip done
2021-07-13 11:11:14,031 DEBUG   HandlerThread:348072 [meta.py:probe():252] probe done
2021-07-13 11:11:14,035 DEBUG   SenderThread:348072 [sender.py:send():179] send: files
2021-07-13 11:11:14,035 INFO    SenderThread:348072 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-13 11:11:14,043 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:11:14,043 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:11:14,170 DEBUG   SenderThread:348072 [sender.py:send():179] send: config
2021-07-13 11:11:14,170 DEBUG   SenderThread:348072 [sender.py:send():179] send: config
2021-07-13 11:11:14,170 DEBUG   SenderThread:348072 [sender.py:send():179] send: config
2021-07-13 11:11:14,515 INFO    Thread-11 :348072 [upload_job.py:push():137] Uploaded file /tmp/tmp64u985tqwandb/1u3tchat-wandb-metadata.json
2021-07-13 11:11:14,745 INFO    Thread-8  :348072 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/wandb-metadata.json
2021-07-13 11:11:14,745 INFO    Thread-8  :348072 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/output.log
2021-07-13 11:11:14,745 INFO    Thread-8  :348072 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/requirements.txt
2021-07-13 11:11:28,753 INFO    Thread-8  :348072 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/output.log
2021-07-13 11:11:29,172 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:11:29,172 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:11:30,754 INFO    Thread-8  :348072 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/output.log
2021-07-13 11:11:42,078 DEBUG   SenderThread:348072 [sender.py:send():179] send: stats
2021-07-13 11:11:43,760 INFO    Thread-8  :348072 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/config.yaml
2021-07-13 11:11:44,303 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:11:44,304 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:11:59,435 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:11:59,436 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:12:12,149 DEBUG   SenderThread:348072 [sender.py:send():179] send: stats
2021-07-13 11:12:14,568 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:12:14,569 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:12:29,703 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:12:29,704 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:12:42,213 DEBUG   SenderThread:348072 [sender.py:send():179] send: stats
2021-07-13 11:12:44,838 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:12:44,839 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:12:59,973 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:12:59,974 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:13:06,802 INFO    Thread-8  :348072 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/output.log
2021-07-13 11:13:12,281 DEBUG   SenderThread:348072 [sender.py:send():179] send: stats
2021-07-13 11:13:15,129 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:13:15,129 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:13:30,277 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:13:30,277 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:13:42,360 DEBUG   SenderThread:348072 [sender.py:send():179] send: stats
2021-07-13 11:13:45,408 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:13:45,409 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:14:00,541 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:14:00,555 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:14:12,440 DEBUG   SenderThread:348072 [sender.py:send():179] send: stats
2021-07-13 11:14:15,687 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:14:15,687 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:14:26,833 INFO    Thread-8  :348072 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/output.log
2021-07-13 11:14:27,502 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:27,503 DEBUG   SenderThread:348072 [sender.py:send():179] send: telemetry
2021-07-13 11:14:27,503 DEBUG   SenderThread:348072 [sender.py:send():179] send: exit
2021-07-13 11:14:27,503 INFO    SenderThread:348072 [sender.py:send_exit():287] handling exit code: 1
2021-07-13 11:14:27,503 INFO    SenderThread:348072 [sender.py:send_exit():295] send defer
2021-07-13 11:14:27,504 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:27,504 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:14:27,504 INFO    HandlerThread:348072 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-13 11:14:27,504 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: defer
2021-07-13 11:14:27,505 INFO    SenderThread:348072 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-13 11:14:27,505 INFO    SenderThread:348072 [sender.py:transition_state():308] send defer: 1
2021-07-13 11:14:27,505 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:14:27,505 INFO    HandlerThread:348072 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-13 11:14:27,574 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: defer
2021-07-13 11:14:27,574 INFO    SenderThread:348072 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-13 11:14:27,574 INFO    SenderThread:348072 [sender.py:transition_state():308] send defer: 2
2021-07-13 11:14:27,575 DEBUG   SenderThread:348072 [sender.py:send():179] send: stats
2021-07-13 11:14:27,575 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:14:27,575 INFO    HandlerThread:348072 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-13 11:14:27,576 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: defer
2021-07-13 11:14:27,576 INFO    SenderThread:348072 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-13 11:14:27,576 INFO    SenderThread:348072 [sender.py:transition_state():308] send defer: 3
2021-07-13 11:14:27,576 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:14:27,576 INFO    HandlerThread:348072 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-13 11:14:27,576 DEBUG   SenderThread:348072 [sender.py:send():179] send: summary
2021-07-13 11:14:27,606 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:27,704 INFO    SenderThread:348072 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 11:14:27,705 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: defer
2021-07-13 11:14:27,705 INFO    SenderThread:348072 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-13 11:14:27,705 INFO    SenderThread:348072 [sender.py:transition_state():308] send defer: 4
2021-07-13 11:14:27,705 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:27,706 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:14:27,706 INFO    HandlerThread:348072 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-13 11:14:27,706 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: defer
2021-07-13 11:14:27,706 INFO    SenderThread:348072 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-13 11:14:27,807 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:27,834 INFO    Thread-8  :348072 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/output.log
2021-07-13 11:14:27,834 INFO    Thread-8  :348072 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/wandb-summary.json
2021-07-13 11:14:27,901 INFO    SenderThread:348072 [sender.py:transition_state():308] send defer: 5
2021-07-13 11:14:27,901 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:27,902 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:14:27,902 INFO    HandlerThread:348072 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-13 11:14:27,902 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: defer
2021-07-13 11:14:27,902 INFO    SenderThread:348072 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-13 11:14:27,902 INFO    SenderThread:348072 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-13 11:14:28,004 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:28,834 INFO    Thread-8  :348072 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/config.yaml
2021-07-13 11:14:28,836 INFO    SenderThread:348072 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files
2021-07-13 11:14:28,836 INFO    SenderThread:348072 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/requirements.txt requirements.txt
2021-07-13 11:14:28,836 INFO    SenderThread:348072 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/output.log output.log
2021-07-13 11:14:28,836 INFO    SenderThread:348072 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/wandb-metadata.json wandb-metadata.json
2021-07-13 11:14:28,836 INFO    SenderThread:348072 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/config.yaml config.yaml
2021-07-13 11:14:28,836 INFO    SenderThread:348072 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/wandb-summary.json wandb-summary.json
2021-07-13 11:14:28,837 INFO    SenderThread:348072 [sender.py:transition_state():308] send defer: 6
2021-07-13 11:14:28,837 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:28,840 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:14:28,840 INFO    HandlerThread:348072 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-13 11:14:28,843 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: defer
2021-07-13 11:14:28,844 INFO    SenderThread:348072 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-13 11:14:28,844 INFO    SenderThread:348072 [file_pusher.py:finish():177] shutting down file pusher
2021-07-13 11:14:28,943 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:28,943 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:29,046 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:29,046 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:29,148 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:29,149 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:29,251 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:29,251 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:29,300 INFO    Thread-14 :348072 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/config.yaml
2021-07-13 11:14:29,309 INFO    Thread-12 :348072 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/requirements.txt
2021-07-13 11:14:29,332 INFO    Thread-15 :348072 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/wandb-summary.json
2021-07-13 11:14:29,353 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:29,354 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:29,401 INFO    Thread-13 :348072 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/files/output.log
2021-07-13 11:14:29,456 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:29,479 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:29,581 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:29,582 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:29,679 INFO    Thread-7  :348072 [sender.py:transition_state():308] send defer: 7
2021-07-13 11:14:29,680 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:14:29,680 INFO    HandlerThread:348072 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-13 11:14:29,680 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: defer
2021-07-13 11:14:29,680 INFO    SenderThread:348072 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-13 11:14:29,684 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:29,949 INFO    SenderThread:348072 [sender.py:transition_state():308] send defer: 8
2021-07-13 11:14:29,949 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:29,950 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:14:29,950 INFO    HandlerThread:348072 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-13 11:14:29,950 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: defer
2021-07-13 11:14:29,950 INFO    SenderThread:348072 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-13 11:14:29,950 INFO    SenderThread:348072 [sender.py:transition_state():308] send defer: 9
2021-07-13 11:14:29,951 DEBUG   SenderThread:348072 [sender.py:send():179] send: final
2021-07-13 11:14:29,951 DEBUG   SenderThread:348072 [sender.py:send():179] send: footer
2021-07-13 11:14:29,951 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:14:29,951 INFO    HandlerThread:348072 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-13 11:14:29,951 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: defer
2021-07-13 11:14:29,951 INFO    SenderThread:348072 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-13 11:14:30,051 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:14:30,052 DEBUG   SenderThread:348072 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:14:30,052 INFO    SenderThread:348072 [file_pusher.py:join():182] waiting for file pusher
2021-07-13 11:14:30,054 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: get_summary
2021-07-13 11:14:30,055 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-13 11:14:30,055 DEBUG   HandlerThread:348072 [handler.py:handle_request():124] handle_request: shutdown
2021-07-13 11:14:30,056 INFO    HandlerThread:348072 [handler.py:finish():638] shutting down handler
2021-07-13 11:14:30,951 INFO    WriterThread:348072 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210713_111111-3cwslju0/run-3cwslju0.wandb
2021-07-13 11:14:31,052 INFO    SenderThread:348072 [sender.py:finish():945] shutting down sender
2021-07-13 11:14:31,052 INFO    SenderThread:348072 [file_pusher.py:finish():177] shutting down file pusher
2021-07-13 11:14:31,052 INFO    SenderThread:348072 [file_pusher.py:join():182] waiting for file pusher
2021-07-13 11:14:31,054 INFO    MainThread:348072 [internal.py:handle_exit():78] Internal process exited