dat
add all
9bd1fec
2021-07-13 00:53:02,112 INFO MainThread:327506 [internal.py:wandb_internal():88] W&B internal server running at pid: 327506, started at: 2021-07-13 00:53:02.112234
2021-07-13 00:53:02,114 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: check_version
2021-07-13 00:53:02,114 INFO WriterThread:327506 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/run-2ilkub1o.wandb
2021-07-13 00:53:02,115 DEBUG SenderThread:327506 [sender.py:send():179] send: header
2021-07-13 00:53:02,116 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: check_version
2021-07-13 00:53:02,154 DEBUG SenderThread:327506 [sender.py:send():179] send: run
2021-07-13 00:53:02,328 INFO SenderThread:327506 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files
2021-07-13 00:53:02,329 INFO SenderThread:327506 [sender.py:_start_run_threads():716] run started: 2ilkub1o with start time 1626137581
2021-07-13 00:53:02,345 DEBUG SenderThread:327506 [sender.py:send():179] send: summary
2021-07-13 00:53:02,345 INFO SenderThread:327506 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 00:53:02,346 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: run_start
2021-07-13 00:53:03,330 INFO Thread-8 :327506 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/wandb-summary.json
2021-07-13 00:53:03,462 DEBUG HandlerThread:327506 [meta.py:__init__():39] meta init
2021-07-13 00:53:03,462 DEBUG HandlerThread:327506 [meta.py:__init__():53] meta init done
2021-07-13 00:53:03,462 DEBUG HandlerThread:327506 [meta.py:probe():210] probe
2021-07-13 00:53:03,463 DEBUG HandlerThread:327506 [meta.py:_setup_git():200] setup git
2021-07-13 00:53:03,492 DEBUG HandlerThread:327506 [meta.py:_setup_git():207] setup git done
2021-07-13 00:53:03,492 DEBUG HandlerThread:327506 [meta.py:_save_pip():57] save pip
2021-07-13 00:53:03,493 DEBUG HandlerThread:327506 [meta.py:_save_pip():71] save pip done
2021-07-13 00:53:03,493 DEBUG HandlerThread:327506 [meta.py:probe():252] probe done
2021-07-13 00:53:03,496 DEBUG SenderThread:327506 [sender.py:send():179] send: files
2021-07-13 00:53:03,496 INFO SenderThread:327506 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-13 00:53:03,504 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:53:03,504 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:53:03,635 DEBUG SenderThread:327506 [sender.py:send():179] send: config
2021-07-13 00:53:03,636 DEBUG SenderThread:327506 [sender.py:send():179] send: config
2021-07-13 00:53:03,636 DEBUG SenderThread:327506 [sender.py:send():179] send: config
2021-07-13 00:53:03,952 INFO Thread-11 :327506 [upload_job.py:push():137] Uploaded file /tmp/tmpi8r4kiyhwandb/3l6ji67i-wandb-metadata.json
2021-07-13 00:53:04,330 INFO Thread-8 :327506 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/wandb-metadata.json
2021-07-13 00:53:04,330 INFO Thread-8 :327506 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/requirements.txt
2021-07-13 00:53:04,330 INFO Thread-8 :327506 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/output.log
2021-07-13 00:53:18,637 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:53:18,637 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:53:20,336 INFO Thread-8 :327506 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/output.log
2021-07-13 00:53:22,336 INFO Thread-8 :327506 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/output.log
2021-07-13 00:53:31,548 DEBUG SenderThread:327506 [sender.py:send():179] send: stats
2021-07-13 00:53:33,340 INFO Thread-8 :327506 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/config.yaml
2021-07-13 00:53:33,769 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:53:33,769 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:53:48,899 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:53:48,899 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:54:01,629 DEBUG SenderThread:327506 [sender.py:send():179] send: stats
2021-07-13 00:54:04,032 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:54:04,032 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:54:12,355 INFO Thread-8 :327506 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/output.log
2021-07-13 00:54:19,178 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:54:19,179 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:54:31,708 DEBUG SenderThread:327506 [sender.py:send():179] send: stats
2021-07-13 00:54:34,599 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:54:34,599 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:54:49,798 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:54:49,798 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:55:01,792 DEBUG SenderThread:327506 [sender.py:send():179] send: stats
2021-07-13 00:55:04,931 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:55:04,931 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:55:20,062 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:55:20,062 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:55:31,868 DEBUG SenderThread:327506 [sender.py:send():179] send: stats
2021-07-13 00:55:35,203 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:55:35,204 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:55:44,391 INFO Thread-8 :327506 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/output.log
2021-07-13 00:55:45,566 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:55:45,567 DEBUG SenderThread:327506 [sender.py:send():179] send: telemetry
2021-07-13 00:55:45,567 DEBUG SenderThread:327506 [sender.py:send():179] send: exit
2021-07-13 00:55:45,567 INFO SenderThread:327506 [sender.py:send_exit():287] handling exit code: 1
2021-07-13 00:55:45,567 INFO SenderThread:327506 [sender.py:send_exit():295] send defer
2021-07-13 00:55:45,567 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:55:45,568 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:55:45,568 INFO HandlerThread:327506 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-13 00:55:45,568 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: defer
2021-07-13 00:55:45,568 INFO SenderThread:327506 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-13 00:55:45,569 INFO SenderThread:327506 [sender.py:transition_state():308] send defer: 1
2021-07-13 00:55:45,569 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:55:45,569 INFO HandlerThread:327506 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-13 00:55:45,601 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: defer
2021-07-13 00:55:45,601 INFO SenderThread:327506 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-13 00:55:45,601 INFO SenderThread:327506 [sender.py:transition_state():308] send defer: 2
2021-07-13 00:55:45,602 DEBUG SenderThread:327506 [sender.py:send():179] send: stats
2021-07-13 00:55:45,602 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:55:45,602 INFO HandlerThread:327506 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-13 00:55:45,602 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: defer
2021-07-13 00:55:45,602 INFO SenderThread:327506 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-13 00:55:45,602 INFO SenderThread:327506 [sender.py:transition_state():308] send defer: 3
2021-07-13 00:55:45,603 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:55:45,603 INFO HandlerThread:327506 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-13 00:55:45,603 DEBUG SenderThread:327506 [sender.py:send():179] send: summary
2021-07-13 00:55:45,603 INFO SenderThread:327506 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 00:55:45,603 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: defer
2021-07-13 00:55:45,604 INFO SenderThread:327506 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-13 00:55:45,604 INFO SenderThread:327506 [sender.py:transition_state():308] send defer: 4
2021-07-13 00:55:45,604 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:55:45,604 INFO HandlerThread:327506 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-13 00:55:45,604 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: defer
2021-07-13 00:55:45,604 INFO SenderThread:327506 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-13 00:55:45,670 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:55:45,784 INFO SenderThread:327506 [sender.py:transition_state():308] send defer: 5
2021-07-13 00:55:45,784 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:55:45,785 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:55:45,785 INFO HandlerThread:327506 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-13 00:55:45,785 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: defer
2021-07-13 00:55:45,785 INFO SenderThread:327506 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-13 00:55:45,786 INFO SenderThread:327506 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-13 00:55:45,887 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:55:46,391 INFO Thread-8 :327506 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/wandb-summary.json
2021-07-13 00:55:46,392 INFO SenderThread:327506 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/config.yaml
2021-07-13 00:55:46,392 INFO SenderThread:327506 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/output.log
2021-07-13 00:55:46,392 INFO SenderThread:327506 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files
2021-07-13 00:55:46,392 INFO SenderThread:327506 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/requirements.txt requirements.txt
2021-07-13 00:55:46,392 INFO SenderThread:327506 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/output.log output.log
2021-07-13 00:55:46,392 INFO SenderThread:327506 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/wandb-metadata.json wandb-metadata.json
2021-07-13 00:55:46,392 INFO SenderThread:327506 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/config.yaml config.yaml
2021-07-13 00:55:46,392 INFO SenderThread:327506 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/wandb-summary.json wandb-summary.json
2021-07-13 00:55:46,393 INFO SenderThread:327506 [sender.py:transition_state():308] send defer: 6
2021-07-13 00:55:46,393 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:55:46,403 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:55:46,403 INFO HandlerThread:327506 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-13 00:55:46,405 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: defer
2021-07-13 00:55:46,405 INFO SenderThread:327506 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-13 00:55:46,405 INFO SenderThread:327506 [file_pusher.py:finish():177] shutting down file pusher
2021-07-13 00:55:46,495 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:55:46,496 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:55:46,598 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:55:46,598 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:55:46,700 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:55:46,700 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:55:46,802 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:55:46,802 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:55:46,867 INFO Thread-14 :327506 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/config.yaml
2021-07-13 00:55:46,874 INFO Thread-15 :327506 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/wandb-summary.json
2021-07-13 00:55:46,876 INFO Thread-13 :327506 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/output.log
2021-07-13 00:55:46,904 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:55:46,905 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:55:46,935 INFO Thread-12 :327506 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/requirements.txt
2021-07-13 00:55:47,007 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:55:47,007 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:55:47,109 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:55:47,109 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:55:47,135 INFO Thread-7 :327506 [sender.py:transition_state():308] send defer: 7
2021-07-13 00:55:47,136 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:55:47,136 INFO HandlerThread:327506 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-13 00:55:47,136 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: defer
2021-07-13 00:55:47,136 INFO SenderThread:327506 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-13 00:55:47,211 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:55:47,415 INFO SenderThread:327506 [sender.py:transition_state():308] send defer: 8
2021-07-13 00:55:47,416 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:55:47,416 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:55:47,416 INFO HandlerThread:327506 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-13 00:55:47,416 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: defer
2021-07-13 00:55:47,417 INFO SenderThread:327506 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-13 00:55:47,417 INFO SenderThread:327506 [sender.py:transition_state():308] send defer: 9
2021-07-13 00:55:47,417 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:55:47,417 INFO HandlerThread:327506 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-13 00:55:47,417 DEBUG SenderThread:327506 [sender.py:send():179] send: final
2021-07-13 00:55:47,417 DEBUG SenderThread:327506 [sender.py:send():179] send: footer
2021-07-13 00:55:47,417 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: defer
2021-07-13 00:55:47,418 INFO SenderThread:327506 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-13 00:55:47,518 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:55:47,518 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:55:47,518 INFO SenderThread:327506 [file_pusher.py:join():182] waiting for file pusher
2021-07-13 00:55:47,520 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: get_summary
2021-07-13 00:55:47,521 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-13 00:55:47,521 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: shutdown
2021-07-13 00:55:47,521 INFO HandlerThread:327506 [handler.py:finish():638] shutting down handler
2021-07-13 00:55:48,418 INFO WriterThread:327506 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/run-2ilkub1o.wandb
2021-07-13 00:55:48,518 INFO SenderThread:327506 [sender.py:finish():945] shutting down sender
2021-07-13 00:55:48,519 INFO SenderThread:327506 [file_pusher.py:finish():177] shutting down file pusher
2021-07-13 00:55:48,519 INFO SenderThread:327506 [file_pusher.py:join():182] waiting for file pusher
2021-07-13 00:55:48,521 INFO MainThread:327506 [internal.py:handle_exit():78] Internal process exited