|
2021-07-13 00:53:02,112 INFO MainThread:327506 [internal.py:wandb_internal():88] W&B internal server running at pid: 327506, started at: 2021-07-13 00:53:02.112234 |
|
2021-07-13 00:53:02,114 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: check_version |
|
2021-07-13 00:53:02,114 INFO WriterThread:327506 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/run-2ilkub1o.wandb |
|
2021-07-13 00:53:02,115 DEBUG SenderThread:327506 [sender.py:send():179] send: header |
|
2021-07-13 00:53:02,116 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: check_version |
|
2021-07-13 00:53:02,154 DEBUG SenderThread:327506 [sender.py:send():179] send: run |
|
2021-07-13 00:53:02,328 INFO SenderThread:327506 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files |
|
2021-07-13 00:53:02,329 INFO SenderThread:327506 [sender.py:_start_run_threads():716] run started: 2ilkub1o with start time 1626137581 |
|
2021-07-13 00:53:02,345 DEBUG SenderThread:327506 [sender.py:send():179] send: summary |
|
2021-07-13 00:53:02,345 INFO SenderThread:327506 [sender.py:_save_file():841] saving file wandb-summary.json with policy end |
|
2021-07-13 00:53:02,346 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: run_start |
|
2021-07-13 00:53:03,330 INFO Thread-8 :327506 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/wandb-summary.json |
|
2021-07-13 00:53:03,462 DEBUG HandlerThread:327506 [meta.py:__init__():39] meta init |
|
2021-07-13 00:53:03,462 DEBUG HandlerThread:327506 [meta.py:__init__():53] meta init done |
|
2021-07-13 00:53:03,462 DEBUG HandlerThread:327506 [meta.py:probe():210] probe |
|
2021-07-13 00:53:03,463 DEBUG HandlerThread:327506 [meta.py:_setup_git():200] setup git |
|
2021-07-13 00:53:03,492 DEBUG HandlerThread:327506 [meta.py:_setup_git():207] setup git done |
|
2021-07-13 00:53:03,492 DEBUG HandlerThread:327506 [meta.py:_save_pip():57] save pip |
|
2021-07-13 00:53:03,493 DEBUG HandlerThread:327506 [meta.py:_save_pip():71] save pip done |
|
2021-07-13 00:53:03,493 DEBUG HandlerThread:327506 [meta.py:probe():252] probe done |
|
2021-07-13 00:53:03,496 DEBUG SenderThread:327506 [sender.py:send():179] send: files |
|
2021-07-13 00:53:03,496 INFO SenderThread:327506 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now |
|
2021-07-13 00:53:03,504 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: stop_status |
|
2021-07-13 00:53:03,504 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: stop_status |
|
2021-07-13 00:53:03,635 DEBUG SenderThread:327506 [sender.py:send():179] send: config |
|
2021-07-13 00:53:03,636 DEBUG SenderThread:327506 [sender.py:send():179] send: config |
|
2021-07-13 00:53:03,636 DEBUG SenderThread:327506 [sender.py:send():179] send: config |
|
2021-07-13 00:53:03,952 INFO Thread-11 :327506 [upload_job.py:push():137] Uploaded file /tmp/tmpi8r4kiyhwandb/3l6ji67i-wandb-metadata.json |
|
2021-07-13 00:53:04,330 INFO Thread-8 :327506 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/wandb-metadata.json |
|
2021-07-13 00:53:04,330 INFO Thread-8 :327506 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/requirements.txt |
|
2021-07-13 00:53:04,330 INFO Thread-8 :327506 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/output.log |
|
2021-07-13 00:53:18,637 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: stop_status |
|
2021-07-13 00:53:18,637 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: stop_status |
|
2021-07-13 00:53:20,336 INFO Thread-8 :327506 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/output.log |
|
2021-07-13 00:53:22,336 INFO Thread-8 :327506 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/output.log |
|
2021-07-13 00:53:31,548 DEBUG SenderThread:327506 [sender.py:send():179] send: stats |
|
2021-07-13 00:53:33,340 INFO Thread-8 :327506 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/config.yaml |
|
2021-07-13 00:53:33,769 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: stop_status |
|
2021-07-13 00:53:33,769 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: stop_status |
|
2021-07-13 00:53:48,899 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: stop_status |
|
2021-07-13 00:53:48,899 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: stop_status |
|
2021-07-13 00:54:01,629 DEBUG SenderThread:327506 [sender.py:send():179] send: stats |
|
2021-07-13 00:54:04,032 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: stop_status |
|
2021-07-13 00:54:04,032 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: stop_status |
|
2021-07-13 00:54:12,355 INFO Thread-8 :327506 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/output.log |
|
2021-07-13 00:54:19,178 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: stop_status |
|
2021-07-13 00:54:19,179 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: stop_status |
|
2021-07-13 00:54:31,708 DEBUG SenderThread:327506 [sender.py:send():179] send: stats |
|
2021-07-13 00:54:34,599 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: stop_status |
|
2021-07-13 00:54:34,599 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: stop_status |
|
2021-07-13 00:54:49,798 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: stop_status |
|
2021-07-13 00:54:49,798 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: stop_status |
|
2021-07-13 00:55:01,792 DEBUG SenderThread:327506 [sender.py:send():179] send: stats |
|
2021-07-13 00:55:04,931 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: stop_status |
|
2021-07-13 00:55:04,931 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: stop_status |
|
2021-07-13 00:55:20,062 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: stop_status |
|
2021-07-13 00:55:20,062 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: stop_status |
|
2021-07-13 00:55:31,868 DEBUG SenderThread:327506 [sender.py:send():179] send: stats |
|
2021-07-13 00:55:35,203 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: stop_status |
|
2021-07-13 00:55:35,204 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: stop_status |
|
2021-07-13 00:55:44,391 INFO Thread-8 :327506 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/output.log |
|
2021-07-13 00:55:45,566 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: poll_exit |
|
2021-07-13 00:55:45,567 DEBUG SenderThread:327506 [sender.py:send():179] send: telemetry |
|
2021-07-13 00:55:45,567 DEBUG SenderThread:327506 [sender.py:send():179] send: exit |
|
2021-07-13 00:55:45,567 INFO SenderThread:327506 [sender.py:send_exit():287] handling exit code: 1 |
|
2021-07-13 00:55:45,567 INFO SenderThread:327506 [sender.py:send_exit():295] send defer |
|
2021-07-13 00:55:45,567 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: poll_exit |
|
2021-07-13 00:55:45,568 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: defer |
|
2021-07-13 00:55:45,568 INFO HandlerThread:327506 [handler.py:handle_request_defer():141] handle defer: 0 |
|
2021-07-13 00:55:45,568 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: defer |
|
2021-07-13 00:55:45,568 INFO SenderThread:327506 [sender.py:send_request_defer():304] handle sender defer: 0 |
|
2021-07-13 00:55:45,569 INFO SenderThread:327506 [sender.py:transition_state():308] send defer: 1 |
|
2021-07-13 00:55:45,569 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: defer |
|
2021-07-13 00:55:45,569 INFO HandlerThread:327506 [handler.py:handle_request_defer():141] handle defer: 1 |
|
2021-07-13 00:55:45,601 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: defer |
|
2021-07-13 00:55:45,601 INFO SenderThread:327506 [sender.py:send_request_defer():304] handle sender defer: 1 |
|
2021-07-13 00:55:45,601 INFO SenderThread:327506 [sender.py:transition_state():308] send defer: 2 |
|
2021-07-13 00:55:45,602 DEBUG SenderThread:327506 [sender.py:send():179] send: stats |
|
2021-07-13 00:55:45,602 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: defer |
|
2021-07-13 00:55:45,602 INFO HandlerThread:327506 [handler.py:handle_request_defer():141] handle defer: 2 |
|
2021-07-13 00:55:45,602 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: defer |
|
2021-07-13 00:55:45,602 INFO SenderThread:327506 [sender.py:send_request_defer():304] handle sender defer: 2 |
|
2021-07-13 00:55:45,602 INFO SenderThread:327506 [sender.py:transition_state():308] send defer: 3 |
|
2021-07-13 00:55:45,603 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: defer |
|
2021-07-13 00:55:45,603 INFO HandlerThread:327506 [handler.py:handle_request_defer():141] handle defer: 3 |
|
2021-07-13 00:55:45,603 DEBUG SenderThread:327506 [sender.py:send():179] send: summary |
|
2021-07-13 00:55:45,603 INFO SenderThread:327506 [sender.py:_save_file():841] saving file wandb-summary.json with policy end |
|
2021-07-13 00:55:45,603 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: defer |
|
2021-07-13 00:55:45,604 INFO SenderThread:327506 [sender.py:send_request_defer():304] handle sender defer: 3 |
|
2021-07-13 00:55:45,604 INFO SenderThread:327506 [sender.py:transition_state():308] send defer: 4 |
|
2021-07-13 00:55:45,604 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: defer |
|
2021-07-13 00:55:45,604 INFO HandlerThread:327506 [handler.py:handle_request_defer():141] handle defer: 4 |
|
2021-07-13 00:55:45,604 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: defer |
|
2021-07-13 00:55:45,604 INFO SenderThread:327506 [sender.py:send_request_defer():304] handle sender defer: 4 |
|
2021-07-13 00:55:45,670 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: poll_exit |
|
2021-07-13 00:55:45,784 INFO SenderThread:327506 [sender.py:transition_state():308] send defer: 5 |
|
2021-07-13 00:55:45,784 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: poll_exit |
|
2021-07-13 00:55:45,785 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: defer |
|
2021-07-13 00:55:45,785 INFO HandlerThread:327506 [handler.py:handle_request_defer():141] handle defer: 5 |
|
2021-07-13 00:55:45,785 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: defer |
|
2021-07-13 00:55:45,785 INFO SenderThread:327506 [sender.py:send_request_defer():304] handle sender defer: 5 |
|
2021-07-13 00:55:45,786 INFO SenderThread:327506 [dir_watcher.py:finish():282] shutting down directory watcher |
|
2021-07-13 00:55:45,887 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: poll_exit |
|
2021-07-13 00:55:46,391 INFO Thread-8 :327506 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/wandb-summary.json |
|
2021-07-13 00:55:46,392 INFO SenderThread:327506 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/config.yaml |
|
2021-07-13 00:55:46,392 INFO SenderThread:327506 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/output.log |
|
2021-07-13 00:55:46,392 INFO SenderThread:327506 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files |
|
2021-07-13 00:55:46,392 INFO SenderThread:327506 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/requirements.txt requirements.txt |
|
2021-07-13 00:55:46,392 INFO SenderThread:327506 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/output.log output.log |
|
2021-07-13 00:55:46,392 INFO SenderThread:327506 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/wandb-metadata.json wandb-metadata.json |
|
2021-07-13 00:55:46,392 INFO SenderThread:327506 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/config.yaml config.yaml |
|
2021-07-13 00:55:46,392 INFO SenderThread:327506 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/wandb-summary.json wandb-summary.json |
|
2021-07-13 00:55:46,393 INFO SenderThread:327506 [sender.py:transition_state():308] send defer: 6 |
|
2021-07-13 00:55:46,393 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: poll_exit |
|
2021-07-13 00:55:46,403 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: defer |
|
2021-07-13 00:55:46,403 INFO HandlerThread:327506 [handler.py:handle_request_defer():141] handle defer: 6 |
|
2021-07-13 00:55:46,405 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: defer |
|
2021-07-13 00:55:46,405 INFO SenderThread:327506 [sender.py:send_request_defer():304] handle sender defer: 6 |
|
2021-07-13 00:55:46,405 INFO SenderThread:327506 [file_pusher.py:finish():177] shutting down file pusher |
|
2021-07-13 00:55:46,495 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: poll_exit |
|
2021-07-13 00:55:46,496 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: poll_exit |
|
2021-07-13 00:55:46,598 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: poll_exit |
|
2021-07-13 00:55:46,598 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: poll_exit |
|
2021-07-13 00:55:46,700 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: poll_exit |
|
2021-07-13 00:55:46,700 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: poll_exit |
|
2021-07-13 00:55:46,802 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: poll_exit |
|
2021-07-13 00:55:46,802 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: poll_exit |
|
2021-07-13 00:55:46,867 INFO Thread-14 :327506 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/config.yaml |
|
2021-07-13 00:55:46,874 INFO Thread-15 :327506 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/wandb-summary.json |
|
2021-07-13 00:55:46,876 INFO Thread-13 :327506 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/output.log |
|
2021-07-13 00:55:46,904 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: poll_exit |
|
2021-07-13 00:55:46,905 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: poll_exit |
|
2021-07-13 00:55:46,935 INFO Thread-12 :327506 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/requirements.txt |
|
2021-07-13 00:55:47,007 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: poll_exit |
|
2021-07-13 00:55:47,007 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: poll_exit |
|
2021-07-13 00:55:47,109 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: poll_exit |
|
2021-07-13 00:55:47,109 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: poll_exit |
|
2021-07-13 00:55:47,135 INFO Thread-7 :327506 [sender.py:transition_state():308] send defer: 7 |
|
2021-07-13 00:55:47,136 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: defer |
|
2021-07-13 00:55:47,136 INFO HandlerThread:327506 [handler.py:handle_request_defer():141] handle defer: 7 |
|
2021-07-13 00:55:47,136 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: defer |
|
2021-07-13 00:55:47,136 INFO SenderThread:327506 [sender.py:send_request_defer():304] handle sender defer: 7 |
|
2021-07-13 00:55:47,211 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: poll_exit |
|
2021-07-13 00:55:47,415 INFO SenderThread:327506 [sender.py:transition_state():308] send defer: 8 |
|
2021-07-13 00:55:47,416 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: poll_exit |
|
2021-07-13 00:55:47,416 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: defer |
|
2021-07-13 00:55:47,416 INFO HandlerThread:327506 [handler.py:handle_request_defer():141] handle defer: 8 |
|
2021-07-13 00:55:47,416 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: defer |
|
2021-07-13 00:55:47,417 INFO SenderThread:327506 [sender.py:send_request_defer():304] handle sender defer: 8 |
|
2021-07-13 00:55:47,417 INFO SenderThread:327506 [sender.py:transition_state():308] send defer: 9 |
|
2021-07-13 00:55:47,417 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: defer |
|
2021-07-13 00:55:47,417 INFO HandlerThread:327506 [handler.py:handle_request_defer():141] handle defer: 9 |
|
2021-07-13 00:55:47,417 DEBUG SenderThread:327506 [sender.py:send():179] send: final |
|
2021-07-13 00:55:47,417 DEBUG SenderThread:327506 [sender.py:send():179] send: footer |
|
2021-07-13 00:55:47,417 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: defer |
|
2021-07-13 00:55:47,418 INFO SenderThread:327506 [sender.py:send_request_defer():304] handle sender defer: 9 |
|
2021-07-13 00:55:47,518 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: poll_exit |
|
2021-07-13 00:55:47,518 DEBUG SenderThread:327506 [sender.py:send_request():193] send_request: poll_exit |
|
2021-07-13 00:55:47,518 INFO SenderThread:327506 [file_pusher.py:join():182] waiting for file pusher |
|
2021-07-13 00:55:47,520 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: get_summary |
|
2021-07-13 00:55:47,521 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: sampled_history |
|
2021-07-13 00:55:47,521 DEBUG HandlerThread:327506 [handler.py:handle_request():124] handle_request: shutdown |
|
2021-07-13 00:55:47,521 INFO HandlerThread:327506 [handler.py:finish():638] shutting down handler |
|
2021-07-13 00:55:48,418 INFO WriterThread:327506 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/run-2ilkub1o.wandb |
|
2021-07-13 00:55:48,518 INFO SenderThread:327506 [sender.py:finish():945] shutting down sender |
|
2021-07-13 00:55:48,519 INFO SenderThread:327506 [file_pusher.py:finish():177] shutting down file pusher |
|
2021-07-13 00:55:48,519 INFO SenderThread:327506 [file_pusher.py:join():182] waiting for file pusher |
|
2021-07-13 00:55:48,521 INFO MainThread:327506 [internal.py:handle_exit():78] Internal process exited |
|
|