File size: 20,364 Bytes
f291f93 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 |
2021-07-13 11:02:13,518 INFO MainThread:345480 [internal.py:wandb_internal():88] W&B internal server running at pid: 345480, started at: 2021-07-13 11:02:13.518424
2021-07-13 11:02:13,520 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: check_version
2021-07-13 11:02:13,520 INFO WriterThread:345480 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/run-594z6oo0.wandb
2021-07-13 11:02:13,521 DEBUG SenderThread:345480 [sender.py:send():179] send: header
2021-07-13 11:02:13,522 DEBUG SenderThread:345480 [sender.py:send_request():193] send_request: check_version
2021-07-13 11:02:13,558 DEBUG SenderThread:345480 [sender.py:send():179] send: run
2021-07-13 11:02:13,741 INFO SenderThread:345480 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files
2021-07-13 11:02:13,742 INFO SenderThread:345480 [sender.py:_start_run_threads():716] run started: 594z6oo0 with start time 1626174132
2021-07-13 11:02:13,742 DEBUG SenderThread:345480 [sender.py:send():179] send: summary
2021-07-13 11:02:13,742 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: run_start
2021-07-13 11:02:13,743 INFO SenderThread:345480 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 11:02:14,743 INFO Thread-8 :345480 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/wandb-summary.json
2021-07-13 11:02:14,868 DEBUG HandlerThread:345480 [meta.py:__init__():39] meta init
2021-07-13 11:02:14,868 DEBUG HandlerThread:345480 [meta.py:__init__():53] meta init done
2021-07-13 11:02:14,868 DEBUG HandlerThread:345480 [meta.py:probe():210] probe
2021-07-13 11:02:14,869 DEBUG HandlerThread:345480 [meta.py:_setup_git():200] setup git
2021-07-13 11:02:14,899 DEBUG HandlerThread:345480 [meta.py:_setup_git():207] setup git done
2021-07-13 11:02:14,899 DEBUG HandlerThread:345480 [meta.py:_save_pip():57] save pip
2021-07-13 11:02:14,899 DEBUG HandlerThread:345480 [meta.py:_save_pip():71] save pip done
2021-07-13 11:02:14,899 DEBUG HandlerThread:345480 [meta.py:probe():252] probe done
2021-07-13 11:02:14,902 DEBUG SenderThread:345480 [sender.py:send():179] send: files
2021-07-13 11:02:14,902 INFO SenderThread:345480 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-13 11:02:14,910 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:02:14,910 DEBUG SenderThread:345480 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:02:15,037 DEBUG SenderThread:345480 [sender.py:send():179] send: config
2021-07-13 11:02:15,038 DEBUG SenderThread:345480 [sender.py:send():179] send: config
2021-07-13 11:02:15,038 DEBUG SenderThread:345480 [sender.py:send():179] send: config
2021-07-13 11:02:15,378 INFO Thread-11 :345480 [upload_job.py:push():137] Uploaded file /tmp/tmpy185wdfywandb/22iauf0r-wandb-metadata.json
2021-07-13 11:02:15,743 INFO Thread-8 :345480 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/wandb-metadata.json
2021-07-13 11:02:15,743 INFO Thread-8 :345480 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/requirements.txt
2021-07-13 11:02:15,743 INFO Thread-8 :345480 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/output.log
2021-07-13 11:02:29,748 INFO Thread-8 :345480 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/output.log
2021-07-13 11:02:30,039 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:02:30,040 DEBUG SenderThread:345480 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:02:31,749 INFO Thread-8 :345480 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/output.log
2021-07-13 11:02:42,950 DEBUG SenderThread:345480 [sender.py:send():179] send: stats
2021-07-13 11:02:44,755 INFO Thread-8 :345480 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/config.yaml
2021-07-13 11:02:45,176 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:02:45,176 DEBUG SenderThread:345480 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:03:00,308 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:03:00,308 DEBUG SenderThread:345480 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:03:13,026 DEBUG SenderThread:345480 [sender.py:send():179] send: stats
2021-07-13 11:03:15,445 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:03:15,445 DEBUG SenderThread:345480 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:03:30,574 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:03:30,575 DEBUG SenderThread:345480 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:03:43,084 DEBUG SenderThread:345480 [sender.py:send():179] send: stats
2021-07-13 11:03:45,707 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:03:45,708 DEBUG SenderThread:345480 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:04:00,845 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:04:00,845 DEBUG SenderThread:345480 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:04:01,785 INFO Thread-8 :345480 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/output.log
2021-07-13 11:04:13,156 DEBUG SenderThread:345480 [sender.py:send():179] send: stats
2021-07-13 11:04:15,994 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:04:15,994 DEBUG SenderThread:345480 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:04:31,136 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:04:31,136 DEBUG SenderThread:345480 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:04:43,236 DEBUG SenderThread:345480 [sender.py:send():179] send: stats
2021-07-13 11:04:46,269 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:04:46,270 DEBUG SenderThread:345480 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:05:01,401 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:05:01,401 DEBUG SenderThread:345480 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:05:13,316 DEBUG SenderThread:345480 [sender.py:send():179] send: stats
2021-07-13 11:05:16,535 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:05:16,536 DEBUG SenderThread:345480 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:05:27,829 INFO Thread-8 :345480 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/output.log
2021-07-13 11:05:28,324 DEBUG SenderThread:345480 [sender.py:send():179] send: telemetry
2021-07-13 11:05:28,325 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:05:28,325 DEBUG SenderThread:345480 [sender.py:send():179] send: exit
2021-07-13 11:05:28,325 INFO SenderThread:345480 [sender.py:send_exit():287] handling exit code: 1
2021-07-13 11:05:28,325 INFO SenderThread:345480 [sender.py:send_exit():295] send defer
2021-07-13 11:05:28,325 DEBUG SenderThread:345480 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:05:28,326 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:05:28,326 INFO HandlerThread:345480 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-13 11:05:28,326 DEBUG SenderThread:345480 [sender.py:send_request():193] send_request: defer
2021-07-13 11:05:28,326 INFO SenderThread:345480 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-13 11:05:28,326 INFO SenderThread:345480 [sender.py:transition_state():308] send defer: 1
2021-07-13 11:05:28,327 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:05:28,327 INFO HandlerThread:345480 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-13 11:05:28,355 DEBUG SenderThread:345480 [sender.py:send_request():193] send_request: defer
2021-07-13 11:05:28,355 INFO SenderThread:345480 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-13 11:05:28,355 INFO SenderThread:345480 [sender.py:transition_state():308] send defer: 2
2021-07-13 11:05:28,355 DEBUG SenderThread:345480 [sender.py:send():179] send: stats
2021-07-13 11:05:28,355 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:05:28,355 INFO HandlerThread:345480 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-13 11:05:28,355 DEBUG SenderThread:345480 [sender.py:send_request():193] send_request: defer
2021-07-13 11:05:28,355 INFO SenderThread:345480 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-13 11:05:28,355 INFO SenderThread:345480 [sender.py:transition_state():308] send defer: 3
2021-07-13 11:05:28,356 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:05:28,356 INFO HandlerThread:345480 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-13 11:05:28,356 DEBUG SenderThread:345480 [sender.py:send():179] send: summary
2021-07-13 11:05:28,356 INFO SenderThread:345480 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 11:05:28,356 DEBUG SenderThread:345480 [sender.py:send_request():193] send_request: defer
2021-07-13 11:05:28,356 INFO SenderThread:345480 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-13 11:05:28,356 INFO SenderThread:345480 [sender.py:transition_state():308] send defer: 4
2021-07-13 11:05:28,357 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:05:28,357 INFO HandlerThread:345480 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-13 11:05:28,357 DEBUG SenderThread:345480 [sender.py:send_request():193] send_request: defer
2021-07-13 11:05:28,357 INFO SenderThread:345480 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-13 11:05:28,428 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:05:28,533 INFO SenderThread:345480 [sender.py:transition_state():308] send defer: 5
2021-07-13 11:05:28,533 DEBUG SenderThread:345480 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:05:28,534 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:05:28,534 INFO HandlerThread:345480 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-13 11:05:28,534 DEBUG SenderThread:345480 [sender.py:send_request():193] send_request: defer
2021-07-13 11:05:28,534 INFO SenderThread:345480 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-13 11:05:28,535 INFO SenderThread:345480 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-13 11:05:28,636 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:05:28,829 INFO Thread-8 :345480 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/wandb-summary.json
2021-07-13 11:05:28,830 INFO SenderThread:345480 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/config.yaml
2021-07-13 11:05:28,830 INFO SenderThread:345480 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/output.log
2021-07-13 11:05:28,830 INFO SenderThread:345480 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files
2021-07-13 11:05:28,830 INFO SenderThread:345480 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/requirements.txt requirements.txt
2021-07-13 11:05:28,830 INFO SenderThread:345480 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/output.log output.log
2021-07-13 11:05:28,831 INFO SenderThread:345480 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/wandb-metadata.json wandb-metadata.json
2021-07-13 11:05:28,831 INFO SenderThread:345480 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/config.yaml config.yaml
2021-07-13 11:05:28,831 INFO SenderThread:345480 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/wandb-summary.json wandb-summary.json
2021-07-13 11:05:28,831 INFO SenderThread:345480 [sender.py:transition_state():308] send defer: 6
2021-07-13 11:05:28,831 DEBUG SenderThread:345480 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:05:28,837 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:05:28,837 INFO HandlerThread:345480 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-13 11:05:28,837 DEBUG SenderThread:345480 [sender.py:send_request():193] send_request: defer
2021-07-13 11:05:28,844 INFO SenderThread:345480 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-13 11:05:28,844 INFO SenderThread:345480 [file_pusher.py:finish():177] shutting down file pusher
2021-07-13 11:05:28,933 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:05:28,934 DEBUG SenderThread:345480 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:05:29,036 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:05:29,036 DEBUG SenderThread:345480 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:05:29,139 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:05:29,139 DEBUG SenderThread:345480 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:05:29,241 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:05:29,241 DEBUG SenderThread:345480 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:05:29,280 INFO Thread-15 :345480 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/wandb-summary.json
2021-07-13 11:05:29,323 INFO Thread-13 :345480 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/output.log
2021-07-13 11:05:29,343 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:05:29,343 DEBUG SenderThread:345480 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:05:29,367 INFO Thread-12 :345480 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/requirements.txt
2021-07-13 11:05:29,389 INFO Thread-14 :345480 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/config.yaml
2021-07-13 11:05:29,445 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:05:29,445 DEBUG SenderThread:345480 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:05:29,548 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:05:29,548 DEBUG SenderThread:345480 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:05:29,589 INFO Thread-7 :345480 [sender.py:transition_state():308] send defer: 7
2021-07-13 11:05:29,590 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:05:29,590 INFO HandlerThread:345480 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-13 11:05:29,590 DEBUG SenderThread:345480 [sender.py:send_request():193] send_request: defer
2021-07-13 11:05:29,590 INFO SenderThread:345480 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-13 11:05:29,650 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:05:29,884 INFO SenderThread:345480 [sender.py:transition_state():308] send defer: 8
2021-07-13 11:05:29,884 DEBUG SenderThread:345480 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:05:29,885 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:05:29,885 INFO HandlerThread:345480 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-13 11:05:29,885 DEBUG SenderThread:345480 [sender.py:send_request():193] send_request: defer
2021-07-13 11:05:29,885 INFO SenderThread:345480 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-13 11:05:29,885 INFO SenderThread:345480 [sender.py:transition_state():308] send defer: 9
2021-07-13 11:05:29,886 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:05:29,886 INFO HandlerThread:345480 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-13 11:05:29,886 DEBUG SenderThread:345480 [sender.py:send():179] send: final
2021-07-13 11:05:29,886 DEBUG SenderThread:345480 [sender.py:send():179] send: footer
2021-07-13 11:05:29,886 DEBUG SenderThread:345480 [sender.py:send_request():193] send_request: defer
2021-07-13 11:05:29,886 INFO SenderThread:345480 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-13 11:05:29,987 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:05:29,987 DEBUG SenderThread:345480 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:05:29,987 INFO SenderThread:345480 [file_pusher.py:join():182] waiting for file pusher
2021-07-13 11:05:29,989 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: get_summary
2021-07-13 11:05:29,990 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-13 11:05:29,991 DEBUG HandlerThread:345480 [handler.py:handle_request():124] handle_request: shutdown
2021-07-13 11:05:29,991 INFO HandlerThread:345480 [handler.py:finish():638] shutting down handler
2021-07-13 11:05:30,886 INFO WriterThread:345480 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/run-594z6oo0.wandb
2021-07-13 11:05:30,987 INFO SenderThread:345480 [sender.py:finish():945] shutting down sender
2021-07-13 11:05:30,988 INFO SenderThread:345480 [file_pusher.py:finish():177] shutting down file pusher
2021-07-13 11:05:30,988 INFO SenderThread:345480 [file_pusher.py:join():182] waiting for file pusher
2021-07-13 11:05:30,990 INFO MainThread:345480 [internal.py:handle_exit():78] Internal process exited
|