File size: 26,788 Bytes
731244f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 |
2021-07-14 23:55:55,747 INFO MainThread:613304 [internal.py:wandb_internal():88] W&B internal server running at pid: 613304, started at: 2021-07-14 23:55:55.746857
2021-07-14 23:55:55,749 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: check_version
2021-07-14 23:55:55,749 INFO WriterThread:613304 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_235555-y01xq728/run-y01xq728.wandb
2021-07-14 23:55:55,750 DEBUG SenderThread:613304 [sender.py:send():179] send: header
2021-07-14 23:55:55,750 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: check_version
2021-07-14 23:55:55,788 DEBUG SenderThread:613304 [sender.py:send():179] send: run
2021-07-14 23:55:55,953 INFO SenderThread:613304 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_235555-y01xq728/files
2021-07-14 23:55:55,954 INFO SenderThread:613304 [sender.py:_start_run_threads():716] run started: y01xq728 with start time 1626306955
2021-07-14 23:55:55,954 DEBUG SenderThread:613304 [sender.py:send():179] send: summary
2021-07-14 23:55:55,954 INFO SenderThread:613304 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 23:55:55,954 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: run_start
2021-07-14 23:55:56,957 INFO Thread-8 :613304 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_235555-y01xq728/files/wandb-summary.json
2021-07-14 23:55:57,055 DEBUG HandlerThread:613304 [meta.py:__init__():39] meta init
2021-07-14 23:55:57,056 DEBUG HandlerThread:613304 [meta.py:__init__():53] meta init done
2021-07-14 23:55:57,056 DEBUG HandlerThread:613304 [meta.py:probe():210] probe
2021-07-14 23:55:57,057 DEBUG HandlerThread:613304 [meta.py:_setup_git():200] setup git
2021-07-14 23:55:57,086 DEBUG HandlerThread:613304 [meta.py:_setup_git():207] setup git done
2021-07-14 23:55:57,087 DEBUG HandlerThread:613304 [meta.py:_save_pip():57] save pip
2021-07-14 23:55:57,087 DEBUG HandlerThread:613304 [meta.py:_save_pip():71] save pip done
2021-07-14 23:55:57,087 DEBUG HandlerThread:613304 [meta.py:probe():252] probe done
2021-07-14 23:55:57,090 DEBUG SenderThread:613304 [sender.py:send():179] send: files
2021-07-14 23:55:57,091 INFO SenderThread:613304 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-14 23:55:57,098 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:55:57,098 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:55:57,230 DEBUG SenderThread:613304 [sender.py:send():179] send: config
2021-07-14 23:55:57,231 DEBUG SenderThread:613304 [sender.py:send():179] send: config
2021-07-14 23:55:57,231 DEBUG SenderThread:613304 [sender.py:send():179] send: config
2021-07-14 23:55:57,551 INFO Thread-11 :613304 [upload_job.py:push():137] Uploaded file /tmp/tmp_7vrmrnzwandb/30914hmv-wandb-metadata.json
2021-07-14 23:55:57,956 INFO Thread-8 :613304 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_235555-y01xq728/files/requirements.txt
2021-07-14 23:55:57,956 INFO Thread-8 :613304 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_235555-y01xq728/files/wandb-metadata.json
2021-07-14 23:55:57,956 INFO Thread-8 :613304 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_235555-y01xq728/files/output.log
2021-07-14 23:56:11,962 INFO Thread-8 :613304 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_235555-y01xq728/files/output.log
2021-07-14 23:56:12,233 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:56:12,233 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:56:13,962 INFO Thread-8 :613304 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_235555-y01xq728/files/output.log
2021-07-14 23:56:25,133 DEBUG SenderThread:613304 [sender.py:send():179] send: stats
2021-07-14 23:56:26,968 INFO Thread-8 :613304 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_235555-y01xq728/files/config.yaml
2021-07-14 23:56:27,371 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:56:27,372 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:56:42,505 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:56:42,505 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:56:55,203 DEBUG SenderThread:613304 [sender.py:send():179] send: stats
2021-07-14 23:56:57,640 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:56:57,641 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:57:12,787 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:57:12,788 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:57:25,275 DEBUG SenderThread:613304 [sender.py:send():179] send: stats
2021-07-14 23:57:27,922 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:57:27,922 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:57:43,054 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:57:43,055 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:57:55,351 DEBUG SenderThread:613304 [sender.py:send():179] send: stats
2021-07-14 23:57:58,186 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:57:58,187 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:58:13,317 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:58:13,318 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:58:25,419 DEBUG SenderThread:613304 [sender.py:send():179] send: stats
2021-07-14 23:58:28,450 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:58:28,450 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:58:43,580 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:58:43,581 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:58:55,493 DEBUG SenderThread:613304 [sender.py:send():179] send: stats
2021-07-14 23:58:58,713 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:58:58,713 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:59:13,843 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:59:13,844 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:59:25,562 DEBUG SenderThread:613304 [sender.py:send():179] send: stats
2021-07-14 23:59:28,973 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:59:28,974 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:59:44,104 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:59:44,104 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:59:55,632 DEBUG SenderThread:613304 [sender.py:send():179] send: stats
2021-07-14 23:59:59,242 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:59:59,243 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:00:14,374 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:00:14,375 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:00:25,703 DEBUG SenderThread:613304 [sender.py:send():179] send: stats
2021-07-15 00:00:29,505 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:00:29,505 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:00:44,637 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:00:44,637 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:00:55,773 DEBUG SenderThread:613304 [sender.py:send():179] send: stats
2021-07-15 00:00:59,774 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:00:59,774 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:01:14,907 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:01:14,907 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:01:25,847 DEBUG SenderThread:613304 [sender.py:send():179] send: stats
2021-07-15 00:01:30,041 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:01:30,042 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:01:45,174 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:01:45,174 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:01:55,920 DEBUG SenderThread:613304 [sender.py:send():179] send: stats
2021-07-15 00:02:00,307 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:02:00,308 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:02:15,449 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:02:15,449 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:02:25,995 DEBUG SenderThread:613304 [sender.py:send():179] send: stats
2021-07-15 00:02:30,588 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:02:30,588 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:02:45,719 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:02:45,720 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:02:56,066 DEBUG SenderThread:613304 [sender.py:send():179] send: stats
2021-07-15 00:03:00,850 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:03:00,850 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:03:15,978 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:03:15,978 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:03:26,139 DEBUG SenderThread:613304 [sender.py:send():179] send: stats
2021-07-15 00:03:31,111 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:03:31,112 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:03:46,244 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:03:46,244 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:03:56,128 INFO Thread-8 :613304 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_235555-y01xq728/files/output.log
2021-07-15 00:03:56,212 DEBUG SenderThread:613304 [sender.py:send():179] send: stats
2021-07-15 00:04:01,389 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:04:01,390 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:04:16,550 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:04:16,551 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:04:26,288 DEBUG SenderThread:613304 [sender.py:send():179] send: stats
2021-07-15 00:04:31,686 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:04:31,686 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:04:46,826 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:04:46,826 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:04:56,371 DEBUG SenderThread:613304 [sender.py:send():179] send: stats
2021-07-15 00:05:01,959 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:05:01,959 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:05:17,092 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:05:17,092 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:05:22,163 INFO Thread-8 :613304 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_235555-y01xq728/files/output.log
2021-07-15 00:05:23,289 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:05:23,289 DEBUG SenderThread:613304 [sender.py:send():179] send: telemetry
2021-07-15 00:05:23,289 DEBUG SenderThread:613304 [sender.py:send():179] send: exit
2021-07-15 00:05:23,289 INFO SenderThread:613304 [sender.py:send_exit():287] handling exit code: 1
2021-07-15 00:05:23,291 INFO SenderThread:613304 [sender.py:send_exit():295] send defer
2021-07-15 00:05:23,291 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:05:23,292 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:05:23,292 INFO HandlerThread:613304 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-15 00:05:23,292 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: defer
2021-07-15 00:05:23,292 INFO SenderThread:613304 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-15 00:05:23,292 INFO SenderThread:613304 [sender.py:transition_state():308] send defer: 1
2021-07-15 00:05:23,293 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:05:23,293 INFO HandlerThread:613304 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-15 00:05:23,339 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: defer
2021-07-15 00:05:23,339 INFO SenderThread:613304 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-15 00:05:23,339 INFO SenderThread:613304 [sender.py:transition_state():308] send defer: 2
2021-07-15 00:05:23,339 DEBUG SenderThread:613304 [sender.py:send():179] send: stats
2021-07-15 00:05:23,340 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:05:23,340 INFO HandlerThread:613304 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-15 00:05:23,340 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: defer
2021-07-15 00:05:23,340 INFO SenderThread:613304 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-15 00:05:23,340 INFO SenderThread:613304 [sender.py:transition_state():308] send defer: 3
2021-07-15 00:05:23,341 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:05:23,341 INFO HandlerThread:613304 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-15 00:05:23,341 DEBUG SenderThread:613304 [sender.py:send():179] send: summary
2021-07-15 00:05:23,342 INFO SenderThread:613304 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-15 00:05:23,342 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: defer
2021-07-15 00:05:23,342 INFO SenderThread:613304 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-15 00:05:23,342 INFO SenderThread:613304 [sender.py:transition_state():308] send defer: 4
2021-07-15 00:05:23,342 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:05:23,342 INFO HandlerThread:613304 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-15 00:05:23,343 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: defer
2021-07-15 00:05:23,343 INFO SenderThread:613304 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-15 00:05:23,393 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:05:23,512 INFO SenderThread:613304 [sender.py:transition_state():308] send defer: 5
2021-07-15 00:05:23,512 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:05:23,513 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:05:23,513 INFO HandlerThread:613304 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-15 00:05:23,513 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: defer
2021-07-15 00:05:23,514 INFO SenderThread:613304 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-15 00:05:23,514 INFO SenderThread:613304 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-15 00:05:23,615 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:05:24,163 INFO Thread-8 :613304 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_235555-y01xq728/files/config.yaml
2021-07-15 00:05:24,164 INFO SenderThread:613304 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_235555-y01xq728/files/output.log
2021-07-15 00:05:24,164 INFO SenderThread:613304 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_235555-y01xq728/files/wandb-summary.json
2021-07-15 00:05:24,164 INFO SenderThread:613304 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210714_235555-y01xq728/files
2021-07-15 00:05:24,165 INFO SenderThread:613304 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_235555-y01xq728/files/requirements.txt requirements.txt
2021-07-15 00:05:24,165 INFO SenderThread:613304 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_235555-y01xq728/files/output.log output.log
2021-07-15 00:05:24,165 INFO SenderThread:613304 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_235555-y01xq728/files/wandb-metadata.json wandb-metadata.json
2021-07-15 00:05:24,165 INFO SenderThread:613304 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_235555-y01xq728/files/config.yaml config.yaml
2021-07-15 00:05:24,165 INFO SenderThread:613304 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_235555-y01xq728/files/wandb-summary.json wandb-summary.json
2021-07-15 00:05:24,166 INFO SenderThread:613304 [sender.py:transition_state():308] send defer: 6
2021-07-15 00:05:24,166 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:05:24,167 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:05:24,167 INFO HandlerThread:613304 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-15 00:05:24,170 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: defer
2021-07-15 00:05:24,170 INFO SenderThread:613304 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-15 00:05:24,170 INFO SenderThread:613304 [file_pusher.py:finish():177] shutting down file pusher
2021-07-15 00:05:24,275 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:05:24,275 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:05:24,377 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:05:24,378 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:05:24,479 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:05:24,480 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:05:24,582 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:05:24,582 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:05:24,625 INFO Thread-15 :613304 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_235555-y01xq728/files/wandb-summary.json
2021-07-15 00:05:24,632 INFO Thread-12 :613304 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_235555-y01xq728/files/requirements.txt
2021-07-15 00:05:24,643 INFO Thread-13 :613304 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_235555-y01xq728/files/output.log
2021-07-15 00:05:24,651 INFO Thread-14 :613304 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_235555-y01xq728/files/config.yaml
2021-07-15 00:05:24,684 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:05:24,684 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:05:24,786 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:05:24,786 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:05:24,851 INFO Thread-7 :613304 [sender.py:transition_state():308] send defer: 7
2021-07-15 00:05:24,852 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:05:24,852 INFO HandlerThread:613304 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-15 00:05:24,852 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: defer
2021-07-15 00:05:24,852 INFO SenderThread:613304 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-15 00:05:24,888 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:05:25,136 INFO SenderThread:613304 [sender.py:transition_state():308] send defer: 8
2021-07-15 00:05:25,136 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:05:25,137 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:05:25,137 INFO HandlerThread:613304 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-15 00:05:25,137 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: defer
2021-07-15 00:05:25,137 INFO SenderThread:613304 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-15 00:05:25,138 INFO SenderThread:613304 [sender.py:transition_state():308] send defer: 9
2021-07-15 00:05:25,138 DEBUG SenderThread:613304 [sender.py:send():179] send: final
2021-07-15 00:05:25,138 DEBUG SenderThread:613304 [sender.py:send():179] send: footer
2021-07-15 00:05:25,138 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:05:25,139 INFO HandlerThread:613304 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-15 00:05:25,139 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: defer
2021-07-15 00:05:25,139 INFO SenderThread:613304 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-15 00:05:25,238 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:05:25,239 DEBUG SenderThread:613304 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:05:25,239 INFO SenderThread:613304 [file_pusher.py:join():182] waiting for file pusher
2021-07-15 00:05:25,240 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: get_summary
2021-07-15 00:05:25,241 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-15 00:05:25,242 DEBUG HandlerThread:613304 [handler.py:handle_request():124] handle_request: shutdown
2021-07-15 00:05:25,242 INFO HandlerThread:613304 [handler.py:finish():638] shutting down handler
2021-07-15 00:05:26,139 INFO WriterThread:613304 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_235555-y01xq728/run-y01xq728.wandb
2021-07-15 00:05:26,239 INFO SenderThread:613304 [sender.py:finish():945] shutting down sender
2021-07-15 00:05:26,239 INFO SenderThread:613304 [file_pusher.py:finish():177] shutting down file pusher
2021-07-15 00:05:26,240 INFO SenderThread:613304 [file_pusher.py:join():182] waiting for file pusher
2021-07-15 00:05:26,242 INFO MainThread:613304 [internal.py:handle_exit():78] Internal process exited
|