File size: 31,185 Bytes
731244f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 |
2021-07-14 23:27:04,669 INFO MainThread:606786 [internal.py:wandb_internal():88] W&B internal server running at pid: 606786, started at: 2021-07-14 23:27:04.669257
2021-07-14 23:27:04,671 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: check_version
2021-07-14 23:27:04,671 INFO WriterThread:606786 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/run-1jijl27o.wandb
2021-07-14 23:27:04,672 DEBUG SenderThread:606786 [sender.py:send():179] send: header
2021-07-14 23:27:04,672 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: check_version
2021-07-14 23:27:04,709 DEBUG SenderThread:606786 [sender.py:send():179] send: run
2021-07-14 23:27:04,879 INFO SenderThread:606786 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files
2021-07-14 23:27:04,880 INFO SenderThread:606786 [sender.py:_start_run_threads():716] run started: 1jijl27o with start time 1626305224
2021-07-14 23:27:04,880 DEBUG SenderThread:606786 [sender.py:send():179] send: summary
2021-07-14 23:27:04,880 INFO SenderThread:606786 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 23:27:04,880 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: run_start
2021-07-14 23:27:05,883 INFO Thread-8 :606786 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/wandb-summary.json
2021-07-14 23:27:05,989 DEBUG HandlerThread:606786 [meta.py:__init__():39] meta init
2021-07-14 23:27:05,989 DEBUG HandlerThread:606786 [meta.py:__init__():53] meta init done
2021-07-14 23:27:05,989 DEBUG HandlerThread:606786 [meta.py:probe():210] probe
2021-07-14 23:27:05,990 DEBUG HandlerThread:606786 [meta.py:_setup_git():200] setup git
2021-07-14 23:27:06,020 DEBUG HandlerThread:606786 [meta.py:_setup_git():207] setup git done
2021-07-14 23:27:06,020 DEBUG HandlerThread:606786 [meta.py:_save_pip():57] save pip
2021-07-14 23:27:06,021 DEBUG HandlerThread:606786 [meta.py:_save_pip():71] save pip done
2021-07-14 23:27:06,021 DEBUG HandlerThread:606786 [meta.py:probe():252] probe done
2021-07-14 23:27:06,024 DEBUG SenderThread:606786 [sender.py:send():179] send: files
2021-07-14 23:27:06,024 INFO SenderThread:606786 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-14 23:27:06,032 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:27:06,033 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:27:06,166 DEBUG SenderThread:606786 [sender.py:send():179] send: config
2021-07-14 23:27:06,166 DEBUG SenderThread:606786 [sender.py:send():179] send: config
2021-07-14 23:27:06,166 DEBUG SenderThread:606786 [sender.py:send():179] send: config
2021-07-14 23:27:06,504 INFO Thread-11 :606786 [upload_job.py:push():137] Uploaded file /tmp/tmpf02i07o_wandb/gludn8x0-wandb-metadata.json
2021-07-14 23:27:06,881 INFO Thread-8 :606786 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/requirements.txt
2021-07-14 23:27:06,881 INFO Thread-8 :606786 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/wandb-metadata.json
2021-07-14 23:27:06,881 INFO Thread-8 :606786 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/output.log
2021-07-14 23:27:20,887 INFO Thread-8 :606786 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/output.log
2021-07-14 23:27:21,168 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:27:21,168 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:27:22,888 INFO Thread-8 :606786 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/output.log
2021-07-14 23:27:34,069 DEBUG SenderThread:606786 [sender.py:send():179] send: stats
2021-07-14 23:27:35,894 INFO Thread-8 :606786 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/config.yaml
2021-07-14 23:27:36,301 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:27:36,302 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:27:51,435 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:27:51,435 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:28:04,145 DEBUG SenderThread:606786 [sender.py:send():179] send: stats
2021-07-14 23:28:06,571 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:28:06,571 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:28:21,704 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:28:21,704 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:28:34,210 DEBUG SenderThread:606786 [sender.py:send():179] send: stats
2021-07-14 23:28:36,838 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:28:36,838 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:28:51,969 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:28:51,970 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:29:04,276 DEBUG SenderThread:606786 [sender.py:send():179] send: stats
2021-07-14 23:29:11,606 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:29:11,606 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:29:26,738 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:29:26,739 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:29:34,341 DEBUG SenderThread:606786 [sender.py:send():179] send: stats
2021-07-14 23:29:41,875 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:29:41,875 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:29:57,008 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:29:57,009 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:30:04,407 DEBUG SenderThread:606786 [sender.py:send():179] send: stats
2021-07-14 23:30:12,141 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:30:12,142 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:30:27,275 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:30:27,276 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:30:34,475 DEBUG SenderThread:606786 [sender.py:send():179] send: stats
2021-07-14 23:30:42,407 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:30:42,407 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:30:57,540 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:30:57,541 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:31:04,540 DEBUG SenderThread:606786 [sender.py:send():179] send: stats
2021-07-14 23:31:12,674 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:31:12,674 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:31:27,806 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:31:27,807 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:31:34,612 DEBUG SenderThread:606786 [sender.py:send():179] send: stats
2021-07-14 23:31:42,940 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:31:42,940 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:31:58,074 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:31:58,075 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:32:04,688 DEBUG SenderThread:606786 [sender.py:send():179] send: stats
2021-07-14 23:32:13,208 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:32:13,208 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:32:28,339 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:32:28,340 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:32:34,767 DEBUG SenderThread:606786 [sender.py:send():179] send: stats
2021-07-14 23:32:43,471 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:32:43,472 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:32:58,604 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:32:58,604 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:33:04,843 DEBUG SenderThread:606786 [sender.py:send():179] send: stats
2021-07-14 23:33:13,739 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:33:13,739 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:33:28,873 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:33:28,874 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:33:34,917 DEBUG SenderThread:606786 [sender.py:send():179] send: stats
2021-07-14 23:33:44,007 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:33:44,007 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:33:59,140 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:33:59,141 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:34:04,989 DEBUG SenderThread:606786 [sender.py:send():179] send: stats
2021-07-14 23:34:14,274 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:34:14,275 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:34:29,406 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:34:29,406 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:34:35,063 DEBUG SenderThread:606786 [sender.py:send():179] send: stats
2021-07-14 23:34:44,538 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:34:44,538 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:34:59,670 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:34:59,671 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:35:05,136 DEBUG SenderThread:606786 [sender.py:send():179] send: stats
2021-07-14 23:35:14,806 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:35:14,806 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:35:29,937 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:35:29,938 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:35:35,212 DEBUG SenderThread:606786 [sender.py:send():179] send: stats
2021-07-14 23:35:45,083 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:35:45,084 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:36:00,215 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:36:00,215 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:36:05,289 DEBUG SenderThread:606786 [sender.py:send():179] send: stats
2021-07-14 23:36:15,359 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:36:15,359 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:36:30,491 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:36:30,492 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:36:35,354 DEBUG SenderThread:606786 [sender.py:send():179] send: stats
2021-07-14 23:36:45,626 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:36:45,626 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:37:00,758 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:37:00,758 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:37:05,418 DEBUG SenderThread:606786 [sender.py:send():179] send: stats
2021-07-14 23:37:15,896 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:37:15,896 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:37:31,030 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:37:31,031 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:37:35,484 DEBUG SenderThread:606786 [sender.py:send():179] send: stats
2021-07-14 23:37:46,162 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:37:46,162 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:38:01,293 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:38:01,294 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:38:05,552 DEBUG SenderThread:606786 [sender.py:send():179] send: stats
2021-07-14 23:38:16,425 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:38:16,425 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:38:31,557 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:38:31,557 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:38:35,624 DEBUG SenderThread:606786 [sender.py:send():179] send: stats
2021-07-14 23:38:46,691 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:38:46,691 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:39:01,823 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:39:01,824 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:39:05,695 DEBUG SenderThread:606786 [sender.py:send():179] send: stats
2021-07-14 23:39:16,955 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:39:16,955 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:39:32,087 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:39:32,088 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:39:35,769 DEBUG SenderThread:606786 [sender.py:send():179] send: stats
2021-07-14 23:39:47,220 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:39:47,221 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:40:02,350 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:40:02,351 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:40:05,843 DEBUG SenderThread:606786 [sender.py:send():179] send: stats
2021-07-14 23:40:17,481 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:40:17,482 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:40:32,615 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:40:32,616 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:40:35,918 DEBUG SenderThread:606786 [sender.py:send():179] send: stats
2021-07-14 23:40:47,205 INFO Thread-8 :606786 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/output.log
2021-07-14 23:40:48,363 DEBUG SenderThread:606786 [sender.py:send():179] send: telemetry
2021-07-14 23:40:48,364 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:40:48,364 DEBUG SenderThread:606786 [sender.py:send():179] send: exit
2021-07-14 23:40:48,364 INFO SenderThread:606786 [sender.py:send_exit():287] handling exit code: 1
2021-07-14 23:40:48,366 INFO SenderThread:606786 [sender.py:send_exit():295] send defer
2021-07-14 23:40:48,366 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:40:48,367 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:40:48,367 INFO HandlerThread:606786 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-14 23:40:48,367 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: defer
2021-07-14 23:40:48,367 INFO SenderThread:606786 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-14 23:40:48,367 INFO SenderThread:606786 [sender.py:transition_state():308] send defer: 1
2021-07-14 23:40:48,368 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:40:48,368 INFO HandlerThread:606786 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-14 23:40:48,446 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: defer
2021-07-14 23:40:48,446 INFO SenderThread:606786 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-14 23:40:48,446 INFO SenderThread:606786 [sender.py:transition_state():308] send defer: 2
2021-07-14 23:40:48,446 DEBUG SenderThread:606786 [sender.py:send():179] send: stats
2021-07-14 23:40:48,447 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:40:48,447 INFO HandlerThread:606786 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-14 23:40:48,447 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: defer
2021-07-14 23:40:48,447 INFO SenderThread:606786 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-14 23:40:48,447 INFO SenderThread:606786 [sender.py:transition_state():308] send defer: 3
2021-07-14 23:40:48,447 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:40:48,448 INFO HandlerThread:606786 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-14 23:40:48,448 DEBUG SenderThread:606786 [sender.py:send():179] send: summary
2021-07-14 23:40:48,448 INFO SenderThread:606786 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 23:40:48,449 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: defer
2021-07-14 23:40:48,449 INFO SenderThread:606786 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-14 23:40:48,449 INFO SenderThread:606786 [sender.py:transition_state():308] send defer: 4
2021-07-14 23:40:48,449 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:40:48,449 INFO HandlerThread:606786 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-14 23:40:48,449 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: defer
2021-07-14 23:40:48,449 INFO SenderThread:606786 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-14 23:40:48,469 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:40:48,629 INFO SenderThread:606786 [sender.py:transition_state():308] send defer: 5
2021-07-14 23:40:48,629 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:40:48,630 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:40:48,630 INFO HandlerThread:606786 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-14 23:40:48,630 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: defer
2021-07-14 23:40:48,630 INFO SenderThread:606786 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-14 23:40:48,630 INFO SenderThread:606786 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-14 23:40:48,732 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:40:49,206 INFO Thread-8 :606786 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/config.yaml
2021-07-14 23:40:49,207 INFO SenderThread:606786 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/output.log
2021-07-14 23:40:49,207 INFO SenderThread:606786 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/wandb-summary.json
2021-07-14 23:40:49,207 INFO SenderThread:606786 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files
2021-07-14 23:40:49,207 INFO SenderThread:606786 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/requirements.txt requirements.txt
2021-07-14 23:40:49,208 INFO SenderThread:606786 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/output.log output.log
2021-07-14 23:40:49,208 INFO SenderThread:606786 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/wandb-metadata.json wandb-metadata.json
2021-07-14 23:40:49,208 INFO SenderThread:606786 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/config.yaml config.yaml
2021-07-14 23:40:49,208 INFO SenderThread:606786 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/wandb-summary.json wandb-summary.json
2021-07-14 23:40:49,209 INFO SenderThread:606786 [sender.py:transition_state():308] send defer: 6
2021-07-14 23:40:49,209 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:40:49,210 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:40:49,210 INFO HandlerThread:606786 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-14 23:40:49,215 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: defer
2021-07-14 23:40:49,216 INFO SenderThread:606786 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-14 23:40:49,216 INFO SenderThread:606786 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 23:40:49,311 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:40:49,311 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:40:49,413 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:40:49,413 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:40:49,515 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:40:49,515 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:40:49,617 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:40:49,618 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:40:49,652 INFO Thread-12 :606786 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/requirements.txt
2021-07-14 23:40:49,653 INFO Thread-14 :606786 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/config.yaml
2021-07-14 23:40:49,698 INFO Thread-15 :606786 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/wandb-summary.json
2021-07-14 23:40:49,719 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:40:49,720 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:40:49,739 INFO Thread-13 :606786 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/files/output.log
2021-07-14 23:40:49,821 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:40:49,822 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:40:49,923 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:40:49,923 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:40:49,940 INFO Thread-7 :606786 [sender.py:transition_state():308] send defer: 7
2021-07-14 23:40:49,940 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:40:49,940 INFO HandlerThread:606786 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-14 23:40:49,940 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: defer
2021-07-14 23:40:49,940 INFO SenderThread:606786 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-14 23:40:50,025 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:40:50,227 INFO SenderThread:606786 [sender.py:transition_state():308] send defer: 8
2021-07-14 23:40:50,228 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:40:50,228 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:40:50,228 INFO HandlerThread:606786 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-14 23:40:50,229 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: defer
2021-07-14 23:40:50,229 INFO SenderThread:606786 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-14 23:40:50,229 INFO SenderThread:606786 [sender.py:transition_state():308] send defer: 9
2021-07-14 23:40:50,229 DEBUG SenderThread:606786 [sender.py:send():179] send: final
2021-07-14 23:40:50,229 DEBUG SenderThread:606786 [sender.py:send():179] send: footer
2021-07-14 23:40:50,230 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:40:50,230 INFO HandlerThread:606786 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-14 23:40:50,230 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: defer
2021-07-14 23:40:50,230 INFO SenderThread:606786 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-14 23:40:50,330 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:40:50,330 DEBUG SenderThread:606786 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:40:50,330 INFO SenderThread:606786 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 23:40:50,332 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: get_summary
2021-07-14 23:40:50,332 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-14 23:40:50,333 DEBUG HandlerThread:606786 [handler.py:handle_request():124] handle_request: shutdown
2021-07-14 23:40:50,333 INFO HandlerThread:606786 [handler.py:finish():638] shutting down handler
2021-07-14 23:40:51,230 INFO WriterThread:606786 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_232703-1jijl27o/run-1jijl27o.wandb
2021-07-14 23:40:51,330 INFO SenderThread:606786 [sender.py:finish():945] shutting down sender
2021-07-14 23:40:51,331 INFO SenderThread:606786 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 23:40:51,331 INFO SenderThread:606786 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 23:40:51,334 INFO MainThread:606786 [internal.py:handle_exit():78] Internal process exited
|