File size: 31,534 Bytes
731244f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 |
2021-07-14 23:11:47,821 INFO MainThread:604064 [internal.py:wandb_internal():88] W&B internal server running at pid: 604064, started at: 2021-07-14 23:11:47.821366
2021-07-14 23:11:47,823 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: check_version
2021-07-14 23:11:47,824 INFO WriterThread:604064 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/run-gkn68kcy.wandb
2021-07-14 23:11:47,824 DEBUG SenderThread:604064 [sender.py:send():179] send: header
2021-07-14 23:11:47,825 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: check_version
2021-07-14 23:11:47,862 DEBUG SenderThread:604064 [sender.py:send():179] send: run
2021-07-14 23:11:48,044 INFO SenderThread:604064 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files
2021-07-14 23:11:48,044 INFO SenderThread:604064 [sender.py:_start_run_threads():716] run started: gkn68kcy with start time 1626304307
2021-07-14 23:11:48,044 DEBUG SenderThread:604064 [sender.py:send():179] send: summary
2021-07-14 23:11:48,044 INFO SenderThread:604064 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 23:11:48,045 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: run_start
2021-07-14 23:11:49,047 INFO Thread-8 :604064 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/wandb-summary.json
2021-07-14 23:11:49,155 DEBUG HandlerThread:604064 [meta.py:__init__():39] meta init
2021-07-14 23:11:49,155 DEBUG HandlerThread:604064 [meta.py:__init__():53] meta init done
2021-07-14 23:11:49,155 DEBUG HandlerThread:604064 [meta.py:probe():210] probe
2021-07-14 23:11:49,156 DEBUG HandlerThread:604064 [meta.py:_setup_git():200] setup git
2021-07-14 23:11:49,187 DEBUG HandlerThread:604064 [meta.py:_setup_git():207] setup git done
2021-07-14 23:11:49,188 DEBUG HandlerThread:604064 [meta.py:_save_pip():57] save pip
2021-07-14 23:11:49,188 DEBUG HandlerThread:604064 [meta.py:_save_pip():71] save pip done
2021-07-14 23:11:49,188 DEBUG HandlerThread:604064 [meta.py:probe():252] probe done
2021-07-14 23:11:49,192 DEBUG SenderThread:604064 [sender.py:send():179] send: files
2021-07-14 23:11:49,192 INFO SenderThread:604064 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-14 23:11:49,199 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:11:49,200 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:11:49,336 DEBUG SenderThread:604064 [sender.py:send():179] send: config
2021-07-14 23:11:49,336 DEBUG SenderThread:604064 [sender.py:send():179] send: config
2021-07-14 23:11:49,337 DEBUG SenderThread:604064 [sender.py:send():179] send: config
2021-07-14 23:11:49,685 INFO Thread-11 :604064 [upload_job.py:push():137] Uploaded file /tmp/tmpnh56hsgtwandb/3jt8aozu-wandb-metadata.json
2021-07-14 23:11:50,046 INFO Thread-8 :604064 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/output.log
2021-07-14 23:11:50,046 INFO Thread-8 :604064 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/requirements.txt
2021-07-14 23:11:50,046 INFO Thread-8 :604064 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/wandb-metadata.json
2021-07-14 23:12:04,051 INFO Thread-8 :604064 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/output.log
2021-07-14 23:12:04,338 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:12:04,339 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:12:17,233 DEBUG SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:12:19,058 INFO Thread-8 :604064 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/config.yaml
2021-07-14 23:12:19,470 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:12:19,471 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:12:34,607 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:12:34,607 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:12:47,304 DEBUG SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:12:49,759 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:12:49,760 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:13:04,899 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:13:04,899 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:13:17,375 DEBUG SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:13:20,032 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:13:20,032 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:13:35,167 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:13:35,168 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:13:47,450 DEBUG SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:13:50,301 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:13:50,301 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:14:05,433 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:14:05,434 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:14:17,527 DEBUG SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:14:20,564 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:14:20,564 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:14:35,694 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:14:35,695 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:14:47,605 DEBUG SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:14:50,827 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:14:50,827 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:15:05,965 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:15:05,965 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:15:17,682 DEBUG SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:15:21,099 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:15:21,100 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:15:36,236 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:15:36,237 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:15:47,752 DEBUG SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:15:51,383 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:15:51,384 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:16:06,514 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:16:06,515 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:16:17,832 DEBUG SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:16:21,647 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:16:21,647 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:16:36,777 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:16:36,778 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:16:47,906 DEBUG SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:16:51,911 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:16:51,911 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:17:07,045 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:17:07,045 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:17:17,984 DEBUG SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:17:22,180 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:17:22,180 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:17:37,312 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:17:37,313 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:17:48,061 DEBUG SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:17:52,447 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:17:52,447 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:18:07,584 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:18:07,584 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:18:18,134 DEBUG SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:18:22,718 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:18:22,718 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:18:37,852 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:18:37,852 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:18:48,202 DEBUG SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:18:52,983 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:18:52,984 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:19:08,118 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:19:08,118 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:19:18,280 DEBUG SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:19:23,249 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:19:23,249 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:19:38,406 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:19:38,407 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:19:48,357 DEBUG SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:19:53,538 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:19:53,539 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:20:08,668 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:20:08,668 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:20:18,431 DEBUG SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:20:23,808 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:20:23,809 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:20:38,943 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:20:38,943 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:20:48,507 DEBUG SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:20:54,079 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:20:54,079 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:21:09,222 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:21:09,223 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:21:18,586 DEBUG SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:21:24,353 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:21:24,354 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:21:39,483 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:21:39,484 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:21:48,663 DEBUG SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:21:55,394 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:21:55,394 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:22:10,527 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:22:10,528 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:22:18,741 DEBUG SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:22:25,659 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:22:25,659 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:22:40,790 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:22:40,790 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:22:48,820 DEBUG SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:22:55,922 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:22:55,923 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:23:11,058 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:23:11,059 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:23:18,897 DEBUG SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:23:26,197 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:23:26,198 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:23:41,329 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:23:41,329 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:23:48,974 DEBUG SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:23:56,463 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:23:56,463 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:24:11,593 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:24:11,593 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:24:19,051 DEBUG SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:24:26,724 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:24:26,724 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:24:41,858 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:24:41,858 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:24:49,130 DEBUG SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:24:56,991 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:24:56,991 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:25:12,121 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:25:12,122 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:25:19,207 DEBUG SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:25:27,253 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:25:27,254 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:25:42,385 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:25:42,386 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:25:49,284 DEBUG SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:25:57,527 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:25:57,527 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:26:06,351 INFO Thread-8 :604064 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/output.log
2021-07-14 23:26:07,081 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:26:07,082 DEBUG SenderThread:604064 [sender.py:send():179] send: telemetry
2021-07-14 23:26:07,082 DEBUG SenderThread:604064 [sender.py:send():179] send: exit
2021-07-14 23:26:07,082 INFO SenderThread:604064 [sender.py:send_exit():287] handling exit code: 1
2021-07-14 23:26:07,083 INFO SenderThread:604064 [sender.py:send_exit():295] send defer
2021-07-14 23:26:07,084 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:26:07,084 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:26:07,085 INFO HandlerThread:604064 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-14 23:26:07,085 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: defer
2021-07-14 23:26:07,085 INFO SenderThread:604064 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-14 23:26:07,085 INFO SenderThread:604064 [sender.py:transition_state():308] send defer: 1
2021-07-14 23:26:07,085 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:26:07,085 INFO HandlerThread:604064 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-14 23:26:07,130 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: defer
2021-07-14 23:26:07,130 INFO SenderThread:604064 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-14 23:26:07,130 INFO SenderThread:604064 [sender.py:transition_state():308] send defer: 2
2021-07-14 23:26:07,131 DEBUG SenderThread:604064 [sender.py:send():179] send: stats
2021-07-14 23:26:07,131 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:26:07,131 INFO HandlerThread:604064 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-14 23:26:07,132 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: defer
2021-07-14 23:26:07,132 INFO SenderThread:604064 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-14 23:26:07,132 INFO SenderThread:604064 [sender.py:transition_state():308] send defer: 3
2021-07-14 23:26:07,132 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:26:07,132 INFO HandlerThread:604064 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-14 23:26:07,132 DEBUG SenderThread:604064 [sender.py:send():179] send: summary
2021-07-14 23:26:07,133 INFO SenderThread:604064 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 23:26:07,133 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: defer
2021-07-14 23:26:07,133 INFO SenderThread:604064 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-14 23:26:07,133 INFO SenderThread:604064 [sender.py:transition_state():308] send defer: 4
2021-07-14 23:26:07,134 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:26:07,134 INFO HandlerThread:604064 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-14 23:26:07,134 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: defer
2021-07-14 23:26:07,134 INFO SenderThread:604064 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-14 23:26:07,228 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:26:07,305 INFO SenderThread:604064 [sender.py:transition_state():308] send defer: 5
2021-07-14 23:26:07,305 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:26:07,306 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:26:07,306 INFO HandlerThread:604064 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-14 23:26:07,306 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: defer
2021-07-14 23:26:07,306 INFO SenderThread:604064 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-14 23:26:07,306 INFO SenderThread:604064 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-14 23:26:07,351 INFO Thread-8 :604064 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/config.yaml
2021-07-14 23:26:07,351 INFO SenderThread:604064 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/output.log
2021-07-14 23:26:07,352 INFO SenderThread:604064 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/wandb-summary.json
2021-07-14 23:26:07,352 INFO SenderThread:604064 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files
2021-07-14 23:26:07,352 INFO SenderThread:604064 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/requirements.txt requirements.txt
2021-07-14 23:26:07,352 INFO SenderThread:604064 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/output.log output.log
2021-07-14 23:26:07,352 INFO SenderThread:604064 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/wandb-metadata.json wandb-metadata.json
2021-07-14 23:26:07,352 INFO SenderThread:604064 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/config.yaml config.yaml
2021-07-14 23:26:07,356 INFO SenderThread:604064 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/wandb-summary.json wandb-summary.json
2021-07-14 23:26:07,359 INFO SenderThread:604064 [sender.py:transition_state():308] send defer: 6
2021-07-14 23:26:07,360 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:26:07,360 INFO HandlerThread:604064 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-14 23:26:07,360 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: defer
2021-07-14 23:26:07,361 INFO SenderThread:604064 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-14 23:26:07,361 INFO SenderThread:604064 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 23:26:07,407 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:26:07,408 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:26:07,513 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:26:07,514 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:26:07,616 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:26:07,616 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:26:07,718 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:26:07,718 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:26:07,820 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:26:07,820 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:26:07,840 INFO Thread-14 :604064 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/config.yaml
2021-07-14 23:26:07,841 INFO Thread-13 :604064 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/output.log
2021-07-14 23:26:07,874 INFO Thread-12 :604064 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/requirements.txt
2021-07-14 23:26:07,875 INFO Thread-15 :604064 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/files/wandb-summary.json
2021-07-14 23:26:07,922 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:26:07,922 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:26:08,024 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:26:08,024 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:26:08,076 INFO Thread-7 :604064 [sender.py:transition_state():308] send defer: 7
2021-07-14 23:26:08,077 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:26:08,077 INFO HandlerThread:604064 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-14 23:26:08,077 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: defer
2021-07-14 23:26:08,077 INFO SenderThread:604064 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-14 23:26:08,126 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:26:08,360 INFO SenderThread:604064 [sender.py:transition_state():308] send defer: 8
2021-07-14 23:26:08,360 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:26:08,361 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:26:08,361 INFO HandlerThread:604064 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-14 23:26:08,361 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: defer
2021-07-14 23:26:08,361 INFO SenderThread:604064 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-14 23:26:08,361 INFO SenderThread:604064 [sender.py:transition_state():308] send defer: 9
2021-07-14 23:26:08,362 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:26:08,362 INFO HandlerThread:604064 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-14 23:26:08,362 DEBUG SenderThread:604064 [sender.py:send():179] send: final
2021-07-14 23:26:08,362 DEBUG SenderThread:604064 [sender.py:send():179] send: footer
2021-07-14 23:26:08,362 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: defer
2021-07-14 23:26:08,362 INFO SenderThread:604064 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-14 23:26:08,462 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:26:08,462 DEBUG SenderThread:604064 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:26:08,463 INFO SenderThread:604064 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 23:26:08,464 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: get_summary
2021-07-14 23:26:08,465 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-14 23:26:08,465 DEBUG HandlerThread:604064 [handler.py:handle_request():124] handle_request: shutdown
2021-07-14 23:26:08,465 INFO HandlerThread:604064 [handler.py:finish():638] shutting down handler
2021-07-14 23:26:09,363 INFO WriterThread:604064 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_231147-gkn68kcy/run-gkn68kcy.wandb
2021-07-14 23:26:09,463 INFO SenderThread:604064 [sender.py:finish():945] shutting down sender
2021-07-14 23:26:09,463 INFO SenderThread:604064 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 23:26:09,463 INFO SenderThread:604064 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 23:26:09,466 INFO MainThread:604064 [internal.py:handle_exit():78] Internal process exited
|