File size: 31,189 Bytes
731244f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 |
2021-07-15 00:37:39,369 INFO MainThread:622170 [internal.py:wandb_internal():88] W&B internal server running at pid: 622170, started at: 2021-07-15 00:37:39.369673
2021-07-15 00:37:39,371 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: check_version
2021-07-15 00:37:39,372 INFO WriterThread:622170 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/run-kj539nne.wandb
2021-07-15 00:37:39,373 DEBUG SenderThread:622170 [sender.py:send():179] send: header
2021-07-15 00:37:39,373 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: check_version
2021-07-15 00:37:39,410 DEBUG SenderThread:622170 [sender.py:send():179] send: run
2021-07-15 00:37:39,578 INFO SenderThread:622170 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files
2021-07-15 00:37:39,578 INFO SenderThread:622170 [sender.py:_start_run_threads():716] run started: kj539nne with start time 1626309458
2021-07-15 00:37:39,578 DEBUG SenderThread:622170 [sender.py:send():179] send: summary
2021-07-15 00:37:39,578 INFO SenderThread:622170 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-15 00:37:39,579 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: run_start
2021-07-15 00:37:40,586 INFO Thread-8 :622170 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/wandb-summary.json
2021-07-15 00:37:40,758 DEBUG HandlerThread:622170 [meta.py:__init__():39] meta init
2021-07-15 00:37:40,758 DEBUG HandlerThread:622170 [meta.py:__init__():53] meta init done
2021-07-15 00:37:40,759 DEBUG HandlerThread:622170 [meta.py:probe():210] probe
2021-07-15 00:37:40,760 DEBUG HandlerThread:622170 [meta.py:_setup_git():200] setup git
2021-07-15 00:37:40,791 DEBUG HandlerThread:622170 [meta.py:_setup_git():207] setup git done
2021-07-15 00:37:40,791 DEBUG HandlerThread:622170 [meta.py:_save_pip():57] save pip
2021-07-15 00:37:40,792 DEBUG HandlerThread:622170 [meta.py:_save_pip():71] save pip done
2021-07-15 00:37:40,792 DEBUG HandlerThread:622170 [meta.py:probe():252] probe done
2021-07-15 00:37:40,796 DEBUG SenderThread:622170 [sender.py:send():179] send: files
2021-07-15 00:37:40,796 INFO SenderThread:622170 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-15 00:37:40,802 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:37:40,802 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:37:40,930 DEBUG SenderThread:622170 [sender.py:send():179] send: config
2021-07-15 00:37:40,931 DEBUG SenderThread:622170 [sender.py:send():179] send: config
2021-07-15 00:37:40,931 DEBUG SenderThread:622170 [sender.py:send():179] send: config
2021-07-15 00:37:41,235 INFO Thread-11 :622170 [upload_job.py:push():137] Uploaded file /tmp/tmpzvfgl2cpwandb/2r82kf3f-wandb-metadata.json
2021-07-15 00:37:41,584 INFO Thread-8 :622170 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/output.log
2021-07-15 00:37:41,585 INFO Thread-8 :622170 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/requirements.txt
2021-07-15 00:37:41,585 INFO Thread-8 :622170 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/wandb-metadata.json
2021-07-15 00:37:55,591 INFO Thread-8 :622170 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/output.log
2021-07-15 00:37:55,933 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:37:55,933 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:37:57,592 INFO Thread-8 :622170 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/output.log
2021-07-15 00:38:08,843 DEBUG SenderThread:622170 [sender.py:send():179] send: stats
2021-07-15 00:38:10,597 INFO Thread-8 :622170 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/config.yaml
2021-07-15 00:38:11,072 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:38:11,073 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:38:26,208 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:38:26,208 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:38:38,927 DEBUG SenderThread:622170 [sender.py:send():179] send: stats
2021-07-15 00:38:41,341 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:38:41,341 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:38:56,476 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:38:56,476 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:39:09,001 DEBUG SenderThread:622170 [sender.py:send():179] send: stats
2021-07-15 00:39:11,609 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:39:11,609 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:39:26,745 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:39:26,746 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:39:39,080 DEBUG SenderThread:622170 [sender.py:send():179] send: stats
2021-07-15 00:39:41,877 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:39:41,877 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:39:57,009 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:39:57,010 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:40:09,159 DEBUG SenderThread:622170 [sender.py:send():179] send: stats
2021-07-15 00:40:12,144 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:40:12,145 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:40:27,280 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:40:27,280 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:40:39,237 DEBUG SenderThread:622170 [sender.py:send():179] send: stats
2021-07-15 00:40:42,413 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:40:42,414 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:40:57,550 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:40:57,551 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:41:09,313 DEBUG SenderThread:622170 [sender.py:send():179] send: stats
2021-07-15 00:41:12,683 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:41:12,684 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:41:27,817 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:41:27,818 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:41:39,376 DEBUG SenderThread:622170 [sender.py:send():179] send: stats
2021-07-15 00:41:42,956 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:41:42,956 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:41:58,089 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:41:58,090 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:42:09,437 DEBUG SenderThread:622170 [sender.py:send():179] send: stats
2021-07-15 00:42:13,226 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:42:13,226 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:42:28,355 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:42:28,356 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:42:39,515 DEBUG SenderThread:622170 [sender.py:send():179] send: stats
2021-07-15 00:42:43,487 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:42:43,488 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:42:58,620 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:42:58,621 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:43:09,593 DEBUG SenderThread:622170 [sender.py:send():179] send: stats
2021-07-15 00:43:13,756 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:43:13,756 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:43:28,889 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:43:28,889 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:43:39,663 DEBUG SenderThread:622170 [sender.py:send():179] send: stats
2021-07-15 00:43:44,020 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:43:44,020 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:43:59,151 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:43:59,151 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:44:09,739 DEBUG SenderThread:622170 [sender.py:send():179] send: stats
2021-07-15 00:44:14,283 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:44:14,284 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:44:29,414 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:44:29,414 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:44:39,817 DEBUG SenderThread:622170 [sender.py:send():179] send: stats
2021-07-15 00:44:44,546 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:44:44,547 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:44:59,680 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:44:59,681 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:45:09,895 DEBUG SenderThread:622170 [sender.py:send():179] send: stats
2021-07-15 00:45:14,816 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:45:14,816 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:45:29,950 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:45:29,950 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:45:39,970 DEBUG SenderThread:622170 [sender.py:send():179] send: stats
2021-07-15 00:45:45,082 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:45:45,082 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:46:00,214 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:46:00,215 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:46:10,046 DEBUG SenderThread:622170 [sender.py:send():179] send: stats
2021-07-15 00:46:15,347 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:46:15,347 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:46:30,482 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:46:30,483 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:46:40,123 DEBUG SenderThread:622170 [sender.py:send():179] send: stats
2021-07-15 00:46:45,616 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:46:45,616 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:47:00,745 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:47:00,745 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:47:10,194 DEBUG SenderThread:622170 [sender.py:send():179] send: stats
2021-07-15 00:47:15,881 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:47:15,881 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:47:31,011 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:47:31,012 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:47:40,259 DEBUG SenderThread:622170 [sender.py:send():179] send: stats
2021-07-15 00:47:46,146 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:47:46,147 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:48:01,282 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:48:01,283 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:48:10,323 DEBUG SenderThread:622170 [sender.py:send():179] send: stats
2021-07-15 00:48:16,413 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:48:16,414 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:48:31,562 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:48:31,563 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:48:40,388 DEBUG SenderThread:622170 [sender.py:send():179] send: stats
2021-07-15 00:48:46,694 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:48:46,695 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:49:02,047 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:49:02,047 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:49:10,453 DEBUG SenderThread:622170 [sender.py:send():179] send: stats
2021-07-15 00:49:17,178 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:49:17,179 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:49:32,311 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:49:32,311 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:49:40,521 DEBUG SenderThread:622170 [sender.py:send():179] send: stats
2021-07-15 00:49:47,444 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:49:47,444 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:50:02,576 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:50:02,576 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:50:10,594 DEBUG SenderThread:622170 [sender.py:send():179] send: stats
2021-07-15 00:50:17,705 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:50:17,706 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:50:32,835 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:50:32,836 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:50:40,670 DEBUG SenderThread:622170 [sender.py:send():179] send: stats
2021-07-15 00:50:47,970 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:50:47,970 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:51:03,105 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:51:03,105 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:51:10,739 DEBUG SenderThread:622170 [sender.py:send():179] send: stats
2021-07-15 00:51:18,234 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:51:18,235 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:51:21,907 INFO Thread-8 :622170 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/output.log
2021-07-15 00:51:23,131 DEBUG SenderThread:622170 [sender.py:send():179] send: telemetry
2021-07-15 00:51:23,132 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:51:23,132 DEBUG SenderThread:622170 [sender.py:send():179] send: exit
2021-07-15 00:51:23,132 INFO SenderThread:622170 [sender.py:send_exit():287] handling exit code: 1
2021-07-15 00:51:23,133 INFO SenderThread:622170 [sender.py:send_exit():295] send defer
2021-07-15 00:51:23,133 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:51:23,134 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:51:23,134 INFO HandlerThread:622170 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-15 00:51:23,134 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: defer
2021-07-15 00:51:23,134 INFO SenderThread:622170 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-15 00:51:23,135 INFO SenderThread:622170 [sender.py:transition_state():308] send defer: 1
2021-07-15 00:51:23,135 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:51:23,135 INFO HandlerThread:622170 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-15 00:51:23,170 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: defer
2021-07-15 00:51:23,170 INFO SenderThread:622170 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-15 00:51:23,170 INFO SenderThread:622170 [sender.py:transition_state():308] send defer: 2
2021-07-15 00:51:23,170 DEBUG SenderThread:622170 [sender.py:send():179] send: stats
2021-07-15 00:51:23,170 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:51:23,171 INFO HandlerThread:622170 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-15 00:51:23,171 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: defer
2021-07-15 00:51:23,171 INFO SenderThread:622170 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-15 00:51:23,171 INFO SenderThread:622170 [sender.py:transition_state():308] send defer: 3
2021-07-15 00:51:23,171 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:51:23,171 INFO HandlerThread:622170 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-15 00:51:23,171 DEBUG SenderThread:622170 [sender.py:send():179] send: summary
2021-07-15 00:51:23,172 INFO SenderThread:622170 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-15 00:51:23,172 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: defer
2021-07-15 00:51:23,172 INFO SenderThread:622170 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-15 00:51:23,172 INFO SenderThread:622170 [sender.py:transition_state():308] send defer: 4
2021-07-15 00:51:23,172 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:51:23,172 INFO HandlerThread:622170 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-15 00:51:23,173 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: defer
2021-07-15 00:51:23,173 INFO SenderThread:622170 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-15 00:51:23,236 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:51:23,352 INFO SenderThread:622170 [sender.py:transition_state():308] send defer: 5
2021-07-15 00:51:23,352 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:51:23,353 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:51:23,353 INFO HandlerThread:622170 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-15 00:51:23,353 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: defer
2021-07-15 00:51:23,353 INFO SenderThread:622170 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-15 00:51:23,353 INFO SenderThread:622170 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-15 00:51:23,454 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:51:23,908 INFO Thread-8 :622170 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/output.log
2021-07-15 00:51:23,909 INFO SenderThread:622170 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/wandb-summary.json
2021-07-15 00:51:23,909 INFO SenderThread:622170 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/config.yaml
2021-07-15 00:51:23,909 INFO SenderThread:622170 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files
2021-07-15 00:51:23,909 INFO SenderThread:622170 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/requirements.txt requirements.txt
2021-07-15 00:51:23,909 INFO SenderThread:622170 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/output.log output.log
2021-07-15 00:51:23,909 INFO SenderThread:622170 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/wandb-metadata.json wandb-metadata.json
2021-07-15 00:51:23,910 INFO SenderThread:622170 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/config.yaml config.yaml
2021-07-15 00:51:23,913 INFO SenderThread:622170 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/wandb-summary.json wandb-summary.json
2021-07-15 00:51:23,913 INFO SenderThread:622170 [sender.py:transition_state():308] send defer: 6
2021-07-15 00:51:23,913 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:51:23,917 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:51:23,917 INFO HandlerThread:622170 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-15 00:51:23,917 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: defer
2021-07-15 00:51:23,917 INFO SenderThread:622170 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-15 00:51:23,917 INFO SenderThread:622170 [file_pusher.py:finish():177] shutting down file pusher
2021-07-15 00:51:24,016 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:51:24,016 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:51:24,118 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:51:24,118 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:51:24,220 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:51:24,220 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:51:24,322 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:51:24,322 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:51:24,352 INFO Thread-15 :622170 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/wandb-summary.json
2021-07-15 00:51:24,355 INFO Thread-12 :622170 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/requirements.txt
2021-07-15 00:51:24,384 INFO Thread-14 :622170 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/config.yaml
2021-07-15 00:51:24,394 INFO Thread-13 :622170 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/files/output.log
2021-07-15 00:51:24,424 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:51:24,424 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:51:24,526 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:51:24,526 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:51:24,595 INFO Thread-7 :622170 [sender.py:transition_state():308] send defer: 7
2021-07-15 00:51:24,595 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:51:24,595 INFO HandlerThread:622170 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-15 00:51:24,595 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: defer
2021-07-15 00:51:24,595 INFO SenderThread:622170 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-15 00:51:24,627 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:51:24,878 INFO SenderThread:622170 [sender.py:transition_state():308] send defer: 8
2021-07-15 00:51:24,879 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:51:24,879 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:51:24,879 INFO HandlerThread:622170 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-15 00:51:24,879 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: defer
2021-07-15 00:51:24,879 INFO SenderThread:622170 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-15 00:51:24,879 INFO SenderThread:622170 [sender.py:transition_state():308] send defer: 9
2021-07-15 00:51:24,880 DEBUG SenderThread:622170 [sender.py:send():179] send: final
2021-07-15 00:51:24,880 DEBUG SenderThread:622170 [sender.py:send():179] send: footer
2021-07-15 00:51:24,880 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:51:24,880 INFO HandlerThread:622170 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-15 00:51:24,880 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: defer
2021-07-15 00:51:24,880 INFO SenderThread:622170 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-15 00:51:24,980 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:51:24,981 DEBUG SenderThread:622170 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:51:24,981 INFO SenderThread:622170 [file_pusher.py:join():182] waiting for file pusher
2021-07-15 00:51:24,982 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: get_summary
2021-07-15 00:51:24,983 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-15 00:51:24,984 DEBUG HandlerThread:622170 [handler.py:handle_request():124] handle_request: shutdown
2021-07-15 00:51:24,984 INFO HandlerThread:622170 [handler.py:finish():638] shutting down handler
2021-07-15 00:51:25,880 INFO WriterThread:622170 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210715_003738-kj539nne/run-kj539nne.wandb
2021-07-15 00:51:25,981 INFO SenderThread:622170 [sender.py:finish():945] shutting down sender
2021-07-15 00:51:25,981 INFO SenderThread:622170 [file_pusher.py:finish():177] shutting down file pusher
2021-07-15 00:51:25,981 INFO SenderThread:622170 [file_pusher.py:join():182] waiting for file pusher
2021-07-15 00:51:25,984 INFO MainThread:622170 [internal.py:handle_exit():78] Internal process exited
|