File size: 55,457 Bytes
8319971 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 |
2022-05-07 17:58:14,107 INFO MainThread:23847 [internal.py:wandb_internal():92] W&B internal server running at pid: 23847, started at: 2022-05-07 17:58:14.106970
2022-05-07 17:58:14,109 INFO WriterThread:23847 [datastore.py:open_for_write():77] open: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/run-3hqzwtal.wandb
2022-05-07 17:58:14,110 DEBUG SenderThread:23847 [sender.py:send():235] send: header
2022-05-07 17:58:14,110 DEBUG SenderThread:23847 [sender.py:send():235] send: run
2022-05-07 17:58:14,188 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: check_version
2022-05-07 17:58:14,188 INFO SenderThread:23847 [dir_watcher.py:__init__():169] watching files in: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files
2022-05-07 17:58:14,188 INFO SenderThread:23847 [sender.py:_start_run_threads():812] run started: 3hqzwtal with start time 1651946293
2022-05-07 17:58:14,188 DEBUG SenderThread:23847 [sender.py:send():235] send: summary
2022-05-07 17:58:14,189 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-07 17:58:14,189 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: check_version
2022-05-07 17:58:14,248 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: run_start
2022-05-07 17:58:15,191 INFO Thread-7 :23847 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-summary.json
2022-05-07 17:58:16,539 DEBUG HandlerThread:23847 [meta.py:__init__():37] meta init
2022-05-07 17:58:16,539 DEBUG HandlerThread:23847 [meta.py:__init__():51] meta init done
2022-05-07 17:58:16,539 DEBUG HandlerThread:23847 [meta.py:probe():211] probe
2022-05-07 17:58:16,541 DEBUG HandlerThread:23847 [meta.py:_setup_git():201] setup git
2022-05-07 17:58:16,575 DEBUG HandlerThread:23847 [meta.py:_setup_git():208] setup git done
2022-05-07 17:58:16,575 DEBUG HandlerThread:23847 [meta.py:_save_pip():55] save pip
2022-05-07 17:58:16,575 DEBUG HandlerThread:23847 [meta.py:_save_pip():69] save pip done
2022-05-07 17:58:16,575 DEBUG HandlerThread:23847 [meta.py:probe():249] probe done
2022-05-07 17:58:16,579 DEBUG SenderThread:23847 [sender.py:send():235] send: files
2022-05-07 17:58:16,579 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-metadata.json with policy now
2022-05-07 17:58:16,585 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: stop_status
2022-05-07 17:58:16,586 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: stop_status
2022-05-07 17:58:16,615 DEBUG SenderThread:23847 [sender.py:send():235] send: telemetry
2022-05-07 17:58:16,860 INFO Thread-11 :23847 [upload_job.py:push():137] Uploaded file /tmp/tmp6ko7nbgiwandb/2qs680he-wandb-metadata.json
2022-05-07 17:58:17,195 INFO Thread-7 :23847 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log
2022-05-07 17:58:17,195 INFO Thread-7 :23847 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/requirements.txt
2022-05-07 17:58:17,196 INFO Thread-7 :23847 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-metadata.json
2022-05-07 17:58:19,196 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log
2022-05-07 17:58:21,197 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log
2022-05-07 17:58:31,201 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log
2022-05-07 17:58:31,677 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: stop_status
2022-05-07 17:58:31,678 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: stop_status
2022-05-07 17:58:33,202 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log
2022-05-07 17:58:44,616 DEBUG SenderThread:23847 [sender.py:send():235] send: stats
2022-05-07 17:58:45,207 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/config.yaml
2022-05-07 17:58:46,930 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: stop_status
2022-05-07 17:58:46,930 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: stop_status
2022-05-07 17:58:57,211 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log
2022-05-07 17:59:01,985 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: stop_status
2022-05-07 17:59:01,986 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: stop_status
2022-05-07 17:59:14,691 DEBUG SenderThread:23847 [sender.py:send():235] send: stats
2022-05-07 17:59:17,476 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: stop_status
2022-05-07 17:59:17,476 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: stop_status
2022-05-07 17:59:19,288 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 17:59:19,378 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 17:59:20,682 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 17:59:20,684 DEBUG SenderThread:23847 [sender.py:send():235] send: history
2022-05-07 17:59:20,685 DEBUG SenderThread:23847 [sender.py:send():235] send: summary
2022-05-07 17:59:20,686 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-07 17:59:20,770 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 17:59:21,221 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log
2022-05-07 17:59:21,222 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-summary.json
2022-05-07 17:59:21,849 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 17:59:21,851 DEBUG SenderThread:23847 [sender.py:send():235] send: history
2022-05-07 17:59:21,852 DEBUG SenderThread:23847 [sender.py:send():235] send: summary
2022-05-07 17:59:21,853 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-07 17:59:21,931 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 17:59:22,222 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-summary.json
2022-05-07 17:59:23,127 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 17:59:23,129 DEBUG SenderThread:23847 [sender.py:send():235] send: history
2022-05-07 17:59:23,130 DEBUG SenderThread:23847 [sender.py:send():235] send: summary
2022-05-07 17:59:23,130 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-07 17:59:23,194 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 17:59:23,222 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log
2022-05-07 17:59:23,222 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-summary.json
2022-05-07 17:59:24,322 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 17:59:24,324 DEBUG SenderThread:23847 [sender.py:send():235] send: history
2022-05-07 17:59:24,325 DEBUG SenderThread:23847 [sender.py:send():235] send: summary
2022-05-07 17:59:24,325 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-07 17:59:24,411 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 17:59:25,223 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log
2022-05-07 17:59:25,223 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-summary.json
2022-05-07 17:59:32,530 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: stop_status
2022-05-07 17:59:32,531 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: stop_status
2022-05-07 17:59:44,766 DEBUG SenderThread:23847 [sender.py:send():235] send: stats
2022-05-07 17:59:47,568 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: stop_status
2022-05-07 17:59:47,569 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: stop_status
2022-05-07 17:59:51,233 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log
2022-05-07 18:00:02,624 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: stop_status
2022-05-07 18:00:02,624 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: stop_status
2022-05-07 18:00:12,240 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log
2022-05-07 18:00:14,847 DEBUG SenderThread:23847 [sender.py:send():235] send: stats
2022-05-07 18:00:17,677 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: stop_status
2022-05-07 18:00:17,677 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: stop_status
2022-05-07 18:00:32,722 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: stop_status
2022-05-07 18:00:32,723 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: stop_status
2022-05-07 18:00:34,050 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:00:34,114 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: log_artifact
2022-05-07 18:00:34,115 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: log_artifact
2022-05-07 18:00:34,218 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:00:34,249 INFO Thread-7 :23847 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/eval/epoch_3_5_f901d5b4b0d1263be066.table.json
2022-05-07 18:00:34,249 INFO Thread-7 :23847 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table
2022-05-07 18:00:34,250 INFO Thread-7 :23847 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media
2022-05-07 18:00:34,250 INFO Thread-7 :23847 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/eval
2022-05-07 18:00:34,574 INFO Thread-13 :23847 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/03/1aa3918a194407ac466420b1ea5405
2022-05-07 18:00:35,249 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log
2022-05-07 18:00:35,435 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:00:35,535 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:00:36,028 INFO SenderThread:23847 [sender.py:send_request_log_artifact():976] logged artifact run-3hqzwtal-evalepoch_3 - {'id': 'QXJ0aWZhY3Q6MTI0MzU4ODAw', 'digest': 'b5c3ea14223fca2fc4c6b3245e57e304', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5ODk5ODky', 'latestArtifact': None}, 'version': 'latest'}
2022-05-07 18:00:36,028 DEBUG SenderThread:23847 [sender.py:send():235] send: files
2022-05-07 18:00:36,028 INFO SenderThread:23847 [sender.py:_save_file():947] saving file media/table/eval/epoch_3_5_f901d5b4b0d1263be066.table.json with policy now
2022-05-07 18:00:36,028 DEBUG SenderThread:23847 [sender.py:send():235] send: history
2022-05-07 18:00:36,029 DEBUG SenderThread:23847 [sender.py:send():235] send: summary
2022-05-07 18:00:36,032 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-07 18:00:36,250 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log
2022-05-07 18:00:36,250 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-summary.json
2022-05-07 18:00:36,279 INFO Thread-14 :23847 [upload_job.py:push():137] Uploaded file /tmp/tmp6ko7nbgiwandb/1go5pz4e-media/table/eval/epoch_3_5_f901d5b4b0d1263be066.table.json
2022-05-07 18:00:36,748 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:00:36,751 DEBUG SenderThread:23847 [sender.py:send():235] send: history
2022-05-07 18:00:36,751 DEBUG SenderThread:23847 [sender.py:send():235] send: summary
2022-05-07 18:00:36,752 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-07 18:00:36,812 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:00:37,250 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-summary.json
2022-05-07 18:00:37,931 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:00:37,932 DEBUG SenderThread:23847 [sender.py:send():235] send: history
2022-05-07 18:00:37,933 DEBUG SenderThread:23847 [sender.py:send():235] send: summary
2022-05-07 18:00:37,934 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-07 18:00:38,012 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:00:38,251 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log
2022-05-07 18:00:38,251 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-summary.json
2022-05-07 18:00:39,225 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:00:39,227 DEBUG SenderThread:23847 [sender.py:send():235] send: history
2022-05-07 18:00:39,228 DEBUG SenderThread:23847 [sender.py:send():235] send: summary
2022-05-07 18:00:39,229 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-07 18:00:39,251 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-summary.json
2022-05-07 18:00:39,285 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:00:40,252 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log
2022-05-07 18:00:40,483 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:00:40,485 DEBUG SenderThread:23847 [sender.py:send():235] send: history
2022-05-07 18:00:40,486 DEBUG SenderThread:23847 [sender.py:send():235] send: summary
2022-05-07 18:00:40,487 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-07 18:00:40,564 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:00:41,252 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-summary.json
2022-05-07 18:00:42,253 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log
2022-05-07 18:00:42,888 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:00:42,945 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: log_artifact
2022-05-07 18:00:42,946 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: log_artifact
2022-05-07 18:00:43,253 INFO Thread-7 :23847 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/eval/epoch_6_10_f901d5b4b0d1263be066.table.json
2022-05-07 18:00:43,254 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/eval
2022-05-07 18:00:43,363 INFO Thread-16 :23847 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/03/1aa3918a194407ac466420b1ea5405
2022-05-07 18:00:43,590 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:00:43,908 INFO SenderThread:23847 [sender.py:send_request_log_artifact():976] logged artifact run-3hqzwtal-evalepoch_6 - {'id': 'QXJ0aWZhY3Q6MTI0MzU4ODI3', 'digest': 'c7eac83a9da52d82139235864a083759', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5ODk5ODk0', 'latestArtifact': None}, 'version': 'latest'}
2022-05-07 18:00:43,908 DEBUG SenderThread:23847 [sender.py:send():235] send: files
2022-05-07 18:00:43,908 INFO SenderThread:23847 [sender.py:_save_file():947] saving file media/table/eval/epoch_6_10_f901d5b4b0d1263be066.table.json with policy now
2022-05-07 18:00:44,166 INFO Thread-17 :23847 [upload_job.py:push():137] Uploaded file /tmp/tmp6ko7nbgiwandb/230mzitx-media/table/eval/epoch_6_10_f901d5b4b0d1263be066.table.json
2022-05-07 18:00:44,254 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log
2022-05-07 18:00:44,709 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:00:44,712 DEBUG SenderThread:23847 [sender.py:send():235] send: history
2022-05-07 18:00:44,712 DEBUG SenderThread:23847 [sender.py:send():235] send: summary
2022-05-07 18:00:44,713 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-07 18:00:44,821 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:00:44,921 DEBUG SenderThread:23847 [sender.py:send():235] send: stats
2022-05-07 18:00:45,254 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-summary.json
2022-05-07 18:00:45,935 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:00:45,937 DEBUG SenderThread:23847 [sender.py:send():235] send: history
2022-05-07 18:00:45,937 DEBUG SenderThread:23847 [sender.py:send():235] send: summary
2022-05-07 18:00:45,939 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-07 18:00:46,017 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:00:46,255 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log
2022-05-07 18:00:46,255 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-summary.json
2022-05-07 18:00:47,263 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:00:47,265 DEBUG SenderThread:23847 [sender.py:send():235] send: history
2022-05-07 18:00:47,266 DEBUG SenderThread:23847 [sender.py:send():235] send: summary
2022-05-07 18:00:47,269 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-07 18:00:47,324 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:00:47,853 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: stop_status
2022-05-07 18:00:47,854 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: stop_status
2022-05-07 18:00:48,256 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log
2022-05-07 18:00:48,256 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-summary.json
2022-05-07 18:00:48,450 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:00:48,452 DEBUG SenderThread:23847 [sender.py:send():235] send: history
2022-05-07 18:00:48,452 DEBUG SenderThread:23847 [sender.py:send():235] send: summary
2022-05-07 18:00:48,453 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-07 18:00:48,528 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:00:49,256 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-summary.json
2022-05-07 18:00:49,681 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:00:49,683 DEBUG SenderThread:23847 [sender.py:send():235] send: history
2022-05-07 18:00:49,683 DEBUG SenderThread:23847 [sender.py:send():235] send: summary
2022-05-07 18:00:49,684 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-07 18:00:49,743 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:00:50,257 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log
2022-05-07 18:00:50,257 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-summary.json
2022-05-07 18:00:52,258 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log
2022-05-07 18:01:02,960 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: stop_status
2022-05-07 18:01:02,961 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: stop_status
2022-05-07 18:01:06,264 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log
2022-05-07 18:01:14,993 DEBUG SenderThread:23847 [sender.py:send():235] send: stats
2022-05-07 18:01:18,011 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: stop_status
2022-05-07 18:01:18,012 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: stop_status
2022-05-07 18:01:18,270 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log
2022-05-07 18:01:29,470 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:01:29,531 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: log_artifact
2022-05-07 18:01:29,531 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: log_artifact
2022-05-07 18:01:29,595 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:01:29,637 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: log_artifact
2022-05-07 18:01:29,735 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:01:29,962 INFO Thread-19 :23847 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/59/70265576a772597adeb57dd33a7a30
2022-05-07 18:01:30,276 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log
2022-05-07 18:01:30,276 INFO Thread-7 :23847 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/eval/epoch_8_incorrect_15_623faa4883d23d9d4194.table.json
2022-05-07 18:01:30,276 INFO Thread-7 :23847 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/eval/epoch_8_15_a7d6e656cdc47be11a4d.table.json
2022-05-07 18:01:30,276 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/eval
2022-05-07 18:01:30,398 INFO SenderThread:23847 [sender.py:send_request_log_artifact():976] logged artifact run-3hqzwtal-evalepoch_8 - {'id': 'QXJ0aWZhY3Q6MTI0MzU5MDgw', 'digest': 'f438f940059d7667deb4a50791726d2c', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5ODk5OTE3', 'latestArtifact': None}, 'version': 'latest'}
2022-05-07 18:01:30,398 DEBUG SenderThread:23847 [sender.py:send():235] send: files
2022-05-07 18:01:30,398 INFO SenderThread:23847 [sender.py:_save_file():947] saving file media/table/eval/epoch_8_15_a7d6e656cdc47be11a4d.table.json with policy now
2022-05-07 18:01:30,399 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: log_artifact
2022-05-07 18:01:30,670 INFO Thread-20 :23847 [upload_job.py:push():137] Uploaded file /tmp/tmp6ko7nbgiwandb/sae1wnru-media/table/eval/epoch_8_15_a7d6e656cdc47be11a4d.table.json
2022-05-07 18:01:30,872 INFO Thread-22 :23847 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/25/29be3032c57f98a337547e6fe497da
2022-05-07 18:01:31,332 INFO SenderThread:23847 [sender.py:send_request_log_artifact():976] logged artifact run-3hqzwtal-evalepoch_8_incorrect - {'id': 'QXJ0aWZhY3Q6MTI0MzU5MDgy', 'digest': '1083b22586f095b594cda26f5e13869f', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5ODk5OTE4', 'latestArtifact': None}, 'version': 'latest'}
2022-05-07 18:01:31,332 DEBUG SenderThread:23847 [sender.py:send():235] send: files
2022-05-07 18:01:31,332 INFO SenderThread:23847 [sender.py:_save_file():947] saving file media/table/eval/epoch_8_incorrect_15_623faa4883d23d9d4194.table.json with policy now
2022-05-07 18:01:31,613 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:01:31,636 INFO Thread-23 :23847 [upload_job.py:push():137] Uploaded file /tmp/tmp6ko7nbgiwandb/3ku2cp9q-media/table/eval/epoch_8_incorrect_15_623faa4883d23d9d4194.table.json
2022-05-07 18:01:31,665 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: log_artifact
2022-05-07 18:01:31,665 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: log_artifact
2022-05-07 18:01:32,277 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log
2022-05-07 18:01:32,277 INFO Thread-7 :23847 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/validation[:90%]/epoch_8_15_b59f86c33aa51dff79ab.table.json
2022-05-07 18:01:32,277 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table
2022-05-07 18:01:32,277 INFO Thread-7 :23847 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/validation[:90%]
2022-05-07 18:01:32,643 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:01:32,679 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: log_artifact
2022-05-07 18:01:32,743 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:01:33,087 INFO Thread-25 :23847 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/af/47f5eac929f7d4f6e6c7d604ae396b
2022-05-07 18:01:33,277 INFO Thread-7 :23847 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/validation[:90%]/epoch_8_incorrect_15_b59f86c33aa51dff79ab.table.json
2022-05-07 18:01:33,278 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/validation[:90%]
2022-05-07 18:01:33,472 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: stop_status
2022-05-07 18:01:33,627 INFO SenderThread:23847 [sender.py:send_request_log_artifact():976] logged artifact run-3hqzwtal-validation90epoch_8 - {'id': 'QXJ0aWZhY3Q6MTI0MzU5MDky', 'digest': '4a0bf6423c6c046a9c499c17c09d3972', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5ODk5OTIw', 'latestArtifact': None}, 'version': 'latest'}
2022-05-07 18:01:33,627 DEBUG SenderThread:23847 [sender.py:send():235] send: files
2022-05-07 18:01:33,627 INFO SenderThread:23847 [sender.py:_save_file():947] saving file media/table/validation[[]:90%]/epoch_8_15_b59f86c33aa51dff79ab.table.json with policy now
2022-05-07 18:01:33,629 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: log_artifact
2022-05-07 18:01:33,924 INFO Thread-26 :23847 [upload_job.py:push():137] Uploaded file /tmp/tmp6ko7nbgiwandb/1xj8yudg-media/table/validation[:90%]/epoch_8_15_b59f86c33aa51dff79ab.table.json
2022-05-07 18:01:34,278 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log
2022-05-07 18:01:34,323 INFO Thread-28 :23847 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/1d/36af48785046dc58b4f025a3f8aea1
2022-05-07 18:01:35,034 INFO SenderThread:23847 [sender.py:send_request_log_artifact():976] logged artifact run-3hqzwtal-validation90epoch_8_incorrect - {'id': 'QXJ0aWZhY3Q6MTI0MzU5MTA0', 'digest': 'b891e308e194fc2dc7887b835c0c4ff7', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5ODk5OTIy', 'latestArtifact': None}, 'version': 'latest'}
2022-05-07 18:01:35,034 DEBUG SenderThread:23847 [sender.py:send():235] send: files
2022-05-07 18:01:35,034 INFO SenderThread:23847 [sender.py:_save_file():947] saving file media/table/validation[[]:90%]/epoch_8_incorrect_15_b59f86c33aa51dff79ab.table.json with policy now
2022-05-07 18:01:35,035 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: stop_status
2022-05-07 18:01:35,167 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:01:35,261 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: log_artifact
2022-05-07 18:01:35,262 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: log_artifact
2022-05-07 18:01:35,278 INFO Thread-7 :23847 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/validation[:95%]/epoch_8_15_a7d6e656cdc47be11a4d.table.json
2022-05-07 18:01:35,279 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table
2022-05-07 18:01:35,279 INFO Thread-7 :23847 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/validation[:95%]
2022-05-07 18:01:35,307 INFO Thread-29 :23847 [upload_job.py:push():137] Uploaded file /tmp/tmp6ko7nbgiwandb/1mnfpbtf-media/table/validation[:90%]/epoch_8_incorrect_15_b59f86c33aa51dff79ab.table.json
2022-05-07 18:01:35,456 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:01:35,513 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: log_artifact
2022-05-07 18:01:35,961 INFO Thread-31 :23847 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/59/70265576a772597adeb57dd33a7a30
2022-05-07 18:01:36,121 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:01:36,149 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: partial_history
2022-05-07 18:01:36,279 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log
2022-05-07 18:01:36,279 INFO Thread-7 :23847 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/validation[:95%]/epoch_8_incorrect_15_623faa4883d23d9d4194.table.json
2022-05-07 18:01:36,279 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/validation[:95%]
2022-05-07 18:01:36,772 INFO SenderThread:23847 [sender.py:send_request_log_artifact():976] logged artifact run-3hqzwtal-validation95epoch_8 - {'id': 'QXJ0aWZhY3Q6MTI0MzU5MTEx', 'digest': 'b7f474680722e0d2994717ec03a92b01', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5ODk5OTIz', 'latestArtifact': None}, 'version': 'latest'}
2022-05-07 18:01:36,773 DEBUG SenderThread:23847 [sender.py:send():235] send: files
2022-05-07 18:01:36,773 INFO SenderThread:23847 [sender.py:_save_file():947] saving file media/table/validation[[]:95%]/epoch_8_15_a7d6e656cdc47be11a4d.table.json with policy now
2022-05-07 18:01:36,774 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: log_artifact
2022-05-07 18:01:37,060 INFO Thread-32 :23847 [upload_job.py:push():137] Uploaded file /tmp/tmp6ko7nbgiwandb/3j6qd40g-media/table/validation[:95%]/epoch_8_15_a7d6e656cdc47be11a4d.table.json
2022-05-07 18:01:38,905 INFO Thread-34 :23847 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/25/29be3032c57f98a337547e6fe497da
2022-05-07 18:01:39,144 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: poll_exit
2022-05-07 18:01:39,281 INFO Thread-7 :23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log
2022-05-07 18:01:40,840 INFO SenderThread:23847 [sender.py:send_request_log_artifact():976] logged artifact run-3hqzwtal-validation95epoch_8_incorrect - {'id': 'QXJ0aWZhY3Q6MTI0MzU5MTIz', 'digest': '009449b4f57a6cf87bce6279ce908e6c', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5ODk5OTI1', 'latestArtifact': None}, 'version': 'latest'}
2022-05-07 18:01:40,840 DEBUG SenderThread:23847 [sender.py:send():235] send: files
2022-05-07 18:01:40,840 INFO SenderThread:23847 [sender.py:_save_file():947] saving file media/table/validation[[]:95%]/epoch_8_incorrect_15_623faa4883d23d9d4194.table.json with policy now
2022-05-07 18:01:40,841 DEBUG SenderThread:23847 [sender.py:send():235] send: telemetry
2022-05-07 18:01:40,841 DEBUG SenderThread:23847 [sender.py:send():235] send: history
2022-05-07 18:01:40,842 DEBUG SenderThread:23847 [sender.py:send():235] send: summary
2022-05-07 18:01:40,843 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-07 18:01:40,843 DEBUG SenderThread:23847 [sender.py:send():235] send: exit
2022-05-07 18:01:40,843 INFO SenderThread:23847 [sender.py:send_exit():371] handling exit code: 0
2022-05-07 18:01:40,845 INFO SenderThread:23847 [sender.py:send_exit():373] handling runtime: 204
2022-05-07 18:01:40,845 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-07 18:01:40,846 INFO SenderThread:23847 [sender.py:send_exit():379] send defer
2022-05-07 18:01:40,846 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: poll_exit
2022-05-07 18:01:40,847 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: defer
2022-05-07 18:01:40,847 INFO HandlerThread:23847 [handler.py:handle_request_defer():164] handle defer: 0
2022-05-07 18:01:40,847 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: defer
2022-05-07 18:01:40,847 INFO SenderThread:23847 [sender.py:send_request_defer():388] handle sender defer: 0
2022-05-07 18:01:40,847 INFO SenderThread:23847 [sender.py:transition_state():392] send defer: 1
2022-05-07 18:01:40,850 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: defer
2022-05-07 18:01:40,850 INFO HandlerThread:23847 [handler.py:handle_request_defer():164] handle defer: 1
2022-05-07 18:01:40,857 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: defer
2022-05-07 18:01:40,857 INFO SenderThread:23847 [sender.py:send_request_defer():388] handle sender defer: 1
2022-05-07 18:01:40,857 INFO SenderThread:23847 [sender.py:transition_state():392] send defer: 2
2022-05-07 18:01:40,857 DEBUG SenderThread:23847 [sender.py:send():235] send: stats
2022-05-07 18:01:40,857 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: defer
2022-05-07 18:01:40,857 INFO HandlerThread:23847 [handler.py:handle_request_defer():164] handle defer: 2
2022-05-07 18:01:40,858 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: defer
2022-05-07 18:01:40,858 INFO SenderThread:23847 [sender.py:send_request_defer():388] handle sender defer: 2
2022-05-07 18:01:40,858 INFO SenderThread:23847 [sender.py:transition_state():392] send defer: 3
2022-05-07 18:01:40,858 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: defer
2022-05-07 18:01:40,858 INFO HandlerThread:23847 [handler.py:handle_request_defer():164] handle defer: 3
2022-05-07 18:01:40,858 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: defer
2022-05-07 18:01:40,858 INFO SenderThread:23847 [sender.py:send_request_defer():388] handle sender defer: 3
2022-05-07 18:01:40,858 INFO SenderThread:23847 [sender.py:transition_state():392] send defer: 4
2022-05-07 18:01:40,859 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: defer
2022-05-07 18:01:40,859 INFO HandlerThread:23847 [handler.py:handle_request_defer():164] handle defer: 4
2022-05-07 18:01:40,859 DEBUG SenderThread:23847 [sender.py:send():235] send: summary
2022-05-07 18:01:40,860 INFO SenderThread:23847 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-05-07 18:01:40,860 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: defer
2022-05-07 18:01:40,861 INFO SenderThread:23847 [sender.py:send_request_defer():388] handle sender defer: 4
2022-05-07 18:01:40,861 INFO SenderThread:23847 [sender.py:transition_state():392] send defer: 5
2022-05-07 18:01:40,861 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: defer
2022-05-07 18:01:40,861 INFO HandlerThread:23847 [handler.py:handle_request_defer():164] handle defer: 5
2022-05-07 18:01:40,861 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: defer
2022-05-07 18:01:40,861 INFO SenderThread:23847 [sender.py:send_request_defer():388] handle sender defer: 5
2022-05-07 18:01:40,913 INFO SenderThread:23847 [sender.py:transition_state():392] send defer: 6
2022-05-07 18:01:40,914 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: defer
2022-05-07 18:01:40,914 INFO HandlerThread:23847 [handler.py:handle_request_defer():164] handle defer: 6
2022-05-07 18:01:40,914 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: defer
2022-05-07 18:01:40,914 INFO SenderThread:23847 [sender.py:send_request_defer():388] handle sender defer: 6
2022-05-07 18:01:40,914 INFO SenderThread:23847 [dir_watcher.py:finish():283] shutting down directory watcher
2022-05-07 18:01:40,948 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: poll_exit
2022-05-07 18:01:41,132 INFO Thread-35 :23847 [upload_job.py:push():137] Uploaded file /tmp/tmp6ko7nbgiwandb/2zz6fan7-media/table/validation[:95%]/epoch_8_incorrect_15_623faa4883d23d9d4194.table.json
2022-05-07 18:01:41,282 INFO SenderThread:23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-summary.json
2022-05-07 18:01:41,282 INFO SenderThread:23847 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/config.yaml
2022-05-07 18:01:41,282 INFO SenderThread:23847 [dir_watcher.py:finish():313] scan: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files
2022-05-07 18:01:41,283 INFO SenderThread:23847 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/requirements.txt requirements.txt
2022-05-07 18:01:41,283 INFO SenderThread:23847 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log output.log
2022-05-07 18:01:41,283 INFO SenderThread:23847 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-metadata.json wandb-metadata.json
2022-05-07 18:01:41,283 INFO SenderThread:23847 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/config.yaml config.yaml
2022-05-07 18:01:41,283 INFO SenderThread:23847 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-summary.json wandb-summary.json
2022-05-07 18:01:41,284 INFO SenderThread:23847 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/validation[:95%]/epoch_8_incorrect_15_623faa4883d23d9d4194.table.json media/table/validation[:95%]/epoch_8_incorrect_15_623faa4883d23d9d4194.table.json
2022-05-07 18:01:41,284 INFO SenderThread:23847 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/validation[:95%]/epoch_8_15_a7d6e656cdc47be11a4d.table.json media/table/validation[:95%]/epoch_8_15_a7d6e656cdc47be11a4d.table.json
2022-05-07 18:01:41,286 INFO SenderThread:23847 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/eval/epoch_6_10_f901d5b4b0d1263be066.table.json media/table/eval/epoch_6_10_f901d5b4b0d1263be066.table.json
2022-05-07 18:01:41,286 INFO SenderThread:23847 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/eval/epoch_8_incorrect_15_623faa4883d23d9d4194.table.json media/table/eval/epoch_8_incorrect_15_623faa4883d23d9d4194.table.json
2022-05-07 18:01:41,287 INFO SenderThread:23847 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/eval/epoch_8_15_a7d6e656cdc47be11a4d.table.json media/table/eval/epoch_8_15_a7d6e656cdc47be11a4d.table.json
2022-05-07 18:01:41,287 INFO SenderThread:23847 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/eval/epoch_3_5_f901d5b4b0d1263be066.table.json media/table/eval/epoch_3_5_f901d5b4b0d1263be066.table.json
2022-05-07 18:01:41,291 INFO SenderThread:23847 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/validation[:90%]/epoch_8_incorrect_15_b59f86c33aa51dff79ab.table.json media/table/validation[:90%]/epoch_8_incorrect_15_b59f86c33aa51dff79ab.table.json
2022-05-07 18:01:41,291 INFO SenderThread:23847 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/media/table/validation[:90%]/epoch_8_15_b59f86c33aa51dff79ab.table.json media/table/validation[:90%]/epoch_8_15_b59f86c33aa51dff79ab.table.json
2022-05-07 18:01:41,292 INFO SenderThread:23847 [sender.py:transition_state():392] send defer: 7
2022-05-07 18:01:41,292 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: poll_exit
2022-05-07 18:01:41,295 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: defer
2022-05-07 18:01:41,295 INFO HandlerThread:23847 [handler.py:handle_request_defer():164] handle defer: 7
2022-05-07 18:01:41,296 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: defer
2022-05-07 18:01:41,296 INFO SenderThread:23847 [sender.py:send_request_defer():388] handle sender defer: 7
2022-05-07 18:01:41,296 INFO SenderThread:23847 [file_pusher.py:finish():145] shutting down file pusher
2022-05-07 18:01:41,396 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: poll_exit
2022-05-07 18:01:41,396 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: poll_exit
2022-05-07 18:01:41,498 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: poll_exit
2022-05-07 18:01:41,498 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: poll_exit
2022-05-07 18:01:41,546 INFO Thread-36 :23847 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/requirements.txt
2022-05-07 18:01:41,568 INFO Thread-38 :23847 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/config.yaml
2022-05-07 18:01:41,600 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: poll_exit
2022-05-07 18:01:41,600 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: poll_exit
2022-05-07 18:01:41,701 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: poll_exit
2022-05-07 18:01:41,702 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: poll_exit
2022-05-07 18:01:41,803 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: poll_exit
2022-05-07 18:01:41,803 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: poll_exit
2022-05-07 18:01:41,842 INFO Thread-37 :23847 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/output.log
2022-05-07 18:01:41,848 INFO Thread-39 :23847 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/files/wandb-summary.json
2022-05-07 18:01:41,905 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: poll_exit
2022-05-07 18:01:41,905 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: poll_exit
2022-05-07 18:01:42,006 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: poll_exit
2022-05-07 18:01:42,007 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: poll_exit
2022-05-07 18:01:42,049 INFO Thread-6 :23847 [sender.py:transition_state():392] send defer: 8
2022-05-07 18:01:42,049 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: defer
2022-05-07 18:01:42,049 INFO HandlerThread:23847 [handler.py:handle_request_defer():164] handle defer: 8
2022-05-07 18:01:42,050 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: defer
2022-05-07 18:01:42,050 INFO SenderThread:23847 [sender.py:send_request_defer():388] handle sender defer: 8
2022-05-07 18:01:42,108 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: poll_exit
2022-05-07 18:01:42,194 INFO SenderThread:23847 [sender.py:transition_state():392] send defer: 9
2022-05-07 18:01:42,194 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: poll_exit
2022-05-07 18:01:42,195 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: defer
2022-05-07 18:01:42,195 INFO HandlerThread:23847 [handler.py:handle_request_defer():164] handle defer: 9
2022-05-07 18:01:42,195 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: defer
2022-05-07 18:01:42,195 INFO SenderThread:23847 [sender.py:send_request_defer():388] handle sender defer: 9
2022-05-07 18:01:42,195 INFO SenderThread:23847 [sender.py:transition_state():392] send defer: 10
2022-05-07 18:01:42,196 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: defer
2022-05-07 18:01:42,196 INFO HandlerThread:23847 [handler.py:handle_request_defer():164] handle defer: 10
2022-05-07 18:01:42,196 DEBUG SenderThread:23847 [sender.py:send():235] send: final
2022-05-07 18:01:42,196 DEBUG SenderThread:23847 [sender.py:send():235] send: footer
2022-05-07 18:01:42,196 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: defer
2022-05-07 18:01:42,196 INFO SenderThread:23847 [sender.py:send_request_defer():388] handle sender defer: 10
2022-05-07 18:01:42,296 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: poll_exit
2022-05-07 18:01:42,296 DEBUG SenderThread:23847 [sender.py:send_request():249] send_request: poll_exit
2022-05-07 18:01:42,296 INFO SenderThread:23847 [file_pusher.py:join():150] waiting for file pusher
2022-05-07 18:01:42,606 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: sampled_history
2022-05-07 18:01:42,608 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: get_summary
2022-05-07 18:01:42,610 DEBUG HandlerThread:23847 [handler.py:handle_request():141] handle_request: shutdown
2022-05-07 18:01:42,610 INFO HandlerThread:23847 [handler.py:finish():778] shutting down handler
2022-05-07 18:01:43,197 INFO WriterThread:23847 [datastore.py:close():281] close: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_175813-3hqzwtal/run-3hqzwtal.wandb
2022-05-07 18:01:43,505 INFO SenderThread:23847 [sender.py:finish():1078] shutting down sender
2022-05-07 18:01:43,505 INFO SenderThread:23847 [file_pusher.py:finish():145] shutting down file pusher
2022-05-07 18:01:43,505 INFO SenderThread:23847 [file_pusher.py:join():150] waiting for file pusher
2022-05-07 18:01:43,507 INFO MainThread:23847 [internal.py:handle_exit():82] Internal process exited
|