| 2022-01-26 12:15:54,828 INFO MainThread:2226380 [internal.py:wandb_internal():87] W&B internal server running at pid: 2226380, started at: 2022-01-26 12:15:54.828545 | |
| 2022-01-26 12:15:54,831 DEBUG HandlerThread:2226380 [handler.py:handle_request():130] handle_request: check_version | |
| 2022-01-26 12:15:54,831 INFO WriterThread:2226380 [datastore.py:open_for_write():77] open: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/run-295enqix.wandb | |
| 2022-01-26 12:15:54,833 DEBUG SenderThread:2226380 [sender.py:send():234] send: header | |
| 2022-01-26 12:15:54,834 DEBUG SenderThread:2226380 [sender.py:send_request():248] send_request: check_version | |
| 2022-01-26 12:15:55,103 DEBUG SenderThread:2226380 [sender.py:send():234] send: run | |
| 2022-01-26 12:15:55,276 INFO SenderThread:2226380 [dir_watcher.py:__init__():169] watching files in: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files | |
| 2022-01-26 12:15:55,276 INFO SenderThread:2226380 [sender.py:_start_run_threads():804] run started: 295enqix with start time 1643217354 | |
| 2022-01-26 12:15:55,276 DEBUG SenderThread:2226380 [sender.py:send():234] send: summary | |
| 2022-01-26 12:15:55,276 INFO SenderThread:2226380 [sender.py:_save_file():939] saving file wandb-summary.json with policy end | |
| 2022-01-26 12:15:55,278 DEBUG HandlerThread:2226380 [handler.py:handle_request():130] handle_request: run_start | |
| 2022-01-26 12:15:55,284 DEBUG HandlerThread:2226380 [meta.py:__init__():40] meta init | |
| 2022-01-26 12:15:55,285 DEBUG HandlerThread:2226380 [meta.py:__init__():54] meta init done | |
| 2022-01-26 12:15:55,285 DEBUG HandlerThread:2226380 [meta.py:probe():214] probe | |
| 2022-01-26 12:15:55,291 DEBUG HandlerThread:2226380 [meta.py:_setup_git():204] setup git | |
| 2022-01-26 12:15:55,309 DEBUG HandlerThread:2226380 [meta.py:_setup_git():211] setup git done | |
| 2022-01-26 12:15:55,309 DEBUG HandlerThread:2226380 [meta.py:_save_pip():58] save pip | |
| 2022-01-26 12:15:55,309 DEBUG HandlerThread:2226380 [meta.py:_save_pip():72] save pip done | |
| 2022-01-26 12:15:55,309 DEBUG HandlerThread:2226380 [meta.py:probe():252] probe done | |
| 2022-01-26 12:15:55,313 DEBUG SenderThread:2226380 [sender.py:send():234] send: files | |
| 2022-01-26 12:15:55,313 INFO SenderThread:2226380 [sender.py:_save_file():939] saving file wandb-metadata.json with policy now | |
| 2022-01-26 12:15:55,322 DEBUG HandlerThread:2226380 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-26 12:15:55,323 DEBUG SenderThread:2226380 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-26 12:15:55,460 DEBUG SenderThread:2226380 [sender.py:send():234] send: config | |
| 2022-01-26 12:15:55,461 DEBUG SenderThread:2226380 [sender.py:send():234] send: metric | |
| 2022-01-26 12:15:55,462 DEBUG SenderThread:2226380 [sender.py:send():234] send: metric | |
| 2022-01-26 12:15:55,462 WARNING SenderThread:2226380 [sender.py:send_metric():897] Seen metric with glob (shouldnt happen) | |
| 2022-01-26 12:15:55,937 INFO Thread-11 :2226380 [upload_job.py:push():137] Uploaded file /tmp/tmp0kjyvd1nwandb/sa83kpbt-wandb-metadata.json | |
| 2022-01-26 12:15:56,277 INFO Thread-8 :2226380 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/wandb-metadata.json | |
| 2022-01-26 12:15:56,278 INFO Thread-8 :2226380 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/requirements.txt | |
| 2022-01-26 12:15:56,278 INFO Thread-8 :2226380 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/output.log | |
| 2022-01-26 12:15:56,278 INFO Thread-8 :2226380 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/wandb-summary.json | |
| 2022-01-26 12:15:58,277 INFO Thread-8 :2226380 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/output.log | |
| 2022-01-26 12:16:10,577 DEBUG HandlerThread:2226380 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-26 12:16:10,577 DEBUG SenderThread:2226380 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-26 12:16:16,285 INFO Thread-8 :2226380 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/output.log | |
| 2022-01-26 12:16:24,276 DEBUG SenderThread:2226380 [sender.py:send():234] send: stats | |
| 2022-01-26 12:16:25,718 DEBUG HandlerThread:2226380 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-26 12:16:25,719 DEBUG SenderThread:2226380 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-26 12:16:26,289 INFO Thread-8 :2226380 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/config.yaml | |
| 2022-01-26 12:16:30,291 INFO Thread-8 :2226380 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/output.log | |
| 2022-01-26 12:16:40,861 DEBUG HandlerThread:2226380 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-26 12:16:40,862 DEBUG SenderThread:2226380 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-26 12:16:44,298 INFO Thread-8 :2226380 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/output.log | |
| 2022-01-26 12:16:55,255 DEBUG SenderThread:2226380 [sender.py:send():234] send: stats | |
| 2022-01-26 12:16:56,119 DEBUG HandlerThread:2226380 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-26 12:16:56,119 DEBUG SenderThread:2226380 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-26 12:16:56,303 INFO Thread-8 :2226380 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/output.log | |
| 2022-01-26 12:17:06,308 INFO Thread-8 :2226380 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/output.log | |
| 2022-01-26 12:17:11,395 DEBUG HandlerThread:2226380 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-26 12:17:11,395 DEBUG SenderThread:2226380 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-26 12:17:16,312 INFO Thread-8 :2226380 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/output.log | |
| 2022-01-26 12:17:26,266 DEBUG SenderThread:2226380 [sender.py:send():234] send: stats | |
| 2022-01-26 12:17:26,317 INFO Thread-8 :2226380 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/output.log | |
| 2022-01-26 12:17:26,631 DEBUG HandlerThread:2226380 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-26 12:17:26,632 DEBUG SenderThread:2226380 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-26 12:17:34,321 INFO Thread-8 :2226380 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/output.log | |
| 2022-01-26 12:17:41,857 DEBUG HandlerThread:2226380 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-26 12:17:41,858 DEBUG SenderThread:2226380 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-26 12:17:42,325 INFO Thread-8 :2226380 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/output.log | |
| 2022-01-26 12:17:50,329 INFO Thread-8 :2226380 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/output.log | |
| 2022-01-26 12:17:56,331 INFO Thread-8 :2226380 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/output.log | |
| 2022-01-26 12:17:57,090 DEBUG HandlerThread:2226380 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-26 12:17:57,090 DEBUG SenderThread:2226380 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-26 12:17:57,233 DEBUG SenderThread:2226380 [sender.py:send():234] send: stats | |
| 2022-01-26 12:18:02,334 INFO Thread-8 :2226380 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/output.log | |
| 2022-01-26 12:18:08,337 INFO Thread-8 :2226380 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/output.log | |
| 2022-01-26 12:18:12,358 DEBUG HandlerThread:2226380 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-26 12:18:12,359 DEBUG SenderThread:2226380 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-26 12:18:13,339 INFO Thread-8 :2226380 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/output.log | |
| 2022-01-26 12:18:27,555 DEBUG HandlerThread:2226380 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-26 12:18:27,555 DEBUG SenderThread:2226380 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-26 12:18:28,215 DEBUG SenderThread:2226380 [sender.py:send():234] send: stats | |
| 2022-01-26 12:18:31,347 INFO Thread-8 :2226380 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/output.log | |
| 2022-01-26 12:18:42,695 DEBUG HandlerThread:2226380 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-26 12:18:42,695 DEBUG SenderThread:2226380 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-26 12:18:45,354 INFO Thread-8 :2226380 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/output.log | |
| 2022-01-26 12:18:57,952 DEBUG HandlerThread:2226380 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-26 12:18:57,952 DEBUG SenderThread:2226380 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-26 12:18:59,199 DEBUG SenderThread:2226380 [sender.py:send():234] send: stats | |
| 2022-01-26 12:18:59,360 INFO Thread-8 :2226380 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/output.log | |
| 2022-01-26 12:19:11,366 INFO Thread-8 :2226380 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/output.log | |
| 2022-01-26 12:19:13,205 DEBUG HandlerThread:2226380 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-26 12:19:13,206 DEBUG SenderThread:2226380 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-26 12:19:23,373 INFO Thread-8 :2226380 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/output.log | |
| 2022-01-26 12:19:28,370 DEBUG HandlerThread:2226380 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-26 12:19:28,371 DEBUG SenderThread:2226380 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-26 12:19:30,192 DEBUG SenderThread:2226380 [sender.py:send():234] send: stats | |
| 2022-01-26 12:19:33,379 INFO Thread-8 :2226380 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/output.log | |
| 2022-01-26 12:19:41,382 INFO Thread-8 :2226380 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/output.log | |
| 2022-01-26 12:19:43,543 DEBUG HandlerThread:2226380 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-26 12:19:43,544 DEBUG SenderThread:2226380 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-26 12:19:51,386 INFO Thread-8 :2226380 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/output.log | |
| 2022-01-26 12:19:58,761 DEBUG HandlerThread:2226380 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-26 12:19:58,761 DEBUG SenderThread:2226380 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-26 12:19:59,389 INFO Thread-8 :2226380 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/output.log | |
| 2022-01-26 12:20:01,184 DEBUG SenderThread:2226380 [sender.py:send():234] send: stats | |
| 2022-01-26 12:20:07,393 INFO Thread-8 :2226380 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/output.log | |
| 2022-01-26 12:20:13,395 INFO Thread-8 :2226380 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/output.log | |
| 2022-01-26 12:20:13,950 DEBUG HandlerThread:2226380 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-26 12:20:13,950 DEBUG SenderThread:2226380 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-26 12:20:19,398 INFO Thread-8 :2226380 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/output.log | |
| 2022-01-26 12:20:25,401 INFO Thread-8 :2226380 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/output.log | |
| 2022-01-26 12:20:29,257 DEBUG HandlerThread:2226380 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-26 12:20:29,257 DEBUG SenderThread:2226380 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-26 12:20:29,402 INFO Thread-8 :2226380 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/output.log | |
| 2022-01-26 12:20:32,140 DEBUG SenderThread:2226380 [sender.py:send():234] send: stats | |
| 2022-01-26 12:20:44,546 DEBUG HandlerThread:2226380 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-26 12:20:44,547 DEBUG SenderThread:2226380 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-26 12:20:47,411 INFO Thread-8 :2226380 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/output.log | |
| 2022-01-26 12:20:59,689 DEBUG HandlerThread:2226380 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-26 12:20:59,690 DEBUG SenderThread:2226380 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-26 12:21:03,070 DEBUG SenderThread:2226380 [sender.py:send():234] send: stats | |
| 2022-01-26 12:21:04,418 INFO Thread-8 :2226380 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/output.log | |
| 2022-01-26 12:21:14,840 DEBUG HandlerThread:2226380 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-26 12:21:14,840 DEBUG SenderThread:2226380 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-26 12:21:18,425 INFO Thread-8 :2226380 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/output.log | |
| 2022-01-26 12:21:32,200 WARNING MainThread:2226380 [internal.py:is_dead():366] Internal process exiting, parent pid 2226158 disappeared | |
| 2022-01-26 12:21:32,200 ERROR MainThread:2226380 [internal.py:wandb_internal():142] Internal process shutdown. | |
| 2022-01-26 12:21:32,483 INFO SenderThread:2226380 [sender.py:finish():1070] shutting down sender | |
| 2022-01-26 12:21:32,484 INFO SenderThread:2226380 [dir_watcher.py:finish():283] shutting down directory watcher | |
| 2022-01-26 12:21:32,492 INFO HandlerThread:2226380 [handler.py:finish():731] shutting down handler | |
| 2022-01-26 12:21:32,503 INFO WriterThread:2226380 [datastore.py:close():281] close: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/run-295enqix.wandb | |
| 2022-01-26 12:21:33,428 INFO SenderThread:2226380 [dir_watcher.py:finish():313] scan: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files | |
| 2022-01-26 12:21:33,429 INFO SenderThread:2226380 [dir_watcher.py:finish():327] scan save: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/config.yaml config.yaml | |
| 2022-01-26 12:21:33,429 INFO SenderThread:2226380 [dir_watcher.py:finish():327] scan save: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/wandb-summary.json wandb-summary.json | |
| 2022-01-26 12:21:33,429 INFO SenderThread:2226380 [dir_watcher.py:finish():327] scan save: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/requirements.txt requirements.txt | |
| 2022-01-26 12:21:33,429 INFO SenderThread:2226380 [dir_watcher.py:finish():327] scan save: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/wandb-metadata.json wandb-metadata.json | |
| 2022-01-26 12:21:33,430 INFO SenderThread:2226380 [dir_watcher.py:finish():327] scan save: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/output.log output.log | |
| 2022-01-26 12:21:33,430 INFO SenderThread:2226380 [file_pusher.py:finish():177] shutting down file pusher | |
| 2022-01-26 12:21:33,430 INFO SenderThread:2226380 [file_pusher.py:join():182] waiting for file pusher | |
| 2022-01-26 12:21:34,064 INFO Thread-14 :2226380 [upload_job.py:push():137] Uploaded file /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/requirements.txt | |
| 2022-01-26 12:21:34,092 INFO Thread-13 :2226380 [upload_job.py:push():137] Uploaded file /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/wandb-summary.json | |
| 2022-01-26 12:21:34,134 INFO Thread-12 :2226380 [upload_job.py:push():137] Uploaded file /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/config.yaml | |
| 2022-01-26 12:21:34,181 INFO Thread-15 :2226380 [upload_job.py:push():137] Uploaded file /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220126_121554-295enqix/files/output.log | |
| 2022-01-26 12:21:34,621 INFO MainThread:2226380 [internal.py:handle_exit():77] Internal process exited | |