2022-05-07 17:42:57,892 INFO MainThread:15930 [internal.py:wandb_internal():92] W&B internal server running at pid: 15930, started at: 2022-05-07 17:42:57.892605 2022-05-07 17:42:57,895 INFO WriterThread:15930 [datastore.py:open_for_write():77] open: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/run-1i5p0k1k.wandb 2022-05-07 17:42:57,895 DEBUG SenderThread:15930 [sender.py:send():235] send: header 2022-05-07 17:42:57,896 DEBUG SenderThread:15930 [sender.py:send():235] send: run 2022-05-07 17:42:58,074 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: check_version 2022-05-07 17:42:58,075 INFO SenderThread:15930 [dir_watcher.py:__init__():169] watching files in: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files 2022-05-07 17:42:58,075 INFO SenderThread:15930 [sender.py:_start_run_threads():812] run started: 1i5p0k1k with start time 1651945377 2022-05-07 17:42:58,075 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:42:58,075 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:42:58,075 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: check_version 2022-05-07 17:42:58,136 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: run_start 2022-05-07 17:42:59,078 INFO Thread-7 :15930 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:43:00,346 DEBUG HandlerThread:15930 [meta.py:__init__():37] meta init 2022-05-07 17:43:00,346 DEBUG HandlerThread:15930 [meta.py:__init__():51] meta init done 2022-05-07 17:43:00,346 DEBUG HandlerThread:15930 [meta.py:probe():211] probe 2022-05-07 17:43:00,347 DEBUG HandlerThread:15930 [meta.py:_setup_git():201] setup git 2022-05-07 17:43:00,376 DEBUG HandlerThread:15930 [meta.py:_setup_git():208] setup git done 2022-05-07 17:43:00,376 DEBUG HandlerThread:15930 [meta.py:_save_pip():55] save pip 2022-05-07 17:43:00,376 DEBUG HandlerThread:15930 [meta.py:_save_pip():69] save pip done 2022-05-07 17:43:00,376 DEBUG HandlerThread:15930 [meta.py:probe():249] probe done 2022-05-07 17:43:00,380 DEBUG SenderThread:15930 [sender.py:send():235] send: files 2022-05-07 17:43:00,380 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-metadata.json with policy now 2022-05-07 17:43:00,386 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 17:43:00,386 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: stop_status 2022-05-07 17:43:00,414 DEBUG SenderThread:15930 [sender.py:send():235] send: telemetry 2022-05-07 17:43:00,671 INFO Thread-11 :15930 [upload_job.py:push():137] Uploaded file /tmp/tmp8t95wtgkwandb/3ix7w5qq-wandb-metadata.json 2022-05-07 17:43:01,088 INFO Thread-7 :15930 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:43:01,088 INFO Thread-7 :15930 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/requirements.txt 2022-05-07 17:43:01,088 INFO Thread-7 :15930 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-metadata.json 2022-05-07 17:43:03,089 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:43:05,090 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:43:15,416 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 17:43:15,416 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: stop_status 2022-05-07 17:43:17,094 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:43:21,095 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:43:23,096 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:43:28,427 DEBUG SenderThread:15930 [sender.py:send():235] send: stats 2022-05-07 17:43:29,098 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/config.yaml 2022-05-07 17:43:30,448 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 17:43:30,449 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: stop_status 2022-05-07 17:43:45,484 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 17:43:45,484 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: stop_status 2022-05-07 17:43:47,106 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:43:58,496 DEBUG SenderThread:15930 [sender.py:send():235] send: stats 2022-05-07 17:44:00,518 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 17:44:00,518 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: stop_status 2022-05-07 17:44:10,603 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:44:10,702 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:44:11,963 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:44:11,966 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:44:11,966 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:44:11,968 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:44:12,046 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:44:12,115 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:44:13,108 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:44:13,110 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:44:13,111 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:44:13,112 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:44:13,115 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:44:13,115 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:44:13,185 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:44:14,116 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:44:14,314 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:44:14,316 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:44:14,317 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:44:14,318 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:44:14,412 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:44:15,116 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:44:15,531 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:44:15,532 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:44:15,533 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:44:15,534 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:44:15,550 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 17:44:15,550 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: stop_status 2022-05-07 17:44:15,626 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:44:16,117 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:44:16,117 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:44:18,117 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:44:28,572 DEBUG SenderThread:15930 [sender.py:send():235] send: stats 2022-05-07 17:44:30,854 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 17:44:30,855 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: stop_status 2022-05-07 17:44:44,128 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:44:45,894 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 17:44:45,895 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: stop_status 2022-05-07 17:44:58,649 DEBUG SenderThread:15930 [sender.py:send():235] send: stats 2022-05-07 17:45:00,932 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 17:45:00,932 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: stop_status 2022-05-07 17:45:04,134 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:45:15,993 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 17:45:15,993 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: stop_status 2022-05-07 17:45:25,618 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:25,742 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: log_artifact 2022-05-07 17:45:25,742 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: log_artifact 2022-05-07 17:45:25,854 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:26,142 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:45:26,142 INFO Thread-7 :15930 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval/epoch_3_5_9e9b5d32dc07660953d8.table.json 2022-05-07 17:45:26,142 INFO Thread-7 :15930 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table 2022-05-07 17:45:26,142 INFO Thread-7 :15930 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media 2022-05-07 17:45:26,142 INFO Thread-7 :15930 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval 2022-05-07 17:45:26,223 INFO Thread-13 :15930 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/c4/b98dad1025f8f2340f391c80ff3526 2022-05-07 17:45:26,948 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:27,020 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:27,328 INFO SenderThread:15930 [sender.py:send_request_log_artifact():976] logged artifact run-1i5p0k1k-evalepoch_3 - {'id': 'QXJ0aWZhY3Q6MTI0MzUzMTY5', 'digest': '62978f223bc3c48bd7f6255ae4020797', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5ODk5MDU4', 'latestArtifact': None}, 'version': 'latest'} 2022-05-07 17:45:27,328 DEBUG SenderThread:15930 [sender.py:send():235] send: files 2022-05-07 17:45:27,328 INFO SenderThread:15930 [sender.py:_save_file():947] saving file media/table/eval/epoch_3_5_9e9b5d32dc07660953d8.table.json with policy now 2022-05-07 17:45:27,329 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:45:27,329 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:45:27,330 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:45:27,738 INFO Thread-14 :15930 [upload_job.py:push():137] Uploaded file /tmp/tmp8t95wtgkwandb/3f4qeuqa-media/table/eval/epoch_3_5_9e9b5d32dc07660953d8.table.json 2022-05-07 17:45:28,142 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:45:28,143 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:45:28,176 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:28,178 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:45:28,178 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:45:28,179 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:45:28,229 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:28,723 DEBUG SenderThread:15930 [sender.py:send():235] send: stats 2022-05-07 17:45:29,143 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:45:29,314 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:29,316 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:45:29,317 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:45:29,317 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:45:29,389 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:30,143 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:45:30,143 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:45:30,569 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:30,571 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:45:30,572 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:45:30,573 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:45:30,619 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:31,114 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 17:45:31,114 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: stop_status 2022-05-07 17:45:31,144 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:45:31,786 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:31,788 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:45:31,789 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:45:31,789 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:45:31,858 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:32,144 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:45:32,144 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:45:34,145 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:45:34,184 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:34,224 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: log_artifact 2022-05-07 17:45:34,224 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: log_artifact 2022-05-07 17:45:34,461 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:34,659 INFO Thread-16 :15930 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/c4/b98dad1025f8f2340f391c80ff3526 2022-05-07 17:45:35,145 INFO Thread-7 :15930 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval/epoch_6_10_9e9b5d32dc07660953d8.table.json 2022-05-07 17:45:35,146 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval 2022-05-07 17:45:35,313 INFO SenderThread:15930 [sender.py:send_request_log_artifact():976] logged artifact run-1i5p0k1k-evalepoch_6 - {'id': 'QXJ0aWZhY3Q6MTI0MzUzMjE5', 'digest': 'b5577517bef1dad00e5b2d8852d641d6', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5ODk5MDYx', 'latestArtifact': None}, 'version': 'latest'} 2022-05-07 17:45:35,314 DEBUG SenderThread:15930 [sender.py:send():235] send: files 2022-05-07 17:45:35,314 INFO SenderThread:15930 [sender.py:_save_file():947] saving file media/table/eval/epoch_6_10_9e9b5d32dc07660953d8.table.json with policy now 2022-05-07 17:45:35,550 INFO Thread-17 :15930 [upload_job.py:push():137] Uploaded file /tmp/tmp8t95wtgkwandb/2c6bno8e-media/table/eval/epoch_6_10_9e9b5d32dc07660953d8.table.json 2022-05-07 17:45:35,560 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:35,562 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:45:35,563 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:45:35,563 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:45:35,627 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:36,146 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:45:36,146 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:45:36,747 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:36,749 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:45:36,750 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:45:36,751 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:45:36,817 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:37,146 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:45:37,999 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:38,001 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:45:38,002 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:45:38,003 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:45:38,046 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:38,147 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:45:38,147 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:45:39,147 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:45:39,153 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:39,155 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:45:39,156 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:45:39,156 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:45:39,223 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:40,148 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:45:40,148 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:45:40,360 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:40,363 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:45:40,363 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:45:40,364 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:45:40,411 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:41,148 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:45:41,148 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:45:42,149 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:45:42,726 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:42,794 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: log_artifact 2022-05-07 17:45:42,794 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: log_artifact 2022-05-07 17:45:42,937 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:43,149 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:45:43,149 INFO Thread-7 :15930 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval/epoch_8_15_9e9b5d32dc07660953d8.table.json 2022-05-07 17:45:43,149 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval 2022-05-07 17:45:43,163 INFO Thread-19 :15930 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/c4/b98dad1025f8f2340f391c80ff3526 2022-05-07 17:45:44,149 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:45:44,154 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:44,223 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:45,150 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:45:45,304 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:45,359 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:46,150 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:45:46,292 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 17:45:46,582 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:46,651 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:47,151 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:45:47,694 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:47,749 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:48,849 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:48,922 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:49,152 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:45:51,153 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:45:51,207 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:51,253 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: log_artifact 2022-05-07 17:45:51,326 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:52,153 INFO Thread-7 :15930 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval/epoch_11_20_9e9b5d32dc07660953d8.table.json 2022-05-07 17:45:52,154 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval 2022-05-07 17:45:52,514 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:52,562 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:53,154 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:45:53,834 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:53,901 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:54,544 INFO SenderThread:15930 [sender.py:send_request_log_artifact():976] logged artifact run-1i5p0k1k-evalepoch_8 - {'id': 'QXJ0aWZhY3Q6MTI0MzUzMjY3', 'digest': 'a7ef8c9adb79bec1a1cf8f5535ab4177', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5ODk5MDY0', 'latestArtifact': None}, 'version': 'latest'} 2022-05-07 17:45:54,544 DEBUG SenderThread:15930 [sender.py:send():235] send: files 2022-05-07 17:45:54,544 INFO SenderThread:15930 [sender.py:_save_file():947] saving file media/table/eval/epoch_8_15_9e9b5d32dc07660953d8.table.json with policy now 2022-05-07 17:45:54,545 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:45:54,545 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:45:54,546 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:45:54,546 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:45:54,547 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:45:54,547 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:45:54,548 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: stop_status 2022-05-07 17:45:54,580 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:45:54,581 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:45:54,581 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:45:54,582 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:45:54,582 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:45:54,583 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:45:54,583 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:45:54,584 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:45:54,584 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:45:54,585 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: log_artifact 2022-05-07 17:45:54,781 INFO Thread-20 :15930 [upload_job.py:push():137] Uploaded file /tmp/tmp8t95wtgkwandb/2fx94wjj-media/table/eval/epoch_8_15_9e9b5d32dc07660953d8.table.json 2022-05-07 17:45:54,913 INFO Thread-22 :15930 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/c4/b98dad1025f8f2340f391c80ff3526 2022-05-07 17:45:55,015 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:55,069 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:55,155 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:45:55,155 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:45:55,411 INFO SenderThread:15930 [sender.py:send_request_log_artifact():976] logged artifact run-1i5p0k1k-evalepoch_11 - {'id': 'QXJ0aWZhY3Q6MTI0MzUzMzMz', 'digest': '42673fe35b3a06b9bc16d39708a92e51', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5ODk5MDY4', 'latestArtifact': None}, 'version': 'latest'} 2022-05-07 17:45:55,411 DEBUG SenderThread:15930 [sender.py:send():235] send: files 2022-05-07 17:45:55,412 INFO SenderThread:15930 [sender.py:_save_file():947] saving file media/table/eval/epoch_11_20_9e9b5d32dc07660953d8.table.json with policy now 2022-05-07 17:45:55,412 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:45:55,415 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:45:55,416 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:45:55,416 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:45:55,417 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:45:55,418 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:45:55,418 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:45:55,419 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:45:55,419 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:45:55,670 INFO Thread-23 :15930 [upload_job.py:push():137] Uploaded file /tmp/tmp8t95wtgkwandb/17x3alub-media/table/eval/epoch_11_20_9e9b5d32dc07660953d8.table.json 2022-05-07 17:45:56,155 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:45:56,251 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:56,253 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:45:56,254 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:45:56,254 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:45:56,319 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:57,156 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:45:57,156 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:45:57,449 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:57,451 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:45:57,451 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:45:57,452 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:45:57,498 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:58,156 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:45:58,793 DEBUG SenderThread:15930 [sender.py:send():235] send: stats 2022-05-07 17:45:59,157 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:45:59,807 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:45:59,847 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: log_artifact 2022-05-07 17:45:59,848 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: log_artifact 2022-05-07 17:46:00,054 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:00,158 INFO Thread-7 :15930 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval/epoch_13_25_9e9b5d32dc07660953d8.table.json 2022-05-07 17:46:00,158 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval 2022-05-07 17:46:01,016 INFO Thread-25 :15930 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/c4/b98dad1025f8f2340f391c80ff3526 2022-05-07 17:46:01,119 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:01,158 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:46:01,192 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:01,478 INFO SenderThread:15930 [sender.py:send_request_log_artifact():976] logged artifact run-1i5p0k1k-evalepoch_13 - {'id': 'QXJ0aWZhY3Q6MTI0MzUzMzY0', 'digest': 'f55899532bd6ce192dd2efa1003d0568', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5ODk5MDcw', 'latestArtifact': None}, 'version': 'latest'} 2022-05-07 17:46:01,479 DEBUG SenderThread:15930 [sender.py:send():235] send: files 2022-05-07 17:46:01,479 INFO SenderThread:15930 [sender.py:_save_file():947] saving file media/table/eval/epoch_13_25_9e9b5d32dc07660953d8.table.json with policy now 2022-05-07 17:46:01,479 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:46:01,480 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:46:01,481 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:46:02,158 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:46:02,385 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:02,387 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:46:02,388 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:46:02,389 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:46:02,435 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:02,779 INFO Thread-26 :15930 [upload_job.py:push():137] Uploaded file /tmp/tmp8t95wtgkwandb/2o0nt7mo-media/table/eval/epoch_13_25_9e9b5d32dc07660953d8.table.json 2022-05-07 17:46:03,159 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:46:03,159 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:46:03,580 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:03,582 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:46:03,583 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:46:03,586 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:46:03,651 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:04,159 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:46:04,785 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:04,788 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:46:04,788 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:46:04,789 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:46:04,836 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:05,160 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:46:05,160 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:46:06,025 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:06,027 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:46:06,027 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:46:06,028 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:46:06,095 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:06,160 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:46:06,399 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 17:46:06,400 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: stop_status 2022-05-07 17:46:07,161 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:46:08,390 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:08,440 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: log_artifact 2022-05-07 17:46:08,440 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: log_artifact 2022-05-07 17:46:08,538 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:08,897 INFO Thread-28 :15930 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/c4/b98dad1025f8f2340f391c80ff3526 2022-05-07 17:46:09,162 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:46:09,162 INFO Thread-7 :15930 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval/epoch_16_30_9e9b5d32dc07660953d8.table.json 2022-05-07 17:46:09,162 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval 2022-05-07 17:46:09,335 INFO SenderThread:15930 [sender.py:send_request_log_artifact():976] logged artifact run-1i5p0k1k-evalepoch_16 - {'id': 'QXJ0aWZhY3Q6MTI0MzUzNDI5', 'digest': 'e7ad5c5540c910d5fba7d7f72a91a4a1', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5ODk5MDc4', 'latestArtifact': None}, 'version': 'latest'} 2022-05-07 17:46:09,335 DEBUG SenderThread:15930 [sender.py:send():235] send: files 2022-05-07 17:46:09,335 INFO SenderThread:15930 [sender.py:_save_file():947] saving file media/table/eval/epoch_16_30_9e9b5d32dc07660953d8.table.json with policy now 2022-05-07 17:46:09,605 INFO Thread-29 :15930 [upload_job.py:push():137] Uploaded file /tmp/tmp8t95wtgkwandb/1fvhs8bz-media/table/eval/epoch_16_30_9e9b5d32dc07660953d8.table.json 2022-05-07 17:46:11,163 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:46:21,480 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 17:46:21,480 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: stop_status 2022-05-07 17:46:28,860 DEBUG SenderThread:15930 [sender.py:send():235] send: stats 2022-05-07 17:46:31,007 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:31,010 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:46:31,010 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:46:31,013 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:46:31,087 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:31,172 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:46:32,339 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:32,341 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:46:32,341 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:46:32,342 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:46:32,423 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:33,173 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:46:33,173 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:46:33,479 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:33,482 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:46:33,482 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:46:33,483 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:46:33,561 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:34,173 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:46:34,173 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:46:34,688 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:34,689 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:46:34,690 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:46:34,691 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:46:34,783 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:35,174 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:46:35,174 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:46:35,897 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:35,899 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:46:35,900 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:46:35,901 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:46:35,979 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:36,174 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:46:36,175 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:46:36,573 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 17:46:36,574 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: stop_status 2022-05-07 17:46:37,175 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:46:38,175 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:46:38,291 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:38,381 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: log_artifact 2022-05-07 17:46:38,382 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: log_artifact 2022-05-07 17:46:38,607 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:39,176 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:46:39,176 INFO Thread-7 :15930 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval/epoch_18_35_9e9b5d32dc07660953d8.table.json 2022-05-07 17:46:39,176 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval 2022-05-07 17:46:39,555 INFO Thread-31 :15930 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/c4/b98dad1025f8f2340f391c80ff3526 2022-05-07 17:46:39,702 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:39,774 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:39,973 INFO SenderThread:15930 [sender.py:send_request_log_artifact():976] logged artifact run-1i5p0k1k-evalepoch_18 - {'id': 'QXJ0aWZhY3Q6MTI0MzUzNzIw', 'digest': 'bc2b60ad488ac57aead98bb982902f5a', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5ODk5MjEz', 'latestArtifact': None}, 'version': 'latest'} 2022-05-07 17:46:39,974 DEBUG SenderThread:15930 [sender.py:send():235] send: files 2022-05-07 17:46:39,974 INFO SenderThread:15930 [sender.py:_save_file():947] saving file media/table/eval/epoch_18_35_9e9b5d32dc07660953d8.table.json with policy now 2022-05-07 17:46:39,974 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:46:39,975 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:46:39,976 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:46:40,176 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:46:40,177 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:46:40,255 INFO Thread-32 :15930 [upload_job.py:push():137] Uploaded file /tmp/tmp8t95wtgkwandb/3mwnmpk2-media/table/eval/epoch_18_35_9e9b5d32dc07660953d8.table.json 2022-05-07 17:46:40,953 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:40,955 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:46:40,956 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:46:40,956 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:46:41,003 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:41,177 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:46:41,177 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:46:42,177 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:46:42,182 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:42,183 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:46:42,184 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:46:42,185 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:46:42,273 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:43,178 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:46:43,178 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:46:43,349 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:43,351 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:46:43,351 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:46:43,352 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:46:43,385 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:44,178 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:46:44,179 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:46:44,415 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:44,417 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:46:44,418 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:46:44,419 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:46:44,487 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:45,179 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:46:45,179 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:46:46,179 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:46:46,806 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:47,030 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: log_artifact 2022-05-07 17:46:47,030 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: log_artifact 2022-05-07 17:46:47,180 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:46:47,180 INFO Thread-7 :15930 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval/epoch_21_40_9e9b5d32dc07660953d8.table.json 2022-05-07 17:46:47,180 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval 2022-05-07 17:46:47,322 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:47,663 INFO Thread-34 :15930 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/c4/b98dad1025f8f2340f391c80ff3526 2022-05-07 17:46:48,088 INFO SenderThread:15930 [sender.py:send_request_log_artifact():976] logged artifact run-1i5p0k1k-evalepoch_21 - {'id': 'QXJ0aWZhY3Q6MTI0MzUzODA4', 'digest': '2c124de00bd9cf16e4e97e2a94668031', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5ODk5MjQ3', 'latestArtifact': None}, 'version': 'latest'} 2022-05-07 17:46:48,088 DEBUG SenderThread:15930 [sender.py:send():235] send: files 2022-05-07 17:46:48,089 INFO SenderThread:15930 [sender.py:_save_file():947] saving file media/table/eval/epoch_21_40_9e9b5d32dc07660953d8.table.json with policy now 2022-05-07 17:46:48,180 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:46:48,317 INFO Thread-35 :15930 [upload_job.py:push():137] Uploaded file /tmp/tmp8t95wtgkwandb/ljrg051o-media/table/eval/epoch_21_40_9e9b5d32dc07660953d8.table.json 2022-05-07 17:46:48,515 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:48,518 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:46:48,518 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:46:48,519 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:46:48,564 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:49,181 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:46:49,653 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:49,655 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:46:49,655 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:46:49,656 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:46:49,747 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:50,181 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:46:50,182 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:46:50,932 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:50,935 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:46:50,935 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:46:50,936 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:46:50,981 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:51,182 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:46:51,765 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 17:46:51,765 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: stop_status 2022-05-07 17:46:52,182 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:46:52,271 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:52,273 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:46:52,274 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:46:52,275 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:46:52,337 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:53,183 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:46:53,353 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:53,355 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:46:53,356 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:46:53,357 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:46:53,390 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:54,183 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:46:54,184 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:46:55,675 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:55,853 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: log_artifact 2022-05-07 17:46:55,853 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: log_artifact 2022-05-07 17:46:55,966 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:56,184 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:46:56,185 INFO Thread-7 :15930 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval/epoch_23_45_9e9b5d32dc07660953d8.table.json 2022-05-07 17:46:56,185 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval 2022-05-07 17:46:56,232 INFO Thread-37 :15930 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/c4/b98dad1025f8f2340f391c80ff3526 2022-05-07 17:46:56,753 INFO SenderThread:15930 [sender.py:send_request_log_artifact():976] logged artifact run-1i5p0k1k-evalepoch_23 - {'id': 'QXJ0aWZhY3Q6MTI0MzUzODgw', 'digest': '02090012ceababc533ceac7285be89b1', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5ODk5MjY5', 'latestArtifact': None}, 'version': 'latest'} 2022-05-07 17:46:56,754 DEBUG SenderThread:15930 [sender.py:send():235] send: files 2022-05-07 17:46:56,754 INFO SenderThread:15930 [sender.py:_save_file():947] saving file media/table/eval/epoch_23_45_9e9b5d32dc07660953d8.table.json with policy now 2022-05-07 17:46:57,032 INFO Thread-38 :15930 [upload_job.py:push():137] Uploaded file /tmp/tmp8t95wtgkwandb/2iu7wlzo-media/table/eval/epoch_23_45_9e9b5d32dc07660953d8.table.json 2022-05-07 17:46:57,128 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:57,131 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:46:57,131 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:46:57,132 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:46:57,185 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:46:57,196 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:58,186 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:46:58,247 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:58,249 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:46:58,250 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:46:58,251 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:46:58,321 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:58,932 DEBUG SenderThread:15930 [sender.py:send():235] send: stats 2022-05-07 17:46:59,186 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:46:59,415 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:46:59,418 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:46:59,418 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:46:59,419 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:46:59,483 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:00,187 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:47:00,187 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:47:00,599 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:00,602 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:47:00,602 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:47:00,604 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:47:00,652 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:01,187 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:47:01,768 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:01,770 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:47:01,771 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:47:01,772 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:47:01,838 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:02,188 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:47:02,188 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:47:04,166 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:04,189 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:47:04,240 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: log_artifact 2022-05-07 17:47:04,240 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: log_artifact 2022-05-07 17:47:04,341 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:05,189 INFO Thread-7 :15930 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval/epoch_26_50_f901d5b4b0d1263be066.table.json 2022-05-07 17:47:05,190 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval 2022-05-07 17:47:05,429 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:05,495 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:06,190 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:47:06,662 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:06,734 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:07,020 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 17:47:07,124 INFO Thread-40 :15930 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/03/1aa3918a194407ac466420b1ea5405 2022-05-07 17:47:07,612 INFO SenderThread:15930 [sender.py:send_request_log_artifact():976] logged artifact run-1i5p0k1k-evalepoch_26 - {'id': 'QXJ0aWZhY3Q6MTI0MzUzOTU4', 'digest': '95d7e1d9457cb6cec7e08222a01ed120', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5ODk5Mjg3', 'latestArtifact': None}, 'version': 'latest'} 2022-05-07 17:47:07,612 DEBUG SenderThread:15930 [sender.py:send():235] send: files 2022-05-07 17:47:07,612 INFO SenderThread:15930 [sender.py:_save_file():947] saving file media/table/eval/epoch_26_50_f901d5b4b0d1263be066.table.json with policy now 2022-05-07 17:47:07,612 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:47:07,613 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:47:07,614 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:47:07,614 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:47:07,615 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:47:07,615 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:47:07,616 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: stop_status 2022-05-07 17:47:07,910 INFO Thread-41 :15930 [upload_job.py:push():137] Uploaded file /tmp/tmp8t95wtgkwandb/3n37n1s7-media/table/eval/epoch_26_50_f901d5b4b0d1263be066.table.json 2022-05-07 17:47:07,917 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:07,919 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:47:07,919 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:47:07,920 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:47:07,971 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:08,193 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:47:08,193 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:47:09,114 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:09,116 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:47:09,116 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:47:09,117 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:47:09,193 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:47:09,193 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:47:09,212 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:10,194 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:47:10,288 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:10,291 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:47:10,291 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:47:10,292 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:47:10,344 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:11,194 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:47:11,195 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:47:12,195 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:47:12,634 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:12,698 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: log_artifact 2022-05-07 17:47:12,698 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: log_artifact 2022-05-07 17:47:12,774 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:13,133 INFO Thread-43 :15930 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/03/1aa3918a194407ac466420b1ea5405 2022-05-07 17:47:13,195 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:47:13,196 INFO Thread-7 :15930 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval/epoch_28_55_f901d5b4b0d1263be066.table.json 2022-05-07 17:47:13,196 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval 2022-05-07 17:47:13,585 INFO SenderThread:15930 [sender.py:send_request_log_artifact():976] logged artifact run-1i5p0k1k-evalepoch_28 - {'id': 'QXJ0aWZhY3Q6MTI0MzUzOTk5', 'digest': '9edba21836fc677e9e8389aaffbe3faa', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5ODk5Mjk0', 'latestArtifact': None}, 'version': 'latest'} 2022-05-07 17:47:13,585 DEBUG SenderThread:15930 [sender.py:send():235] send: files 2022-05-07 17:47:13,585 INFO SenderThread:15930 [sender.py:_save_file():947] saving file media/table/eval/epoch_28_55_f901d5b4b0d1263be066.table.json with policy now 2022-05-07 17:47:13,847 INFO Thread-44 :15930 [upload_job.py:push():137] Uploaded file /tmp/tmp8t95wtgkwandb/3552emff-media/table/eval/epoch_28_55_f901d5b4b0d1263be066.table.json 2022-05-07 17:47:13,921 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:13,924 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:47:13,924 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:47:13,925 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:47:13,994 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:14,196 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:47:14,196 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:47:15,110 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:15,113 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:47:15,113 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:47:15,114 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:47:15,166 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:15,196 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:47:15,197 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:47:16,197 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:47:16,401 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:16,403 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:47:16,404 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:47:16,405 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:47:16,471 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:17,198 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:47:17,198 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:47:17,512 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:17,514 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:47:17,515 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:47:17,516 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:47:17,568 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:18,199 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:47:18,199 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:47:18,686 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:18,688 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:47:18,688 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:47:18,689 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:47:18,759 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:19,199 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:47:19,199 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:47:20,200 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:47:21,075 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:21,200 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:47:21,206 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: log_artifact 2022-05-07 17:47:21,207 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: log_artifact 2022-05-07 17:47:21,316 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:21,735 INFO Thread-46 :15930 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/03/1aa3918a194407ac466420b1ea5405 2022-05-07 17:47:22,118 INFO SenderThread:15930 [sender.py:send_request_log_artifact():976] logged artifact run-1i5p0k1k-evalepoch_31 - {'id': 'QXJ0aWZhY3Q6MTI0MzU0MDUy', 'digest': '870c902cebca02cb584853c36d67bce4', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5ODk5Mjk3', 'latestArtifact': None}, 'version': 'latest'} 2022-05-07 17:47:22,118 DEBUG SenderThread:15930 [sender.py:send():235] send: files 2022-05-07 17:47:22,118 INFO SenderThread:15930 [sender.py:_save_file():947] saving file media/table/eval/epoch_31_60_f901d5b4b0d1263be066.table.json with policy now 2022-05-07 17:47:22,201 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:47:22,201 INFO Thread-7 :15930 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval/epoch_31_60_f901d5b4b0d1263be066.table.json 2022-05-07 17:47:22,201 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval 2022-05-07 17:47:22,395 INFO Thread-47 :15930 [upload_job.py:push():137] Uploaded file /tmp/tmp8t95wtgkwandb/3raqeosp-media/table/eval/epoch_31_60_f901d5b4b0d1263be066.table.json 2022-05-07 17:47:22,442 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:22,444 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:47:22,445 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:47:22,446 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:47:22,497 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:22,953 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 17:47:22,953 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: stop_status 2022-05-07 17:47:23,201 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:47:23,201 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:47:23,647 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:23,649 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:47:23,650 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:47:23,650 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:47:23,719 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:24,202 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:47:24,202 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:47:24,858 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:24,860 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:47:24,861 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:47:24,862 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:47:24,905 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:25,202 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:47:25,202 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:47:26,077 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:26,080 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:47:26,080 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:47:26,081 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:47:26,147 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:26,203 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:47:27,203 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:47:27,235 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:27,237 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:47:27,237 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:47:27,238 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:47:27,289 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:28,204 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:47:29,005 DEBUG SenderThread:15930 [sender.py:send():235] send: stats 2022-05-07 17:47:29,205 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:47:29,624 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:29,965 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: log_artifact 2022-05-07 17:47:29,966 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: log_artifact 2022-05-07 17:47:30,208 INFO Thread-7 :15930 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval/epoch_33_65_92ae240f2472ae79ad31.table.json 2022-05-07 17:47:30,209 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval 2022-05-07 17:47:30,504 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:31,209 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:47:31,613 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:31,685 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:32,829 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:32,879 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:47:33,210 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:47:34,306 INFO Thread-49 :15930 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/00/66783abbd98b29e02682346c82293a 2022-05-07 17:47:34,830 INFO SenderThread:15930 [sender.py:send_request_log_artifact():976] logged artifact run-1i5p0k1k-evalepoch_33 - {'id': 'QXJ0aWZhY3Q6MTI0MzU0MTIx', 'digest': '49689cb3677eb1348a7e7c52d0115f1a', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5ODk5MzAw', 'latestArtifact': None}, 'version': 'latest'} 2022-05-07 17:47:34,830 DEBUG SenderThread:15930 [sender.py:send():235] send: files 2022-05-07 17:47:34,830 INFO SenderThread:15930 [sender.py:_save_file():947] saving file media/table/eval/epoch_33_65_92ae240f2472ae79ad31.table.json with policy now 2022-05-07 17:47:34,831 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:47:34,832 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:47:34,833 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:47:34,833 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:47:34,833 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:47:34,834 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:47:35,097 INFO Thread-50 :15930 [upload_job.py:push():137] Uploaded file /tmp/tmp8t95wtgkwandb/26g6gbke-media/table/eval/epoch_33_65_92ae240f2472ae79ad31.table.json 2022-05-07 17:47:35,211 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:47:35,211 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:47:38,016 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 17:47:38,016 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: stop_status 2022-05-07 17:47:50,220 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:47:53,049 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 17:47:53,049 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: stop_status 2022-05-07 17:47:59,083 DEBUG SenderThread:15930 [sender.py:send():235] send: stats 2022-05-07 17:48:02,226 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:48:08,097 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 17:48:08,097 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: stop_status 2022-05-07 17:48:13,316 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:48:13,407 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: log_artifact 2022-05-07 17:48:13,408 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: log_artifact 2022-05-07 17:48:13,468 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:48:13,694 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: log_artifact 2022-05-07 17:48:13,759 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:48:13,800 INFO Thread-52 :15930 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/af/47f5eac929f7d4f6e6c7d604ae396b 2022-05-07 17:48:14,212 INFO SenderThread:15930 [sender.py:send_request_log_artifact():976] logged artifact run-1i5p0k1k-validation90epoch_34 - {'id': 'QXJ0aWZhY3Q6MTI0MzU0NDY4', 'digest': 'ba45b3df01cdd17c73e05b3c1d7bba5f', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5ODk5MzM0', 'latestArtifact': None}, 'version': 'latest'} 2022-05-07 17:48:14,213 DEBUG SenderThread:15930 [sender.py:send():235] send: files 2022-05-07 17:48:14,213 INFO SenderThread:15930 [sender.py:_save_file():947] saving file media/table/validation[[]:90%]/epoch_34_67_b59f86c33aa51dff79ab.table.json with policy now 2022-05-07 17:48:14,214 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: log_artifact 2022-05-07 17:48:14,233 INFO Thread-7 :15930 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/validation[:90%]/epoch_34_67_b59f86c33aa51dff79ab.table.json 2022-05-07 17:48:14,233 INFO Thread-7 :15930 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/validation[:90%]/epoch_34_incorrect_67_b59f86c33aa51dff79ab.table.json 2022-05-07 17:48:14,234 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table 2022-05-07 17:48:14,234 INFO Thread-7 :15930 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/validation[:90%] 2022-05-07 17:48:14,580 INFO Thread-55 :15930 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/1d/36af48785046dc58b4f025a3f8aea1 2022-05-07 17:48:14,588 INFO Thread-53 :15930 [upload_job.py:push():137] Uploaded file /tmp/tmp8t95wtgkwandb/1jkdf40c-media/table/validation[:90%]/epoch_34_67_b59f86c33aa51dff79ab.table.json 2022-05-07 17:48:15,069 INFO SenderThread:15930 [sender.py:send_request_log_artifact():976] logged artifact run-1i5p0k1k-validation90epoch_34_incorrect - {'id': 'QXJ0aWZhY3Q6MTI0MzU0NDcz', 'digest': '6701b1c3375bf20b0e9b415806d9d998', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5ODk5MzM1', 'latestArtifact': None}, 'version': 'latest'} 2022-05-07 17:48:15,069 DEBUG SenderThread:15930 [sender.py:send():235] send: files 2022-05-07 17:48:15,069 INFO SenderThread:15930 [sender.py:_save_file():947] saving file media/table/validation[[]:90%]/epoch_34_incorrect_67_b59f86c33aa51dff79ab.table.json with policy now 2022-05-07 17:48:15,234 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:48:15,317 INFO Thread-56 :15930 [upload_job.py:push():137] Uploaded file /tmp/tmp8t95wtgkwandb/1sz99zcl-media/table/validation[:90%]/epoch_34_incorrect_67_b59f86c33aa51dff79ab.table.json 2022-05-07 17:48:16,044 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:48:16,116 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: log_artifact 2022-05-07 17:48:16,116 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: log_artifact 2022-05-07 17:48:16,235 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:48:16,235 INFO Thread-7 :15930 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/validation[:95%]/epoch_34_67_a7d6e656cdc47be11a4d.table.json 2022-05-07 17:48:16,239 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table 2022-05-07 17:48:16,239 INFO Thread-7 :15930 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/validation[:95%] 2022-05-07 17:48:16,255 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:48:16,327 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: log_artifact 2022-05-07 17:48:16,404 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:48:16,449 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 17:48:16,555 INFO Thread-58 :15930 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/59/70265576a772597adeb57dd33a7a30 2022-05-07 17:48:17,066 INFO SenderThread:15930 [sender.py:send_request_log_artifact():976] logged artifact run-1i5p0k1k-validation95epoch_34 - {'id': 'QXJ0aWZhY3Q6MTI0MzU0NDgz', 'digest': '99d8d46e06e71378993d6dc9793d3722', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5ODk5MzM2', 'latestArtifact': None}, 'version': 'latest'} 2022-05-07 17:48:17,066 DEBUG SenderThread:15930 [sender.py:send():235] send: files 2022-05-07 17:48:17,066 INFO SenderThread:15930 [sender.py:_save_file():947] saving file media/table/validation[[]:95%]/epoch_34_67_a7d6e656cdc47be11a4d.table.json with policy now 2022-05-07 17:48:17,067 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: log_artifact 2022-05-07 17:48:17,235 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:48:17,235 INFO Thread-7 :15930 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/validation[:95%]/epoch_34_incorrect_67_623faa4883d23d9d4194.table.json 2022-05-07 17:48:17,236 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/validation[:95%] 2022-05-07 17:48:17,323 INFO Thread-59 :15930 [upload_job.py:push():137] Uploaded file /tmp/tmp8t95wtgkwandb/3jrmrrlo-media/table/validation[:95%]/epoch_34_67_a7d6e656cdc47be11a4d.table.json 2022-05-07 17:48:18,236 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:48:19,358 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: poll_exit 2022-05-07 17:48:20,237 INFO Thread-7 :15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:48:24,459 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: poll_exit 2022-05-07 17:48:25,071 INFO Thread-61 :15930 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/25/29be3032c57f98a337547e6fe497da 2022-05-07 17:48:25,498 INFO SenderThread:15930 [sender.py:send_request_log_artifact():976] logged artifact run-1i5p0k1k-validation95epoch_34_incorrect - {'id': 'QXJ0aWZhY3Q6MTI0MzU0NTQ3', 'digest': '9e5b046266c0c5f196b4992086c08ea8', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5ODk5MzM4', 'latestArtifact': None}, 'version': 'latest'} 2022-05-07 17:48:25,499 DEBUG SenderThread:15930 [sender.py:send():235] send: files 2022-05-07 17:48:25,499 INFO SenderThread:15930 [sender.py:_save_file():947] saving file media/table/validation[[]:95%]/epoch_34_incorrect_67_623faa4883d23d9d4194.table.json with policy now 2022-05-07 17:48:25,499 DEBUG SenderThread:15930 [sender.py:send():235] send: telemetry 2022-05-07 17:48:25,500 DEBUG SenderThread:15930 [sender.py:send():235] send: history 2022-05-07 17:48:25,500 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:48:25,501 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:48:25,502 DEBUG SenderThread:15930 [sender.py:send():235] send: exit 2022-05-07 17:48:25,502 INFO SenderThread:15930 [sender.py:send_exit():371] handling exit code: 0 2022-05-07 17:48:25,502 INFO SenderThread:15930 [sender.py:send_exit():373] handling runtime: 321 2022-05-07 17:48:25,503 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:48:25,503 INFO SenderThread:15930 [sender.py:send_exit():379] send defer 2022-05-07 17:48:25,503 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: poll_exit 2022-05-07 17:48:25,504 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: poll_exit 2022-05-07 17:48:25,504 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: defer 2022-05-07 17:48:25,504 INFO HandlerThread:15930 [handler.py:handle_request_defer():164] handle defer: 0 2022-05-07 17:48:25,505 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: defer 2022-05-07 17:48:25,505 INFO SenderThread:15930 [sender.py:send_request_defer():388] handle sender defer: 0 2022-05-07 17:48:25,505 INFO SenderThread:15930 [sender.py:transition_state():392] send defer: 1 2022-05-07 17:48:25,507 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: defer 2022-05-07 17:48:25,508 INFO HandlerThread:15930 [handler.py:handle_request_defer():164] handle defer: 1 2022-05-07 17:48:25,550 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: defer 2022-05-07 17:48:25,550 INFO SenderThread:15930 [sender.py:send_request_defer():388] handle sender defer: 1 2022-05-07 17:48:25,551 INFO SenderThread:15930 [sender.py:transition_state():392] send defer: 2 2022-05-07 17:48:25,551 DEBUG SenderThread:15930 [sender.py:send():235] send: stats 2022-05-07 17:48:25,551 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: defer 2022-05-07 17:48:25,552 INFO HandlerThread:15930 [handler.py:handle_request_defer():164] handle defer: 2 2022-05-07 17:48:25,552 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: defer 2022-05-07 17:48:25,552 INFO SenderThread:15930 [sender.py:send_request_defer():388] handle sender defer: 2 2022-05-07 17:48:25,552 INFO SenderThread:15930 [sender.py:transition_state():392] send defer: 3 2022-05-07 17:48:25,552 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: defer 2022-05-07 17:48:25,552 INFO HandlerThread:15930 [handler.py:handle_request_defer():164] handle defer: 3 2022-05-07 17:48:25,553 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: defer 2022-05-07 17:48:25,553 INFO SenderThread:15930 [sender.py:send_request_defer():388] handle sender defer: 3 2022-05-07 17:48:25,553 INFO SenderThread:15930 [sender.py:transition_state():392] send defer: 4 2022-05-07 17:48:25,555 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: defer 2022-05-07 17:48:25,555 INFO HandlerThread:15930 [handler.py:handle_request_defer():164] handle defer: 4 2022-05-07 17:48:25,556 DEBUG SenderThread:15930 [sender.py:send():235] send: summary 2022-05-07 17:48:25,557 INFO SenderThread:15930 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 17:48:25,558 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: defer 2022-05-07 17:48:25,558 INFO SenderThread:15930 [sender.py:send_request_defer():388] handle sender defer: 4 2022-05-07 17:48:25,558 INFO SenderThread:15930 [sender.py:transition_state():392] send defer: 5 2022-05-07 17:48:25,558 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: defer 2022-05-07 17:48:25,558 INFO HandlerThread:15930 [handler.py:handle_request_defer():164] handle defer: 5 2022-05-07 17:48:25,558 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: defer 2022-05-07 17:48:25,558 INFO SenderThread:15930 [sender.py:send_request_defer():388] handle sender defer: 5 2022-05-07 17:48:25,606 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: poll_exit 2022-05-07 17:48:25,623 INFO SenderThread:15930 [sender.py:transition_state():392] send defer: 6 2022-05-07 17:48:25,623 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: poll_exit 2022-05-07 17:48:25,624 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: defer 2022-05-07 17:48:25,624 INFO HandlerThread:15930 [handler.py:handle_request_defer():164] handle defer: 6 2022-05-07 17:48:25,624 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: defer 2022-05-07 17:48:25,624 INFO SenderThread:15930 [sender.py:send_request_defer():388] handle sender defer: 6 2022-05-07 17:48:25,624 INFO SenderThread:15930 [dir_watcher.py:finish():283] shutting down directory watcher 2022-05-07 17:48:25,725 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: poll_exit 2022-05-07 17:48:25,761 INFO Thread-62 :15930 [upload_job.py:push():137] Uploaded file /tmp/tmp8t95wtgkwandb/2lizbf1i-media/table/validation[:95%]/epoch_34_incorrect_67_623faa4883d23d9d4194.table.json 2022-05-07 17:48:26,239 INFO SenderThread:15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:48:26,239 INFO SenderThread:15930 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/config.yaml 2022-05-07 17:48:26,239 INFO SenderThread:15930 [dir_watcher.py:finish():313] scan: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files 2022-05-07 17:48:26,240 INFO SenderThread:15930 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/requirements.txt requirements.txt 2022-05-07 17:48:26,240 INFO SenderThread:15930 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log output.log 2022-05-07 17:48:26,240 INFO SenderThread:15930 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-metadata.json wandb-metadata.json 2022-05-07 17:48:26,240 INFO SenderThread:15930 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/config.yaml config.yaml 2022-05-07 17:48:26,240 INFO SenderThread:15930 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json wandb-summary.json 2022-05-07 17:48:26,248 INFO SenderThread:15930 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/validation[:95%]/epoch_34_incorrect_67_623faa4883d23d9d4194.table.json media/table/validation[:95%]/epoch_34_incorrect_67_623faa4883d23d9d4194.table.json 2022-05-07 17:48:26,251 INFO SenderThread:15930 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/validation[:95%]/epoch_34_67_a7d6e656cdc47be11a4d.table.json media/table/validation[:95%]/epoch_34_67_a7d6e656cdc47be11a4d.table.json 2022-05-07 17:48:26,251 INFO SenderThread:15930 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval/epoch_13_25_9e9b5d32dc07660953d8.table.json media/table/eval/epoch_13_25_9e9b5d32dc07660953d8.table.json 2022-05-07 17:48:26,251 INFO SenderThread:15930 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval/epoch_23_45_9e9b5d32dc07660953d8.table.json media/table/eval/epoch_23_45_9e9b5d32dc07660953d8.table.json 2022-05-07 17:48:26,251 INFO SenderThread:15930 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval/epoch_33_65_92ae240f2472ae79ad31.table.json media/table/eval/epoch_33_65_92ae240f2472ae79ad31.table.json 2022-05-07 17:48:26,251 INFO SenderThread:15930 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval/epoch_8_15_9e9b5d32dc07660953d8.table.json media/table/eval/epoch_8_15_9e9b5d32dc07660953d8.table.json 2022-05-07 17:48:26,251 INFO SenderThread:15930 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval/epoch_28_55_f901d5b4b0d1263be066.table.json media/table/eval/epoch_28_55_f901d5b4b0d1263be066.table.json 2022-05-07 17:48:26,252 INFO SenderThread:15930 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval/epoch_16_30_9e9b5d32dc07660953d8.table.json media/table/eval/epoch_16_30_9e9b5d32dc07660953d8.table.json 2022-05-07 17:48:26,252 INFO SenderThread:15930 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval/epoch_6_10_9e9b5d32dc07660953d8.table.json media/table/eval/epoch_6_10_9e9b5d32dc07660953d8.table.json 2022-05-07 17:48:26,252 INFO SenderThread:15930 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval/epoch_11_20_9e9b5d32dc07660953d8.table.json media/table/eval/epoch_11_20_9e9b5d32dc07660953d8.table.json 2022-05-07 17:48:26,252 INFO SenderThread:15930 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval/epoch_3_5_9e9b5d32dc07660953d8.table.json media/table/eval/epoch_3_5_9e9b5d32dc07660953d8.table.json 2022-05-07 17:48:26,252 INFO SenderThread:15930 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval/epoch_31_60_f901d5b4b0d1263be066.table.json media/table/eval/epoch_31_60_f901d5b4b0d1263be066.table.json 2022-05-07 17:48:26,252 INFO SenderThread:15930 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval/epoch_26_50_f901d5b4b0d1263be066.table.json media/table/eval/epoch_26_50_f901d5b4b0d1263be066.table.json 2022-05-07 17:48:26,252 INFO SenderThread:15930 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval/epoch_18_35_9e9b5d32dc07660953d8.table.json media/table/eval/epoch_18_35_9e9b5d32dc07660953d8.table.json 2022-05-07 17:48:26,252 INFO SenderThread:15930 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/eval/epoch_21_40_9e9b5d32dc07660953d8.table.json media/table/eval/epoch_21_40_9e9b5d32dc07660953d8.table.json 2022-05-07 17:48:26,252 INFO SenderThread:15930 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/validation[:90%]/epoch_34_67_b59f86c33aa51dff79ab.table.json media/table/validation[:90%]/epoch_34_67_b59f86c33aa51dff79ab.table.json 2022-05-07 17:48:26,252 INFO SenderThread:15930 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/media/table/validation[:90%]/epoch_34_incorrect_67_b59f86c33aa51dff79ab.table.json media/table/validation[:90%]/epoch_34_incorrect_67_b59f86c33aa51dff79ab.table.json 2022-05-07 17:48:26,252 INFO SenderThread:15930 [sender.py:transition_state():392] send defer: 7 2022-05-07 17:48:26,252 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: poll_exit 2022-05-07 17:48:26,253 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: defer 2022-05-07 17:48:26,253 INFO HandlerThread:15930 [handler.py:handle_request_defer():164] handle defer: 7 2022-05-07 17:48:26,253 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: defer 2022-05-07 17:48:26,253 INFO SenderThread:15930 [sender.py:send_request_defer():388] handle sender defer: 7 2022-05-07 17:48:26,253 INFO SenderThread:15930 [file_pusher.py:finish():145] shutting down file pusher 2022-05-07 17:48:26,354 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: poll_exit 2022-05-07 17:48:26,354 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: poll_exit 2022-05-07 17:48:26,456 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: poll_exit 2022-05-07 17:48:26,456 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: poll_exit 2022-05-07 17:48:26,477 INFO Thread-63 :15930 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/requirements.txt 2022-05-07 17:48:26,503 INFO Thread-64 :15930 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/output.log 2022-05-07 17:48:26,505 INFO Thread-66 :15930 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/wandb-summary.json 2022-05-07 17:48:26,511 INFO Thread-65 :15930 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/files/config.yaml 2022-05-07 17:48:26,558 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: poll_exit 2022-05-07 17:48:26,558 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: poll_exit 2022-05-07 17:48:26,659 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: poll_exit 2022-05-07 17:48:26,660 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: poll_exit 2022-05-07 17:48:26,712 INFO Thread-6 :15930 [sender.py:transition_state():392] send defer: 8 2022-05-07 17:48:26,712 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: defer 2022-05-07 17:48:26,712 INFO HandlerThread:15930 [handler.py:handle_request_defer():164] handle defer: 8 2022-05-07 17:48:26,712 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: defer 2022-05-07 17:48:26,713 INFO SenderThread:15930 [sender.py:send_request_defer():388] handle sender defer: 8 2022-05-07 17:48:26,761 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: poll_exit 2022-05-07 17:48:26,887 INFO SenderThread:15930 [sender.py:transition_state():392] send defer: 9 2022-05-07 17:48:26,887 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: poll_exit 2022-05-07 17:48:26,888 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: defer 2022-05-07 17:48:26,888 INFO HandlerThread:15930 [handler.py:handle_request_defer():164] handle defer: 9 2022-05-07 17:48:26,888 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: defer 2022-05-07 17:48:26,888 INFO SenderThread:15930 [sender.py:send_request_defer():388] handle sender defer: 9 2022-05-07 17:48:26,888 INFO SenderThread:15930 [sender.py:transition_state():392] send defer: 10 2022-05-07 17:48:26,888 DEBUG SenderThread:15930 [sender.py:send():235] send: final 2022-05-07 17:48:26,888 DEBUG SenderThread:15930 [sender.py:send():235] send: footer 2022-05-07 17:48:26,889 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: defer 2022-05-07 17:48:26,889 INFO HandlerThread:15930 [handler.py:handle_request_defer():164] handle defer: 10 2022-05-07 17:48:26,889 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: defer 2022-05-07 17:48:26,889 INFO SenderThread:15930 [sender.py:send_request_defer():388] handle sender defer: 10 2022-05-07 17:48:26,989 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: poll_exit 2022-05-07 17:48:26,989 DEBUG SenderThread:15930 [sender.py:send_request():249] send_request: poll_exit 2022-05-07 17:48:26,989 INFO SenderThread:15930 [file_pusher.py:join():150] waiting for file pusher 2022-05-07 17:48:27,153 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: sampled_history 2022-05-07 17:48:27,155 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: get_summary 2022-05-07 17:48:27,157 DEBUG HandlerThread:15930 [handler.py:handle_request():141] handle_request: shutdown 2022-05-07 17:48:27,157 INFO HandlerThread:15930 [handler.py:finish():778] shutting down handler 2022-05-07 17:48:27,889 INFO WriterThread:15930 [datastore.py:close():281] close: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_174257-1i5p0k1k/run-1i5p0k1k.wandb 2022-05-07 17:48:28,052 INFO SenderThread:15930 [sender.py:finish():1078] shutting down sender 2022-05-07 17:48:28,052 INFO SenderThread:15930 [file_pusher.py:finish():145] shutting down file pusher 2022-05-07 17:48:28,052 INFO SenderThread:15930 [file_pusher.py:join():150] waiting for file pusher 2022-05-07 17:48:28,053 INFO MainThread:15930 [internal.py:handle_exit():82] Internal process exited