2022-05-07 18:04:52,597 INFO MainThread:26902 [internal.py:wandb_internal():92] W&B internal server running at pid: 26902, started at: 2022-05-07 18:04:52.597521 2022-05-07 18:04:52,599 INFO WriterThread:26902 [datastore.py:open_for_write():77] open: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/run-1hmxxk4r.wandb 2022-05-07 18:04:52,600 DEBUG SenderThread:26902 [sender.py:send():235] send: header 2022-05-07 18:04:52,601 DEBUG SenderThread:26902 [sender.py:send():235] send: run 2022-05-07 18:04:52,668 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: check_version 2022-05-07 18:04:52,668 INFO SenderThread:26902 [dir_watcher.py:__init__():169] watching files in: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files 2022-05-07 18:04:52,669 INFO SenderThread:26902 [sender.py:_start_run_threads():812] run started: 1hmxxk4r with start time 1651946691 2022-05-07 18:04:52,669 DEBUG SenderThread:26902 [sender.py:send():235] send: summary 2022-05-07 18:04:52,669 INFO SenderThread:26902 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 18:04:52,669 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: check_version 2022-05-07 18:04:52,729 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: run_start 2022-05-07 18:04:53,670 INFO Thread-7 :26902 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/wandb-summary.json 2022-05-07 18:04:54,896 DEBUG HandlerThread:26902 [meta.py:__init__():37] meta init 2022-05-07 18:04:54,896 DEBUG HandlerThread:26902 [meta.py:__init__():51] meta init done 2022-05-07 18:04:54,896 DEBUG HandlerThread:26902 [meta.py:probe():211] probe 2022-05-07 18:04:54,897 DEBUG HandlerThread:26902 [meta.py:_setup_git():201] setup git 2022-05-07 18:04:54,926 DEBUG HandlerThread:26902 [meta.py:_setup_git():208] setup git done 2022-05-07 18:04:54,926 DEBUG HandlerThread:26902 [meta.py:_save_pip():55] save pip 2022-05-07 18:04:54,927 DEBUG HandlerThread:26902 [meta.py:_save_pip():69] save pip done 2022-05-07 18:04:54,927 DEBUG HandlerThread:26902 [meta.py:probe():249] probe done 2022-05-07 18:04:54,930 DEBUG SenderThread:26902 [sender.py:send():235] send: files 2022-05-07 18:04:54,930 INFO SenderThread:26902 [sender.py:_save_file():947] saving file wandb-metadata.json with policy now 2022-05-07 18:04:54,940 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 18:04:54,940 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: stop_status 2022-05-07 18:04:54,982 DEBUG SenderThread:26902 [sender.py:send():235] send: telemetry 2022-05-07 18:04:55,398 INFO Thread-11 :26902 [upload_job.py:push():137] Uploaded file /tmp/tmpsjzlge6lwandb/2f8amrj2-wandb-metadata.json 2022-05-07 18:04:55,677 INFO Thread-7 :26902 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/output.log 2022-05-07 18:04:55,677 INFO Thread-7 :26902 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/requirements.txt 2022-05-07 18:04:55,677 INFO Thread-7 :26902 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/wandb-metadata.json 2022-05-07 18:04:57,677 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/output.log 2022-05-07 18:04:59,678 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/output.log 2022-05-07 18:05:09,682 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/output.log 2022-05-07 18:05:10,025 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 18:05:10,025 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: stop_status 2022-05-07 18:05:11,683 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/output.log 2022-05-07 18:05:22,976 DEBUG SenderThread:26902 [sender.py:send():235] send: stats 2022-05-07 18:05:23,687 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/config.yaml 2022-05-07 18:05:25,070 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 18:05:25,070 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: stop_status 2022-05-07 18:05:35,691 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/output.log 2022-05-07 18:05:40,115 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 18:05:40,116 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: stop_status 2022-05-07 18:05:53,051 DEBUG SenderThread:26902 [sender.py:send():235] send: stats 2022-05-07 18:05:55,189 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 18:05:55,189 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: stop_status 2022-05-07 18:05:57,605 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:05:57,705 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:05:58,956 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:05:58,959 DEBUG SenderThread:26902 [sender.py:send():235] send: history 2022-05-07 18:05:58,959 DEBUG SenderThread:26902 [sender.py:send():235] send: summary 2022-05-07 18:05:58,960 INFO SenderThread:26902 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 18:05:59,046 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:05:59,699 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/output.log 2022-05-07 18:05:59,700 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/wandb-summary.json 2022-05-07 18:06:00,087 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:06:00,089 DEBUG SenderThread:26902 [sender.py:send():235] send: history 2022-05-07 18:06:00,090 DEBUG SenderThread:26902 [sender.py:send():235] send: summary 2022-05-07 18:06:00,090 INFO SenderThread:26902 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 18:06:00,169 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:06:00,700 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/wandb-summary.json 2022-05-07 18:06:01,353 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:06:01,355 DEBUG SenderThread:26902 [sender.py:send():235] send: history 2022-05-07 18:06:01,355 DEBUG SenderThread:26902 [sender.py:send():235] send: summary 2022-05-07 18:06:01,356 INFO SenderThread:26902 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 18:06:01,424 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:06:01,700 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/output.log 2022-05-07 18:06:01,700 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/wandb-summary.json 2022-05-07 18:06:02,523 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:06:02,525 DEBUG SenderThread:26902 [sender.py:send():235] send: history 2022-05-07 18:06:02,526 DEBUG SenderThread:26902 [sender.py:send():235] send: summary 2022-05-07 18:06:02,526 INFO SenderThread:26902 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 18:06:02,625 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:06:02,701 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/wandb-summary.json 2022-05-07 18:06:03,701 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/output.log 2022-05-07 18:06:10,221 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 18:06:10,222 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: stop_status 2022-05-07 18:06:23,127 DEBUG SenderThread:26902 [sender.py:send():235] send: stats 2022-05-07 18:06:25,260 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 18:06:25,260 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: stop_status 2022-05-07 18:06:29,711 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/output.log 2022-05-07 18:06:40,313 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 18:06:40,313 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: stop_status 2022-05-07 18:06:49,718 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/output.log 2022-05-07 18:06:53,205 DEBUG SenderThread:26902 [sender.py:send():235] send: stats 2022-05-07 18:06:55,362 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 18:06:55,362 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: stop_status 2022-05-07 18:07:10,397 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 18:07:10,397 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: stop_status 2022-05-07 18:07:12,830 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:07:12,896 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: log_artifact 2022-05-07 18:07:12,896 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: log_artifact 2022-05-07 18:07:13,055 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:07:13,459 INFO Thread-13 :26902 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/03/1aa3918a194407ac466420b1ea5405 2022-05-07 18:07:13,727 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/output.log 2022-05-07 18:07:13,727 INFO Thread-7 :26902 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/media/table/eval/step_5_5_f901d5b4b0d1263be066.table.json 2022-05-07 18:07:13,727 INFO Thread-7 :26902 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/media/table/eval 2022-05-07 18:07:13,727 INFO Thread-7 :26902 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/media/table 2022-05-07 18:07:13,727 INFO Thread-7 :26902 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/media 2022-05-07 18:07:13,897 INFO SenderThread:26902 [sender.py:send_request_log_artifact():976] logged artifact run-1hmxxk4r-evalstep_5 - {'id': 'QXJ0aWZhY3Q6MTI0MzYwNzYy', 'digest': '2d5ec9fbd3f8be2743afb20637dbd249', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5OTAwMDE4', 'latestArtifact': None}, 'version': 'latest'} 2022-05-07 18:07:13,898 DEBUG SenderThread:26902 [sender.py:send():235] send: files 2022-05-07 18:07:13,898 INFO SenderThread:26902 [sender.py:_save_file():947] saving file media/table/eval/step_5_5_f901d5b4b0d1263be066.table.json with policy now 2022-05-07 18:07:14,152 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:07:14,154 DEBUG SenderThread:26902 [sender.py:send():235] send: history 2022-05-07 18:07:14,155 DEBUG SenderThread:26902 [sender.py:send():235] send: summary 2022-05-07 18:07:14,156 INFO SenderThread:26902 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 18:07:14,251 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:07:14,290 INFO Thread-14 :26902 [upload_job.py:push():137] Uploaded file /tmp/tmpsjzlge6lwandb/3bemxkkv-media/table/eval/step_5_5_f901d5b4b0d1263be066.table.json 2022-05-07 18:07:14,727 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/wandb-summary.json 2022-05-07 18:07:15,387 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:07:15,388 DEBUG SenderThread:26902 [sender.py:send():235] send: history 2022-05-07 18:07:15,389 DEBUG SenderThread:26902 [sender.py:send():235] send: summary 2022-05-07 18:07:15,390 INFO SenderThread:26902 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 18:07:15,442 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:07:15,728 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/output.log 2022-05-07 18:07:15,728 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/wandb-summary.json 2022-05-07 18:07:16,555 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:07:16,557 DEBUG SenderThread:26902 [sender.py:send():235] send: history 2022-05-07 18:07:16,558 DEBUG SenderThread:26902 [sender.py:send():235] send: summary 2022-05-07 18:07:16,559 INFO SenderThread:26902 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 18:07:16,635 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:07:16,728 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/wandb-summary.json 2022-05-07 18:07:17,728 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/output.log 2022-05-07 18:07:17,815 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:07:17,817 DEBUG SenderThread:26902 [sender.py:send():235] send: history 2022-05-07 18:07:17,818 DEBUG SenderThread:26902 [sender.py:send():235] send: summary 2022-05-07 18:07:17,819 INFO SenderThread:26902 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 18:07:17,874 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:07:18,729 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/wandb-summary.json 2022-05-07 18:07:19,033 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:07:19,035 DEBUG SenderThread:26902 [sender.py:send():235] send: history 2022-05-07 18:07:19,036 DEBUG SenderThread:26902 [sender.py:send():235] send: summary 2022-05-07 18:07:19,037 INFO SenderThread:26902 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 18:07:19,109 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:07:19,730 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/output.log 2022-05-07 18:07:19,730 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/wandb-summary.json 2022-05-07 18:07:20,730 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/output.log 2022-05-07 18:07:21,377 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:07:21,467 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: log_artifact 2022-05-07 18:07:21,468 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: log_artifact 2022-05-07 18:07:21,581 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:07:21,731 INFO Thread-7 :26902 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/media/table/eval/step_10_10_f901d5b4b0d1263be066.table.json 2022-05-07 18:07:21,731 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/media/table/eval 2022-05-07 18:07:21,825 INFO Thread-16 :26902 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/03/1aa3918a194407ac466420b1ea5405 2022-05-07 18:07:22,503 INFO SenderThread:26902 [sender.py:send_request_log_artifact():976] logged artifact run-1hmxxk4r-evalstep_10 - {'id': 'QXJ0aWZhY3Q6MTI0MzYwODAz', 'digest': 'cb3536ee0532719f7aa50e9dd15a9285', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5OTAwMDIw', 'latestArtifact': None}, 'version': 'latest'} 2022-05-07 18:07:22,503 DEBUG SenderThread:26902 [sender.py:send():235] send: files 2022-05-07 18:07:22,503 INFO SenderThread:26902 [sender.py:_save_file():947] saving file media/table/eval/step_10_10_f901d5b4b0d1263be066.table.json with policy now 2022-05-07 18:07:22,655 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:07:22,657 DEBUG SenderThread:26902 [sender.py:send():235] send: history 2022-05-07 18:07:22,658 DEBUG SenderThread:26902 [sender.py:send():235] send: summary 2022-05-07 18:07:22,659 INFO SenderThread:26902 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 18:07:22,717 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:07:22,731 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/output.log 2022-05-07 18:07:22,731 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/wandb-summary.json 2022-05-07 18:07:22,744 INFO Thread-17 :26902 [upload_job.py:push():137] Uploaded file /tmp/tmpsjzlge6lwandb/3rnvtsey-media/table/eval/step_10_10_f901d5b4b0d1263be066.table.json 2022-05-07 18:07:23,283 DEBUG SenderThread:26902 [sender.py:send():235] send: stats 2022-05-07 18:07:23,832 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:07:23,834 DEBUG SenderThread:26902 [sender.py:send():235] send: history 2022-05-07 18:07:23,834 DEBUG SenderThread:26902 [sender.py:send():235] send: summary 2022-05-07 18:07:23,835 INFO SenderThread:26902 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 18:07:23,936 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:07:24,732 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/output.log 2022-05-07 18:07:24,732 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/wandb-summary.json 2022-05-07 18:07:25,137 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:07:25,139 DEBUG SenderThread:26902 [sender.py:send():235] send: history 2022-05-07 18:07:25,140 DEBUG SenderThread:26902 [sender.py:send():235] send: summary 2022-05-07 18:07:25,143 INFO SenderThread:26902 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 18:07:25,195 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:07:25,501 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 18:07:25,501 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: stop_status 2022-05-07 18:07:25,733 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/wandb-summary.json 2022-05-07 18:07:26,315 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:07:26,317 DEBUG SenderThread:26902 [sender.py:send():235] send: history 2022-05-07 18:07:26,318 DEBUG SenderThread:26902 [sender.py:send():235] send: summary 2022-05-07 18:07:26,318 INFO SenderThread:26902 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 18:07:26,395 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:07:26,733 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/output.log 2022-05-07 18:07:26,733 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/wandb-summary.json 2022-05-07 18:07:27,588 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:07:27,591 DEBUG SenderThread:26902 [sender.py:send():235] send: history 2022-05-07 18:07:27,591 DEBUG SenderThread:26902 [sender.py:send():235] send: summary 2022-05-07 18:07:27,592 INFO SenderThread:26902 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 18:07:27,644 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:07:27,734 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/wandb-summary.json 2022-05-07 18:07:28,734 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/output.log 2022-05-07 18:07:30,735 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/output.log 2022-05-07 18:07:40,581 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 18:07:40,581 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: stop_status 2022-05-07 18:07:44,741 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/output.log 2022-05-07 18:07:53,359 DEBUG SenderThread:26902 [sender.py:send():235] send: stats 2022-05-07 18:07:56,245 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 18:07:56,246 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: stop_status 2022-05-07 18:07:56,747 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/output.log 2022-05-07 18:08:09,459 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:08:09,797 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: log_artifact 2022-05-07 18:08:09,797 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: log_artifact 2022-05-07 18:08:09,874 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:08:09,916 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: log_artifact 2022-05-07 18:08:10,168 INFO Thread-19 :26902 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/59/70265576a772597adeb57dd33a7a30 2022-05-07 18:08:10,263 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:08:10,617 INFO SenderThread:26902 [sender.py:send_request_log_artifact():976] logged artifact run-1hmxxk4r-evalstep_15 - {'id': 'QXJ0aWZhY3Q6MTI0MzYxMDU4', 'digest': '8d24fe6b92e960ee996cec30d52a381c', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5OTAwMDQ5', 'latestArtifact': None}, 'version': 'latest'} 2022-05-07 18:08:10,618 DEBUG SenderThread:26902 [sender.py:send():235] send: files 2022-05-07 18:08:10,618 INFO SenderThread:26902 [sender.py:_save_file():947] saving file media/table/eval/step_15_15_a7d6e656cdc47be11a4d.table.json with policy now 2022-05-07 18:08:10,618 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: log_artifact 2022-05-07 18:08:10,753 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/output.log 2022-05-07 18:08:10,754 INFO Thread-7 :26902 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/media/table/eval/step_15_15_a7d6e656cdc47be11a4d.table.json 2022-05-07 18:08:10,754 INFO Thread-7 :26902 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/media/table/eval/final_generation_incorrect_15_623faa4883d23d9d4194.table.json 2022-05-07 18:08:10,754 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/media/table/eval 2022-05-07 18:08:10,843 INFO Thread-20 :26902 [upload_job.py:push():137] Uploaded file /tmp/tmpsjzlge6lwandb/oks0bbg6-media/table/eval/step_15_15_a7d6e656cdc47be11a4d.table.json 2022-05-07 18:08:11,323 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: stop_status 2022-05-07 18:08:11,325 INFO Thread-22 :26902 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/25/29be3032c57f98a337547e6fe497da 2022-05-07 18:08:11,795 INFO SenderThread:26902 [sender.py:send_request_log_artifact():976] logged artifact run-1hmxxk4r-evalfinal_generation_incorrect - {'id': 'QXJ0aWZhY3Q6MTI0MzYxMDY1', 'digest': '3f942e4248698c8f75ed857c64bedc43', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5OTAwMDUy', 'latestArtifact': None}, 'version': 'latest'} 2022-05-07 18:08:11,796 DEBUG SenderThread:26902 [sender.py:send():235] send: files 2022-05-07 18:08:11,796 INFO SenderThread:26902 [sender.py:_save_file():947] saving file media/table/eval/final_generation_incorrect_15_623faa4883d23d9d4194.table.json with policy now 2022-05-07 18:08:11,797 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: stop_status 2022-05-07 18:08:12,051 INFO Thread-23 :26902 [upload_job.py:push():137] Uploaded file /tmp/tmpsjzlge6lwandb/1fhz95yu-media/table/eval/final_generation_incorrect_15_623faa4883d23d9d4194.table.json 2022-05-07 18:08:12,161 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:08:12,236 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: log_artifact 2022-05-07 18:08:12,236 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: log_artifact 2022-05-07 18:08:12,295 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:08:12,332 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: log_artifact 2022-05-07 18:08:12,417 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:08:12,652 INFO Thread-25 :26902 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/af/47f5eac929f7d4f6e6c7d604ae396b 2022-05-07 18:08:12,754 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/output.log 2022-05-07 18:08:12,755 INFO Thread-7 :26902 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/media/table/validation[:90%]/step_15_15_b59f86c33aa51dff79ab.table.json 2022-05-07 18:08:12,755 INFO Thread-7 :26902 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/media/table/validation[:90%]/final_generation_incorrect_15_b59f86c33aa51dff79ab.table.json 2022-05-07 18:08:12,755 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/media/table 2022-05-07 18:08:12,755 INFO Thread-7 :26902 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/media/table/validation[:90%] 2022-05-07 18:08:14,756 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/output.log 2022-05-07 18:08:14,815 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:08:14,895 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: log_artifact 2022-05-07 18:08:14,978 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:08:15,033 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: log_artifact 2022-05-07 18:08:15,094 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:08:15,126 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: partial_history 2022-05-07 18:08:15,756 INFO Thread-7 :26902 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/media/table/validation[:95%]/step_15_15_a7d6e656cdc47be11a4d.table.json 2022-05-07 18:08:15,756 INFO Thread-7 :26902 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/media/table/validation[:95%]/final_generation_incorrect_15_623faa4883d23d9d4194.table.json 2022-05-07 18:08:15,757 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/media/table 2022-05-07 18:08:15,757 INFO Thread-7 :26902 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/media/table/validation[:95%] 2022-05-07 18:08:16,346 INFO SenderThread:26902 [sender.py:send_request_log_artifact():976] logged artifact run-1hmxxk4r-validation90step_15 - {'id': 'QXJ0aWZhY3Q6MTI0MzYxMDcy', 'digest': 'b9155c81fbadd3b4b2be871801f24639', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5OTAwMDU0', 'latestArtifact': None}, 'version': 'latest'} 2022-05-07 18:08:16,346 DEBUG SenderThread:26902 [sender.py:send():235] send: files 2022-05-07 18:08:16,346 INFO SenderThread:26902 [sender.py:_save_file():947] saving file media/table/validation[[]:90%]/step_15_15_b59f86c33aa51dff79ab.table.json with policy now 2022-05-07 18:08:16,347 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: log_artifact 2022-05-07 18:08:16,634 INFO Thread-26 :26902 [upload_job.py:push():137] Uploaded file /tmp/tmpsjzlge6lwandb/12dtfern-media/table/validation[:90%]/step_15_15_b59f86c33aa51dff79ab.table.json 2022-05-07 18:08:16,757 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/output.log 2022-05-07 18:08:16,787 INFO Thread-28 :26902 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/1d/36af48785046dc58b4f025a3f8aea1 2022-05-07 18:08:17,264 INFO SenderThread:26902 [sender.py:send_request_log_artifact():976] logged artifact run-1hmxxk4r-validation90final_generation_incorrect - {'id': 'QXJ0aWZhY3Q6MTI0MzYxMDk2', 'digest': 'f7512f30bf4c456be7b1a5f6c70e0a72', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5OTAwMDYw', 'latestArtifact': None}, 'version': 'latest'} 2022-05-07 18:08:17,264 DEBUG SenderThread:26902 [sender.py:send():235] send: files 2022-05-07 18:08:17,264 INFO SenderThread:26902 [sender.py:_save_file():947] saving file media/table/validation[[]:90%]/final_generation_incorrect_15_b59f86c33aa51dff79ab.table.json with policy now 2022-05-07 18:08:17,265 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: log_artifact 2022-05-07 18:08:17,532 INFO Thread-29 :26902 [upload_job.py:push():137] Uploaded file /tmp/tmpsjzlge6lwandb/19rzf7rd-media/table/validation[:90%]/final_generation_incorrect_15_b59f86c33aa51dff79ab.table.json 2022-05-07 18:08:17,619 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: poll_exit 2022-05-07 18:08:17,758 INFO Thread-7 :26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/output.log 2022-05-07 18:08:20,172 INFO Thread-31 :26902 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/59/70265576a772597adeb57dd33a7a30 2022-05-07 18:08:20,575 INFO SenderThread:26902 [sender.py:send_request_log_artifact():976] logged artifact run-1hmxxk4r-validation95step_15 - {'id': 'QXJ0aWZhY3Q6MTI0MzYxMTA4', 'digest': 'afc1c295cc63a04d7548709360e09946', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5OTAwMDY0', 'latestArtifact': None}, 'version': 'latest'} 2022-05-07 18:08:20,575 DEBUG SenderThread:26902 [sender.py:send():235] send: files 2022-05-07 18:08:20,575 INFO SenderThread:26902 [sender.py:_save_file():947] saving file media/table/validation[[]:95%]/step_15_15_a7d6e656cdc47be11a4d.table.json with policy now 2022-05-07 18:08:20,576 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: log_artifact 2022-05-07 18:08:20,838 INFO Thread-32 :26902 [upload_job.py:push():137] Uploaded file /tmp/tmpsjzlge6lwandb/1c81wuqh-media/table/validation[:95%]/step_15_15_a7d6e656cdc47be11a4d.table.json 2022-05-07 18:08:21,497 INFO Thread-34 :26902 [upload_job.py:push():95] Uploaded file /home/sanchitgandhi/.cache/wandb/artifacts/obj/md5/25/29be3032c57f98a337547e6fe497da 2022-05-07 18:08:21,949 INFO SenderThread:26902 [sender.py:send_request_log_artifact():976] logged artifact run-1hmxxk4r-validation95final_generation_incorrect - {'id': 'QXJ0aWZhY3Q6MTI0MzYxMTE4', 'digest': 'c3b06e384aabea2991e3811071232ab4', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE5OTAwMDY1', 'latestArtifact': None}, 'version': 'latest'} 2022-05-07 18:08:21,949 DEBUG SenderThread:26902 [sender.py:send():235] send: files 2022-05-07 18:08:21,950 INFO SenderThread:26902 [sender.py:_save_file():947] saving file media/table/validation[[]:95%]/final_generation_incorrect_15_623faa4883d23d9d4194.table.json with policy now 2022-05-07 18:08:21,950 DEBUG SenderThread:26902 [sender.py:send():235] send: telemetry 2022-05-07 18:08:21,950 DEBUG SenderThread:26902 [sender.py:send():235] send: history 2022-05-07 18:08:21,951 DEBUG SenderThread:26902 [sender.py:send():235] send: summary 2022-05-07 18:08:21,952 INFO SenderThread:26902 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 18:08:21,952 DEBUG SenderThread:26902 [sender.py:send():235] send: exit 2022-05-07 18:08:21,953 INFO SenderThread:26902 [sender.py:send_exit():371] handling exit code: 0 2022-05-07 18:08:21,953 INFO SenderThread:26902 [sender.py:send_exit():373] handling runtime: 204 2022-05-07 18:08:21,954 INFO SenderThread:26902 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 18:08:21,954 INFO SenderThread:26902 [sender.py:send_exit():379] send defer 2022-05-07 18:08:21,954 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: poll_exit 2022-05-07 18:08:21,955 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: defer 2022-05-07 18:08:21,955 INFO HandlerThread:26902 [handler.py:handle_request_defer():164] handle defer: 0 2022-05-07 18:08:21,958 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: defer 2022-05-07 18:08:21,958 INFO SenderThread:26902 [sender.py:send_request_defer():388] handle sender defer: 0 2022-05-07 18:08:21,958 INFO SenderThread:26902 [sender.py:transition_state():392] send defer: 1 2022-05-07 18:08:21,958 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: defer 2022-05-07 18:08:21,958 INFO HandlerThread:26902 [handler.py:handle_request_defer():164] handle defer: 1 2022-05-07 18:08:22,033 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: defer 2022-05-07 18:08:22,033 INFO SenderThread:26902 [sender.py:send_request_defer():388] handle sender defer: 1 2022-05-07 18:08:22,033 INFO SenderThread:26902 [sender.py:transition_state():392] send defer: 2 2022-05-07 18:08:22,033 DEBUG SenderThread:26902 [sender.py:send():235] send: stats 2022-05-07 18:08:22,033 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: defer 2022-05-07 18:08:22,034 INFO HandlerThread:26902 [handler.py:handle_request_defer():164] handle defer: 2 2022-05-07 18:08:22,034 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: defer 2022-05-07 18:08:22,034 INFO SenderThread:26902 [sender.py:send_request_defer():388] handle sender defer: 2 2022-05-07 18:08:22,034 INFO SenderThread:26902 [sender.py:transition_state():392] send defer: 3 2022-05-07 18:08:22,034 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: defer 2022-05-07 18:08:22,034 INFO HandlerThread:26902 [handler.py:handle_request_defer():164] handle defer: 3 2022-05-07 18:08:22,034 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: defer 2022-05-07 18:08:22,034 INFO SenderThread:26902 [sender.py:send_request_defer():388] handle sender defer: 3 2022-05-07 18:08:22,034 INFO SenderThread:26902 [sender.py:transition_state():392] send defer: 4 2022-05-07 18:08:22,034 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: defer 2022-05-07 18:08:22,034 INFO HandlerThread:26902 [handler.py:handle_request_defer():164] handle defer: 4 2022-05-07 18:08:22,035 DEBUG SenderThread:26902 [sender.py:send():235] send: summary 2022-05-07 18:08:22,036 INFO SenderThread:26902 [sender.py:_save_file():947] saving file wandb-summary.json with policy end 2022-05-07 18:08:22,036 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: defer 2022-05-07 18:08:22,036 INFO SenderThread:26902 [sender.py:send_request_defer():388] handle sender defer: 4 2022-05-07 18:08:22,036 INFO SenderThread:26902 [sender.py:transition_state():392] send defer: 5 2022-05-07 18:08:22,036 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: defer 2022-05-07 18:08:22,036 INFO HandlerThread:26902 [handler.py:handle_request_defer():164] handle defer: 5 2022-05-07 18:08:22,036 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: defer 2022-05-07 18:08:22,037 INFO SenderThread:26902 [sender.py:send_request_defer():388] handle sender defer: 5 2022-05-07 18:08:22,056 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: poll_exit 2022-05-07 18:08:22,373 INFO SenderThread:26902 [sender.py:transition_state():392] send defer: 6 2022-05-07 18:08:22,373 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: poll_exit 2022-05-07 18:08:22,373 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: defer 2022-05-07 18:08:22,373 INFO HandlerThread:26902 [handler.py:handle_request_defer():164] handle defer: 6 2022-05-07 18:08:22,374 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: defer 2022-05-07 18:08:22,374 INFO SenderThread:26902 [sender.py:send_request_defer():388] handle sender defer: 6 2022-05-07 18:08:22,374 INFO SenderThread:26902 [dir_watcher.py:finish():283] shutting down directory watcher 2022-05-07 18:08:22,475 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: poll_exit 2022-05-07 18:08:22,759 INFO SenderThread:26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/config.yaml 2022-05-07 18:08:22,760 INFO SenderThread:26902 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/wandb-summary.json 2022-05-07 18:08:22,760 INFO SenderThread:26902 [dir_watcher.py:finish():313] scan: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files 2022-05-07 18:08:22,760 INFO SenderThread:26902 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/requirements.txt requirements.txt 2022-05-07 18:08:22,760 INFO SenderThread:26902 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/output.log output.log 2022-05-07 18:08:22,760 INFO SenderThread:26902 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/wandb-metadata.json wandb-metadata.json 2022-05-07 18:08:22,760 INFO SenderThread:26902 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/config.yaml config.yaml 2022-05-07 18:08:22,761 INFO SenderThread:26902 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/wandb-summary.json wandb-summary.json 2022-05-07 18:08:22,763 INFO SenderThread:26902 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/media/table/validation[:95%]/step_15_15_a7d6e656cdc47be11a4d.table.json media/table/validation[:95%]/step_15_15_a7d6e656cdc47be11a4d.table.json 2022-05-07 18:08:22,766 INFO SenderThread:26902 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/media/table/validation[:95%]/final_generation_incorrect_15_623faa4883d23d9d4194.table.json media/table/validation[:95%]/final_generation_incorrect_15_623faa4883d23d9d4194.table.json 2022-05-07 18:08:22,768 INFO SenderThread:26902 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/media/table/eval/step_15_15_a7d6e656cdc47be11a4d.table.json media/table/eval/step_15_15_a7d6e656cdc47be11a4d.table.json 2022-05-07 18:08:22,768 INFO SenderThread:26902 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/media/table/eval/final_generation_incorrect_15_623faa4883d23d9d4194.table.json media/table/eval/final_generation_incorrect_15_623faa4883d23d9d4194.table.json 2022-05-07 18:08:22,768 INFO SenderThread:26902 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/media/table/eval/step_10_10_f901d5b4b0d1263be066.table.json media/table/eval/step_10_10_f901d5b4b0d1263be066.table.json 2022-05-07 18:08:22,768 INFO SenderThread:26902 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/media/table/eval/step_5_5_f901d5b4b0d1263be066.table.json media/table/eval/step_5_5_f901d5b4b0d1263be066.table.json 2022-05-07 18:08:22,769 INFO SenderThread:26902 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/media/table/validation[:90%]/step_15_15_b59f86c33aa51dff79ab.table.json media/table/validation[:90%]/step_15_15_b59f86c33aa51dff79ab.table.json 2022-05-07 18:08:22,769 INFO SenderThread:26902 [dir_watcher.py:finish():327] scan save: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/media/table/validation[:90%]/final_generation_incorrect_15_b59f86c33aa51dff79ab.table.json media/table/validation[:90%]/final_generation_incorrect_15_b59f86c33aa51dff79ab.table.json 2022-05-07 18:08:22,769 INFO SenderThread:26902 [sender.py:transition_state():392] send defer: 7 2022-05-07 18:08:22,769 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: poll_exit 2022-05-07 18:08:22,773 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: defer 2022-05-07 18:08:22,773 INFO HandlerThread:26902 [handler.py:handle_request_defer():164] handle defer: 7 2022-05-07 18:08:22,773 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: defer 2022-05-07 18:08:22,773 INFO SenderThread:26902 [sender.py:send_request_defer():388] handle sender defer: 7 2022-05-07 18:08:22,773 INFO SenderThread:26902 [file_pusher.py:finish():145] shutting down file pusher 2022-05-07 18:08:22,873 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: poll_exit 2022-05-07 18:08:22,873 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: poll_exit 2022-05-07 18:08:22,900 INFO Thread-35 :26902 [upload_job.py:push():137] Uploaded file /tmp/tmpsjzlge6lwandb/2pgocaqp-media/table/validation[:95%]/final_generation_incorrect_15_623faa4883d23d9d4194.table.json 2022-05-07 18:08:22,975 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: poll_exit 2022-05-07 18:08:22,975 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: poll_exit 2022-05-07 18:08:23,007 INFO Thread-37 :26902 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/output.log 2022-05-07 18:08:23,011 INFO Thread-39 :26902 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/wandb-summary.json 2022-05-07 18:08:23,029 INFO Thread-36 :26902 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/requirements.txt 2022-05-07 18:08:23,077 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: poll_exit 2022-05-07 18:08:23,077 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: poll_exit 2022-05-07 18:08:23,179 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: poll_exit 2022-05-07 18:08:23,179 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: poll_exit 2022-05-07 18:08:23,280 INFO Thread-38 :26902 [upload_job.py:push():137] Uploaded file /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/files/config.yaml 2022-05-07 18:08:23,281 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: poll_exit 2022-05-07 18:08:23,281 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: poll_exit 2022-05-07 18:08:23,383 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: poll_exit 2022-05-07 18:08:23,383 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: poll_exit 2022-05-07 18:08:23,481 INFO Thread-6 :26902 [sender.py:transition_state():392] send defer: 8 2022-05-07 18:08:23,481 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: defer 2022-05-07 18:08:23,481 INFO HandlerThread:26902 [handler.py:handle_request_defer():164] handle defer: 8 2022-05-07 18:08:23,482 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: defer 2022-05-07 18:08:23,482 INFO SenderThread:26902 [sender.py:send_request_defer():388] handle sender defer: 8 2022-05-07 18:08:23,485 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: poll_exit 2022-05-07 18:08:23,645 INFO SenderThread:26902 [sender.py:transition_state():392] send defer: 9 2022-05-07 18:08:23,645 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: poll_exit 2022-05-07 18:08:23,645 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: defer 2022-05-07 18:08:23,645 INFO HandlerThread:26902 [handler.py:handle_request_defer():164] handle defer: 9 2022-05-07 18:08:23,645 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: defer 2022-05-07 18:08:23,646 INFO SenderThread:26902 [sender.py:send_request_defer():388] handle sender defer: 9 2022-05-07 18:08:23,646 INFO SenderThread:26902 [sender.py:transition_state():392] send defer: 10 2022-05-07 18:08:23,646 DEBUG SenderThread:26902 [sender.py:send():235] send: final 2022-05-07 18:08:23,646 DEBUG SenderThread:26902 [sender.py:send():235] send: footer 2022-05-07 18:08:23,646 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: defer 2022-05-07 18:08:23,646 INFO HandlerThread:26902 [handler.py:handle_request_defer():164] handle defer: 10 2022-05-07 18:08:23,646 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: defer 2022-05-07 18:08:23,647 INFO SenderThread:26902 [sender.py:send_request_defer():388] handle sender defer: 10 2022-05-07 18:08:23,747 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: poll_exit 2022-05-07 18:08:23,747 DEBUG SenderThread:26902 [sender.py:send_request():249] send_request: poll_exit 2022-05-07 18:08:23,747 INFO SenderThread:26902 [file_pusher.py:join():150] waiting for file pusher 2022-05-07 18:08:23,874 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: sampled_history 2022-05-07 18:08:23,875 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: get_summary 2022-05-07 18:08:23,876 DEBUG HandlerThread:26902 [handler.py:handle_request():141] handle_request: shutdown 2022-05-07 18:08:23,876 INFO HandlerThread:26902 [handler.py:finish():778] shutting down handler 2022-05-07 18:08:24,647 INFO WriterThread:26902 [datastore.py:close():281] close: /home/sanchitgandhi/flax-wav2vec2-2-bart-dummy/wandb/run-20220507_180451-1hmxxk4r/run-1hmxxk4r.wandb 2022-05-07 18:08:24,772 INFO SenderThread:26902 [sender.py:finish():1078] shutting down sender 2022-05-07 18:08:24,772 INFO SenderThread:26902 [file_pusher.py:finish():145] shutting down file pusher 2022-05-07 18:08:24,772 INFO SenderThread:26902 [file_pusher.py:join():150] waiting for file pusher 2022-05-07 18:08:24,852 INFO MainThread:26902 [internal.py:handle_exit():82] Internal process exited