sanchit-gandhi's picture
Training in progress, step 500
a5afdd5
raw
history blame
17 kB
2022-03-01 01:29:45,420 INFO MainThread:236851 [internal.py:wandb_internal():89] W&B internal server running at pid: 236851, started at: 2022-03-01 01:29:45.420003
2022-03-01 01:29:45,422 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: check_version
2022-03-01 01:29:45,422 INFO WriterThread:236851 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/run-25hk0gke.wandb
2022-03-01 01:29:45,424 DEBUG SenderThread:236851 [sender.py:send():235] send: header
2022-03-01 01:29:45,424 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: check_version
2022-03-01 01:29:45,489 DEBUG SenderThread:236851 [sender.py:send():235] send: run
2022-03-01 01:29:45,576 INFO SenderThread:236851 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files
2022-03-01 01:29:45,576 INFO SenderThread:236851 [sender.py:_start_run_threads():809] run started: 25hk0gke with start time 1646098184
2022-03-01 01:29:45,576 DEBUG SenderThread:236851 [sender.py:send():235] send: summary
2022-03-01 01:29:45,576 INFO SenderThread:236851 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 01:29:45,577 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: run_start
2022-03-01 01:29:45,582 DEBUG HandlerThread:236851 [meta.py:__init__():36] meta init
2022-03-01 01:29:45,582 DEBUG HandlerThread:236851 [meta.py:__init__():50] meta init done
2022-03-01 01:29:45,582 DEBUG HandlerThread:236851 [meta.py:probe():210] probe
2022-03-01 01:29:45,588 DEBUG HandlerThread:236851 [meta.py:_setup_git():200] setup git
2022-03-01 01:29:45,603 DEBUG HandlerThread:236851 [meta.py:_setup_git():207] setup git done
2022-03-01 01:29:45,603 DEBUG HandlerThread:236851 [meta.py:_save_pip():54] save pip
2022-03-01 01:29:45,604 DEBUG HandlerThread:236851 [meta.py:_save_pip():68] save pip done
2022-03-01 01:29:45,604 DEBUG HandlerThread:236851 [meta.py:probe():248] probe done
2022-03-01 01:29:45,684 DEBUG SenderThread:236851 [sender.py:send():235] send: files
2022-03-01 01:29:45,684 INFO SenderThread:236851 [sender.py:_save_file():944] saving file wandb-metadata.json with policy now
2022-03-01 01:29:45,689 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: stop_status
2022-03-01 01:29:45,690 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: stop_status
2022-03-01 01:29:45,729 DEBUG SenderThread:236851 [sender.py:send():235] send: config
2022-03-01 01:29:45,731 DEBUG SenderThread:236851 [sender.py:send():235] send: metric
2022-03-01 01:29:45,731 DEBUG SenderThread:236851 [sender.py:send():235] send: metric
2022-03-01 01:29:45,731 WARNING SenderThread:236851 [sender.py:send_metric():902] Seen metric with glob (shouldnt happen)
2022-03-01 01:29:45,969 INFO Thread-11 :236851 [upload_job.py:push():137] Uploaded file /tmp/tmpdzeo2dm2wandb/3813kelu-wandb-metadata.json
2022-03-01 01:29:46,578 INFO Thread-8 :236851 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/requirements.txt
2022-03-01 01:29:46,578 INFO Thread-8 :236851 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/wandb-summary.json
2022-03-01 01:29:46,578 INFO Thread-8 :236851 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/wandb-metadata.json
2022-03-01 01:29:46,578 INFO Thread-8 :236851 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/output.log
2022-03-01 01:29:48,577 INFO Thread-8 :236851 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/output.log
2022-03-01 01:29:52,578 INFO Thread-8 :236851 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/output.log
2022-03-01 01:29:54,442 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:29:54,442 DEBUG SenderThread:236851 [sender.py:send():235] send: telemetry
2022-03-01 01:29:54,443 DEBUG SenderThread:236851 [sender.py:send():235] send: exit
2022-03-01 01:29:54,443 INFO SenderThread:236851 [sender.py:send_exit():371] handling exit code: 1
2022-03-01 01:29:54,443 INFO SenderThread:236851 [sender.py:send_exit():373] handling runtime: 8
2022-03-01 01:29:54,443 INFO SenderThread:236851 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 01:29:54,443 INFO SenderThread:236851 [sender.py:send_exit():379] send defer
2022-03-01 01:29:54,444 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:29:54,444 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:29:54,444 INFO HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 0
2022-03-01 01:29:54,444 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: defer
2022-03-01 01:29:54,445 INFO SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 0
2022-03-01 01:29:54,445 INFO SenderThread:236851 [sender.py:transition_state():392] send defer: 1
2022-03-01 01:29:54,445 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:29:54,445 INFO HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 1
2022-03-01 01:29:54,515 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: defer
2022-03-01 01:29:54,515 INFO SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 1
2022-03-01 01:29:54,515 INFO SenderThread:236851 [sender.py:transition_state():392] send defer: 2
2022-03-01 01:29:54,515 DEBUG SenderThread:236851 [sender.py:send():235] send: stats
2022-03-01 01:29:54,516 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:29:54,516 INFO HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 2
2022-03-01 01:29:54,516 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: defer
2022-03-01 01:29:54,516 INFO SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 2
2022-03-01 01:29:54,516 INFO SenderThread:236851 [sender.py:transition_state():392] send defer: 3
2022-03-01 01:29:54,517 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:29:54,517 INFO HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 3
2022-03-01 01:29:54,517 DEBUG SenderThread:236851 [sender.py:send():235] send: summary
2022-03-01 01:29:54,517 INFO SenderThread:236851 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 01:29:54,518 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: defer
2022-03-01 01:29:54,518 INFO SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 3
2022-03-01 01:29:54,518 INFO SenderThread:236851 [sender.py:transition_state():392] send defer: 4
2022-03-01 01:29:54,518 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:29:54,518 INFO HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 4
2022-03-01 01:29:54,518 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: defer
2022-03-01 01:29:54,518 INFO SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 4
2022-03-01 01:29:54,545 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:29:54,579 INFO Thread-8 :236851 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/wandb-summary.json
2022-03-01 01:29:54,579 INFO Thread-8 :236851 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/output.log
2022-03-01 01:29:54,803 INFO SenderThread:236851 [sender.py:transition_state():392] send defer: 5
2022-03-01 01:29:54,803 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:29:54,804 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:29:54,804 INFO HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 5
2022-03-01 01:29:54,804 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: defer
2022-03-01 01:29:54,804 INFO SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 5
2022-03-01 01:29:54,804 INFO SenderThread:236851 [dir_watcher.py:finish():283] shutting down directory watcher
2022-03-01 01:29:54,905 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:29:55,579 INFO Thread-8 :236851 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/config.yaml
2022-03-01 01:29:55,580 INFO SenderThread:236851 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files
2022-03-01 01:29:55,580 INFO SenderThread:236851 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/wandb-metadata.json wandb-metadata.json
2022-03-01 01:29:55,580 INFO SenderThread:236851 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/output.log output.log
2022-03-01 01:29:55,581 INFO SenderThread:236851 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/wandb-summary.json wandb-summary.json
2022-03-01 01:29:55,581 INFO SenderThread:236851 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/requirements.txt requirements.txt
2022-03-01 01:29:55,586 INFO SenderThread:236851 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/config.yaml config.yaml
2022-03-01 01:29:55,586 INFO SenderThread:236851 [sender.py:transition_state():392] send defer: 6
2022-03-01 01:29:55,587 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:29:55,588 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:29:55,588 INFO HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 6
2022-03-01 01:29:55,591 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: defer
2022-03-01 01:29:55,591 INFO SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 6
2022-03-01 01:29:55,592 INFO SenderThread:236851 [file_pusher.py:finish():177] shutting down file pusher
2022-03-01 01:29:55,689 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:29:55,689 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:29:55,791 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:29:55,791 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:29:55,856 INFO Thread-14 :236851 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/requirements.txt
2022-03-01 01:29:55,860 INFO Thread-13 :236851 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/wandb-summary.json
2022-03-01 01:29:55,885 INFO Thread-15 :236851 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/config.yaml
2022-03-01 01:29:55,893 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:29:55,893 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:29:55,941 INFO Thread-12 :236851 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/output.log
2022-03-01 01:29:55,994 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:29:55,995 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:29:56,096 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:29:56,096 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:29:56,141 INFO Thread-7 :236851 [sender.py:transition_state():392] send defer: 7
2022-03-01 01:29:56,142 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:29:56,142 INFO HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 7
2022-03-01 01:29:56,142 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: defer
2022-03-01 01:29:56,142 INFO SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 7
2022-03-01 01:29:56,188 INFO SenderThread:236851 [sender.py:transition_state():392] send defer: 8
2022-03-01 01:29:56,188 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:29:56,188 INFO HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 8
2022-03-01 01:29:56,188 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: defer
2022-03-01 01:29:56,188 INFO SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 8
2022-03-01 01:29:56,189 INFO SenderThread:236851 [sender.py:transition_state():392] send defer: 9
2022-03-01 01:29:56,189 DEBUG SenderThread:236851 [sender.py:send():235] send: final
2022-03-01 01:29:56,189 DEBUG SenderThread:236851 [sender.py:send():235] send: footer
2022-03-01 01:29:56,189 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:29:56,189 INFO HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 9
2022-03-01 01:29:56,190 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: defer
2022-03-01 01:29:56,190 INFO SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 9
2022-03-01 01:29:56,197 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:29:56,198 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:29:56,198 INFO SenderThread:236851 [file_pusher.py:join():182] waiting for file pusher
2022-03-01 01:29:56,258 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: get_summary
2022-03-01 01:29:56,259 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: sampled_history
2022-03-01 01:29:56,259 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: shutdown
2022-03-01 01:29:56,259 INFO HandlerThread:236851 [handler.py:finish():739] shutting down handler
2022-03-01 01:29:57,190 INFO WriterThread:236851 [datastore.py:close():281] close: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/run-25hk0gke.wandb
2022-03-01 01:29:57,257 INFO SenderThread:236851 [sender.py:finish():1075] shutting down sender
2022-03-01 01:29:57,257 INFO SenderThread:236851 [file_pusher.py:finish():177] shutting down file pusher
2022-03-01 01:29:57,257 INFO SenderThread:236851 [file_pusher.py:join():182] waiting for file pusher
2022-03-01 01:29:57,259 INFO MainThread:236851 [internal.py:handle_exit():79] Internal process exited