File size: 16,996 Bytes
a5afdd5 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 |
2022-03-01 01:29:45,420 INFO MainThread:236851 [internal.py:wandb_internal():89] W&B internal server running at pid: 236851, started at: 2022-03-01 01:29:45.420003
2022-03-01 01:29:45,422 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: check_version
2022-03-01 01:29:45,422 INFO WriterThread:236851 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/run-25hk0gke.wandb
2022-03-01 01:29:45,424 DEBUG SenderThread:236851 [sender.py:send():235] send: header
2022-03-01 01:29:45,424 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: check_version
2022-03-01 01:29:45,489 DEBUG SenderThread:236851 [sender.py:send():235] send: run
2022-03-01 01:29:45,576 INFO SenderThread:236851 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files
2022-03-01 01:29:45,576 INFO SenderThread:236851 [sender.py:_start_run_threads():809] run started: 25hk0gke with start time 1646098184
2022-03-01 01:29:45,576 DEBUG SenderThread:236851 [sender.py:send():235] send: summary
2022-03-01 01:29:45,576 INFO SenderThread:236851 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 01:29:45,577 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: run_start
2022-03-01 01:29:45,582 DEBUG HandlerThread:236851 [meta.py:__init__():36] meta init
2022-03-01 01:29:45,582 DEBUG HandlerThread:236851 [meta.py:__init__():50] meta init done
2022-03-01 01:29:45,582 DEBUG HandlerThread:236851 [meta.py:probe():210] probe
2022-03-01 01:29:45,588 DEBUG HandlerThread:236851 [meta.py:_setup_git():200] setup git
2022-03-01 01:29:45,603 DEBUG HandlerThread:236851 [meta.py:_setup_git():207] setup git done
2022-03-01 01:29:45,603 DEBUG HandlerThread:236851 [meta.py:_save_pip():54] save pip
2022-03-01 01:29:45,604 DEBUG HandlerThread:236851 [meta.py:_save_pip():68] save pip done
2022-03-01 01:29:45,604 DEBUG HandlerThread:236851 [meta.py:probe():248] probe done
2022-03-01 01:29:45,684 DEBUG SenderThread:236851 [sender.py:send():235] send: files
2022-03-01 01:29:45,684 INFO SenderThread:236851 [sender.py:_save_file():944] saving file wandb-metadata.json with policy now
2022-03-01 01:29:45,689 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: stop_status
2022-03-01 01:29:45,690 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: stop_status
2022-03-01 01:29:45,729 DEBUG SenderThread:236851 [sender.py:send():235] send: config
2022-03-01 01:29:45,731 DEBUG SenderThread:236851 [sender.py:send():235] send: metric
2022-03-01 01:29:45,731 DEBUG SenderThread:236851 [sender.py:send():235] send: metric
2022-03-01 01:29:45,731 WARNING SenderThread:236851 [sender.py:send_metric():902] Seen metric with glob (shouldnt happen)
2022-03-01 01:29:45,969 INFO Thread-11 :236851 [upload_job.py:push():137] Uploaded file /tmp/tmpdzeo2dm2wandb/3813kelu-wandb-metadata.json
2022-03-01 01:29:46,578 INFO Thread-8 :236851 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/requirements.txt
2022-03-01 01:29:46,578 INFO Thread-8 :236851 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/wandb-summary.json
2022-03-01 01:29:46,578 INFO Thread-8 :236851 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/wandb-metadata.json
2022-03-01 01:29:46,578 INFO Thread-8 :236851 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/output.log
2022-03-01 01:29:48,577 INFO Thread-8 :236851 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/output.log
2022-03-01 01:29:52,578 INFO Thread-8 :236851 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/output.log
2022-03-01 01:29:54,442 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:29:54,442 DEBUG SenderThread:236851 [sender.py:send():235] send: telemetry
2022-03-01 01:29:54,443 DEBUG SenderThread:236851 [sender.py:send():235] send: exit
2022-03-01 01:29:54,443 INFO SenderThread:236851 [sender.py:send_exit():371] handling exit code: 1
2022-03-01 01:29:54,443 INFO SenderThread:236851 [sender.py:send_exit():373] handling runtime: 8
2022-03-01 01:29:54,443 INFO SenderThread:236851 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 01:29:54,443 INFO SenderThread:236851 [sender.py:send_exit():379] send defer
2022-03-01 01:29:54,444 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:29:54,444 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:29:54,444 INFO HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 0
2022-03-01 01:29:54,444 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: defer
2022-03-01 01:29:54,445 INFO SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 0
2022-03-01 01:29:54,445 INFO SenderThread:236851 [sender.py:transition_state():392] send defer: 1
2022-03-01 01:29:54,445 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:29:54,445 INFO HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 1
2022-03-01 01:29:54,515 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: defer
2022-03-01 01:29:54,515 INFO SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 1
2022-03-01 01:29:54,515 INFO SenderThread:236851 [sender.py:transition_state():392] send defer: 2
2022-03-01 01:29:54,515 DEBUG SenderThread:236851 [sender.py:send():235] send: stats
2022-03-01 01:29:54,516 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:29:54,516 INFO HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 2
2022-03-01 01:29:54,516 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: defer
2022-03-01 01:29:54,516 INFO SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 2
2022-03-01 01:29:54,516 INFO SenderThread:236851 [sender.py:transition_state():392] send defer: 3
2022-03-01 01:29:54,517 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:29:54,517 INFO HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 3
2022-03-01 01:29:54,517 DEBUG SenderThread:236851 [sender.py:send():235] send: summary
2022-03-01 01:29:54,517 INFO SenderThread:236851 [sender.py:_save_file():944] saving file wandb-summary.json with policy end
2022-03-01 01:29:54,518 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: defer
2022-03-01 01:29:54,518 INFO SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 3
2022-03-01 01:29:54,518 INFO SenderThread:236851 [sender.py:transition_state():392] send defer: 4
2022-03-01 01:29:54,518 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:29:54,518 INFO HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 4
2022-03-01 01:29:54,518 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: defer
2022-03-01 01:29:54,518 INFO SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 4
2022-03-01 01:29:54,545 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:29:54,579 INFO Thread-8 :236851 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/wandb-summary.json
2022-03-01 01:29:54,579 INFO Thread-8 :236851 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/output.log
2022-03-01 01:29:54,803 INFO SenderThread:236851 [sender.py:transition_state():392] send defer: 5
2022-03-01 01:29:54,803 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:29:54,804 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:29:54,804 INFO HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 5
2022-03-01 01:29:54,804 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: defer
2022-03-01 01:29:54,804 INFO SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 5
2022-03-01 01:29:54,804 INFO SenderThread:236851 [dir_watcher.py:finish():283] shutting down directory watcher
2022-03-01 01:29:54,905 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:29:55,579 INFO Thread-8 :236851 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/config.yaml
2022-03-01 01:29:55,580 INFO SenderThread:236851 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files
2022-03-01 01:29:55,580 INFO SenderThread:236851 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/wandb-metadata.json wandb-metadata.json
2022-03-01 01:29:55,580 INFO SenderThread:236851 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/output.log output.log
2022-03-01 01:29:55,581 INFO SenderThread:236851 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/wandb-summary.json wandb-summary.json
2022-03-01 01:29:55,581 INFO SenderThread:236851 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/requirements.txt requirements.txt
2022-03-01 01:29:55,586 INFO SenderThread:236851 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/config.yaml config.yaml
2022-03-01 01:29:55,586 INFO SenderThread:236851 [sender.py:transition_state():392] send defer: 6
2022-03-01 01:29:55,587 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:29:55,588 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:29:55,588 INFO HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 6
2022-03-01 01:29:55,591 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: defer
2022-03-01 01:29:55,591 INFO SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 6
2022-03-01 01:29:55,592 INFO SenderThread:236851 [file_pusher.py:finish():177] shutting down file pusher
2022-03-01 01:29:55,689 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:29:55,689 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:29:55,791 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:29:55,791 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:29:55,856 INFO Thread-14 :236851 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/requirements.txt
2022-03-01 01:29:55,860 INFO Thread-13 :236851 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/wandb-summary.json
2022-03-01 01:29:55,885 INFO Thread-15 :236851 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/config.yaml
2022-03-01 01:29:55,893 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:29:55,893 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:29:55,941 INFO Thread-12 :236851 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/files/output.log
2022-03-01 01:29:55,994 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:29:55,995 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:29:56,096 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:29:56,096 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:29:56,141 INFO Thread-7 :236851 [sender.py:transition_state():392] send defer: 7
2022-03-01 01:29:56,142 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:29:56,142 INFO HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 7
2022-03-01 01:29:56,142 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: defer
2022-03-01 01:29:56,142 INFO SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 7
2022-03-01 01:29:56,188 INFO SenderThread:236851 [sender.py:transition_state():392] send defer: 8
2022-03-01 01:29:56,188 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:29:56,188 INFO HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 8
2022-03-01 01:29:56,188 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: defer
2022-03-01 01:29:56,188 INFO SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 8
2022-03-01 01:29:56,189 INFO SenderThread:236851 [sender.py:transition_state():392] send defer: 9
2022-03-01 01:29:56,189 DEBUG SenderThread:236851 [sender.py:send():235] send: final
2022-03-01 01:29:56,189 DEBUG SenderThread:236851 [sender.py:send():235] send: footer
2022-03-01 01:29:56,189 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: defer
2022-03-01 01:29:56,189 INFO HandlerThread:236851 [handler.py:handle_request_defer():154] handle defer: 9
2022-03-01 01:29:56,190 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: defer
2022-03-01 01:29:56,190 INFO SenderThread:236851 [sender.py:send_request_defer():388] handle sender defer: 9
2022-03-01 01:29:56,197 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: poll_exit
2022-03-01 01:29:56,198 DEBUG SenderThread:236851 [sender.py:send_request():249] send_request: poll_exit
2022-03-01 01:29:56,198 INFO SenderThread:236851 [file_pusher.py:join():182] waiting for file pusher
2022-03-01 01:29:56,258 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: get_summary
2022-03-01 01:29:56,259 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: sampled_history
2022-03-01 01:29:56,259 DEBUG HandlerThread:236851 [handler.py:handle_request():131] handle_request: shutdown
2022-03-01 01:29:56,259 INFO HandlerThread:236851 [handler.py:finish():739] shutting down handler
2022-03-01 01:29:57,190 INFO WriterThread:236851 [datastore.py:close():281] close: /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220301_012944-25hk0gke/run-25hk0gke.wandb
2022-03-01 01:29:57,257 INFO SenderThread:236851 [sender.py:finish():1075] shutting down sender
2022-03-01 01:29:57,257 INFO SenderThread:236851 [file_pusher.py:finish():177] shutting down file pusher
2022-03-01 01:29:57,257 INFO SenderThread:236851 [file_pusher.py:join():182] waiting for file pusher
2022-03-01 01:29:57,259 INFO MainThread:236851 [internal.py:handle_exit():79] Internal process exited
|