File size: 23,295 Bytes
4f730f1 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 |
2022-03-23 17:15:39,798 INFO MainThread:16634 [internal.py:wandb_internal():92] W&B internal server running at pid: 16634, started at: 2022-03-23 17:15:39.798196
2022-03-23 17:15:39,829 INFO WriterThread:16634 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/run-5b3vz6lx.wandb
2022-03-23 17:15:39,830 DEBUG SenderThread:16634 [sender.py:send():235] send: header
2022-03-23 17:15:39,830 DEBUG SenderThread:16634 [sender.py:send():235] send: run
2022-03-23 17:15:40,142 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: check_version
2022-03-23 17:15:40,142 INFO SenderThread:16634 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files
2022-03-23 17:15:40,143 INFO SenderThread:16634 [sender.py:_start_run_threads():812] run started: 5b3vz6lx with start time 1648055739
2022-03-23 17:15:40,143 DEBUG SenderThread:16634 [sender.py:send():235] send: summary
2022-03-23 17:15:40,143 INFO SenderThread:16634 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-03-23 17:15:40,143 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: check_version
2022-03-23 17:15:40,202 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: run_start
2022-03-23 17:15:41,144 INFO Thread-7 :16634 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/wandb-summary.json
2022-03-23 17:15:42,639 DEBUG HandlerThread:16634 [meta.py:__init__():37] meta init
2022-03-23 17:15:42,639 DEBUG HandlerThread:16634 [meta.py:__init__():51] meta init done
2022-03-23 17:15:42,639 DEBUG HandlerThread:16634 [meta.py:probe():211] probe
2022-03-23 17:15:42,641 DEBUG HandlerThread:16634 [meta.py:_setup_git():201] setup git
2022-03-23 17:15:42,676 DEBUG HandlerThread:16634 [meta.py:_setup_git():208] setup git done
2022-03-23 17:15:42,676 DEBUG HandlerThread:16634 [meta.py:_save_pip():55] save pip
2022-03-23 17:15:42,677 DEBUG HandlerThread:16634 [meta.py:_save_pip():69] save pip done
2022-03-23 17:15:42,677 DEBUG HandlerThread:16634 [meta.py:probe():249] probe done
2022-03-23 17:15:42,680 DEBUG SenderThread:16634 [sender.py:send():235] send: files
2022-03-23 17:15:42,681 INFO SenderThread:16634 [sender.py:_save_file():947] saving file wandb-metadata.json with policy now
2022-03-23 17:15:42,686 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status
2022-03-23 17:15:42,687 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: stop_status
2022-03-23 17:15:42,740 DEBUG SenderThread:16634 [sender.py:send():235] send: telemetry
2022-03-23 17:15:43,147 INFO Thread-7 :16634 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/requirements.txt
2022-03-23 17:15:43,147 INFO Thread-7 :16634 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/output.log
2022-03-23 17:15:43,147 INFO Thread-7 :16634 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/wandb-metadata.json
2022-03-23 17:15:43,573 INFO Thread-11 :16634 [upload_job.py:push():137] Uploaded file /tmp/tmpcgnu3sa6wandb/1lmj2vet-wandb-metadata.json
2022-03-23 17:15:45,148 INFO Thread-7 :16634 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/output.log
2022-03-23 17:15:57,742 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status
2022-03-23 17:15:57,742 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: stop_status
2022-03-23 17:16:01,154 INFO Thread-7 :16634 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/output.log
2022-03-23 17:16:05,156 INFO Thread-7 :16634 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/output.log
2022-03-23 17:16:09,158 INFO Thread-7 :16634 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/output.log
2022-03-23 17:16:10,158 INFO Thread-7 :16634 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/config.yaml
2022-03-23 17:16:10,726 DEBUG SenderThread:16634 [sender.py:send():235] send: stats
2022-03-23 17:16:12,777 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status
2022-03-23 17:16:12,777 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: stop_status
2022-03-23 17:16:13,159 INFO Thread-7 :16634 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/output.log
2022-03-23 17:16:17,161 INFO Thread-7 :16634 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/output.log
2022-03-23 17:16:27,852 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status
2022-03-23 17:16:27,852 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: stop_status
2022-03-23 17:16:40,804 DEBUG SenderThread:16634 [sender.py:send():235] send: stats
2022-03-23 17:16:42,916 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status
2022-03-23 17:16:42,916 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: stop_status
2022-03-23 17:16:57,968 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status
2022-03-23 17:16:57,968 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: stop_status
2022-03-23 17:17:10,883 DEBUG SenderThread:16634 [sender.py:send():235] send: stats
2022-03-23 17:17:13,013 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status
2022-03-23 17:17:13,013 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: stop_status
2022-03-23 17:17:28,060 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status
2022-03-23 17:17:28,060 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: stop_status
2022-03-23 17:17:40,963 DEBUG SenderThread:16634 [sender.py:send():235] send: stats
2022-03-23 17:17:43,134 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status
2022-03-23 17:17:43,134 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: stop_status
2022-03-23 17:17:58,179 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status
2022-03-23 17:17:58,179 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: stop_status
2022-03-23 17:18:11,043 DEBUG SenderThread:16634 [sender.py:send():235] send: stats
2022-03-23 17:18:13,217 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status
2022-03-23 17:18:13,217 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: stop_status
2022-03-23 17:18:28,336 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status
2022-03-23 17:18:28,337 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: stop_status
2022-03-23 17:18:41,123 DEBUG SenderThread:16634 [sender.py:send():235] send: stats
2022-03-23 17:18:43,408 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status
2022-03-23 17:18:43,408 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: stop_status
2022-03-23 17:18:58,446 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status
2022-03-23 17:18:58,447 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: stop_status
2022-03-23 17:19:11,201 DEBUG SenderThread:16634 [sender.py:send():235] send: stats
2022-03-23 17:19:13,488 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status
2022-03-23 17:19:13,489 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: stop_status
2022-03-23 17:19:28,533 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status
2022-03-23 17:19:28,533 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: stop_status
2022-03-23 17:19:37,253 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: partial_history
2022-03-23 17:19:37,254 DEBUG SenderThread:16634 [sender.py:send():235] send: telemetry
2022-03-23 17:19:38,243 INFO Thread-7 :16634 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/output.log
2022-03-23 17:19:39,594 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-23 17:19:39,594 DEBUG SenderThread:16634 [sender.py:send():235] send: exit
2022-03-23 17:19:39,595 INFO SenderThread:16634 [sender.py:send_exit():371] handling exit code: 1
2022-03-23 17:19:39,596 INFO SenderThread:16634 [sender.py:send_exit():373] handling runtime: 239
2022-03-23 17:19:39,596 INFO SenderThread:16634 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-03-23 17:19:39,597 INFO SenderThread:16634 [sender.py:send_exit():379] send defer
2022-03-23 17:19:39,597 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: poll_exit
2022-03-23 17:19:39,597 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: defer
2022-03-23 17:19:39,598 INFO HandlerThread:16634 [handler.py:handle_request_defer():164] handle defer: 0
2022-03-23 17:19:39,598 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: defer
2022-03-23 17:19:39,598 INFO SenderThread:16634 [sender.py:send_request_defer():388] handle sender defer: 0
2022-03-23 17:19:39,598 INFO SenderThread:16634 [sender.py:transition_state():392] send defer: 1
2022-03-23 17:19:39,598 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: defer
2022-03-23 17:19:39,598 INFO HandlerThread:16634 [handler.py:handle_request_defer():164] handle defer: 1
2022-03-23 17:19:39,673 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: defer
2022-03-23 17:19:39,673 INFO SenderThread:16634 [sender.py:send_request_defer():388] handle sender defer: 1
2022-03-23 17:19:39,673 INFO SenderThread:16634 [sender.py:transition_state():392] send defer: 2
2022-03-23 17:19:39,673 DEBUG SenderThread:16634 [sender.py:send():235] send: stats
2022-03-23 17:19:39,674 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: defer
2022-03-23 17:19:39,674 INFO HandlerThread:16634 [handler.py:handle_request_defer():164] handle defer: 2
2022-03-23 17:19:39,674 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: defer
2022-03-23 17:19:39,674 INFO SenderThread:16634 [sender.py:send_request_defer():388] handle sender defer: 2
2022-03-23 17:19:39,674 INFO SenderThread:16634 [sender.py:transition_state():392] send defer: 3
2022-03-23 17:19:39,674 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: defer
2022-03-23 17:19:39,674 INFO HandlerThread:16634 [handler.py:handle_request_defer():164] handle defer: 3
2022-03-23 17:19:39,674 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: defer
2022-03-23 17:19:39,675 INFO SenderThread:16634 [sender.py:send_request_defer():388] handle sender defer: 3
2022-03-23 17:19:39,675 INFO SenderThread:16634 [sender.py:transition_state():392] send defer: 4
2022-03-23 17:19:39,675 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: defer
2022-03-23 17:19:39,675 INFO HandlerThread:16634 [handler.py:handle_request_defer():164] handle defer: 4
2022-03-23 17:19:39,675 DEBUG SenderThread:16634 [sender.py:send():235] send: summary
2022-03-23 17:19:39,675 INFO SenderThread:16634 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-03-23 17:19:39,676 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: defer
2022-03-23 17:19:39,676 INFO SenderThread:16634 [sender.py:send_request_defer():388] handle sender defer: 4
2022-03-23 17:19:39,676 INFO SenderThread:16634 [sender.py:transition_state():392] send defer: 5
2022-03-23 17:19:39,676 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: defer
2022-03-23 17:19:39,676 INFO HandlerThread:16634 [handler.py:handle_request_defer():164] handle defer: 5
2022-03-23 17:19:39,676 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: defer
2022-03-23 17:19:39,676 INFO SenderThread:16634 [sender.py:send_request_defer():388] handle sender defer: 5
2022-03-23 17:19:39,699 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-23 17:19:39,863 INFO SenderThread:16634 [sender.py:transition_state():392] send defer: 6
2022-03-23 17:19:39,863 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: poll_exit
2022-03-23 17:19:39,864 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: defer
2022-03-23 17:19:39,864 INFO HandlerThread:16634 [handler.py:handle_request_defer():164] handle defer: 6
2022-03-23 17:19:39,864 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: defer
2022-03-23 17:19:39,864 INFO SenderThread:16634 [sender.py:send_request_defer():388] handle sender defer: 6
2022-03-23 17:19:39,864 INFO SenderThread:16634 [dir_watcher.py:finish():283] shutting down directory watcher
2022-03-23 17:19:39,965 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-23 17:19:40,244 INFO SenderThread:16634 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/output.log
2022-03-23 17:19:40,244 INFO SenderThread:16634 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/config.yaml
2022-03-23 17:19:40,244 INFO SenderThread:16634 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/wandb-summary.json
2022-03-23 17:19:40,244 INFO SenderThread:16634 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files
2022-03-23 17:19:40,245 INFO SenderThread:16634 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/requirements.txt requirements.txt
2022-03-23 17:19:40,245 INFO SenderThread:16634 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/output.log output.log
2022-03-23 17:19:40,245 INFO SenderThread:16634 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/wandb-metadata.json wandb-metadata.json
2022-03-23 17:19:40,245 INFO SenderThread:16634 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/config.yaml config.yaml
2022-03-23 17:19:40,245 INFO SenderThread:16634 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/wandb-summary.json wandb-summary.json
2022-03-23 17:19:40,245 INFO SenderThread:16634 [sender.py:transition_state():392] send defer: 7
2022-03-23 17:19:40,246 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: poll_exit
2022-03-23 17:19:40,247 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: defer
2022-03-23 17:19:40,247 INFO HandlerThread:16634 [handler.py:handle_request_defer():164] handle defer: 7
2022-03-23 17:19:40,250 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: defer
2022-03-23 17:19:40,250 INFO SenderThread:16634 [sender.py:send_request_defer():388] handle sender defer: 7
2022-03-23 17:19:40,250 INFO SenderThread:16634 [file_pusher.py:finish():145] shutting down file pusher
2022-03-23 17:19:40,348 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-23 17:19:40,349 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: poll_exit
2022-03-23 17:19:40,450 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-23 17:19:40,450 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: poll_exit
2022-03-23 17:19:40,518 INFO Thread-14 :16634 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/config.yaml
2022-03-23 17:19:40,552 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-23 17:19:40,552 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: poll_exit
2022-03-23 17:19:40,561 INFO Thread-12 :16634 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/requirements.txt
2022-03-23 17:19:40,570 INFO Thread-13 :16634 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/output.log
2022-03-23 17:19:40,589 INFO Thread-15 :16634 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/wandb-summary.json
2022-03-23 17:19:40,654 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-23 17:19:40,654 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: poll_exit
2022-03-23 17:19:40,755 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-23 17:19:40,756 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: poll_exit
2022-03-23 17:19:40,789 INFO Thread-6 :16634 [sender.py:transition_state():392] send defer: 8
2022-03-23 17:19:40,790 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: defer
2022-03-23 17:19:40,790 INFO HandlerThread:16634 [handler.py:handle_request_defer():164] handle defer: 8
2022-03-23 17:19:40,790 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: defer
2022-03-23 17:19:40,790 INFO SenderThread:16634 [sender.py:send_request_defer():388] handle sender defer: 8
2022-03-23 17:19:40,857 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-23 17:19:40,915 INFO SenderThread:16634 [sender.py:transition_state():392] send defer: 9
2022-03-23 17:19:40,915 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: poll_exit
2022-03-23 17:19:40,915 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: defer
2022-03-23 17:19:40,916 INFO HandlerThread:16634 [handler.py:handle_request_defer():164] handle defer: 9
2022-03-23 17:19:40,916 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: defer
2022-03-23 17:19:40,916 INFO SenderThread:16634 [sender.py:send_request_defer():388] handle sender defer: 9
2022-03-23 17:19:40,916 INFO SenderThread:16634 [sender.py:transition_state():392] send defer: 10
2022-03-23 17:19:40,916 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: defer
2022-03-23 17:19:40,916 INFO HandlerThread:16634 [handler.py:handle_request_defer():164] handle defer: 10
2022-03-23 17:19:40,917 DEBUG SenderThread:16634 [sender.py:send():235] send: final
2022-03-23 17:19:40,917 DEBUG SenderThread:16634 [sender.py:send():235] send: footer
2022-03-23 17:19:40,917 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: defer
2022-03-23 17:19:40,917 INFO SenderThread:16634 [sender.py:send_request_defer():388] handle sender defer: 10
2022-03-23 17:19:41,017 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-23 17:19:41,017 DEBUG SenderThread:16634 [sender.py:send_request():249] send_request: poll_exit
2022-03-23 17:19:41,017 INFO SenderThread:16634 [file_pusher.py:join():150] waiting for file pusher
2022-03-23 17:19:41,172 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: sampled_history
2022-03-23 17:19:41,173 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: get_summary
2022-03-23 17:19:41,174 DEBUG HandlerThread:16634 [handler.py:handle_request():141] handle_request: shutdown
2022-03-23 17:19:41,174 INFO HandlerThread:16634 [handler.py:finish():778] shutting down handler
2022-03-23 17:19:41,917 INFO WriterThread:16634 [datastore.py:close():281] close: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/run-5b3vz6lx.wandb
2022-03-23 17:19:42,071 INFO SenderThread:16634 [sender.py:finish():1078] shutting down sender
2022-03-23 17:19:42,071 INFO SenderThread:16634 [file_pusher.py:finish():145] shutting down file pusher
2022-03-23 17:19:42,071 INFO SenderThread:16634 [file_pusher.py:join():150] waiting for file pusher
2022-03-23 17:19:42,110 INFO MainThread:16634 [internal.py:handle_exit():82] Internal process exited
|