File size: 24,131 Bytes
4f730f1 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 |
2022-03-24 10:38:21,870 INFO MainThread:94821 [internal.py:wandb_internal():92] W&B internal server running at pid: 94821, started at: 2022-03-24 10:38:21.870480
2022-03-24 10:38:21,897 INFO WriterThread:94821 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/run-m1uuspzx.wandb
2022-03-24 10:38:21,898 DEBUG SenderThread:94821 [sender.py:send():235] send: header
2022-03-24 10:38:21,898 DEBUG SenderThread:94821 [sender.py:send():235] send: run
2022-03-24 10:38:21,964 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: check_version
2022-03-24 10:38:21,965 INFO SenderThread:94821 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files
2022-03-24 10:38:21,965 INFO SenderThread:94821 [sender.py:_start_run_threads():812] run started: m1uuspzx with start time 1648118301
2022-03-24 10:38:21,965 DEBUG SenderThread:94821 [sender.py:send():235] send: summary
2022-03-24 10:38:21,965 INFO SenderThread:94821 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-03-24 10:38:21,965 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: check_version
2022-03-24 10:38:22,024 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: run_start
2022-03-24 10:38:23,028 INFO Thread-7 :94821 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/wandb-summary.json
2022-03-24 10:38:24,159 DEBUG HandlerThread:94821 [meta.py:__init__():37] meta init
2022-03-24 10:38:24,159 DEBUG HandlerThread:94821 [meta.py:__init__():51] meta init done
2022-03-24 10:38:24,159 DEBUG HandlerThread:94821 [meta.py:probe():211] probe
2022-03-24 10:38:24,160 DEBUG HandlerThread:94821 [meta.py:_setup_git():201] setup git
2022-03-24 10:38:24,193 DEBUG HandlerThread:94821 [meta.py:_setup_git():208] setup git done
2022-03-24 10:38:24,193 DEBUG HandlerThread:94821 [meta.py:_save_pip():55] save pip
2022-03-24 10:38:24,193 DEBUG HandlerThread:94821 [meta.py:_save_pip():69] save pip done
2022-03-24 10:38:24,193 DEBUG HandlerThread:94821 [meta.py:probe():249] probe done
2022-03-24 10:38:24,197 DEBUG SenderThread:94821 [sender.py:send():235] send: files
2022-03-24 10:38:24,197 INFO SenderThread:94821 [sender.py:_save_file():947] saving file wandb-metadata.json with policy now
2022-03-24 10:38:24,202 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:38:24,203 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:38:24,253 DEBUG SenderThread:94821 [sender.py:send():235] send: telemetry
2022-03-24 10:38:24,436 INFO Thread-11 :94821 [upload_job.py:push():137] Uploaded file /tmp/tmp7e9jep7fwandb/2kq1u3tk-wandb-metadata.json
2022-03-24 10:38:25,080 INFO Thread-7 :94821 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/wandb-metadata.json
2022-03-24 10:38:25,081 INFO Thread-7 :94821 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/output.log
2022-03-24 10:38:25,081 INFO Thread-7 :94821 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/requirements.txt
2022-03-24 10:38:27,081 INFO Thread-7 :94821 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/output.log
2022-03-24 10:38:39,255 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:38:39,255 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:38:43,087 INFO Thread-7 :94821 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/output.log
2022-03-24 10:38:47,089 INFO Thread-7 :94821 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/output.log
2022-03-24 10:38:49,090 INFO Thread-7 :94821 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/output.log
2022-03-24 10:38:52,237 DEBUG SenderThread:94821 [sender.py:send():235] send: stats
2022-03-24 10:38:53,091 INFO Thread-7 :94821 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/output.log
2022-03-24 10:38:53,092 INFO Thread-7 :94821 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/config.yaml
2022-03-24 10:38:54,291 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:38:54,291 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:38:57,093 INFO Thread-7 :94821 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/output.log
2022-03-24 10:39:09,352 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:39:09,353 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:39:22,309 DEBUG SenderThread:94821 [sender.py:send():235] send: stats
2022-03-24 10:39:24,449 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:39:24,449 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:39:39,531 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:39:39,531 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:39:52,382 DEBUG SenderThread:94821 [sender.py:send():235] send: stats
2022-03-24 10:39:54,601 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:39:54,602 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:40:09,640 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:40:09,640 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:40:22,457 DEBUG SenderThread:94821 [sender.py:send():235] send: stats
2022-03-24 10:40:24,688 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:40:24,688 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:40:39,754 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:40:39,759 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:40:52,532 DEBUG SenderThread:94821 [sender.py:send():235] send: stats
2022-03-24 10:40:54,817 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:40:54,817 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:41:09,853 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:41:09,854 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:41:22,604 DEBUG SenderThread:94821 [sender.py:send():235] send: stats
2022-03-24 10:41:24,889 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:41:24,890 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:41:39,927 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:41:39,927 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:41:52,678 DEBUG SenderThread:94821 [sender.py:send():235] send: stats
2022-03-24 10:41:54,965 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:41:54,965 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:42:10,012 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:42:10,013 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:42:22,756 DEBUG SenderThread:94821 [sender.py:send():235] send: stats
2022-03-24 10:42:25,051 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:42:25,051 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:42:40,092 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:42:40,093 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:42:52,828 DEBUG SenderThread:94821 [sender.py:send():235] send: stats
2022-03-24 10:42:55,293 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:42:55,294 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:42:56,754 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: partial_history
2022-03-24 10:42:56,754 DEBUG SenderThread:94821 [sender.py:send():235] send: telemetry
2022-03-24 10:42:58,184 INFO Thread-7 :94821 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/output.log
2022-03-24 10:42:59,214 DEBUG SenderThread:94821 [sender.py:send():235] send: exit
2022-03-24 10:42:59,215 INFO SenderThread:94821 [sender.py:send_exit():371] handling exit code: 1
2022-03-24 10:42:59,217 INFO SenderThread:94821 [sender.py:send_exit():373] handling runtime: 277
2022-03-24 10:42:59,217 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-24 10:42:59,218 INFO SenderThread:94821 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-03-24 10:42:59,218 INFO SenderThread:94821 [sender.py:send_exit():379] send defer
2022-03-24 10:42:59,218 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: poll_exit
2022-03-24 10:42:59,219 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: defer
2022-03-24 10:42:59,219 INFO HandlerThread:94821 [handler.py:handle_request_defer():164] handle defer: 0
2022-03-24 10:42:59,220 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: defer
2022-03-24 10:42:59,220 INFO SenderThread:94821 [sender.py:send_request_defer():388] handle sender defer: 0
2022-03-24 10:42:59,220 INFO SenderThread:94821 [sender.py:transition_state():392] send defer: 1
2022-03-24 10:42:59,220 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: defer
2022-03-24 10:42:59,220 INFO HandlerThread:94821 [handler.py:handle_request_defer():164] handle defer: 1
2022-03-24 10:42:59,244 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: defer
2022-03-24 10:42:59,245 INFO SenderThread:94821 [sender.py:send_request_defer():388] handle sender defer: 1
2022-03-24 10:42:59,245 INFO SenderThread:94821 [sender.py:transition_state():392] send defer: 2
2022-03-24 10:42:59,245 DEBUG SenderThread:94821 [sender.py:send():235] send: stats
2022-03-24 10:42:59,245 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: defer
2022-03-24 10:42:59,245 INFO HandlerThread:94821 [handler.py:handle_request_defer():164] handle defer: 2
2022-03-24 10:42:59,246 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: defer
2022-03-24 10:42:59,246 INFO SenderThread:94821 [sender.py:send_request_defer():388] handle sender defer: 2
2022-03-24 10:42:59,246 INFO SenderThread:94821 [sender.py:transition_state():392] send defer: 3
2022-03-24 10:42:59,246 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: defer
2022-03-24 10:42:59,246 INFO HandlerThread:94821 [handler.py:handle_request_defer():164] handle defer: 3
2022-03-24 10:42:59,246 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: defer
2022-03-24 10:42:59,246 INFO SenderThread:94821 [sender.py:send_request_defer():388] handle sender defer: 3
2022-03-24 10:42:59,246 INFO SenderThread:94821 [sender.py:transition_state():392] send defer: 4
2022-03-24 10:42:59,247 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: defer
2022-03-24 10:42:59,247 INFO HandlerThread:94821 [handler.py:handle_request_defer():164] handle defer: 4
2022-03-24 10:42:59,247 DEBUG SenderThread:94821 [sender.py:send():235] send: summary
2022-03-24 10:42:59,247 INFO SenderThread:94821 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-03-24 10:42:59,247 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: defer
2022-03-24 10:42:59,247 INFO SenderThread:94821 [sender.py:send_request_defer():388] handle sender defer: 4
2022-03-24 10:42:59,248 INFO SenderThread:94821 [sender.py:transition_state():392] send defer: 5
2022-03-24 10:42:59,248 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: defer
2022-03-24 10:42:59,248 INFO HandlerThread:94821 [handler.py:handle_request_defer():164] handle defer: 5
2022-03-24 10:42:59,248 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: defer
2022-03-24 10:42:59,248 INFO SenderThread:94821 [sender.py:send_request_defer():388] handle sender defer: 5
2022-03-24 10:42:59,321 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-24 10:42:59,347 INFO SenderThread:94821 [sender.py:transition_state():392] send defer: 6
2022-03-24 10:42:59,348 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: poll_exit
2022-03-24 10:42:59,348 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: defer
2022-03-24 10:42:59,348 INFO HandlerThread:94821 [handler.py:handle_request_defer():164] handle defer: 6
2022-03-24 10:42:59,348 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: defer
2022-03-24 10:42:59,349 INFO SenderThread:94821 [sender.py:send_request_defer():388] handle sender defer: 6
2022-03-24 10:42:59,349 INFO SenderThread:94821 [dir_watcher.py:finish():283] shutting down directory watcher
2022-03-24 10:42:59,449 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-24 10:43:00,184 INFO Thread-7 :94821 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/output.log
2022-03-24 10:43:00,185 INFO SenderThread:94821 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/wandb-summary.json
2022-03-24 10:43:00,185 INFO SenderThread:94821 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/config.yaml
2022-03-24 10:43:00,186 INFO SenderThread:94821 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files
2022-03-24 10:43:00,186 INFO SenderThread:94821 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/requirements.txt requirements.txt
2022-03-24 10:43:00,186 INFO SenderThread:94821 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/output.log output.log
2022-03-24 10:43:00,186 INFO SenderThread:94821 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/wandb-metadata.json wandb-metadata.json
2022-03-24 10:43:00,186 INFO SenderThread:94821 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/config.yaml config.yaml
2022-03-24 10:43:00,192 INFO SenderThread:94821 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/wandb-summary.json wandb-summary.json
2022-03-24 10:43:00,193 INFO SenderThread:94821 [sender.py:transition_state():392] send defer: 7
2022-03-24 10:43:00,193 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: poll_exit
2022-03-24 10:43:00,199 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: defer
2022-03-24 10:43:00,199 INFO HandlerThread:94821 [handler.py:handle_request_defer():164] handle defer: 7
2022-03-24 10:43:00,199 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: defer
2022-03-24 10:43:00,200 INFO SenderThread:94821 [sender.py:send_request_defer():388] handle sender defer: 7
2022-03-24 10:43:00,200 INFO SenderThread:94821 [file_pusher.py:finish():145] shutting down file pusher
2022-03-24 10:43:00,295 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-24 10:43:00,295 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: poll_exit
2022-03-24 10:43:00,397 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-24 10:43:00,397 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: poll_exit
2022-03-24 10:43:00,453 INFO Thread-14 :94821 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/config.yaml
2022-03-24 10:43:00,465 INFO Thread-12 :94821 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/requirements.txt
2022-03-24 10:43:00,471 INFO Thread-13 :94821 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/output.log
2022-03-24 10:43:00,499 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-24 10:43:00,499 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: poll_exit
2022-03-24 10:43:00,562 INFO Thread-15 :94821 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/wandb-summary.json
2022-03-24 10:43:00,600 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-24 10:43:00,601 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: poll_exit
2022-03-24 10:43:00,702 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-24 10:43:00,703 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: poll_exit
2022-03-24 10:43:00,762 INFO Thread-6 :94821 [sender.py:transition_state():392] send defer: 8
2022-03-24 10:43:00,763 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: defer
2022-03-24 10:43:00,763 INFO HandlerThread:94821 [handler.py:handle_request_defer():164] handle defer: 8
2022-03-24 10:43:00,763 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: defer
2022-03-24 10:43:00,763 INFO SenderThread:94821 [sender.py:send_request_defer():388] handle sender defer: 8
2022-03-24 10:43:00,804 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-24 10:43:00,864 INFO SenderThread:94821 [sender.py:transition_state():392] send defer: 9
2022-03-24 10:43:00,864 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: poll_exit
2022-03-24 10:43:00,865 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: defer
2022-03-24 10:43:00,865 INFO HandlerThread:94821 [handler.py:handle_request_defer():164] handle defer: 9
2022-03-24 10:43:00,865 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: defer
2022-03-24 10:43:00,865 INFO SenderThread:94821 [sender.py:send_request_defer():388] handle sender defer: 9
2022-03-24 10:43:00,865 INFO SenderThread:94821 [sender.py:transition_state():392] send defer: 10
2022-03-24 10:43:00,866 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: defer
2022-03-24 10:43:00,866 INFO HandlerThread:94821 [handler.py:handle_request_defer():164] handle defer: 10
2022-03-24 10:43:00,866 DEBUG SenderThread:94821 [sender.py:send():235] send: final
2022-03-24 10:43:00,866 DEBUG SenderThread:94821 [sender.py:send():235] send: footer
2022-03-24 10:43:00,866 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: defer
2022-03-24 10:43:00,866 INFO SenderThread:94821 [sender.py:send_request_defer():388] handle sender defer: 10
2022-03-24 10:43:00,966 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-24 10:43:00,966 DEBUG SenderThread:94821 [sender.py:send_request():249] send_request: poll_exit
2022-03-24 10:43:00,966 INFO SenderThread:94821 [file_pusher.py:join():150] waiting for file pusher
2022-03-24 10:43:01,129 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: sampled_history
2022-03-24 10:43:01,130 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: get_summary
2022-03-24 10:43:01,131 DEBUG HandlerThread:94821 [handler.py:handle_request():141] handle_request: shutdown
2022-03-24 10:43:01,131 INFO HandlerThread:94821 [handler.py:finish():778] shutting down handler
2022-03-24 10:43:01,866 INFO WriterThread:94821 [datastore.py:close():281] close: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/run-m1uuspzx.wandb
2022-03-24 10:43:02,028 INFO SenderThread:94821 [sender.py:finish():1078] shutting down sender
2022-03-24 10:43:02,028 INFO SenderThread:94821 [file_pusher.py:finish():145] shutting down file pusher
2022-03-24 10:43:02,028 INFO SenderThread:94821 [file_pusher.py:join():150] waiting for file pusher
2022-03-24 10:43:02,031 INFO MainThread:94821 [internal.py:handle_exit():82] Internal process exited
|