File size: 24,131 Bytes
4f730f1
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
2022-03-24 10:38:21,870 INFO    MainThread:94821 [internal.py:wandb_internal():92] W&B internal server running at pid: 94821, started at: 2022-03-24 10:38:21.870480
2022-03-24 10:38:21,897 INFO    WriterThread:94821 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/run-m1uuspzx.wandb
2022-03-24 10:38:21,898 DEBUG   SenderThread:94821 [sender.py:send():235] send: header
2022-03-24 10:38:21,898 DEBUG   SenderThread:94821 [sender.py:send():235] send: run
2022-03-24 10:38:21,964 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: check_version
2022-03-24 10:38:21,965 INFO    SenderThread:94821 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files
2022-03-24 10:38:21,965 INFO    SenderThread:94821 [sender.py:_start_run_threads():812] run started: m1uuspzx with start time 1648118301
2022-03-24 10:38:21,965 DEBUG   SenderThread:94821 [sender.py:send():235] send: summary
2022-03-24 10:38:21,965 INFO    SenderThread:94821 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-03-24 10:38:21,965 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: check_version
2022-03-24 10:38:22,024 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: run_start
2022-03-24 10:38:23,028 INFO    Thread-7  :94821 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/wandb-summary.json
2022-03-24 10:38:24,159 DEBUG   HandlerThread:94821 [meta.py:__init__():37] meta init
2022-03-24 10:38:24,159 DEBUG   HandlerThread:94821 [meta.py:__init__():51] meta init done
2022-03-24 10:38:24,159 DEBUG   HandlerThread:94821 [meta.py:probe():211] probe
2022-03-24 10:38:24,160 DEBUG   HandlerThread:94821 [meta.py:_setup_git():201] setup git
2022-03-24 10:38:24,193 DEBUG   HandlerThread:94821 [meta.py:_setup_git():208] setup git done
2022-03-24 10:38:24,193 DEBUG   HandlerThread:94821 [meta.py:_save_pip():55] save pip
2022-03-24 10:38:24,193 DEBUG   HandlerThread:94821 [meta.py:_save_pip():69] save pip done
2022-03-24 10:38:24,193 DEBUG   HandlerThread:94821 [meta.py:probe():249] probe done
2022-03-24 10:38:24,197 DEBUG   SenderThread:94821 [sender.py:send():235] send: files
2022-03-24 10:38:24,197 INFO    SenderThread:94821 [sender.py:_save_file():947] saving file wandb-metadata.json with policy now
2022-03-24 10:38:24,202 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:38:24,203 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:38:24,253 DEBUG   SenderThread:94821 [sender.py:send():235] send: telemetry
2022-03-24 10:38:24,436 INFO    Thread-11 :94821 [upload_job.py:push():137] Uploaded file /tmp/tmp7e9jep7fwandb/2kq1u3tk-wandb-metadata.json
2022-03-24 10:38:25,080 INFO    Thread-7  :94821 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/wandb-metadata.json
2022-03-24 10:38:25,081 INFO    Thread-7  :94821 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/output.log
2022-03-24 10:38:25,081 INFO    Thread-7  :94821 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/requirements.txt
2022-03-24 10:38:27,081 INFO    Thread-7  :94821 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/output.log
2022-03-24 10:38:39,255 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:38:39,255 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:38:43,087 INFO    Thread-7  :94821 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/output.log
2022-03-24 10:38:47,089 INFO    Thread-7  :94821 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/output.log
2022-03-24 10:38:49,090 INFO    Thread-7  :94821 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/output.log
2022-03-24 10:38:52,237 DEBUG   SenderThread:94821 [sender.py:send():235] send: stats
2022-03-24 10:38:53,091 INFO    Thread-7  :94821 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/output.log
2022-03-24 10:38:53,092 INFO    Thread-7  :94821 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/config.yaml
2022-03-24 10:38:54,291 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:38:54,291 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:38:57,093 INFO    Thread-7  :94821 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/output.log
2022-03-24 10:39:09,352 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:39:09,353 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:39:22,309 DEBUG   SenderThread:94821 [sender.py:send():235] send: stats
2022-03-24 10:39:24,449 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:39:24,449 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:39:39,531 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:39:39,531 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:39:52,382 DEBUG   SenderThread:94821 [sender.py:send():235] send: stats
2022-03-24 10:39:54,601 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:39:54,602 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:40:09,640 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:40:09,640 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:40:22,457 DEBUG   SenderThread:94821 [sender.py:send():235] send: stats
2022-03-24 10:40:24,688 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:40:24,688 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:40:39,754 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:40:39,759 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:40:52,532 DEBUG   SenderThread:94821 [sender.py:send():235] send: stats
2022-03-24 10:40:54,817 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:40:54,817 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:41:09,853 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:41:09,854 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:41:22,604 DEBUG   SenderThread:94821 [sender.py:send():235] send: stats
2022-03-24 10:41:24,889 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:41:24,890 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:41:39,927 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:41:39,927 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:41:52,678 DEBUG   SenderThread:94821 [sender.py:send():235] send: stats
2022-03-24 10:41:54,965 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:41:54,965 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:42:10,012 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:42:10,013 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:42:22,756 DEBUG   SenderThread:94821 [sender.py:send():235] send: stats
2022-03-24 10:42:25,051 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:42:25,051 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:42:40,092 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:42:40,093 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:42:52,828 DEBUG   SenderThread:94821 [sender.py:send():235] send: stats
2022-03-24 10:42:55,293 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: stop_status
2022-03-24 10:42:55,294 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: stop_status
2022-03-24 10:42:56,754 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: partial_history
2022-03-24 10:42:56,754 DEBUG   SenderThread:94821 [sender.py:send():235] send: telemetry
2022-03-24 10:42:58,184 INFO    Thread-7  :94821 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/output.log
2022-03-24 10:42:59,214 DEBUG   SenderThread:94821 [sender.py:send():235] send: exit
2022-03-24 10:42:59,215 INFO    SenderThread:94821 [sender.py:send_exit():371] handling exit code: 1
2022-03-24 10:42:59,217 INFO    SenderThread:94821 [sender.py:send_exit():373] handling runtime: 277
2022-03-24 10:42:59,217 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-24 10:42:59,218 INFO    SenderThread:94821 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-03-24 10:42:59,218 INFO    SenderThread:94821 [sender.py:send_exit():379] send defer
2022-03-24 10:42:59,218 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: poll_exit
2022-03-24 10:42:59,219 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: defer
2022-03-24 10:42:59,219 INFO    HandlerThread:94821 [handler.py:handle_request_defer():164] handle defer: 0
2022-03-24 10:42:59,220 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: defer
2022-03-24 10:42:59,220 INFO    SenderThread:94821 [sender.py:send_request_defer():388] handle sender defer: 0
2022-03-24 10:42:59,220 INFO    SenderThread:94821 [sender.py:transition_state():392] send defer: 1
2022-03-24 10:42:59,220 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: defer
2022-03-24 10:42:59,220 INFO    HandlerThread:94821 [handler.py:handle_request_defer():164] handle defer: 1
2022-03-24 10:42:59,244 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: defer
2022-03-24 10:42:59,245 INFO    SenderThread:94821 [sender.py:send_request_defer():388] handle sender defer: 1
2022-03-24 10:42:59,245 INFO    SenderThread:94821 [sender.py:transition_state():392] send defer: 2
2022-03-24 10:42:59,245 DEBUG   SenderThread:94821 [sender.py:send():235] send: stats
2022-03-24 10:42:59,245 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: defer
2022-03-24 10:42:59,245 INFO    HandlerThread:94821 [handler.py:handle_request_defer():164] handle defer: 2
2022-03-24 10:42:59,246 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: defer
2022-03-24 10:42:59,246 INFO    SenderThread:94821 [sender.py:send_request_defer():388] handle sender defer: 2
2022-03-24 10:42:59,246 INFO    SenderThread:94821 [sender.py:transition_state():392] send defer: 3
2022-03-24 10:42:59,246 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: defer
2022-03-24 10:42:59,246 INFO    HandlerThread:94821 [handler.py:handle_request_defer():164] handle defer: 3
2022-03-24 10:42:59,246 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: defer
2022-03-24 10:42:59,246 INFO    SenderThread:94821 [sender.py:send_request_defer():388] handle sender defer: 3
2022-03-24 10:42:59,246 INFO    SenderThread:94821 [sender.py:transition_state():392] send defer: 4
2022-03-24 10:42:59,247 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: defer
2022-03-24 10:42:59,247 INFO    HandlerThread:94821 [handler.py:handle_request_defer():164] handle defer: 4
2022-03-24 10:42:59,247 DEBUG   SenderThread:94821 [sender.py:send():235] send: summary
2022-03-24 10:42:59,247 INFO    SenderThread:94821 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-03-24 10:42:59,247 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: defer
2022-03-24 10:42:59,247 INFO    SenderThread:94821 [sender.py:send_request_defer():388] handle sender defer: 4
2022-03-24 10:42:59,248 INFO    SenderThread:94821 [sender.py:transition_state():392] send defer: 5
2022-03-24 10:42:59,248 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: defer
2022-03-24 10:42:59,248 INFO    HandlerThread:94821 [handler.py:handle_request_defer():164] handle defer: 5
2022-03-24 10:42:59,248 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: defer
2022-03-24 10:42:59,248 INFO    SenderThread:94821 [sender.py:send_request_defer():388] handle sender defer: 5
2022-03-24 10:42:59,321 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-24 10:42:59,347 INFO    SenderThread:94821 [sender.py:transition_state():392] send defer: 6
2022-03-24 10:42:59,348 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: poll_exit
2022-03-24 10:42:59,348 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: defer
2022-03-24 10:42:59,348 INFO    HandlerThread:94821 [handler.py:handle_request_defer():164] handle defer: 6
2022-03-24 10:42:59,348 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: defer
2022-03-24 10:42:59,349 INFO    SenderThread:94821 [sender.py:send_request_defer():388] handle sender defer: 6
2022-03-24 10:42:59,349 INFO    SenderThread:94821 [dir_watcher.py:finish():283] shutting down directory watcher
2022-03-24 10:42:59,449 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-24 10:43:00,184 INFO    Thread-7  :94821 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/output.log
2022-03-24 10:43:00,185 INFO    SenderThread:94821 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/wandb-summary.json
2022-03-24 10:43:00,185 INFO    SenderThread:94821 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/config.yaml
2022-03-24 10:43:00,186 INFO    SenderThread:94821 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files
2022-03-24 10:43:00,186 INFO    SenderThread:94821 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/requirements.txt requirements.txt
2022-03-24 10:43:00,186 INFO    SenderThread:94821 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/output.log output.log
2022-03-24 10:43:00,186 INFO    SenderThread:94821 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/wandb-metadata.json wandb-metadata.json
2022-03-24 10:43:00,186 INFO    SenderThread:94821 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/config.yaml config.yaml
2022-03-24 10:43:00,192 INFO    SenderThread:94821 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/wandb-summary.json wandb-summary.json
2022-03-24 10:43:00,193 INFO    SenderThread:94821 [sender.py:transition_state():392] send defer: 7
2022-03-24 10:43:00,193 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: poll_exit
2022-03-24 10:43:00,199 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: defer
2022-03-24 10:43:00,199 INFO    HandlerThread:94821 [handler.py:handle_request_defer():164] handle defer: 7
2022-03-24 10:43:00,199 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: defer
2022-03-24 10:43:00,200 INFO    SenderThread:94821 [sender.py:send_request_defer():388] handle sender defer: 7
2022-03-24 10:43:00,200 INFO    SenderThread:94821 [file_pusher.py:finish():145] shutting down file pusher
2022-03-24 10:43:00,295 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-24 10:43:00,295 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: poll_exit
2022-03-24 10:43:00,397 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-24 10:43:00,397 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: poll_exit
2022-03-24 10:43:00,453 INFO    Thread-14 :94821 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/config.yaml
2022-03-24 10:43:00,465 INFO    Thread-12 :94821 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/requirements.txt
2022-03-24 10:43:00,471 INFO    Thread-13 :94821 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/output.log
2022-03-24 10:43:00,499 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-24 10:43:00,499 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: poll_exit
2022-03-24 10:43:00,562 INFO    Thread-15 :94821 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/files/wandb-summary.json
2022-03-24 10:43:00,600 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-24 10:43:00,601 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: poll_exit
2022-03-24 10:43:00,702 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-24 10:43:00,703 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: poll_exit
2022-03-24 10:43:00,762 INFO    Thread-6  :94821 [sender.py:transition_state():392] send defer: 8
2022-03-24 10:43:00,763 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: defer
2022-03-24 10:43:00,763 INFO    HandlerThread:94821 [handler.py:handle_request_defer():164] handle defer: 8
2022-03-24 10:43:00,763 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: defer
2022-03-24 10:43:00,763 INFO    SenderThread:94821 [sender.py:send_request_defer():388] handle sender defer: 8
2022-03-24 10:43:00,804 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-24 10:43:00,864 INFO    SenderThread:94821 [sender.py:transition_state():392] send defer: 9
2022-03-24 10:43:00,864 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: poll_exit
2022-03-24 10:43:00,865 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: defer
2022-03-24 10:43:00,865 INFO    HandlerThread:94821 [handler.py:handle_request_defer():164] handle defer: 9
2022-03-24 10:43:00,865 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: defer
2022-03-24 10:43:00,865 INFO    SenderThread:94821 [sender.py:send_request_defer():388] handle sender defer: 9
2022-03-24 10:43:00,865 INFO    SenderThread:94821 [sender.py:transition_state():392] send defer: 10
2022-03-24 10:43:00,866 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: defer
2022-03-24 10:43:00,866 INFO    HandlerThread:94821 [handler.py:handle_request_defer():164] handle defer: 10
2022-03-24 10:43:00,866 DEBUG   SenderThread:94821 [sender.py:send():235] send: final
2022-03-24 10:43:00,866 DEBUG   SenderThread:94821 [sender.py:send():235] send: footer
2022-03-24 10:43:00,866 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: defer
2022-03-24 10:43:00,866 INFO    SenderThread:94821 [sender.py:send_request_defer():388] handle sender defer: 10
2022-03-24 10:43:00,966 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-24 10:43:00,966 DEBUG   SenderThread:94821 [sender.py:send_request():249] send_request: poll_exit
2022-03-24 10:43:00,966 INFO    SenderThread:94821 [file_pusher.py:join():150] waiting for file pusher
2022-03-24 10:43:01,129 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: sampled_history
2022-03-24 10:43:01,130 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: get_summary
2022-03-24 10:43:01,131 DEBUG   HandlerThread:94821 [handler.py:handle_request():141] handle_request: shutdown
2022-03-24 10:43:01,131 INFO    HandlerThread:94821 [handler.py:finish():778] shutting down handler
2022-03-24 10:43:01,866 INFO    WriterThread:94821 [datastore.py:close():281] close: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220324_103821-m1uuspzx/run-m1uuspzx.wandb
2022-03-24 10:43:02,028 INFO    SenderThread:94821 [sender.py:finish():1078] shutting down sender
2022-03-24 10:43:02,028 INFO    SenderThread:94821 [file_pusher.py:finish():145] shutting down file pusher
2022-03-24 10:43:02,028 INFO    SenderThread:94821 [file_pusher.py:join():150] waiting for file pusher
2022-03-24 10:43:02,031 INFO    MainThread:94821 [internal.py:handle_exit():82] Internal process exited