File size: 23,295 Bytes
4f730f1
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
2022-03-23 17:15:39,798 INFO    MainThread:16634 [internal.py:wandb_internal():92] W&B internal server running at pid: 16634, started at: 2022-03-23 17:15:39.798196
2022-03-23 17:15:39,829 INFO    WriterThread:16634 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/run-5b3vz6lx.wandb
2022-03-23 17:15:39,830 DEBUG   SenderThread:16634 [sender.py:send():235] send: header
2022-03-23 17:15:39,830 DEBUG   SenderThread:16634 [sender.py:send():235] send: run
2022-03-23 17:15:40,142 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: check_version
2022-03-23 17:15:40,142 INFO    SenderThread:16634 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files
2022-03-23 17:15:40,143 INFO    SenderThread:16634 [sender.py:_start_run_threads():812] run started: 5b3vz6lx with start time 1648055739
2022-03-23 17:15:40,143 DEBUG   SenderThread:16634 [sender.py:send():235] send: summary
2022-03-23 17:15:40,143 INFO    SenderThread:16634 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-03-23 17:15:40,143 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: check_version
2022-03-23 17:15:40,202 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: run_start
2022-03-23 17:15:41,144 INFO    Thread-7  :16634 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/wandb-summary.json
2022-03-23 17:15:42,639 DEBUG   HandlerThread:16634 [meta.py:__init__():37] meta init
2022-03-23 17:15:42,639 DEBUG   HandlerThread:16634 [meta.py:__init__():51] meta init done
2022-03-23 17:15:42,639 DEBUG   HandlerThread:16634 [meta.py:probe():211] probe
2022-03-23 17:15:42,641 DEBUG   HandlerThread:16634 [meta.py:_setup_git():201] setup git
2022-03-23 17:15:42,676 DEBUG   HandlerThread:16634 [meta.py:_setup_git():208] setup git done
2022-03-23 17:15:42,676 DEBUG   HandlerThread:16634 [meta.py:_save_pip():55] save pip
2022-03-23 17:15:42,677 DEBUG   HandlerThread:16634 [meta.py:_save_pip():69] save pip done
2022-03-23 17:15:42,677 DEBUG   HandlerThread:16634 [meta.py:probe():249] probe done
2022-03-23 17:15:42,680 DEBUG   SenderThread:16634 [sender.py:send():235] send: files
2022-03-23 17:15:42,681 INFO    SenderThread:16634 [sender.py:_save_file():947] saving file wandb-metadata.json with policy now
2022-03-23 17:15:42,686 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status
2022-03-23 17:15:42,687 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: stop_status
2022-03-23 17:15:42,740 DEBUG   SenderThread:16634 [sender.py:send():235] send: telemetry
2022-03-23 17:15:43,147 INFO    Thread-7  :16634 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/requirements.txt
2022-03-23 17:15:43,147 INFO    Thread-7  :16634 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/output.log
2022-03-23 17:15:43,147 INFO    Thread-7  :16634 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/wandb-metadata.json
2022-03-23 17:15:43,573 INFO    Thread-11 :16634 [upload_job.py:push():137] Uploaded file /tmp/tmpcgnu3sa6wandb/1lmj2vet-wandb-metadata.json
2022-03-23 17:15:45,148 INFO    Thread-7  :16634 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/output.log
2022-03-23 17:15:57,742 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status
2022-03-23 17:15:57,742 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: stop_status
2022-03-23 17:16:01,154 INFO    Thread-7  :16634 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/output.log
2022-03-23 17:16:05,156 INFO    Thread-7  :16634 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/output.log
2022-03-23 17:16:09,158 INFO    Thread-7  :16634 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/output.log
2022-03-23 17:16:10,158 INFO    Thread-7  :16634 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/config.yaml
2022-03-23 17:16:10,726 DEBUG   SenderThread:16634 [sender.py:send():235] send: stats
2022-03-23 17:16:12,777 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status
2022-03-23 17:16:12,777 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: stop_status
2022-03-23 17:16:13,159 INFO    Thread-7  :16634 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/output.log
2022-03-23 17:16:17,161 INFO    Thread-7  :16634 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/output.log
2022-03-23 17:16:27,852 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status
2022-03-23 17:16:27,852 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: stop_status
2022-03-23 17:16:40,804 DEBUG   SenderThread:16634 [sender.py:send():235] send: stats
2022-03-23 17:16:42,916 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status
2022-03-23 17:16:42,916 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: stop_status
2022-03-23 17:16:57,968 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status
2022-03-23 17:16:57,968 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: stop_status
2022-03-23 17:17:10,883 DEBUG   SenderThread:16634 [sender.py:send():235] send: stats
2022-03-23 17:17:13,013 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status
2022-03-23 17:17:13,013 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: stop_status
2022-03-23 17:17:28,060 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status
2022-03-23 17:17:28,060 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: stop_status
2022-03-23 17:17:40,963 DEBUG   SenderThread:16634 [sender.py:send():235] send: stats
2022-03-23 17:17:43,134 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status
2022-03-23 17:17:43,134 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: stop_status
2022-03-23 17:17:58,179 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status
2022-03-23 17:17:58,179 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: stop_status
2022-03-23 17:18:11,043 DEBUG   SenderThread:16634 [sender.py:send():235] send: stats
2022-03-23 17:18:13,217 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status
2022-03-23 17:18:13,217 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: stop_status
2022-03-23 17:18:28,336 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status
2022-03-23 17:18:28,337 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: stop_status
2022-03-23 17:18:41,123 DEBUG   SenderThread:16634 [sender.py:send():235] send: stats
2022-03-23 17:18:43,408 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status
2022-03-23 17:18:43,408 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: stop_status
2022-03-23 17:18:58,446 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status
2022-03-23 17:18:58,447 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: stop_status
2022-03-23 17:19:11,201 DEBUG   SenderThread:16634 [sender.py:send():235] send: stats
2022-03-23 17:19:13,488 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status
2022-03-23 17:19:13,489 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: stop_status
2022-03-23 17:19:28,533 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: stop_status
2022-03-23 17:19:28,533 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: stop_status
2022-03-23 17:19:37,253 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: partial_history
2022-03-23 17:19:37,254 DEBUG   SenderThread:16634 [sender.py:send():235] send: telemetry
2022-03-23 17:19:38,243 INFO    Thread-7  :16634 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/output.log
2022-03-23 17:19:39,594 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-23 17:19:39,594 DEBUG   SenderThread:16634 [sender.py:send():235] send: exit
2022-03-23 17:19:39,595 INFO    SenderThread:16634 [sender.py:send_exit():371] handling exit code: 1
2022-03-23 17:19:39,596 INFO    SenderThread:16634 [sender.py:send_exit():373] handling runtime: 239
2022-03-23 17:19:39,596 INFO    SenderThread:16634 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-03-23 17:19:39,597 INFO    SenderThread:16634 [sender.py:send_exit():379] send defer
2022-03-23 17:19:39,597 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: poll_exit
2022-03-23 17:19:39,597 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: defer
2022-03-23 17:19:39,598 INFO    HandlerThread:16634 [handler.py:handle_request_defer():164] handle defer: 0
2022-03-23 17:19:39,598 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: defer
2022-03-23 17:19:39,598 INFO    SenderThread:16634 [sender.py:send_request_defer():388] handle sender defer: 0
2022-03-23 17:19:39,598 INFO    SenderThread:16634 [sender.py:transition_state():392] send defer: 1
2022-03-23 17:19:39,598 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: defer
2022-03-23 17:19:39,598 INFO    HandlerThread:16634 [handler.py:handle_request_defer():164] handle defer: 1
2022-03-23 17:19:39,673 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: defer
2022-03-23 17:19:39,673 INFO    SenderThread:16634 [sender.py:send_request_defer():388] handle sender defer: 1
2022-03-23 17:19:39,673 INFO    SenderThread:16634 [sender.py:transition_state():392] send defer: 2
2022-03-23 17:19:39,673 DEBUG   SenderThread:16634 [sender.py:send():235] send: stats
2022-03-23 17:19:39,674 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: defer
2022-03-23 17:19:39,674 INFO    HandlerThread:16634 [handler.py:handle_request_defer():164] handle defer: 2
2022-03-23 17:19:39,674 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: defer
2022-03-23 17:19:39,674 INFO    SenderThread:16634 [sender.py:send_request_defer():388] handle sender defer: 2
2022-03-23 17:19:39,674 INFO    SenderThread:16634 [sender.py:transition_state():392] send defer: 3
2022-03-23 17:19:39,674 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: defer
2022-03-23 17:19:39,674 INFO    HandlerThread:16634 [handler.py:handle_request_defer():164] handle defer: 3
2022-03-23 17:19:39,674 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: defer
2022-03-23 17:19:39,675 INFO    SenderThread:16634 [sender.py:send_request_defer():388] handle sender defer: 3
2022-03-23 17:19:39,675 INFO    SenderThread:16634 [sender.py:transition_state():392] send defer: 4
2022-03-23 17:19:39,675 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: defer
2022-03-23 17:19:39,675 INFO    HandlerThread:16634 [handler.py:handle_request_defer():164] handle defer: 4
2022-03-23 17:19:39,675 DEBUG   SenderThread:16634 [sender.py:send():235] send: summary
2022-03-23 17:19:39,675 INFO    SenderThread:16634 [sender.py:_save_file():947] saving file wandb-summary.json with policy end
2022-03-23 17:19:39,676 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: defer
2022-03-23 17:19:39,676 INFO    SenderThread:16634 [sender.py:send_request_defer():388] handle sender defer: 4
2022-03-23 17:19:39,676 INFO    SenderThread:16634 [sender.py:transition_state():392] send defer: 5
2022-03-23 17:19:39,676 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: defer
2022-03-23 17:19:39,676 INFO    HandlerThread:16634 [handler.py:handle_request_defer():164] handle defer: 5
2022-03-23 17:19:39,676 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: defer
2022-03-23 17:19:39,676 INFO    SenderThread:16634 [sender.py:send_request_defer():388] handle sender defer: 5
2022-03-23 17:19:39,699 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-23 17:19:39,863 INFO    SenderThread:16634 [sender.py:transition_state():392] send defer: 6
2022-03-23 17:19:39,863 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: poll_exit
2022-03-23 17:19:39,864 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: defer
2022-03-23 17:19:39,864 INFO    HandlerThread:16634 [handler.py:handle_request_defer():164] handle defer: 6
2022-03-23 17:19:39,864 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: defer
2022-03-23 17:19:39,864 INFO    SenderThread:16634 [sender.py:send_request_defer():388] handle sender defer: 6
2022-03-23 17:19:39,864 INFO    SenderThread:16634 [dir_watcher.py:finish():283] shutting down directory watcher
2022-03-23 17:19:39,965 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-23 17:19:40,244 INFO    SenderThread:16634 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/output.log
2022-03-23 17:19:40,244 INFO    SenderThread:16634 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/config.yaml
2022-03-23 17:19:40,244 INFO    SenderThread:16634 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/wandb-summary.json
2022-03-23 17:19:40,244 INFO    SenderThread:16634 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files
2022-03-23 17:19:40,245 INFO    SenderThread:16634 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/requirements.txt requirements.txt
2022-03-23 17:19:40,245 INFO    SenderThread:16634 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/output.log output.log
2022-03-23 17:19:40,245 INFO    SenderThread:16634 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/wandb-metadata.json wandb-metadata.json
2022-03-23 17:19:40,245 INFO    SenderThread:16634 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/config.yaml config.yaml
2022-03-23 17:19:40,245 INFO    SenderThread:16634 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/wandb-summary.json wandb-summary.json
2022-03-23 17:19:40,245 INFO    SenderThread:16634 [sender.py:transition_state():392] send defer: 7
2022-03-23 17:19:40,246 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: poll_exit
2022-03-23 17:19:40,247 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: defer
2022-03-23 17:19:40,247 INFO    HandlerThread:16634 [handler.py:handle_request_defer():164] handle defer: 7
2022-03-23 17:19:40,250 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: defer
2022-03-23 17:19:40,250 INFO    SenderThread:16634 [sender.py:send_request_defer():388] handle sender defer: 7
2022-03-23 17:19:40,250 INFO    SenderThread:16634 [file_pusher.py:finish():145] shutting down file pusher
2022-03-23 17:19:40,348 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-23 17:19:40,349 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: poll_exit
2022-03-23 17:19:40,450 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-23 17:19:40,450 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: poll_exit
2022-03-23 17:19:40,518 INFO    Thread-14 :16634 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/config.yaml
2022-03-23 17:19:40,552 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-23 17:19:40,552 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: poll_exit
2022-03-23 17:19:40,561 INFO    Thread-12 :16634 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/requirements.txt
2022-03-23 17:19:40,570 INFO    Thread-13 :16634 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/output.log
2022-03-23 17:19:40,589 INFO    Thread-15 :16634 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/files/wandb-summary.json
2022-03-23 17:19:40,654 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-23 17:19:40,654 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: poll_exit
2022-03-23 17:19:40,755 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-23 17:19:40,756 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: poll_exit
2022-03-23 17:19:40,789 INFO    Thread-6  :16634 [sender.py:transition_state():392] send defer: 8
2022-03-23 17:19:40,790 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: defer
2022-03-23 17:19:40,790 INFO    HandlerThread:16634 [handler.py:handle_request_defer():164] handle defer: 8
2022-03-23 17:19:40,790 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: defer
2022-03-23 17:19:40,790 INFO    SenderThread:16634 [sender.py:send_request_defer():388] handle sender defer: 8
2022-03-23 17:19:40,857 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-23 17:19:40,915 INFO    SenderThread:16634 [sender.py:transition_state():392] send defer: 9
2022-03-23 17:19:40,915 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: poll_exit
2022-03-23 17:19:40,915 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: defer
2022-03-23 17:19:40,916 INFO    HandlerThread:16634 [handler.py:handle_request_defer():164] handle defer: 9
2022-03-23 17:19:40,916 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: defer
2022-03-23 17:19:40,916 INFO    SenderThread:16634 [sender.py:send_request_defer():388] handle sender defer: 9
2022-03-23 17:19:40,916 INFO    SenderThread:16634 [sender.py:transition_state():392] send defer: 10
2022-03-23 17:19:40,916 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: defer
2022-03-23 17:19:40,916 INFO    HandlerThread:16634 [handler.py:handle_request_defer():164] handle defer: 10
2022-03-23 17:19:40,917 DEBUG   SenderThread:16634 [sender.py:send():235] send: final
2022-03-23 17:19:40,917 DEBUG   SenderThread:16634 [sender.py:send():235] send: footer
2022-03-23 17:19:40,917 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: defer
2022-03-23 17:19:40,917 INFO    SenderThread:16634 [sender.py:send_request_defer():388] handle sender defer: 10
2022-03-23 17:19:41,017 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: poll_exit
2022-03-23 17:19:41,017 DEBUG   SenderThread:16634 [sender.py:send_request():249] send_request: poll_exit
2022-03-23 17:19:41,017 INFO    SenderThread:16634 [file_pusher.py:join():150] waiting for file pusher
2022-03-23 17:19:41,172 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: sampled_history
2022-03-23 17:19:41,173 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: get_summary
2022-03-23 17:19:41,174 DEBUG   HandlerThread:16634 [handler.py:handle_request():141] handle_request: shutdown
2022-03-23 17:19:41,174 INFO    HandlerThread:16634 [handler.py:finish():778] shutting down handler
2022-03-23 17:19:41,917 INFO    WriterThread:16634 [datastore.py:close():281] close: /home/sanchit_huggingface_co/flax-wav2vec2-2-bart-large-cnn-gradient-accumulation/wandb/run-20220323_171538-5b3vz6lx/run-5b3vz6lx.wandb
2022-03-23 17:19:42,071 INFO    SenderThread:16634 [sender.py:finish():1078] shutting down sender
2022-03-23 17:19:42,071 INFO    SenderThread:16634 [file_pusher.py:finish():145] shutting down file pusher
2022-03-23 17:19:42,071 INFO    SenderThread:16634 [file_pusher.py:join():150] waiting for file pusher
2022-03-23 17:19:42,110 INFO    MainThread:16634 [internal.py:handle_exit():82] Internal process exited