File size: 26,850 Bytes
9915204
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
731244f
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
2021-07-14 22:58:21,337 INFO    MainThread:601574 [internal.py:wandb_internal():88] W&B internal server running at pid: 601574, started at: 2021-07-14 22:58:21.336704
2021-07-14 22:58:21,339 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: check_version
2021-07-14 22:58:21,339 INFO    WriterThread:601574 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/run-1dpoijkp.wandb
2021-07-14 22:58:21,340 DEBUG   SenderThread:601574 [sender.py:send():179] send: header
2021-07-14 22:58:21,340 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: check_version
2021-07-14 22:58:21,377 DEBUG   SenderThread:601574 [sender.py:send():179] send: run
2021-07-14 22:58:21,602 INFO    SenderThread:601574 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files
2021-07-14 22:58:21,602 INFO    SenderThread:601574 [sender.py:_start_run_threads():716] run started: 1dpoijkp with start time 1626303500
2021-07-14 22:58:21,602 DEBUG   SenderThread:601574 [sender.py:send():179] send: summary
2021-07-14 22:58:21,603 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: run_start
2021-07-14 22:58:21,603 INFO    SenderThread:601574 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 22:58:22,605 INFO    Thread-8  :601574 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/wandb-summary.json
2021-07-14 22:58:22,701 DEBUG   HandlerThread:601574 [meta.py:__init__():39] meta init
2021-07-14 22:58:22,701 DEBUG   HandlerThread:601574 [meta.py:__init__():53] meta init done
2021-07-14 22:58:22,701 DEBUG   HandlerThread:601574 [meta.py:probe():210] probe
2021-07-14 22:58:22,702 DEBUG   HandlerThread:601574 [meta.py:_setup_git():200] setup git
2021-07-14 22:58:22,732 DEBUG   HandlerThread:601574 [meta.py:_setup_git():207] setup git done
2021-07-14 22:58:22,732 DEBUG   HandlerThread:601574 [meta.py:_save_pip():57] save pip
2021-07-14 22:58:22,732 DEBUG   HandlerThread:601574 [meta.py:_save_pip():71] save pip done
2021-07-14 22:58:22,733 DEBUG   HandlerThread:601574 [meta.py:probe():252] probe done
2021-07-14 22:58:22,736 DEBUG   SenderThread:601574 [sender.py:send():179] send: files
2021-07-14 22:58:22,736 INFO    SenderThread:601574 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-14 22:58:22,742 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:58:22,742 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:58:22,871 DEBUG   SenderThread:601574 [sender.py:send():179] send: config
2021-07-14 22:58:22,872 DEBUG   SenderThread:601574 [sender.py:send():179] send: config
2021-07-14 22:58:22,872 DEBUG   SenderThread:601574 [sender.py:send():179] send: config
2021-07-14 22:58:23,214 INFO    Thread-11 :601574 [upload_job.py:push():137] Uploaded file /tmp/tmpg5fs3m8gwandb/35jlequ6-wandb-metadata.json
2021-07-14 22:58:23,603 INFO    Thread-8  :601574 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/output.log
2021-07-14 22:58:23,604 INFO    Thread-8  :601574 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/requirements.txt
2021-07-14 22:58:23,604 INFO    Thread-8  :601574 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/wandb-metadata.json
2021-07-14 22:58:37,609 INFO    Thread-8  :601574 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/output.log
2021-07-14 22:58:37,873 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:58:37,874 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:58:50,784 DEBUG   SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 22:58:52,614 INFO    Thread-8  :601574 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/config.yaml
2021-07-14 22:58:53,006 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:58:53,006 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:59:08,141 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:59:08,141 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:59:20,866 DEBUG   SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 22:59:23,276 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:59:23,277 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:59:38,415 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:59:38,415 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:59:50,943 DEBUG   SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 22:59:53,547 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:59:53,547 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:00:08,680 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:00:08,680 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:00:21,020 DEBUG   SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 23:00:23,810 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:00:23,811 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:00:38,944 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:00:38,945 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:00:51,098 DEBUG   SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 23:00:54,080 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:00:54,080 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:01:09,212 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:01:09,212 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:01:21,176 DEBUG   SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 23:01:24,345 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:01:24,346 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:01:39,477 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:01:39,478 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:01:51,254 DEBUG   SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 23:01:54,612 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:01:54,612 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:02:09,744 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:02:09,744 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:02:21,332 DEBUG   SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 23:02:24,932 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:02:24,932 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:02:40,066 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:02:40,067 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:02:51,409 DEBUG   SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 23:02:55,209 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:02:55,209 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:03:10,341 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:03:10,341 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:03:21,484 DEBUG   SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 23:03:25,483 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:03:25,483 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:03:40,615 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:03:40,615 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:03:51,547 DEBUG   SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 23:03:56,377 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:03:56,377 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:04:11,520 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:04:11,520 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:04:21,610 DEBUG   SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 23:04:26,652 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:04:26,652 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:04:41,781 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:04:41,781 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:04:51,676 DEBUG   SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 23:04:56,913 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:04:56,913 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:05:13,026 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:05:13,027 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:05:21,744 DEBUG   SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 23:05:28,237 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:05:28,238 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:05:44,049 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:05:44,049 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:05:51,809 DEBUG   SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 23:05:59,179 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:05:59,180 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:06:14,311 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:06:14,311 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:06:21,764 INFO    Thread-8  :601574 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/output.log
2021-07-14 23:06:21,877 DEBUG   SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 23:06:29,456 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:06:29,457 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:06:44,749 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:06:44,749 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:06:51,950 DEBUG   SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 23:06:59,941 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:06:59,941 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:07:15,073 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:07:15,077 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:07:22,025 DEBUG   SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 23:07:30,221 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:07:30,221 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:07:45,363 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:07:45,364 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:07:50,438 DEBUG   SenderThread:601574 [sender.py:send():179] send: telemetry
2021-07-14 23:07:50,439 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:07:50,439 DEBUG   SenderThread:601574 [sender.py:send():179] send: exit
2021-07-14 23:07:50,439 INFO    SenderThread:601574 [sender.py:send_exit():287] handling exit code: 1
2021-07-14 23:07:50,440 INFO    SenderThread:601574 [sender.py:send_exit():295] send defer
2021-07-14 23:07:50,440 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:07:50,441 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:07:50,441 INFO    HandlerThread:601574 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-14 23:07:50,441 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: defer
2021-07-14 23:07:50,441 INFO    SenderThread:601574 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-14 23:07:50,441 INFO    SenderThread:601574 [sender.py:transition_state():308] send defer: 1
2021-07-14 23:07:50,442 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:07:50,442 INFO    HandlerThread:601574 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-14 23:07:50,497 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: defer
2021-07-14 23:07:50,497 INFO    SenderThread:601574 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-14 23:07:50,497 INFO    SenderThread:601574 [sender.py:transition_state():308] send defer: 2
2021-07-14 23:07:50,497 DEBUG   SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 23:07:50,498 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:07:50,498 INFO    HandlerThread:601574 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-14 23:07:50,498 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: defer
2021-07-14 23:07:50,498 INFO    SenderThread:601574 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-14 23:07:50,498 INFO    SenderThread:601574 [sender.py:transition_state():308] send defer: 3
2021-07-14 23:07:50,499 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:07:50,499 INFO    HandlerThread:601574 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-14 23:07:50,499 DEBUG   SenderThread:601574 [sender.py:send():179] send: summary
2021-07-14 23:07:50,499 INFO    SenderThread:601574 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 23:07:50,499 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: defer
2021-07-14 23:07:50,499 INFO    SenderThread:601574 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-14 23:07:50,499 INFO    SenderThread:601574 [sender.py:transition_state():308] send defer: 4
2021-07-14 23:07:50,500 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:07:50,500 INFO    HandlerThread:601574 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-14 23:07:50,500 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: defer
2021-07-14 23:07:50,500 INFO    SenderThread:601574 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-14 23:07:50,543 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:07:50,746 INFO    SenderThread:601574 [sender.py:transition_state():308] send defer: 5
2021-07-14 23:07:50,746 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:07:50,747 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:07:50,747 INFO    HandlerThread:601574 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-14 23:07:50,747 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: defer
2021-07-14 23:07:50,747 INFO    SenderThread:601574 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-14 23:07:50,747 INFO    SenderThread:601574 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-14 23:07:50,787 INFO    SenderThread:601574 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/output.log
2021-07-14 23:07:50,787 INFO    SenderThread:601574 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/config.yaml
2021-07-14 23:07:50,787 INFO    SenderThread:601574 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/wandb-summary.json
2021-07-14 23:07:50,788 INFO    SenderThread:601574 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files
2021-07-14 23:07:50,788 INFO    SenderThread:601574 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/requirements.txt requirements.txt
2021-07-14 23:07:50,788 INFO    SenderThread:601574 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/output.log output.log
2021-07-14 23:07:50,788 INFO    SenderThread:601574 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/wandb-metadata.json wandb-metadata.json
2021-07-14 23:07:50,788 INFO    SenderThread:601574 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/config.yaml config.yaml
2021-07-14 23:07:50,789 INFO    SenderThread:601574 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/wandb-summary.json wandb-summary.json
2021-07-14 23:07:50,792 INFO    SenderThread:601574 [sender.py:transition_state():308] send defer: 6
2021-07-14 23:07:50,792 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:07:50,792 INFO    HandlerThread:601574 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-14 23:07:50,795 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: defer
2021-07-14 23:07:50,795 INFO    SenderThread:601574 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-14 23:07:50,795 INFO    SenderThread:601574 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 23:07:50,848 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:07:50,849 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:07:50,951 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:07:50,951 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:07:51,053 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:07:51,053 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:07:51,155 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:07:51,155 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:07:51,257 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:07:51,257 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:07:51,284 INFO    Thread-12 :601574 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/requirements.txt
2021-07-14 23:07:51,311 INFO    Thread-14 :601574 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/config.yaml
2021-07-14 23:07:51,322 INFO    Thread-13 :601574 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/output.log
2021-07-14 23:07:51,359 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:07:51,359 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:07:51,461 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:07:51,461 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:07:51,466 INFO    Thread-15 :601574 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/wandb-summary.json
2021-07-14 23:07:51,563 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:07:51,563 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:07:51,665 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:07:51,665 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:07:51,666 INFO    Thread-7  :601574 [sender.py:transition_state():308] send defer: 7
2021-07-14 23:07:51,667 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:07:51,667 INFO    HandlerThread:601574 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-14 23:07:51,667 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: defer
2021-07-14 23:07:51,667 INFO    SenderThread:601574 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-14 23:07:51,766 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:07:51,941 INFO    SenderThread:601574 [sender.py:transition_state():308] send defer: 8
2021-07-14 23:07:51,941 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:07:51,942 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:07:51,942 INFO    HandlerThread:601574 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-14 23:07:51,942 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: defer
2021-07-14 23:07:51,942 INFO    SenderThread:601574 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-14 23:07:51,942 INFO    SenderThread:601574 [sender.py:transition_state():308] send defer: 9
2021-07-14 23:07:51,942 DEBUG   SenderThread:601574 [sender.py:send():179] send: final
2021-07-14 23:07:51,942 DEBUG   SenderThread:601574 [sender.py:send():179] send: footer
2021-07-14 23:07:51,943 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:07:51,943 INFO    HandlerThread:601574 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-14 23:07:51,943 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: defer
2021-07-14 23:07:51,943 INFO    SenderThread:601574 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-14 23:07:52,043 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:07:52,043 DEBUG   SenderThread:601574 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:07:52,044 INFO    SenderThread:601574 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 23:07:52,045 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: get_summary
2021-07-14 23:07:52,045 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-14 23:07:52,046 DEBUG   HandlerThread:601574 [handler.py:handle_request():124] handle_request: shutdown
2021-07-14 23:07:52,046 INFO    HandlerThread:601574 [handler.py:finish():638] shutting down handler
2021-07-14 23:07:52,943 INFO    WriterThread:601574 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/run-1dpoijkp.wandb
2021-07-14 23:07:53,044 INFO    SenderThread:601574 [sender.py:finish():945] shutting down sender
2021-07-14 23:07:53,044 INFO    SenderThread:601574 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 23:07:53,044 INFO    SenderThread:601574 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 23:07:53,047 INFO    MainThread:601574 [internal.py:handle_exit():78] Internal process exited