File size: 27,099 Bytes
f291f93
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
2021-07-14 17:55:20,938 INFO    MainThread:545287 [internal.py:wandb_internal():88] W&B internal server running at pid: 545287, started at: 2021-07-14 17:55:20.938091
2021-07-14 17:55:20,940 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: check_version
2021-07-14 17:55:20,941 INFO    WriterThread:545287 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/run-t843i2it.wandb
2021-07-14 17:55:20,942 DEBUG   SenderThread:545287 [sender.py:send():179] send: header
2021-07-14 17:55:20,942 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: check_version
2021-07-14 17:55:20,985 DEBUG   SenderThread:545287 [sender.py:send():179] send: run
2021-07-14 17:55:21,157 INFO    SenderThread:545287 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files
2021-07-14 17:55:21,158 INFO    SenderThread:545287 [sender.py:_start_run_threads():716] run started: t843i2it with start time 1626285318
2021-07-14 17:55:21,158 DEBUG   SenderThread:545287 [sender.py:send():179] send: summary
2021-07-14 17:55:21,158 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: run_start
2021-07-14 17:55:21,159 INFO    SenderThread:545287 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 17:55:24,792 INFO    Thread-8  :545287 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/wandb-summary.json
2021-07-14 17:55:26,883 INFO    Thread-8  :545287 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/output.log
2021-07-14 17:55:32,372 DEBUG   HandlerThread:545287 [meta.py:__init__():39] meta init
2021-07-14 17:55:32,372 DEBUG   HandlerThread:545287 [meta.py:__init__():53] meta init done
2021-07-14 17:55:32,372 DEBUG   HandlerThread:545287 [meta.py:probe():210] probe
2021-07-14 17:55:32,373 DEBUG   HandlerThread:545287 [meta.py:_setup_git():200] setup git
2021-07-14 17:55:32,402 DEBUG   HandlerThread:545287 [meta.py:_setup_git():207] setup git done
2021-07-14 17:55:32,402 DEBUG   HandlerThread:545287 [meta.py:_save_pip():57] save pip
2021-07-14 17:55:32,403 DEBUG   HandlerThread:545287 [meta.py:_save_pip():71] save pip done
2021-07-14 17:55:32,403 DEBUG   HandlerThread:545287 [meta.py:probe():252] probe done
2021-07-14 17:55:32,406 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 17:55:32,406 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 17:55:32,544 DEBUG   SenderThread:545287 [sender.py:send():179] send: config
2021-07-14 17:55:32,545 DEBUG   SenderThread:545287 [sender.py:send():179] send: config
2021-07-14 17:55:32,545 DEBUG   SenderThread:545287 [sender.py:send():179] send: config
2021-07-14 17:55:32,545 DEBUG   SenderThread:545287 [sender.py:send():179] send: files
2021-07-14 17:55:32,545 INFO    SenderThread:545287 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-14 17:55:32,952 INFO    Thread-8  :545287 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/wandb-metadata.json
2021-07-14 17:55:32,953 INFO    Thread-8  :545287 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/requirements.txt
2021-07-14 17:55:33,181 INFO    Thread-11 :545287 [upload_job.py:push():137] Uploaded file /tmp/tmpldoi9p0fwandb/1bn7qo44-wandb-metadata.json
2021-07-14 17:55:42,956 INFO    Thread-8  :545287 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/output.log
2021-07-14 17:55:44,957 INFO    Thread-8  :545287 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/output.log
2021-07-14 17:55:47,548 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 17:55:47,549 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 17:55:51,960 INFO    Thread-8  :545287 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/config.yaml
2021-07-14 17:56:00,458 DEBUG   SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 17:56:02,681 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 17:56:02,681 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 17:56:17,816 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 17:56:17,816 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 17:56:30,528 DEBUG   SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 17:56:32,958 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 17:56:32,959 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 17:56:48,094 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 17:56:48,094 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 17:57:00,620 DEBUG   SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 17:57:03,228 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 17:57:03,229 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 17:57:18,364 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 17:57:18,364 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 17:57:30,693 DEBUG   SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 17:57:33,502 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 17:57:33,502 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 17:57:48,640 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 17:57:48,640 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 17:58:00,768 DEBUG   SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 17:58:03,776 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 17:58:03,776 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 17:58:18,928 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 17:58:18,928 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 17:58:30,841 DEBUG   SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 17:58:34,063 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 17:58:34,063 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 17:58:49,196 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 17:58:49,196 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 17:59:00,919 DEBUG   SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 17:59:04,329 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 17:59:04,330 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 17:59:19,466 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 17:59:19,467 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 17:59:30,993 DEBUG   SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 17:59:34,601 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 17:59:34,601 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 17:59:49,736 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 17:59:49,736 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:00:01,070 DEBUG   SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 18:00:04,866 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:00:04,867 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:00:20,000 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:00:20,001 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:00:31,138 DEBUG   SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 18:00:35,144 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:00:35,144 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:00:50,279 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:00:50,280 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:01:01,215 DEBUG   SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 18:01:05,414 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:01:05,414 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:01:20,564 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:01:20,565 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:01:31,294 DEBUG   SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 18:01:35,699 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:01:35,699 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:01:50,829 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:01:50,830 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:02:01,371 DEBUG   SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 18:02:05,966 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:02:05,967 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:02:21,099 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:02:21,099 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:02:31,448 DEBUG   SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 18:02:36,234 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:02:36,235 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:02:51,370 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:02:51,370 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:03:01,521 DEBUG   SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 18:03:06,821 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:03:06,821 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:03:22,036 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:03:22,036 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:03:31,600 DEBUG   SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 18:03:33,147 INFO    Thread-8  :545287 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/output.log
2021-07-14 18:03:37,189 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:03:37,190 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:03:52,336 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:03:52,337 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:04:01,678 DEBUG   SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 18:04:07,470 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:04:07,470 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:04:22,603 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:04:22,604 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:04:31,760 DEBUG   SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 18:04:37,735 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:04:37,735 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:04:52,873 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:04:52,873 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:05:01,843 DEBUG   SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 18:05:08,009 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:05:08,010 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:05:21,189 INFO    Thread-8  :545287 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/output.log
2021-07-14 18:05:22,751 DEBUG   SenderThread:545287 [sender.py:send():179] send: telemetry
2021-07-14 18:05:22,752 DEBUG   SenderThread:545287 [sender.py:send():179] send: exit
2021-07-14 18:05:22,752 INFO    SenderThread:545287 [sender.py:send_exit():287] handling exit code: 1
2021-07-14 18:05:22,753 INFO    SenderThread:545287 [sender.py:send_exit():295] send defer
2021-07-14 18:05:22,753 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:05:22,754 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:05:22,754 INFO    HandlerThread:545287 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-14 18:05:22,754 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:05:22,754 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: defer
2021-07-14 18:05:22,754 INFO    SenderThread:545287 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-14 18:05:22,754 INFO    SenderThread:545287 [sender.py:transition_state():308] send defer: 1
2021-07-14 18:05:22,755 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:05:22,755 INFO    HandlerThread:545287 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-14 18:05:22,795 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: defer
2021-07-14 18:05:22,795 INFO    SenderThread:545287 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-14 18:05:22,795 INFO    SenderThread:545287 [sender.py:transition_state():308] send defer: 2
2021-07-14 18:05:22,796 DEBUG   SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 18:05:22,796 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:05:22,796 INFO    HandlerThread:545287 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-14 18:05:22,796 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: defer
2021-07-14 18:05:22,797 INFO    SenderThread:545287 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-14 18:05:22,797 INFO    SenderThread:545287 [sender.py:transition_state():308] send defer: 3
2021-07-14 18:05:22,797 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:05:22,797 INFO    HandlerThread:545287 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-14 18:05:22,797 DEBUG   SenderThread:545287 [sender.py:send():179] send: summary
2021-07-14 18:05:22,797 INFO    SenderThread:545287 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 18:05:22,798 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: defer
2021-07-14 18:05:22,798 INFO    SenderThread:545287 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-14 18:05:22,798 INFO    SenderThread:545287 [sender.py:transition_state():308] send defer: 4
2021-07-14 18:05:22,798 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:05:22,798 INFO    HandlerThread:545287 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-14 18:05:22,798 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: defer
2021-07-14 18:05:22,798 INFO    SenderThread:545287 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-14 18:05:22,856 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:05:22,985 INFO    SenderThread:545287 [sender.py:transition_state():308] send defer: 5
2021-07-14 18:05:22,985 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:05:22,986 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:05:22,986 INFO    HandlerThread:545287 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-14 18:05:22,986 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: defer
2021-07-14 18:05:22,986 INFO    SenderThread:545287 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-14 18:05:22,986 INFO    SenderThread:545287 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-14 18:05:23,087 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:05:23,190 INFO    Thread-8  :545287 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/output.log
2021-07-14 18:05:23,190 INFO    SenderThread:545287 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/wandb-summary.json
2021-07-14 18:05:23,190 INFO    SenderThread:545287 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/config.yaml
2021-07-14 18:05:23,191 INFO    SenderThread:545287 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files
2021-07-14 18:05:23,191 INFO    SenderThread:545287 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/requirements.txt requirements.txt
2021-07-14 18:05:23,191 INFO    SenderThread:545287 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/output.log output.log
2021-07-14 18:05:23,195 INFO    SenderThread:545287 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/wandb-metadata.json wandb-metadata.json
2021-07-14 18:05:23,195 INFO    SenderThread:545287 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/config.yaml config.yaml
2021-07-14 18:05:23,195 INFO    SenderThread:545287 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/wandb-summary.json wandb-summary.json
2021-07-14 18:05:23,195 INFO    SenderThread:545287 [sender.py:transition_state():308] send defer: 6
2021-07-14 18:05:23,195 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:05:23,196 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:05:23,196 INFO    HandlerThread:545287 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-14 18:05:23,197 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: defer
2021-07-14 18:05:23,197 INFO    SenderThread:545287 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-14 18:05:23,197 INFO    SenderThread:545287 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 18:05:23,299 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:05:23,299 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:05:23,401 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:05:23,401 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:05:23,503 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:05:23,503 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:05:23,605 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:05:23,605 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:05:23,648 INFO    Thread-12 :545287 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/requirements.txt
2021-07-14 18:05:23,655 INFO    Thread-13 :545287 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/output.log
2021-07-14 18:05:23,658 INFO    Thread-15 :545287 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/wandb-summary.json
2021-07-14 18:05:23,661 INFO    Thread-14 :545287 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/config.yaml
2021-07-14 18:05:23,708 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:05:23,708 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:05:23,810 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:05:23,810 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:05:23,861 INFO    Thread-7  :545287 [sender.py:transition_state():308] send defer: 7
2021-07-14 18:05:23,862 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:05:23,862 INFO    HandlerThread:545287 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-14 18:05:23,862 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: defer
2021-07-14 18:05:23,862 INFO    SenderThread:545287 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-14 18:05:23,912 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:05:24,152 INFO    SenderThread:545287 [sender.py:transition_state():308] send defer: 8
2021-07-14 18:05:24,152 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:05:24,153 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:05:24,153 INFO    HandlerThread:545287 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-14 18:05:24,154 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: defer
2021-07-14 18:05:24,154 INFO    SenderThread:545287 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-14 18:05:24,154 INFO    SenderThread:545287 [sender.py:transition_state():308] send defer: 9
2021-07-14 18:05:24,154 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:05:24,155 INFO    HandlerThread:545287 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-14 18:05:24,155 DEBUG   SenderThread:545287 [sender.py:send():179] send: final
2021-07-14 18:05:24,155 DEBUG   SenderThread:545287 [sender.py:send():179] send: footer
2021-07-14 18:05:24,155 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: defer
2021-07-14 18:05:24,155 INFO    SenderThread:545287 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-14 18:05:24,255 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:05:24,255 DEBUG   SenderThread:545287 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:05:24,255 INFO    SenderThread:545287 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 18:05:24,257 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: get_summary
2021-07-14 18:05:24,257 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-14 18:05:24,258 DEBUG   HandlerThread:545287 [handler.py:handle_request():124] handle_request: shutdown
2021-07-14 18:05:24,258 INFO    HandlerThread:545287 [handler.py:finish():638] shutting down handler
2021-07-14 18:05:25,155 INFO    WriterThread:545287 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/run-t843i2it.wandb
2021-07-14 18:05:25,256 INFO    SenderThread:545287 [sender.py:finish():945] shutting down sender
2021-07-14 18:05:25,256 INFO    SenderThread:545287 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 18:05:25,256 INFO    SenderThread:545287 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 18:05:25,275 INFO    MainThread:545287 [internal.py:handle_exit():78] Internal process exited