File size: 27,129 Bytes
f291f93
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
2021-07-14 18:19:55,251 INFO    MainThread:549018 [internal.py:wandb_internal():88] W&B internal server running at pid: 549018, started at: 2021-07-14 18:19:55.251471
2021-07-14 18:19:55,253 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: check_version
2021-07-14 18:19:55,254 INFO    WriterThread:549018 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/run-351hrfn2.wandb
2021-07-14 18:19:55,255 DEBUG   SenderThread:549018 [sender.py:send():179] send: header
2021-07-14 18:19:55,255 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: check_version
2021-07-14 18:19:55,291 DEBUG   SenderThread:549018 [sender.py:send():179] send: run
2021-07-14 18:19:55,471 INFO    SenderThread:549018 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files
2021-07-14 18:19:55,471 INFO    SenderThread:549018 [sender.py:_start_run_threads():716] run started: 351hrfn2 with start time 1626286794
2021-07-14 18:19:55,471 DEBUG   SenderThread:549018 [sender.py:send():179] send: summary
2021-07-14 18:19:55,471 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: run_start
2021-07-14 18:19:55,472 INFO    SenderThread:549018 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 18:19:56,473 INFO    Thread-8  :549018 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/wandb-summary.json
2021-07-14 18:19:56,606 DEBUG   HandlerThread:549018 [meta.py:__init__():39] meta init
2021-07-14 18:19:56,606 DEBUG   HandlerThread:549018 [meta.py:__init__():53] meta init done
2021-07-14 18:19:56,606 DEBUG   HandlerThread:549018 [meta.py:probe():210] probe
2021-07-14 18:19:56,607 DEBUG   HandlerThread:549018 [meta.py:_setup_git():200] setup git
2021-07-14 18:19:56,636 DEBUG   HandlerThread:549018 [meta.py:_setup_git():207] setup git done
2021-07-14 18:19:56,636 DEBUG   HandlerThread:549018 [meta.py:_save_pip():57] save pip
2021-07-14 18:19:56,636 DEBUG   HandlerThread:549018 [meta.py:_save_pip():71] save pip done
2021-07-14 18:19:56,636 DEBUG   HandlerThread:549018 [meta.py:probe():252] probe done
2021-07-14 18:19:56,640 DEBUG   SenderThread:549018 [sender.py:send():179] send: files
2021-07-14 18:19:56,640 INFO    SenderThread:549018 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-14 18:19:56,648 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:19:56,648 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:19:56,776 DEBUG   SenderThread:549018 [sender.py:send():179] send: config
2021-07-14 18:19:56,777 DEBUG   SenderThread:549018 [sender.py:send():179] send: config
2021-07-14 18:19:56,777 DEBUG   SenderThread:549018 [sender.py:send():179] send: config
2021-07-14 18:19:57,091 INFO    Thread-11 :549018 [upload_job.py:push():137] Uploaded file /tmp/tmpup88esztwandb/nma268qr-wandb-metadata.json
2021-07-14 18:19:57,473 INFO    Thread-8  :549018 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/wandb-metadata.json
2021-07-14 18:19:57,473 INFO    Thread-8  :549018 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/output.log
2021-07-14 18:19:57,473 INFO    Thread-8  :549018 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/requirements.txt
2021-07-14 18:20:11,779 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:20:11,779 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:20:15,481 INFO    Thread-8  :549018 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/output.log
2021-07-14 18:20:24,687 DEBUG   SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:20:26,485 INFO    Thread-8  :549018 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/config.yaml
2021-07-14 18:20:26,913 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:20:26,913 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:20:42,047 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:20:42,048 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:20:54,758 DEBUG   SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:20:57,181 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:20:57,182 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:21:12,314 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:21:12,315 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:21:24,825 DEBUG   SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:21:27,445 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:21:27,446 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:21:42,580 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:21:42,580 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:21:54,891 DEBUG   SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:21:57,716 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:21:57,716 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:22:12,849 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:22:12,849 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:22:24,963 DEBUG   SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:22:27,983 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:22:27,984 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:22:43,121 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:22:43,121 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:22:55,039 DEBUG   SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:22:58,254 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:22:58,254 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:23:13,385 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:23:13,385 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:23:25,114 DEBUG   SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:23:28,517 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:23:28,517 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:23:43,648 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:23:43,649 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:23:55,185 DEBUG   SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:23:58,785 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:23:58,785 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:24:13,918 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:24:13,918 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:24:25,261 DEBUG   SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:24:29,053 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:24:29,053 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:24:44,185 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:24:44,185 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:24:55,335 DEBUG   SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:24:59,318 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:24:59,318 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:25:14,450 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:25:14,450 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:25:25,409 DEBUG   SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:25:29,584 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:25:29,584 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:25:44,713 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:25:44,713 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:25:55,479 DEBUG   SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:25:59,845 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:25:59,845 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:26:14,976 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:26:14,977 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:26:25,544 DEBUG   SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:26:30,114 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:26:30,114 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:26:45,247 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:26:45,247 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:26:55,606 DEBUG   SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:27:00,379 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:27:00,380 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:27:15,542 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:27:15,542 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:27:25,670 DEBUG   SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:27:30,683 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:27:30,684 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:27:45,816 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:27:45,817 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:27:55,730 DEBUG   SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:28:00,949 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:28:00,949 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:28:05,678 INFO    Thread-8  :549018 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/output.log
2021-07-14 18:28:16,097 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:28:16,097 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:28:25,799 DEBUG   SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:28:31,288 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:28:31,288 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:28:46,425 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:28:46,425 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:28:55,882 DEBUG   SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:29:01,559 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:29:01,559 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:29:16,690 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:29:16,690 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:29:25,961 DEBUG   SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:29:31,823 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:29:31,823 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:29:43,722 INFO    Thread-8  :549018 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/output.log
2021-07-14 18:29:44,729 DEBUG   SenderThread:549018 [sender.py:send():179] send: telemetry
2021-07-14 18:29:44,730 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:29:44,730 DEBUG   SenderThread:549018 [sender.py:send():179] send: exit
2021-07-14 18:29:44,730 INFO    SenderThread:549018 [sender.py:send_exit():287] handling exit code: 1
2021-07-14 18:29:44,731 INFO    SenderThread:549018 [sender.py:send_exit():295] send defer
2021-07-14 18:29:44,731 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:29:44,731 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:29:44,731 INFO    HandlerThread:549018 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-14 18:29:44,732 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: defer
2021-07-14 18:29:44,732 INFO    SenderThread:549018 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-14 18:29:44,732 INFO    SenderThread:549018 [sender.py:transition_state():308] send defer: 1
2021-07-14 18:29:44,732 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:29:44,732 INFO    HandlerThread:549018 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-14 18:29:44,807 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: defer
2021-07-14 18:29:44,807 INFO    SenderThread:549018 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-14 18:29:44,807 INFO    SenderThread:549018 [sender.py:transition_state():308] send defer: 2
2021-07-14 18:29:44,807 DEBUG   SenderThread:549018 [sender.py:send():179] send: stats
2021-07-14 18:29:44,807 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:29:44,807 INFO    HandlerThread:549018 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-14 18:29:44,808 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: defer
2021-07-14 18:29:44,808 INFO    SenderThread:549018 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-14 18:29:44,808 INFO    SenderThread:549018 [sender.py:transition_state():308] send defer: 3
2021-07-14 18:29:44,808 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:29:44,808 INFO    HandlerThread:549018 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-14 18:29:44,808 DEBUG   SenderThread:549018 [sender.py:send():179] send: summary
2021-07-14 18:29:44,808 INFO    SenderThread:549018 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 18:29:44,809 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: defer
2021-07-14 18:29:44,809 INFO    SenderThread:549018 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-14 18:29:44,809 INFO    SenderThread:549018 [sender.py:transition_state():308] send defer: 4
2021-07-14 18:29:44,809 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:29:44,809 INFO    HandlerThread:549018 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-14 18:29:44,809 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: defer
2021-07-14 18:29:44,809 INFO    SenderThread:549018 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-14 18:29:44,833 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:29:44,983 INFO    SenderThread:549018 [sender.py:transition_state():308] send defer: 5
2021-07-14 18:29:44,983 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:29:44,984 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:29:44,984 INFO    HandlerThread:549018 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-14 18:29:44,984 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: defer
2021-07-14 18:29:44,984 INFO    SenderThread:549018 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-14 18:29:44,984 INFO    SenderThread:549018 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-14 18:29:45,085 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:29:45,722 INFO    Thread-8  :549018 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/output.log
2021-07-14 18:29:45,723 INFO    SenderThread:549018 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/wandb-summary.json
2021-07-14 18:29:45,723 INFO    SenderThread:549018 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/config.yaml
2021-07-14 18:29:45,724 INFO    SenderThread:549018 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files
2021-07-14 18:29:45,724 INFO    SenderThread:549018 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/requirements.txt requirements.txt
2021-07-14 18:29:45,724 INFO    SenderThread:549018 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/output.log output.log
2021-07-14 18:29:45,725 INFO    SenderThread:549018 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/wandb-metadata.json wandb-metadata.json
2021-07-14 18:29:45,725 INFO    SenderThread:549018 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/config.yaml config.yaml
2021-07-14 18:29:45,725 INFO    SenderThread:549018 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/wandb-summary.json wandb-summary.json
2021-07-14 18:29:45,731 INFO    SenderThread:549018 [sender.py:transition_state():308] send defer: 6
2021-07-14 18:29:45,731 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:29:45,735 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:29:45,735 INFO    HandlerThread:549018 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-14 18:29:45,738 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: defer
2021-07-14 18:29:45,738 INFO    SenderThread:549018 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-14 18:29:45,738 INFO    SenderThread:549018 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 18:29:45,833 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:29:45,834 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:29:45,935 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:29:45,935 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:29:46,037 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:29:46,037 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:29:46,139 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:29:46,140 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:29:46,204 INFO    Thread-15 :549018 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/wandb-summary.json
2021-07-14 18:29:46,205 INFO    Thread-12 :549018 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/requirements.txt
2021-07-14 18:29:46,218 INFO    Thread-14 :549018 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/config.yaml
2021-07-14 18:29:46,241 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:29:46,242 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:29:46,287 INFO    Thread-13 :549018 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/files/output.log
2021-07-14 18:29:46,343 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:29:46,344 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:29:46,446 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:29:46,446 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:29:46,488 INFO    Thread-7  :549018 [sender.py:transition_state():308] send defer: 7
2021-07-14 18:29:46,489 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:29:46,489 INFO    HandlerThread:549018 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-14 18:29:46,489 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: defer
2021-07-14 18:29:46,489 INFO    SenderThread:549018 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-14 18:29:46,548 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:29:46,780 INFO    SenderThread:549018 [sender.py:transition_state():308] send defer: 8
2021-07-14 18:29:46,780 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:29:46,780 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:29:46,780 INFO    HandlerThread:549018 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-14 18:29:46,780 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: defer
2021-07-14 18:29:46,781 INFO    SenderThread:549018 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-14 18:29:46,781 INFO    SenderThread:549018 [sender.py:transition_state():308] send defer: 9
2021-07-14 18:29:46,781 DEBUG   SenderThread:549018 [sender.py:send():179] send: final
2021-07-14 18:29:46,781 DEBUG   SenderThread:549018 [sender.py:send():179] send: footer
2021-07-14 18:29:46,781 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:29:46,782 INFO    HandlerThread:549018 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-14 18:29:46,782 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: defer
2021-07-14 18:29:46,782 INFO    SenderThread:549018 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-14 18:29:46,882 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:29:46,882 DEBUG   SenderThread:549018 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:29:46,882 INFO    SenderThread:549018 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 18:29:46,883 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: get_summary
2021-07-14 18:29:46,884 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-14 18:29:46,884 DEBUG   HandlerThread:549018 [handler.py:handle_request():124] handle_request: shutdown
2021-07-14 18:29:46,885 INFO    HandlerThread:549018 [handler.py:finish():638] shutting down handler
2021-07-14 18:29:47,782 INFO    WriterThread:549018 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_181954-351hrfn2/run-351hrfn2.wandb
2021-07-14 18:29:47,882 INFO    SenderThread:549018 [sender.py:finish():945] shutting down sender
2021-07-14 18:29:47,882 INFO    SenderThread:549018 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 18:29:47,882 INFO    SenderThread:549018 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 18:29:47,885 INFO    MainThread:549018 [internal.py:handle_exit():78] Internal process exited