File size: 25,524 Bytes
731244f
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
2021-07-15 00:17:36,936 INFO    MainThread:617849 [internal.py:wandb_internal():88] W&B internal server running at pid: 617849, started at: 2021-07-15 00:17:36.936178
2021-07-15 00:17:36,938 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: check_version
2021-07-15 00:17:36,939 DEBUG   SenderThread:617849 [sender.py:send():179] send: header
2021-07-15 00:17:36,939 INFO    WriterThread:617849 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/run-u185degl.wandb
2021-07-15 00:17:36,939 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: check_version
2021-07-15 00:17:36,977 DEBUG   SenderThread:617849 [sender.py:send():179] send: run
2021-07-15 00:17:37,151 INFO    SenderThread:617849 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files
2021-07-15 00:17:37,151 INFO    SenderThread:617849 [sender.py:_start_run_threads():716] run started: u185degl with start time 1626308256
2021-07-15 00:17:37,151 DEBUG   SenderThread:617849 [sender.py:send():179] send: summary
2021-07-15 00:17:37,151 INFO    SenderThread:617849 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-15 00:17:37,151 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: run_start
2021-07-15 00:17:38,153 INFO    Thread-8  :617849 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/wandb-summary.json
2021-07-15 00:17:38,283 DEBUG   HandlerThread:617849 [meta.py:__init__():39] meta init
2021-07-15 00:17:38,284 DEBUG   HandlerThread:617849 [meta.py:__init__():53] meta init done
2021-07-15 00:17:38,284 DEBUG   HandlerThread:617849 [meta.py:probe():210] probe
2021-07-15 00:17:38,285 DEBUG   HandlerThread:617849 [meta.py:_setup_git():200] setup git
2021-07-15 00:17:38,315 DEBUG   HandlerThread:617849 [meta.py:_setup_git():207] setup git done
2021-07-15 00:17:38,315 DEBUG   HandlerThread:617849 [meta.py:_save_pip():57] save pip
2021-07-15 00:17:38,316 DEBUG   HandlerThread:617849 [meta.py:_save_pip():71] save pip done
2021-07-15 00:17:38,316 DEBUG   HandlerThread:617849 [meta.py:probe():252] probe done
2021-07-15 00:17:38,319 DEBUG   SenderThread:617849 [sender.py:send():179] send: files
2021-07-15 00:17:38,319 INFO    SenderThread:617849 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-15 00:17:38,325 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:17:38,325 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:17:38,454 DEBUG   SenderThread:617849 [sender.py:send():179] send: config
2021-07-15 00:17:38,454 DEBUG   SenderThread:617849 [sender.py:send():179] send: config
2021-07-15 00:17:38,454 DEBUG   SenderThread:617849 [sender.py:send():179] send: config
2021-07-15 00:17:38,776 INFO    Thread-11 :617849 [upload_job.py:push():137] Uploaded file /tmp/tmpxo1vs5f9wandb/odzm0bpa-wandb-metadata.json
2021-07-15 00:17:39,152 INFO    Thread-8  :617849 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/requirements.txt
2021-07-15 00:17:39,152 INFO    Thread-8  :617849 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/wandb-metadata.json
2021-07-15 00:17:39,152 INFO    Thread-8  :617849 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/output.log
2021-07-15 00:17:53,157 INFO    Thread-8  :617849 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/output.log
2021-07-15 00:17:53,456 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:17:53,456 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:18:06,368 DEBUG   SenderThread:617849 [sender.py:send():179] send: stats
2021-07-15 00:18:08,163 INFO    Thread-8  :617849 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/config.yaml
2021-07-15 00:18:08,591 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:18:08,592 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:18:23,724 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:18:23,725 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:18:36,448 DEBUG   SenderThread:617849 [sender.py:send():179] send: stats
2021-07-15 00:18:38,860 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:18:38,860 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:18:53,994 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:18:53,994 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:19:06,524 DEBUG   SenderThread:617849 [sender.py:send():179] send: stats
2021-07-15 00:19:09,126 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:19:09,127 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:19:24,259 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:19:24,260 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:19:36,600 DEBUG   SenderThread:617849 [sender.py:send():179] send: stats
2021-07-15 00:19:39,407 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:19:39,407 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:19:54,540 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:19:54,540 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:20:06,663 DEBUG   SenderThread:617849 [sender.py:send():179] send: stats
2021-07-15 00:20:09,672 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:20:09,672 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:20:24,807 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:20:24,807 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:20:36,726 DEBUG   SenderThread:617849 [sender.py:send():179] send: stats
2021-07-15 00:20:39,938 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:20:39,938 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:20:55,070 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:20:55,070 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:21:06,791 DEBUG   SenderThread:617849 [sender.py:send():179] send: stats
2021-07-15 00:21:10,200 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:21:10,200 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:21:25,331 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:21:25,331 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:21:36,863 DEBUG   SenderThread:617849 [sender.py:send():179] send: stats
2021-07-15 00:21:40,462 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:21:40,462 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:21:55,592 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:21:55,593 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:22:06,940 DEBUG   SenderThread:617849 [sender.py:send():179] send: stats
2021-07-15 00:22:10,734 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:22:10,735 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:22:25,867 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:22:25,867 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:22:37,017 DEBUG   SenderThread:617849 [sender.py:send():179] send: stats
2021-07-15 00:22:40,998 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:22:40,998 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:22:56,126 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:22:56,126 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:23:07,091 DEBUG   SenderThread:617849 [sender.py:send():179] send: stats
2021-07-15 00:23:11,257 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:23:11,257 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:23:26,389 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:23:26,389 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:23:37,156 DEBUG   SenderThread:617849 [sender.py:send():179] send: stats
2021-07-15 00:23:41,522 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:23:41,522 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:23:56,651 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:23:56,652 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:24:07,226 DEBUG   SenderThread:617849 [sender.py:send():179] send: stats
2021-07-15 00:24:11,783 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:24:11,783 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:24:26,932 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:24:26,932 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:24:27,307 INFO    Thread-8  :617849 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/output.log
2021-07-15 00:24:37,304 DEBUG   SenderThread:617849 [sender.py:send():179] send: stats
2021-07-15 00:24:42,077 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:24:42,078 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:24:57,226 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:24:57,227 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:25:07,379 DEBUG   SenderThread:617849 [sender.py:send():179] send: stats
2021-07-15 00:25:12,357 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:25:12,357 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:25:27,488 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:25:27,489 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:25:37,452 DEBUG   SenderThread:617849 [sender.py:send():179] send: stats
2021-07-15 00:25:42,619 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:25:42,619 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:25:54,343 INFO    Thread-8  :617849 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/output.log
2021-07-15 00:25:54,412 DEBUG   SenderThread:617849 [sender.py:send():179] send: telemetry
2021-07-15 00:25:54,412 DEBUG   SenderThread:617849 [sender.py:send():179] send: exit
2021-07-15 00:25:54,412 INFO    SenderThread:617849 [sender.py:send_exit():287] handling exit code: 1
2021-07-15 00:25:54,412 INFO    SenderThread:617849 [sender.py:send_exit():295] send defer
2021-07-15 00:25:54,413 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:25:54,413 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:25:54,413 INFO    HandlerThread:617849 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-15 00:25:54,413 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:25:54,413 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: defer
2021-07-15 00:25:54,414 INFO    SenderThread:617849 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-15 00:25:54,414 INFO    SenderThread:617849 [sender.py:transition_state():308] send defer: 1
2021-07-15 00:25:54,414 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:25:54,414 INFO    HandlerThread:617849 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-15 00:25:54,493 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: defer
2021-07-15 00:25:54,493 INFO    SenderThread:617849 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-15 00:25:54,493 INFO    SenderThread:617849 [sender.py:transition_state():308] send defer: 2
2021-07-15 00:25:54,493 DEBUG   SenderThread:617849 [sender.py:send():179] send: stats
2021-07-15 00:25:54,494 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:25:54,494 INFO    HandlerThread:617849 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-15 00:25:54,494 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: defer
2021-07-15 00:25:54,494 INFO    SenderThread:617849 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-15 00:25:54,494 INFO    SenderThread:617849 [sender.py:transition_state():308] send defer: 3
2021-07-15 00:25:54,494 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:25:54,494 INFO    HandlerThread:617849 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-15 00:25:54,494 DEBUG   SenderThread:617849 [sender.py:send():179] send: summary
2021-07-15 00:25:54,495 INFO    SenderThread:617849 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-15 00:25:54,495 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: defer
2021-07-15 00:25:54,495 INFO    SenderThread:617849 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-15 00:25:54,495 INFO    SenderThread:617849 [sender.py:transition_state():308] send defer: 4
2021-07-15 00:25:54,495 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:25:54,495 INFO    HandlerThread:617849 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-15 00:25:54,495 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: defer
2021-07-15 00:25:54,495 INFO    SenderThread:617849 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-15 00:25:54,516 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:25:54,687 INFO    SenderThread:617849 [sender.py:transition_state():308] send defer: 5
2021-07-15 00:25:54,688 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:25:54,688 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:25:54,688 INFO    HandlerThread:617849 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-15 00:25:54,688 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: defer
2021-07-15 00:25:54,688 INFO    SenderThread:617849 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-15 00:25:54,688 INFO    SenderThread:617849 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-15 00:25:54,790 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:25:55,343 INFO    Thread-8  :617849 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/output.log
2021-07-15 00:25:55,344 INFO    SenderThread:617849 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/config.yaml
2021-07-15 00:25:55,344 INFO    SenderThread:617849 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/wandb-summary.json
2021-07-15 00:25:55,344 INFO    SenderThread:617849 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files
2021-07-15 00:25:55,344 INFO    SenderThread:617849 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/requirements.txt requirements.txt
2021-07-15 00:25:55,344 INFO    SenderThread:617849 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/output.log output.log
2021-07-15 00:25:55,344 INFO    SenderThread:617849 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/wandb-metadata.json wandb-metadata.json
2021-07-15 00:25:55,344 INFO    SenderThread:617849 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/config.yaml config.yaml
2021-07-15 00:25:55,345 INFO    SenderThread:617849 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/wandb-summary.json wandb-summary.json
2021-07-15 00:25:55,345 INFO    SenderThread:617849 [sender.py:transition_state():308] send defer: 6
2021-07-15 00:25:55,345 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:25:55,346 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:25:55,346 INFO    HandlerThread:617849 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-15 00:25:55,349 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: defer
2021-07-15 00:25:55,349 INFO    SenderThread:617849 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-15 00:25:55,349 INFO    SenderThread:617849 [file_pusher.py:finish():177] shutting down file pusher
2021-07-15 00:25:55,447 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:25:55,448 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:25:55,550 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:25:55,550 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:25:55,652 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:25:55,652 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:25:55,754 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:25:55,754 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:25:55,810 INFO    Thread-13 :617849 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/output.log
2021-07-15 00:25:55,821 INFO    Thread-15 :617849 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/wandb-summary.json
2021-07-15 00:25:55,827 INFO    Thread-12 :617849 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/requirements.txt
2021-07-15 00:25:55,856 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:25:55,856 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:25:55,869 INFO    Thread-14 :617849 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/config.yaml
2021-07-15 00:25:55,958 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:25:55,958 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:25:56,060 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:25:56,060 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:25:56,069 INFO    Thread-7  :617849 [sender.py:transition_state():308] send defer: 7
2021-07-15 00:25:56,070 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:25:56,070 INFO    HandlerThread:617849 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-15 00:25:56,070 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: defer
2021-07-15 00:25:56,070 INFO    SenderThread:617849 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-15 00:25:56,162 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:25:56,354 INFO    SenderThread:617849 [sender.py:transition_state():308] send defer: 8
2021-07-15 00:25:56,354 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:25:56,355 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:25:56,355 INFO    HandlerThread:617849 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-15 00:25:56,355 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: defer
2021-07-15 00:25:56,355 INFO    SenderThread:617849 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-15 00:25:56,355 INFO    SenderThread:617849 [sender.py:transition_state():308] send defer: 9
2021-07-15 00:25:56,356 DEBUG   SenderThread:617849 [sender.py:send():179] send: final
2021-07-15 00:25:56,356 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:25:56,356 DEBUG   SenderThread:617849 [sender.py:send():179] send: footer
2021-07-15 00:25:56,356 INFO    HandlerThread:617849 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-15 00:25:56,356 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: defer
2021-07-15 00:25:56,356 INFO    SenderThread:617849 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-15 00:25:56,456 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:25:56,456 DEBUG   SenderThread:617849 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:25:56,456 INFO    SenderThread:617849 [file_pusher.py:join():182] waiting for file pusher
2021-07-15 00:25:56,458 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: get_summary
2021-07-15 00:25:56,458 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-15 00:25:56,459 DEBUG   HandlerThread:617849 [handler.py:handle_request():124] handle_request: shutdown
2021-07-15 00:25:56,459 INFO    HandlerThread:617849 [handler.py:finish():638] shutting down handler
2021-07-15 00:25:57,356 INFO    WriterThread:617849 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/run-u185degl.wandb
2021-07-15 00:25:57,457 INFO    SenderThread:617849 [sender.py:finish():945] shutting down sender
2021-07-15 00:25:57,457 INFO    SenderThread:617849 [file_pusher.py:finish():177] shutting down file pusher
2021-07-15 00:25:57,457 INFO    SenderThread:617849 [file_pusher.py:join():182] waiting for file pusher
2021-07-15 00:25:57,462 INFO    MainThread:617849 [internal.py:handle_exit():78] Internal process exited