File size: 20,364 Bytes
f291f93
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
2021-07-13 11:02:13,518 INFO    MainThread:345480 [internal.py:wandb_internal():88] W&B internal server running at pid: 345480, started at: 2021-07-13 11:02:13.518424
2021-07-13 11:02:13,520 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: check_version
2021-07-13 11:02:13,520 INFO    WriterThread:345480 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/run-594z6oo0.wandb
2021-07-13 11:02:13,521 DEBUG   SenderThread:345480 [sender.py:send():179] send: header
2021-07-13 11:02:13,522 DEBUG   SenderThread:345480 [sender.py:send_request():193] send_request: check_version
2021-07-13 11:02:13,558 DEBUG   SenderThread:345480 [sender.py:send():179] send: run
2021-07-13 11:02:13,741 INFO    SenderThread:345480 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files
2021-07-13 11:02:13,742 INFO    SenderThread:345480 [sender.py:_start_run_threads():716] run started: 594z6oo0 with start time 1626174132
2021-07-13 11:02:13,742 DEBUG   SenderThread:345480 [sender.py:send():179] send: summary
2021-07-13 11:02:13,742 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: run_start
2021-07-13 11:02:13,743 INFO    SenderThread:345480 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 11:02:14,743 INFO    Thread-8  :345480 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/wandb-summary.json
2021-07-13 11:02:14,868 DEBUG   HandlerThread:345480 [meta.py:__init__():39] meta init
2021-07-13 11:02:14,868 DEBUG   HandlerThread:345480 [meta.py:__init__():53] meta init done
2021-07-13 11:02:14,868 DEBUG   HandlerThread:345480 [meta.py:probe():210] probe
2021-07-13 11:02:14,869 DEBUG   HandlerThread:345480 [meta.py:_setup_git():200] setup git
2021-07-13 11:02:14,899 DEBUG   HandlerThread:345480 [meta.py:_setup_git():207] setup git done
2021-07-13 11:02:14,899 DEBUG   HandlerThread:345480 [meta.py:_save_pip():57] save pip
2021-07-13 11:02:14,899 DEBUG   HandlerThread:345480 [meta.py:_save_pip():71] save pip done
2021-07-13 11:02:14,899 DEBUG   HandlerThread:345480 [meta.py:probe():252] probe done
2021-07-13 11:02:14,902 DEBUG   SenderThread:345480 [sender.py:send():179] send: files
2021-07-13 11:02:14,902 INFO    SenderThread:345480 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-13 11:02:14,910 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:02:14,910 DEBUG   SenderThread:345480 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:02:15,037 DEBUG   SenderThread:345480 [sender.py:send():179] send: config
2021-07-13 11:02:15,038 DEBUG   SenderThread:345480 [sender.py:send():179] send: config
2021-07-13 11:02:15,038 DEBUG   SenderThread:345480 [sender.py:send():179] send: config
2021-07-13 11:02:15,378 INFO    Thread-11 :345480 [upload_job.py:push():137] Uploaded file /tmp/tmpy185wdfywandb/22iauf0r-wandb-metadata.json
2021-07-13 11:02:15,743 INFO    Thread-8  :345480 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/wandb-metadata.json
2021-07-13 11:02:15,743 INFO    Thread-8  :345480 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/requirements.txt
2021-07-13 11:02:15,743 INFO    Thread-8  :345480 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/output.log
2021-07-13 11:02:29,748 INFO    Thread-8  :345480 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/output.log
2021-07-13 11:02:30,039 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:02:30,040 DEBUG   SenderThread:345480 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:02:31,749 INFO    Thread-8  :345480 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/output.log
2021-07-13 11:02:42,950 DEBUG   SenderThread:345480 [sender.py:send():179] send: stats
2021-07-13 11:02:44,755 INFO    Thread-8  :345480 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/config.yaml
2021-07-13 11:02:45,176 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:02:45,176 DEBUG   SenderThread:345480 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:03:00,308 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:03:00,308 DEBUG   SenderThread:345480 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:03:13,026 DEBUG   SenderThread:345480 [sender.py:send():179] send: stats
2021-07-13 11:03:15,445 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:03:15,445 DEBUG   SenderThread:345480 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:03:30,574 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:03:30,575 DEBUG   SenderThread:345480 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:03:43,084 DEBUG   SenderThread:345480 [sender.py:send():179] send: stats
2021-07-13 11:03:45,707 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:03:45,708 DEBUG   SenderThread:345480 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:04:00,845 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:04:00,845 DEBUG   SenderThread:345480 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:04:01,785 INFO    Thread-8  :345480 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/output.log
2021-07-13 11:04:13,156 DEBUG   SenderThread:345480 [sender.py:send():179] send: stats
2021-07-13 11:04:15,994 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:04:15,994 DEBUG   SenderThread:345480 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:04:31,136 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:04:31,136 DEBUG   SenderThread:345480 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:04:43,236 DEBUG   SenderThread:345480 [sender.py:send():179] send: stats
2021-07-13 11:04:46,269 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:04:46,270 DEBUG   SenderThread:345480 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:05:01,401 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:05:01,401 DEBUG   SenderThread:345480 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:05:13,316 DEBUG   SenderThread:345480 [sender.py:send():179] send: stats
2021-07-13 11:05:16,535 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 11:05:16,536 DEBUG   SenderThread:345480 [sender.py:send_request():193] send_request: stop_status
2021-07-13 11:05:27,829 INFO    Thread-8  :345480 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/output.log
2021-07-13 11:05:28,324 DEBUG   SenderThread:345480 [sender.py:send():179] send: telemetry
2021-07-13 11:05:28,325 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:05:28,325 DEBUG   SenderThread:345480 [sender.py:send():179] send: exit
2021-07-13 11:05:28,325 INFO    SenderThread:345480 [sender.py:send_exit():287] handling exit code: 1
2021-07-13 11:05:28,325 INFO    SenderThread:345480 [sender.py:send_exit():295] send defer
2021-07-13 11:05:28,325 DEBUG   SenderThread:345480 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:05:28,326 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:05:28,326 INFO    HandlerThread:345480 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-13 11:05:28,326 DEBUG   SenderThread:345480 [sender.py:send_request():193] send_request: defer
2021-07-13 11:05:28,326 INFO    SenderThread:345480 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-13 11:05:28,326 INFO    SenderThread:345480 [sender.py:transition_state():308] send defer: 1
2021-07-13 11:05:28,327 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:05:28,327 INFO    HandlerThread:345480 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-13 11:05:28,355 DEBUG   SenderThread:345480 [sender.py:send_request():193] send_request: defer
2021-07-13 11:05:28,355 INFO    SenderThread:345480 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-13 11:05:28,355 INFO    SenderThread:345480 [sender.py:transition_state():308] send defer: 2
2021-07-13 11:05:28,355 DEBUG   SenderThread:345480 [sender.py:send():179] send: stats
2021-07-13 11:05:28,355 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:05:28,355 INFO    HandlerThread:345480 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-13 11:05:28,355 DEBUG   SenderThread:345480 [sender.py:send_request():193] send_request: defer
2021-07-13 11:05:28,355 INFO    SenderThread:345480 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-13 11:05:28,355 INFO    SenderThread:345480 [sender.py:transition_state():308] send defer: 3
2021-07-13 11:05:28,356 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:05:28,356 INFO    HandlerThread:345480 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-13 11:05:28,356 DEBUG   SenderThread:345480 [sender.py:send():179] send: summary
2021-07-13 11:05:28,356 INFO    SenderThread:345480 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 11:05:28,356 DEBUG   SenderThread:345480 [sender.py:send_request():193] send_request: defer
2021-07-13 11:05:28,356 INFO    SenderThread:345480 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-13 11:05:28,356 INFO    SenderThread:345480 [sender.py:transition_state():308] send defer: 4
2021-07-13 11:05:28,357 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:05:28,357 INFO    HandlerThread:345480 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-13 11:05:28,357 DEBUG   SenderThread:345480 [sender.py:send_request():193] send_request: defer
2021-07-13 11:05:28,357 INFO    SenderThread:345480 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-13 11:05:28,428 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:05:28,533 INFO    SenderThread:345480 [sender.py:transition_state():308] send defer: 5
2021-07-13 11:05:28,533 DEBUG   SenderThread:345480 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:05:28,534 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:05:28,534 INFO    HandlerThread:345480 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-13 11:05:28,534 DEBUG   SenderThread:345480 [sender.py:send_request():193] send_request: defer
2021-07-13 11:05:28,534 INFO    SenderThread:345480 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-13 11:05:28,535 INFO    SenderThread:345480 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-13 11:05:28,636 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:05:28,829 INFO    Thread-8  :345480 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/wandb-summary.json
2021-07-13 11:05:28,830 INFO    SenderThread:345480 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/config.yaml
2021-07-13 11:05:28,830 INFO    SenderThread:345480 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/output.log
2021-07-13 11:05:28,830 INFO    SenderThread:345480 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files
2021-07-13 11:05:28,830 INFO    SenderThread:345480 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/requirements.txt requirements.txt
2021-07-13 11:05:28,830 INFO    SenderThread:345480 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/output.log output.log
2021-07-13 11:05:28,831 INFO    SenderThread:345480 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/wandb-metadata.json wandb-metadata.json
2021-07-13 11:05:28,831 INFO    SenderThread:345480 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/config.yaml config.yaml
2021-07-13 11:05:28,831 INFO    SenderThread:345480 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/wandb-summary.json wandb-summary.json
2021-07-13 11:05:28,831 INFO    SenderThread:345480 [sender.py:transition_state():308] send defer: 6
2021-07-13 11:05:28,831 DEBUG   SenderThread:345480 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:05:28,837 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:05:28,837 INFO    HandlerThread:345480 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-13 11:05:28,837 DEBUG   SenderThread:345480 [sender.py:send_request():193] send_request: defer
2021-07-13 11:05:28,844 INFO    SenderThread:345480 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-13 11:05:28,844 INFO    SenderThread:345480 [file_pusher.py:finish():177] shutting down file pusher
2021-07-13 11:05:28,933 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:05:28,934 DEBUG   SenderThread:345480 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:05:29,036 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:05:29,036 DEBUG   SenderThread:345480 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:05:29,139 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:05:29,139 DEBUG   SenderThread:345480 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:05:29,241 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:05:29,241 DEBUG   SenderThread:345480 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:05:29,280 INFO    Thread-15 :345480 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/wandb-summary.json
2021-07-13 11:05:29,323 INFO    Thread-13 :345480 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/output.log
2021-07-13 11:05:29,343 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:05:29,343 DEBUG   SenderThread:345480 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:05:29,367 INFO    Thread-12 :345480 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/requirements.txt
2021-07-13 11:05:29,389 INFO    Thread-14 :345480 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/files/config.yaml
2021-07-13 11:05:29,445 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:05:29,445 DEBUG   SenderThread:345480 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:05:29,548 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:05:29,548 DEBUG   SenderThread:345480 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:05:29,589 INFO    Thread-7  :345480 [sender.py:transition_state():308] send defer: 7
2021-07-13 11:05:29,590 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:05:29,590 INFO    HandlerThread:345480 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-13 11:05:29,590 DEBUG   SenderThread:345480 [sender.py:send_request():193] send_request: defer
2021-07-13 11:05:29,590 INFO    SenderThread:345480 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-13 11:05:29,650 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:05:29,884 INFO    SenderThread:345480 [sender.py:transition_state():308] send defer: 8
2021-07-13 11:05:29,884 DEBUG   SenderThread:345480 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:05:29,885 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:05:29,885 INFO    HandlerThread:345480 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-13 11:05:29,885 DEBUG   SenderThread:345480 [sender.py:send_request():193] send_request: defer
2021-07-13 11:05:29,885 INFO    SenderThread:345480 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-13 11:05:29,885 INFO    SenderThread:345480 [sender.py:transition_state():308] send defer: 9
2021-07-13 11:05:29,886 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: defer
2021-07-13 11:05:29,886 INFO    HandlerThread:345480 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-13 11:05:29,886 DEBUG   SenderThread:345480 [sender.py:send():179] send: final
2021-07-13 11:05:29,886 DEBUG   SenderThread:345480 [sender.py:send():179] send: footer
2021-07-13 11:05:29,886 DEBUG   SenderThread:345480 [sender.py:send_request():193] send_request: defer
2021-07-13 11:05:29,886 INFO    SenderThread:345480 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-13 11:05:29,987 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 11:05:29,987 DEBUG   SenderThread:345480 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 11:05:29,987 INFO    SenderThread:345480 [file_pusher.py:join():182] waiting for file pusher
2021-07-13 11:05:29,989 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: get_summary
2021-07-13 11:05:29,990 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-13 11:05:29,991 DEBUG   HandlerThread:345480 [handler.py:handle_request():124] handle_request: shutdown
2021-07-13 11:05:29,991 INFO    HandlerThread:345480 [handler.py:finish():638] shutting down handler
2021-07-13 11:05:30,886 INFO    WriterThread:345480 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210713_110212-594z6oo0/run-594z6oo0.wandb
2021-07-13 11:05:30,987 INFO    SenderThread:345480 [sender.py:finish():945] shutting down sender
2021-07-13 11:05:30,988 INFO    SenderThread:345480 [file_pusher.py:finish():177] shutting down file pusher
2021-07-13 11:05:30,988 INFO    SenderThread:345480 [file_pusher.py:join():182] waiting for file pusher
2021-07-13 11:05:30,990 INFO    MainThread:345480 [internal.py:handle_exit():78] Internal process exited