File size: 20,079 Bytes
f291f93
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
2021-07-13 13:16:16,698 INFO    MainThread:363863 [internal.py:wandb_internal():88] W&B internal server running at pid: 363863, started at: 2021-07-13 13:16:16.698045
2021-07-13 13:16:16,700 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: check_version
2021-07-13 13:16:16,700 INFO    WriterThread:363863 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/run-w761mg4c.wandb
2021-07-13 13:16:16,701 DEBUG   SenderThread:363863 [sender.py:send():179] send: header
2021-07-13 13:16:16,701 DEBUG   SenderThread:363863 [sender.py:send_request():193] send_request: check_version
2021-07-13 13:16:16,739 DEBUG   SenderThread:363863 [sender.py:send():179] send: run
2021-07-13 13:16:16,916 INFO    SenderThread:363863 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files
2021-07-13 13:16:16,916 INFO    SenderThread:363863 [sender.py:_start_run_threads():716] run started: w761mg4c with start time 1626182176
2021-07-13 13:16:16,916 DEBUG   SenderThread:363863 [sender.py:send():179] send: summary
2021-07-13 13:16:16,917 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: run_start
2021-07-13 13:16:16,917 INFO    SenderThread:363863 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 13:16:17,918 INFO    Thread-8  :363863 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/wandb-summary.json
2021-07-13 13:16:18,041 DEBUG   HandlerThread:363863 [meta.py:__init__():39] meta init
2021-07-13 13:16:18,041 DEBUG   HandlerThread:363863 [meta.py:__init__():53] meta init done
2021-07-13 13:16:18,041 DEBUG   HandlerThread:363863 [meta.py:probe():210] probe
2021-07-13 13:16:18,042 DEBUG   HandlerThread:363863 [meta.py:_setup_git():200] setup git
2021-07-13 13:16:18,071 DEBUG   HandlerThread:363863 [meta.py:_setup_git():207] setup git done
2021-07-13 13:16:18,071 DEBUG   HandlerThread:363863 [meta.py:_save_pip():57] save pip
2021-07-13 13:16:18,072 DEBUG   HandlerThread:363863 [meta.py:_save_pip():71] save pip done
2021-07-13 13:16:18,072 DEBUG   HandlerThread:363863 [meta.py:probe():252] probe done
2021-07-13 13:16:18,075 DEBUG   SenderThread:363863 [sender.py:send():179] send: files
2021-07-13 13:16:18,075 INFO    SenderThread:363863 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-13 13:16:18,083 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 13:16:18,083 DEBUG   SenderThread:363863 [sender.py:send_request():193] send_request: stop_status
2021-07-13 13:16:18,213 DEBUG   SenderThread:363863 [sender.py:send():179] send: config
2021-07-13 13:16:18,214 DEBUG   SenderThread:363863 [sender.py:send():179] send: config
2021-07-13 13:16:18,214 DEBUG   SenderThread:363863 [sender.py:send():179] send: config
2021-07-13 13:16:18,530 INFO    Thread-11 :363863 [upload_job.py:push():137] Uploaded file /tmp/tmpe18zfifmwandb/okixm8uo-wandb-metadata.json
2021-07-13 13:16:18,918 INFO    Thread-8  :363863 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/output.log
2021-07-13 13:16:18,918 INFO    Thread-8  :363863 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/wandb-metadata.json
2021-07-13 13:16:18,918 INFO    Thread-8  :363863 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/requirements.txt
2021-07-13 13:16:33,215 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 13:16:33,215 DEBUG   SenderThread:363863 [sender.py:send_request():193] send_request: stop_status
2021-07-13 13:16:34,924 INFO    Thread-8  :363863 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/output.log
2021-07-13 13:16:46,124 DEBUG   SenderThread:363863 [sender.py:send():179] send: stats
2021-07-13 13:16:47,928 INFO    Thread-8  :363863 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/config.yaml
2021-07-13 13:16:48,349 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 13:16:48,350 DEBUG   SenderThread:363863 [sender.py:send_request():193] send_request: stop_status
2021-07-13 13:17:03,483 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 13:17:03,484 DEBUG   SenderThread:363863 [sender.py:send_request():193] send_request: stop_status
2021-07-13 13:17:16,202 DEBUG   SenderThread:363863 [sender.py:send():179] send: stats
2021-07-13 13:17:18,616 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 13:17:18,617 DEBUG   SenderThread:363863 [sender.py:send_request():193] send_request: stop_status
2021-07-13 13:17:33,748 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 13:17:33,749 DEBUG   SenderThread:363863 [sender.py:send_request():193] send_request: stop_status
2021-07-13 13:17:46,277 DEBUG   SenderThread:363863 [sender.py:send():179] send: stats
2021-07-13 13:17:48,883 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 13:17:48,883 DEBUG   SenderThread:363863 [sender.py:send_request():193] send_request: stop_status
2021-07-13 13:18:04,023 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 13:18:04,023 DEBUG   SenderThread:363863 [sender.py:send_request():193] send_request: stop_status
2021-07-13 13:18:04,956 INFO    Thread-8  :363863 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/output.log
2021-07-13 13:18:16,347 DEBUG   SenderThread:363863 [sender.py:send():179] send: stats
2021-07-13 13:18:19,179 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 13:18:19,179 DEBUG   SenderThread:363863 [sender.py:send_request():193] send_request: stop_status
2021-07-13 13:18:34,324 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 13:18:34,325 DEBUG   SenderThread:363863 [sender.py:send_request():193] send_request: stop_status
2021-07-13 13:18:46,418 DEBUG   SenderThread:363863 [sender.py:send():179] send: stats
2021-07-13 13:18:49,456 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 13:18:49,456 DEBUG   SenderThread:363863 [sender.py:send_request():193] send_request: stop_status
2021-07-13 13:19:04,586 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 13:19:04,586 DEBUG   SenderThread:363863 [sender.py:send_request():193] send_request: stop_status
2021-07-13 13:19:16,489 DEBUG   SenderThread:363863 [sender.py:send():179] send: stats
2021-07-13 13:19:19,718 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 13:19:19,718 DEBUG   SenderThread:363863 [sender.py:send_request():193] send_request: stop_status
2021-07-13 13:19:34,851 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 13:19:34,852 DEBUG   SenderThread:363863 [sender.py:send_request():193] send_request: stop_status
2021-07-13 13:19:46,568 DEBUG   SenderThread:363863 [sender.py:send():179] send: stats
2021-07-13 13:19:49,985 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 13:19:49,985 DEBUG   SenderThread:363863 [sender.py:send_request():193] send_request: stop_status
2021-07-13 13:20:00,715 DEBUG   SenderThread:363863 [sender.py:send():179] send: telemetry
2021-07-13 13:20:00,715 DEBUG   SenderThread:363863 [sender.py:send():179] send: exit
2021-07-13 13:20:00,715 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 13:20:00,715 INFO    SenderThread:363863 [sender.py:send_exit():287] handling exit code: 1
2021-07-13 13:20:00,716 INFO    SenderThread:363863 [sender.py:send_exit():295] send defer
2021-07-13 13:20:00,716 DEBUG   SenderThread:363863 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 13:20:00,716 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: defer
2021-07-13 13:20:00,716 INFO    HandlerThread:363863 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-13 13:20:00,717 DEBUG   SenderThread:363863 [sender.py:send_request():193] send_request: defer
2021-07-13 13:20:00,717 INFO    SenderThread:363863 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-13 13:20:00,717 INFO    SenderThread:363863 [sender.py:transition_state():308] send defer: 1
2021-07-13 13:20:00,717 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: defer
2021-07-13 13:20:00,717 INFO    HandlerThread:363863 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-13 13:20:00,805 DEBUG   SenderThread:363863 [sender.py:send_request():193] send_request: defer
2021-07-13 13:20:00,805 INFO    SenderThread:363863 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-13 13:20:00,805 INFO    SenderThread:363863 [sender.py:transition_state():308] send defer: 2
2021-07-13 13:20:00,806 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: defer
2021-07-13 13:20:00,806 INFO    HandlerThread:363863 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-13 13:20:00,806 DEBUG   SenderThread:363863 [sender.py:send():179] send: stats
2021-07-13 13:20:00,806 DEBUG   SenderThread:363863 [sender.py:send_request():193] send_request: defer
2021-07-13 13:20:00,806 INFO    SenderThread:363863 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-13 13:20:00,806 INFO    SenderThread:363863 [sender.py:transition_state():308] send defer: 3
2021-07-13 13:20:00,806 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: defer
2021-07-13 13:20:00,807 INFO    HandlerThread:363863 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-13 13:20:00,807 DEBUG   SenderThread:363863 [sender.py:send():179] send: summary
2021-07-13 13:20:00,807 INFO    SenderThread:363863 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 13:20:00,807 DEBUG   SenderThread:363863 [sender.py:send_request():193] send_request: defer
2021-07-13 13:20:00,807 INFO    SenderThread:363863 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-13 13:20:00,807 INFO    SenderThread:363863 [sender.py:transition_state():308] send defer: 4
2021-07-13 13:20:00,808 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: defer
2021-07-13 13:20:00,808 INFO    HandlerThread:363863 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-13 13:20:00,808 DEBUG   SenderThread:363863 [sender.py:send_request():193] send_request: defer
2021-07-13 13:20:00,808 INFO    SenderThread:363863 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-13 13:20:00,818 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 13:20:00,985 INFO    SenderThread:363863 [sender.py:transition_state():308] send defer: 5
2021-07-13 13:20:00,986 DEBUG   SenderThread:363863 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 13:20:00,986 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: defer
2021-07-13 13:20:00,986 INFO    HandlerThread:363863 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-13 13:20:00,987 DEBUG   SenderThread:363863 [sender.py:send_request():193] send_request: defer
2021-07-13 13:20:00,987 INFO    SenderThread:363863 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-13 13:20:00,987 INFO    SenderThread:363863 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-13 13:20:00,998 INFO    Thread-8  :363863 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/output.log
2021-07-13 13:20:00,999 INFO    SenderThread:363863 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/config.yaml
2021-07-13 13:20:00,999 INFO    SenderThread:363863 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/wandb-summary.json
2021-07-13 13:20:00,999 INFO    SenderThread:363863 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files
2021-07-13 13:20:00,999 INFO    SenderThread:363863 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/requirements.txt requirements.txt
2021-07-13 13:20:00,999 INFO    SenderThread:363863 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/output.log output.log
2021-07-13 13:20:00,999 INFO    SenderThread:363863 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/wandb-metadata.json wandb-metadata.json
2021-07-13 13:20:01,000 INFO    SenderThread:363863 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/config.yaml config.yaml
2021-07-13 13:20:01,000 INFO    SenderThread:363863 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/wandb-summary.json wandb-summary.json
2021-07-13 13:20:01,000 INFO    SenderThread:363863 [sender.py:transition_state():308] send defer: 6
2021-07-13 13:20:01,012 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: defer
2021-07-13 13:20:01,013 INFO    HandlerThread:363863 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-13 13:20:01,013 DEBUG   SenderThread:363863 [sender.py:send_request():193] send_request: defer
2021-07-13 13:20:01,013 INFO    SenderThread:363863 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-13 13:20:01,013 INFO    SenderThread:363863 [file_pusher.py:finish():177] shutting down file pusher
2021-07-13 13:20:01,088 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 13:20:01,088 DEBUG   SenderThread:363863 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 13:20:01,190 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 13:20:01,191 DEBUG   SenderThread:363863 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 13:20:01,292 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 13:20:01,293 DEBUG   SenderThread:363863 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 13:20:01,395 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 13:20:01,395 DEBUG   SenderThread:363863 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 13:20:01,443 INFO    Thread-13 :363863 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/output.log
2021-07-13 13:20:01,467 INFO    Thread-15 :363863 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/wandb-summary.json
2021-07-13 13:20:01,468 INFO    Thread-14 :363863 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/config.yaml
2021-07-13 13:20:01,474 INFO    Thread-12 :363863 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/files/requirements.txt
2021-07-13 13:20:01,497 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 13:20:01,497 DEBUG   SenderThread:363863 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 13:20:01,599 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 13:20:01,599 DEBUG   SenderThread:363863 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 13:20:01,674 INFO    Thread-7  :363863 [sender.py:transition_state():308] send defer: 7
2021-07-13 13:20:01,675 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: defer
2021-07-13 13:20:01,675 INFO    HandlerThread:363863 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-13 13:20:01,675 DEBUG   SenderThread:363863 [sender.py:send_request():193] send_request: defer
2021-07-13 13:20:01,675 INFO    SenderThread:363863 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-13 13:20:01,701 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 13:20:01,941 INFO    SenderThread:363863 [sender.py:transition_state():308] send defer: 8
2021-07-13 13:20:01,942 DEBUG   SenderThread:363863 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 13:20:01,942 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: defer
2021-07-13 13:20:01,942 INFO    HandlerThread:363863 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-13 13:20:01,942 DEBUG   SenderThread:363863 [sender.py:send_request():193] send_request: defer
2021-07-13 13:20:01,943 INFO    SenderThread:363863 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-13 13:20:01,943 INFO    SenderThread:363863 [sender.py:transition_state():308] send defer: 9
2021-07-13 13:20:01,943 DEBUG   SenderThread:363863 [sender.py:send():179] send: final
2021-07-13 13:20:01,943 DEBUG   SenderThread:363863 [sender.py:send():179] send: footer
2021-07-13 13:20:01,943 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: defer
2021-07-13 13:20:01,943 INFO    HandlerThread:363863 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-13 13:20:01,944 DEBUG   SenderThread:363863 [sender.py:send_request():193] send_request: defer
2021-07-13 13:20:01,944 INFO    SenderThread:363863 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-13 13:20:02,044 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 13:20:02,044 DEBUG   SenderThread:363863 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 13:20:02,044 INFO    SenderThread:363863 [file_pusher.py:join():182] waiting for file pusher
2021-07-13 13:20:02,046 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: get_summary
2021-07-13 13:20:02,046 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-13 13:20:02,047 DEBUG   HandlerThread:363863 [handler.py:handle_request():124] handle_request: shutdown
2021-07-13 13:20:02,047 INFO    HandlerThread:363863 [handler.py:finish():638] shutting down handler
2021-07-13 13:20:02,944 INFO    WriterThread:363863 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210713_131615-w761mg4c/run-w761mg4c.wandb
2021-07-13 13:20:03,044 INFO    SenderThread:363863 [sender.py:finish():945] shutting down sender
2021-07-13 13:20:03,044 INFO    SenderThread:363863 [file_pusher.py:finish():177] shutting down file pusher
2021-07-13 13:20:03,045 INFO    SenderThread:363863 [file_pusher.py:join():182] waiting for file pusher
2021-07-13 13:20:03,047 INFO    MainThread:363863 [internal.py:handle_exit():78] Internal process exited