File size: 19,609 Bytes
9bd1fec
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
2021-07-13 00:49:11,524 INFO    MainThread:325318 [internal.py:wandb_internal():88] W&B internal server running at pid: 325318, started at: 2021-07-13 00:49:11.523864
2021-07-13 00:49:11,526 DEBUG   HandlerThread:325318 [handler.py:handle_request():124] handle_request: check_version
2021-07-13 00:49:11,526 INFO    WriterThread:325318 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/run-3mu9pog5.wandb
2021-07-13 00:49:11,527 DEBUG   SenderThread:325318 [sender.py:send():179] send: header
2021-07-13 00:49:11,527 DEBUG   SenderThread:325318 [sender.py:send_request():193] send_request: check_version
2021-07-13 00:49:11,564 DEBUG   SenderThread:325318 [sender.py:send():179] send: run
2021-07-13 00:49:11,738 INFO    SenderThread:325318 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files
2021-07-13 00:49:11,739 INFO    SenderThread:325318 [sender.py:_start_run_threads():716] run started: 3mu9pog5 with start time 1626137350
2021-07-13 00:49:11,739 DEBUG   SenderThread:325318 [sender.py:send():179] send: summary
2021-07-13 00:49:11,739 INFO    SenderThread:325318 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 00:49:11,739 DEBUG   HandlerThread:325318 [handler.py:handle_request():124] handle_request: run_start
2021-07-13 00:49:12,741 INFO    Thread-8  :325318 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/wandb-summary.json
2021-07-13 00:49:12,868 DEBUG   HandlerThread:325318 [meta.py:__init__():39] meta init
2021-07-13 00:49:12,868 DEBUG   HandlerThread:325318 [meta.py:__init__():53] meta init done
2021-07-13 00:49:12,868 DEBUG   HandlerThread:325318 [meta.py:probe():210] probe
2021-07-13 00:49:12,870 DEBUG   HandlerThread:325318 [meta.py:_setup_git():200] setup git
2021-07-13 00:49:12,899 DEBUG   HandlerThread:325318 [meta.py:_setup_git():207] setup git done
2021-07-13 00:49:12,899 DEBUG   HandlerThread:325318 [meta.py:_save_pip():57] save pip
2021-07-13 00:49:12,899 DEBUG   HandlerThread:325318 [meta.py:_save_pip():71] save pip done
2021-07-13 00:49:12,899 DEBUG   HandlerThread:325318 [meta.py:probe():252] probe done
2021-07-13 00:49:12,903 DEBUG   SenderThread:325318 [sender.py:send():179] send: files
2021-07-13 00:49:12,903 INFO    SenderThread:325318 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-13 00:49:12,910 DEBUG   HandlerThread:325318 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:49:12,911 DEBUG   SenderThread:325318 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:49:13,042 DEBUG   SenderThread:325318 [sender.py:send():179] send: config
2021-07-13 00:49:13,043 DEBUG   SenderThread:325318 [sender.py:send():179] send: config
2021-07-13 00:49:13,043 DEBUG   SenderThread:325318 [sender.py:send():179] send: config
2021-07-13 00:49:13,348 INFO    Thread-11 :325318 [upload_job.py:push():137] Uploaded file /tmp/tmpkvnk9e30wandb/65yetzns-wandb-metadata.json
2021-07-13 00:49:13,741 INFO    Thread-8  :325318 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/output.log
2021-07-13 00:49:13,741 INFO    Thread-8  :325318 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/requirements.txt
2021-07-13 00:49:13,741 INFO    Thread-8  :325318 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/wandb-metadata.json
2021-07-13 00:49:28,044 DEBUG   HandlerThread:325318 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:49:28,044 DEBUG   SenderThread:325318 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:49:29,748 INFO    Thread-8  :325318 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/output.log
2021-07-13 00:49:31,749 INFO    Thread-8  :325318 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/output.log
2021-07-13 00:49:40,952 DEBUG   SenderThread:325318 [sender.py:send():179] send: stats
2021-07-13 00:49:42,754 INFO    Thread-8  :325318 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/config.yaml
2021-07-13 00:49:43,176 DEBUG   HandlerThread:325318 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:49:43,177 DEBUG   SenderThread:325318 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:49:58,307 DEBUG   HandlerThread:325318 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:49:58,307 DEBUG   SenderThread:325318 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:50:11,029 DEBUG   SenderThread:325318 [sender.py:send():179] send: stats
2021-07-13 00:50:13,441 DEBUG   HandlerThread:325318 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:50:13,442 DEBUG   SenderThread:325318 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:50:21,769 INFO    Thread-8  :325318 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/output.log
2021-07-13 00:50:28,590 DEBUG   HandlerThread:325318 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:50:28,590 DEBUG   SenderThread:325318 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:50:41,106 DEBUG   SenderThread:325318 [sender.py:send():179] send: stats
2021-07-13 00:50:43,758 DEBUG   HandlerThread:325318 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:50:43,759 DEBUG   SenderThread:325318 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:50:58,908 DEBUG   HandlerThread:325318 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:50:58,909 DEBUG   SenderThread:325318 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:51:11,187 DEBUG   SenderThread:325318 [sender.py:send():179] send: stats
2021-07-13 00:51:14,040 DEBUG   HandlerThread:325318 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:51:14,041 DEBUG   SenderThread:325318 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:51:29,172 DEBUG   HandlerThread:325318 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:51:29,173 DEBUG   SenderThread:325318 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:51:41,267 DEBUG   SenderThread:325318 [sender.py:send():179] send: stats
2021-07-13 00:51:44,303 DEBUG   HandlerThread:325318 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:51:44,304 DEBUG   SenderThread:325318 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:51:53,809 INFO    Thread-8  :325318 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/output.log
2021-07-13 00:51:54,323 DEBUG   HandlerThread:325318 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:51:54,323 DEBUG   SenderThread:325318 [sender.py:send():179] send: telemetry
2021-07-13 00:51:54,323 DEBUG   SenderThread:325318 [sender.py:send():179] send: exit
2021-07-13 00:51:54,323 INFO    SenderThread:325318 [sender.py:send_exit():287] handling exit code: 1
2021-07-13 00:51:54,323 INFO    SenderThread:325318 [sender.py:send_exit():295] send defer
2021-07-13 00:51:54,323 DEBUG   SenderThread:325318 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:51:54,324 DEBUG   HandlerThread:325318 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:51:54,324 INFO    HandlerThread:325318 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-13 00:51:54,324 DEBUG   SenderThread:325318 [sender.py:send_request():193] send_request: defer
2021-07-13 00:51:54,324 INFO    SenderThread:325318 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-13 00:51:54,324 INFO    SenderThread:325318 [sender.py:transition_state():308] send defer: 1
2021-07-13 00:51:54,325 DEBUG   HandlerThread:325318 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:51:54,325 INFO    HandlerThread:325318 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-13 00:51:54,400 DEBUG   SenderThread:325318 [sender.py:send_request():193] send_request: defer
2021-07-13 00:51:54,400 INFO    SenderThread:325318 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-13 00:51:54,400 INFO    SenderThread:325318 [sender.py:transition_state():308] send defer: 2
2021-07-13 00:51:54,401 DEBUG   HandlerThread:325318 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:51:54,401 INFO    HandlerThread:325318 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-13 00:51:54,401 DEBUG   SenderThread:325318 [sender.py:send():179] send: stats
2021-07-13 00:51:54,401 DEBUG   SenderThread:325318 [sender.py:send_request():193] send_request: defer
2021-07-13 00:51:54,401 INFO    SenderThread:325318 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-13 00:51:54,401 INFO    SenderThread:325318 [sender.py:transition_state():308] send defer: 3
2021-07-13 00:51:54,402 DEBUG   HandlerThread:325318 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:51:54,402 INFO    HandlerThread:325318 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-13 00:51:54,402 DEBUG   SenderThread:325318 [sender.py:send():179] send: summary
2021-07-13 00:51:54,402 INFO    SenderThread:325318 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 00:51:54,403 DEBUG   SenderThread:325318 [sender.py:send_request():193] send_request: defer
2021-07-13 00:51:54,403 INFO    SenderThread:325318 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-13 00:51:54,403 INFO    SenderThread:325318 [sender.py:transition_state():308] send defer: 4
2021-07-13 00:51:54,403 DEBUG   HandlerThread:325318 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:51:54,403 INFO    HandlerThread:325318 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-13 00:51:54,403 DEBUG   SenderThread:325318 [sender.py:send_request():193] send_request: defer
2021-07-13 00:51:54,403 INFO    SenderThread:325318 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-13 00:51:54,426 DEBUG   HandlerThread:325318 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:51:54,590 INFO    SenderThread:325318 [sender.py:transition_state():308] send defer: 5
2021-07-13 00:51:54,590 DEBUG   SenderThread:325318 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:51:54,591 DEBUG   HandlerThread:325318 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:51:54,591 INFO    HandlerThread:325318 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-13 00:51:54,591 DEBUG   SenderThread:325318 [sender.py:send_request():193] send_request: defer
2021-07-13 00:51:54,591 INFO    SenderThread:325318 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-13 00:51:54,591 INFO    SenderThread:325318 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-13 00:51:54,693 DEBUG   HandlerThread:325318 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:51:54,809 INFO    Thread-8  :325318 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/output.log
2021-07-13 00:51:54,810 INFO    SenderThread:325318 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/config.yaml
2021-07-13 00:51:54,810 INFO    SenderThread:325318 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/wandb-summary.json
2021-07-13 00:51:54,810 INFO    SenderThread:325318 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files
2021-07-13 00:51:54,810 INFO    SenderThread:325318 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/requirements.txt requirements.txt
2021-07-13 00:51:54,810 INFO    SenderThread:325318 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/output.log output.log
2021-07-13 00:51:54,811 INFO    SenderThread:325318 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/wandb-metadata.json wandb-metadata.json
2021-07-13 00:51:54,811 INFO    SenderThread:325318 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/config.yaml config.yaml
2021-07-13 00:51:54,811 INFO    SenderThread:325318 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/wandb-summary.json wandb-summary.json
2021-07-13 00:51:54,811 INFO    SenderThread:325318 [sender.py:transition_state():308] send defer: 6
2021-07-13 00:51:54,811 DEBUG   SenderThread:325318 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:51:54,812 DEBUG   HandlerThread:325318 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:51:54,812 INFO    HandlerThread:325318 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-13 00:51:54,812 DEBUG   SenderThread:325318 [sender.py:send_request():193] send_request: defer
2021-07-13 00:51:54,814 INFO    SenderThread:325318 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-13 00:51:54,814 INFO    SenderThread:325318 [file_pusher.py:finish():177] shutting down file pusher
2021-07-13 00:51:54,913 DEBUG   HandlerThread:325318 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:51:54,914 DEBUG   SenderThread:325318 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:51:55,016 DEBUG   HandlerThread:325318 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:51:55,016 DEBUG   SenderThread:325318 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:51:55,118 DEBUG   HandlerThread:325318 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:51:55,118 DEBUG   SenderThread:325318 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:51:55,220 DEBUG   HandlerThread:325318 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:51:55,220 DEBUG   SenderThread:325318 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:51:55,257 INFO    Thread-14 :325318 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/config.yaml
2021-07-13 00:51:55,266 INFO    Thread-12 :325318 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/requirements.txt
2021-07-13 00:51:55,277 INFO    Thread-13 :325318 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/output.log
2021-07-13 00:51:55,288 INFO    Thread-15 :325318 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/files/wandb-summary.json
2021-07-13 00:51:55,322 DEBUG   HandlerThread:325318 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:51:55,322 DEBUG   SenderThread:325318 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:51:55,424 DEBUG   HandlerThread:325318 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:51:55,425 DEBUG   SenderThread:325318 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:51:55,489 INFO    Thread-7  :325318 [sender.py:transition_state():308] send defer: 7
2021-07-13 00:51:55,489 DEBUG   HandlerThread:325318 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:51:55,489 INFO    HandlerThread:325318 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-13 00:51:55,489 DEBUG   SenderThread:325318 [sender.py:send_request():193] send_request: defer
2021-07-13 00:51:55,490 INFO    SenderThread:325318 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-13 00:51:55,526 DEBUG   HandlerThread:325318 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:51:55,771 INFO    SenderThread:325318 [sender.py:transition_state():308] send defer: 8
2021-07-13 00:51:55,772 DEBUG   SenderThread:325318 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:51:55,772 DEBUG   HandlerThread:325318 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:51:55,772 INFO    HandlerThread:325318 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-13 00:51:55,772 DEBUG   SenderThread:325318 [sender.py:send_request():193] send_request: defer
2021-07-13 00:51:55,772 INFO    SenderThread:325318 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-13 00:51:55,773 INFO    SenderThread:325318 [sender.py:transition_state():308] send defer: 9
2021-07-13 00:51:55,773 DEBUG   HandlerThread:325318 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:51:55,773 INFO    HandlerThread:325318 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-13 00:51:55,773 DEBUG   SenderThread:325318 [sender.py:send():179] send: final
2021-07-13 00:51:55,773 DEBUG   SenderThread:325318 [sender.py:send():179] send: footer
2021-07-13 00:51:55,773 DEBUG   SenderThread:325318 [sender.py:send_request():193] send_request: defer
2021-07-13 00:51:55,773 INFO    SenderThread:325318 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-13 00:51:55,874 DEBUG   HandlerThread:325318 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:51:55,874 DEBUG   SenderThread:325318 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:51:55,874 INFO    SenderThread:325318 [file_pusher.py:join():182] waiting for file pusher
2021-07-13 00:51:55,876 DEBUG   HandlerThread:325318 [handler.py:handle_request():124] handle_request: get_summary
2021-07-13 00:51:55,877 DEBUG   HandlerThread:325318 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-13 00:51:55,877 DEBUG   HandlerThread:325318 [handler.py:handle_request():124] handle_request: shutdown
2021-07-13 00:51:55,877 INFO    HandlerThread:325318 [handler.py:finish():638] shutting down handler
2021-07-13 00:51:56,774 INFO    WriterThread:325318 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210713_004910-3mu9pog5/run-3mu9pog5.wandb
2021-07-13 00:51:56,875 INFO    SenderThread:325318 [sender.py:finish():945] shutting down sender
2021-07-13 00:51:56,875 INFO    SenderThread:325318 [file_pusher.py:finish():177] shutting down file pusher
2021-07-13 00:51:56,875 INFO    SenderThread:325318 [file_pusher.py:join():182] waiting for file pusher
2021-07-13 00:51:56,877 INFO    MainThread:325318 [internal.py:handle_exit():78] Internal process exited