File size: 19,829 Bytes
9bd1fec
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
2021-07-13 00:53:02,112 INFO    MainThread:327506 [internal.py:wandb_internal():88] W&B internal server running at pid: 327506, started at: 2021-07-13 00:53:02.112234
2021-07-13 00:53:02,114 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: check_version
2021-07-13 00:53:02,114 INFO    WriterThread:327506 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/run-2ilkub1o.wandb
2021-07-13 00:53:02,115 DEBUG   SenderThread:327506 [sender.py:send():179] send: header
2021-07-13 00:53:02,116 DEBUG   SenderThread:327506 [sender.py:send_request():193] send_request: check_version
2021-07-13 00:53:02,154 DEBUG   SenderThread:327506 [sender.py:send():179] send: run
2021-07-13 00:53:02,328 INFO    SenderThread:327506 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files
2021-07-13 00:53:02,329 INFO    SenderThread:327506 [sender.py:_start_run_threads():716] run started: 2ilkub1o with start time 1626137581
2021-07-13 00:53:02,345 DEBUG   SenderThread:327506 [sender.py:send():179] send: summary
2021-07-13 00:53:02,345 INFO    SenderThread:327506 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 00:53:02,346 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: run_start
2021-07-13 00:53:03,330 INFO    Thread-8  :327506 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/wandb-summary.json
2021-07-13 00:53:03,462 DEBUG   HandlerThread:327506 [meta.py:__init__():39] meta init
2021-07-13 00:53:03,462 DEBUG   HandlerThread:327506 [meta.py:__init__():53] meta init done
2021-07-13 00:53:03,462 DEBUG   HandlerThread:327506 [meta.py:probe():210] probe
2021-07-13 00:53:03,463 DEBUG   HandlerThread:327506 [meta.py:_setup_git():200] setup git
2021-07-13 00:53:03,492 DEBUG   HandlerThread:327506 [meta.py:_setup_git():207] setup git done
2021-07-13 00:53:03,492 DEBUG   HandlerThread:327506 [meta.py:_save_pip():57] save pip
2021-07-13 00:53:03,493 DEBUG   HandlerThread:327506 [meta.py:_save_pip():71] save pip done
2021-07-13 00:53:03,493 DEBUG   HandlerThread:327506 [meta.py:probe():252] probe done
2021-07-13 00:53:03,496 DEBUG   SenderThread:327506 [sender.py:send():179] send: files
2021-07-13 00:53:03,496 INFO    SenderThread:327506 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-13 00:53:03,504 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:53:03,504 DEBUG   SenderThread:327506 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:53:03,635 DEBUG   SenderThread:327506 [sender.py:send():179] send: config
2021-07-13 00:53:03,636 DEBUG   SenderThread:327506 [sender.py:send():179] send: config
2021-07-13 00:53:03,636 DEBUG   SenderThread:327506 [sender.py:send():179] send: config
2021-07-13 00:53:03,952 INFO    Thread-11 :327506 [upload_job.py:push():137] Uploaded file /tmp/tmpi8r4kiyhwandb/3l6ji67i-wandb-metadata.json
2021-07-13 00:53:04,330 INFO    Thread-8  :327506 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/wandb-metadata.json
2021-07-13 00:53:04,330 INFO    Thread-8  :327506 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/requirements.txt
2021-07-13 00:53:04,330 INFO    Thread-8  :327506 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/output.log
2021-07-13 00:53:18,637 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:53:18,637 DEBUG   SenderThread:327506 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:53:20,336 INFO    Thread-8  :327506 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/output.log
2021-07-13 00:53:22,336 INFO    Thread-8  :327506 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/output.log
2021-07-13 00:53:31,548 DEBUG   SenderThread:327506 [sender.py:send():179] send: stats
2021-07-13 00:53:33,340 INFO    Thread-8  :327506 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/config.yaml
2021-07-13 00:53:33,769 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:53:33,769 DEBUG   SenderThread:327506 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:53:48,899 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:53:48,899 DEBUG   SenderThread:327506 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:54:01,629 DEBUG   SenderThread:327506 [sender.py:send():179] send: stats
2021-07-13 00:54:04,032 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:54:04,032 DEBUG   SenderThread:327506 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:54:12,355 INFO    Thread-8  :327506 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/output.log
2021-07-13 00:54:19,178 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:54:19,179 DEBUG   SenderThread:327506 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:54:31,708 DEBUG   SenderThread:327506 [sender.py:send():179] send: stats
2021-07-13 00:54:34,599 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:54:34,599 DEBUG   SenderThread:327506 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:54:49,798 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:54:49,798 DEBUG   SenderThread:327506 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:55:01,792 DEBUG   SenderThread:327506 [sender.py:send():179] send: stats
2021-07-13 00:55:04,931 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:55:04,931 DEBUG   SenderThread:327506 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:55:20,062 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:55:20,062 DEBUG   SenderThread:327506 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:55:31,868 DEBUG   SenderThread:327506 [sender.py:send():179] send: stats
2021-07-13 00:55:35,203 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: stop_status
2021-07-13 00:55:35,204 DEBUG   SenderThread:327506 [sender.py:send_request():193] send_request: stop_status
2021-07-13 00:55:44,391 INFO    Thread-8  :327506 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/output.log
2021-07-13 00:55:45,566 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:55:45,567 DEBUG   SenderThread:327506 [sender.py:send():179] send: telemetry
2021-07-13 00:55:45,567 DEBUG   SenderThread:327506 [sender.py:send():179] send: exit
2021-07-13 00:55:45,567 INFO    SenderThread:327506 [sender.py:send_exit():287] handling exit code: 1
2021-07-13 00:55:45,567 INFO    SenderThread:327506 [sender.py:send_exit():295] send defer
2021-07-13 00:55:45,567 DEBUG   SenderThread:327506 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:55:45,568 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:55:45,568 INFO    HandlerThread:327506 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-13 00:55:45,568 DEBUG   SenderThread:327506 [sender.py:send_request():193] send_request: defer
2021-07-13 00:55:45,568 INFO    SenderThread:327506 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-13 00:55:45,569 INFO    SenderThread:327506 [sender.py:transition_state():308] send defer: 1
2021-07-13 00:55:45,569 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:55:45,569 INFO    HandlerThread:327506 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-13 00:55:45,601 DEBUG   SenderThread:327506 [sender.py:send_request():193] send_request: defer
2021-07-13 00:55:45,601 INFO    SenderThread:327506 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-13 00:55:45,601 INFO    SenderThread:327506 [sender.py:transition_state():308] send defer: 2
2021-07-13 00:55:45,602 DEBUG   SenderThread:327506 [sender.py:send():179] send: stats
2021-07-13 00:55:45,602 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:55:45,602 INFO    HandlerThread:327506 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-13 00:55:45,602 DEBUG   SenderThread:327506 [sender.py:send_request():193] send_request: defer
2021-07-13 00:55:45,602 INFO    SenderThread:327506 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-13 00:55:45,602 INFO    SenderThread:327506 [sender.py:transition_state():308] send defer: 3
2021-07-13 00:55:45,603 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:55:45,603 INFO    HandlerThread:327506 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-13 00:55:45,603 DEBUG   SenderThread:327506 [sender.py:send():179] send: summary
2021-07-13 00:55:45,603 INFO    SenderThread:327506 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-13 00:55:45,603 DEBUG   SenderThread:327506 [sender.py:send_request():193] send_request: defer
2021-07-13 00:55:45,604 INFO    SenderThread:327506 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-13 00:55:45,604 INFO    SenderThread:327506 [sender.py:transition_state():308] send defer: 4
2021-07-13 00:55:45,604 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:55:45,604 INFO    HandlerThread:327506 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-13 00:55:45,604 DEBUG   SenderThread:327506 [sender.py:send_request():193] send_request: defer
2021-07-13 00:55:45,604 INFO    SenderThread:327506 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-13 00:55:45,670 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:55:45,784 INFO    SenderThread:327506 [sender.py:transition_state():308] send defer: 5
2021-07-13 00:55:45,784 DEBUG   SenderThread:327506 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:55:45,785 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:55:45,785 INFO    HandlerThread:327506 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-13 00:55:45,785 DEBUG   SenderThread:327506 [sender.py:send_request():193] send_request: defer
2021-07-13 00:55:45,785 INFO    SenderThread:327506 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-13 00:55:45,786 INFO    SenderThread:327506 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-13 00:55:45,887 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:55:46,391 INFO    Thread-8  :327506 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/wandb-summary.json
2021-07-13 00:55:46,392 INFO    SenderThread:327506 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/config.yaml
2021-07-13 00:55:46,392 INFO    SenderThread:327506 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/output.log
2021-07-13 00:55:46,392 INFO    SenderThread:327506 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files
2021-07-13 00:55:46,392 INFO    SenderThread:327506 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/requirements.txt requirements.txt
2021-07-13 00:55:46,392 INFO    SenderThread:327506 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/output.log output.log
2021-07-13 00:55:46,392 INFO    SenderThread:327506 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/wandb-metadata.json wandb-metadata.json
2021-07-13 00:55:46,392 INFO    SenderThread:327506 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/config.yaml config.yaml
2021-07-13 00:55:46,392 INFO    SenderThread:327506 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/wandb-summary.json wandb-summary.json
2021-07-13 00:55:46,393 INFO    SenderThread:327506 [sender.py:transition_state():308] send defer: 6
2021-07-13 00:55:46,393 DEBUG   SenderThread:327506 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:55:46,403 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:55:46,403 INFO    HandlerThread:327506 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-13 00:55:46,405 DEBUG   SenderThread:327506 [sender.py:send_request():193] send_request: defer
2021-07-13 00:55:46,405 INFO    SenderThread:327506 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-13 00:55:46,405 INFO    SenderThread:327506 [file_pusher.py:finish():177] shutting down file pusher
2021-07-13 00:55:46,495 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:55:46,496 DEBUG   SenderThread:327506 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:55:46,598 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:55:46,598 DEBUG   SenderThread:327506 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:55:46,700 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:55:46,700 DEBUG   SenderThread:327506 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:55:46,802 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:55:46,802 DEBUG   SenderThread:327506 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:55:46,867 INFO    Thread-14 :327506 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/config.yaml
2021-07-13 00:55:46,874 INFO    Thread-15 :327506 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/wandb-summary.json
2021-07-13 00:55:46,876 INFO    Thread-13 :327506 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/output.log
2021-07-13 00:55:46,904 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:55:46,905 DEBUG   SenderThread:327506 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:55:46,935 INFO    Thread-12 :327506 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/files/requirements.txt
2021-07-13 00:55:47,007 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:55:47,007 DEBUG   SenderThread:327506 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:55:47,109 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:55:47,109 DEBUG   SenderThread:327506 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:55:47,135 INFO    Thread-7  :327506 [sender.py:transition_state():308] send defer: 7
2021-07-13 00:55:47,136 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:55:47,136 INFO    HandlerThread:327506 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-13 00:55:47,136 DEBUG   SenderThread:327506 [sender.py:send_request():193] send_request: defer
2021-07-13 00:55:47,136 INFO    SenderThread:327506 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-13 00:55:47,211 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:55:47,415 INFO    SenderThread:327506 [sender.py:transition_state():308] send defer: 8
2021-07-13 00:55:47,416 DEBUG   SenderThread:327506 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:55:47,416 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:55:47,416 INFO    HandlerThread:327506 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-13 00:55:47,416 DEBUG   SenderThread:327506 [sender.py:send_request():193] send_request: defer
2021-07-13 00:55:47,417 INFO    SenderThread:327506 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-13 00:55:47,417 INFO    SenderThread:327506 [sender.py:transition_state():308] send defer: 9
2021-07-13 00:55:47,417 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: defer
2021-07-13 00:55:47,417 INFO    HandlerThread:327506 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-13 00:55:47,417 DEBUG   SenderThread:327506 [sender.py:send():179] send: final
2021-07-13 00:55:47,417 DEBUG   SenderThread:327506 [sender.py:send():179] send: footer
2021-07-13 00:55:47,417 DEBUG   SenderThread:327506 [sender.py:send_request():193] send_request: defer
2021-07-13 00:55:47,418 INFO    SenderThread:327506 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-13 00:55:47,518 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-13 00:55:47,518 DEBUG   SenderThread:327506 [sender.py:send_request():193] send_request: poll_exit
2021-07-13 00:55:47,518 INFO    SenderThread:327506 [file_pusher.py:join():182] waiting for file pusher
2021-07-13 00:55:47,520 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: get_summary
2021-07-13 00:55:47,521 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-13 00:55:47,521 DEBUG   HandlerThread:327506 [handler.py:handle_request():124] handle_request: shutdown
2021-07-13 00:55:47,521 INFO    HandlerThread:327506 [handler.py:finish():638] shutting down handler
2021-07-13 00:55:48,418 INFO    WriterThread:327506 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210713_005301-2ilkub1o/run-2ilkub1o.wandb
2021-07-13 00:55:48,518 INFO    SenderThread:327506 [sender.py:finish():945] shutting down sender
2021-07-13 00:55:48,519 INFO    SenderThread:327506 [file_pusher.py:finish():177] shutting down file pusher
2021-07-13 00:55:48,519 INFO    SenderThread:327506 [file_pusher.py:join():182] waiting for file pusher
2021-07-13 00:55:48,521 INFO    MainThread:327506 [internal.py:handle_exit():78] Internal process exited