File size: 22,077 Bytes
9c3de9e
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
2022-01-14 21:28:56,265 INFO    MainThread:8253 [internal.py:wandb_internal():87] W&B internal server running at pid: 8253, started at: 2022-01-14 21:28:56.265129
2022-01-14 21:28:56,268 DEBUG   SenderThread:8253 [sender.py:send():234] send: header
2022-01-14 21:28:56,268 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: check_version
2022-01-14 21:28:56,268 INFO    WriterThread:8253 [datastore.py:open_for_write():77] open: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/run-32qdb4k5.wandb
2022-01-14 21:28:56,268 DEBUG   SenderThread:8253 [sender.py:send_request():248] send_request: check_version
2022-01-14 21:28:56,352 DEBUG   SenderThread:8253 [sender.py:send():234] send: run
2022-01-14 21:28:56,515 INFO    SenderThread:8253 [dir_watcher.py:__init__():169] watching files in: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files
2022-01-14 21:28:56,515 INFO    SenderThread:8253 [sender.py:_start_run_threads():804] run started: 32qdb4k5 with start time 1642195735
2022-01-14 21:28:56,515 DEBUG   SenderThread:8253 [sender.py:send():234] send: summary
2022-01-14 21:28:56,515 INFO    SenderThread:8253 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-14 21:28:56,515 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: run_start
2022-01-14 21:28:57,561 INFO    Thread-8  :8253 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/wandb-summary.json
2022-01-14 21:28:58,974 DEBUG   HandlerThread:8253 [meta.py:__init__():40] meta init
2022-01-14 21:28:58,974 DEBUG   HandlerThread:8253 [meta.py:__init__():54] meta init done
2022-01-14 21:28:58,974 DEBUG   HandlerThread:8253 [meta.py:probe():214] probe
2022-01-14 21:28:58,975 DEBUG   HandlerThread:8253 [meta.py:_setup_git():204] setup git
2022-01-14 21:28:59,006 DEBUG   HandlerThread:8253 [meta.py:_setup_git():211] setup git done
2022-01-14 21:28:59,006 DEBUG   HandlerThread:8253 [meta.py:_save_code():92] save code
2022-01-14 21:28:59,018 DEBUG   HandlerThread:8253 [meta.py:_save_code():113] save code done
2022-01-14 21:28:59,018 DEBUG   HandlerThread:8253 [meta.py:_save_patches():130] save patches
2022-01-14 21:28:59,561 INFO    Thread-8  :8253 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/code/run_mlm_flax.py
2022-01-14 21:28:59,562 INFO    Thread-8  :8253 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/code
2022-01-14 21:29:01,562 INFO    Thread-8  :8253 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/output.log
2022-01-14 21:29:03,563 INFO    Thread-8  :8253 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/output.log
2022-01-14 21:29:03,563 INFO    Thread-8  :8253 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/diff.patch
2022-01-14 21:29:08,274 ERROR   HandlerThread:8253 [meta.py:_save_patches():171] Error generating diff: Command '['git', 'diff', '--submodule=diff', 'HEAD']' timed out after 5 seconds
2022-01-14 21:29:08,274 DEBUG   HandlerThread:8253 [meta.py:_save_patches():172] save patches done
2022-01-14 21:29:08,274 DEBUG   HandlerThread:8253 [meta.py:_save_pip():58] save pip
2022-01-14 21:29:08,275 DEBUG   HandlerThread:8253 [meta.py:_save_pip():72] save pip done
2022-01-14 21:29:08,275 DEBUG   HandlerThread:8253 [meta.py:probe():252] probe done
2022-01-14 21:29:08,283 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 21:29:08,284 DEBUG   SenderThread:8253 [sender.py:send_request():248] send_request: stop_status
2022-01-14 21:29:08,566 DEBUG   SenderThread:8253 [sender.py:send():234] send: config
2022-01-14 21:29:08,566 DEBUG   SenderThread:8253 [sender.py:send():234] send: config
2022-01-14 21:29:08,566 DEBUG   SenderThread:8253 [sender.py:send():234] send: config
2022-01-14 21:29:08,567 DEBUG   SenderThread:8253 [sender.py:send():234] send: files
2022-01-14 21:29:08,567 INFO    SenderThread:8253 [sender.py:_save_file():939] saving file wandb-metadata.json with policy now
2022-01-14 21:29:08,567 INFO    SenderThread:8253 [sender.py:_save_file():939] saving file code/run_mlm_flax.py with policy now
2022-01-14 21:29:08,571 INFO    Thread-8  :8253 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/requirements.txt
2022-01-14 21:29:08,571 INFO    Thread-8  :8253 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/wandb-metadata.json
2022-01-14 21:29:09,032 INFO    Thread-12 :8253 [upload_job.py:push():137] Uploaded file /tmp/tmpdg54qv_0wandb/w1tibuxq-code/run_mlm_flax.py
2022-01-14 21:29:09,069 INFO    Thread-11 :8253 [upload_job.py:push():137] Uploaded file /tmp/tmpdg54qv_0wandb/35h4ryp5-wandb-metadata.json
2022-01-14 21:29:09,571 INFO    Thread-8  :8253 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/output.log
2022-01-14 21:29:21,520 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 21:29:21,521 DEBUG   SenderThread:8253 [sender.py:send_request():248] send_request: stop_status
2022-01-14 21:29:27,049 DEBUG   SenderThread:8253 [sender.py:send():234] send: stats
2022-01-14 21:29:27,579 INFO    Thread-8  :8253 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/config.yaml
2022-01-14 21:29:36,656 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 21:29:36,657 DEBUG   SenderThread:8253 [sender.py:send_request():248] send_request: stop_status
2022-01-14 21:29:51,794 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 21:29:51,795 DEBUG   SenderThread:8253 [sender.py:send_request():248] send_request: stop_status
2022-01-14 21:29:57,119 DEBUG   SenderThread:8253 [sender.py:send():234] send: stats
2022-01-14 21:29:58,591 INFO    Thread-8  :8253 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/output.log
2022-01-14 21:30:06,945 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 21:30:06,945 DEBUG   SenderThread:8253 [sender.py:send_request():248] send_request: stop_status
2022-01-14 21:30:22,126 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 21:30:22,127 DEBUG   SenderThread:8253 [sender.py:send_request():248] send_request: stop_status
2022-01-14 21:30:27,189 DEBUG   SenderThread:8253 [sender.py:send():234] send: stats
2022-01-14 21:30:37,330 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 21:30:37,330 DEBUG   SenderThread:8253 [sender.py:send_request():248] send_request: stop_status
2022-01-14 21:30:52,532 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 21:30:52,532 DEBUG   SenderThread:8253 [sender.py:send_request():248] send_request: stop_status
2022-01-14 21:30:57,257 DEBUG   SenderThread:8253 [sender.py:send():234] send: stats
2022-01-14 21:31:07,691 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 21:31:07,692 DEBUG   SenderThread:8253 [sender.py:send_request():248] send_request: stop_status
2022-01-14 21:31:22,944 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 21:31:22,945 DEBUG   SenderThread:8253 [sender.py:send_request():248] send_request: stop_status
2022-01-14 21:31:27,323 DEBUG   SenderThread:8253 [sender.py:send():234] send: stats
2022-01-14 21:31:38,085 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 21:31:38,086 DEBUG   SenderThread:8253 [sender.py:send_request():248] send_request: stop_status
2022-01-14 21:31:53,231 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 21:31:53,231 DEBUG   SenderThread:8253 [sender.py:send_request():248] send_request: stop_status
2022-01-14 21:31:57,395 DEBUG   SenderThread:8253 [sender.py:send():234] send: stats
2022-01-14 21:32:08,366 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 21:32:08,367 DEBUG   SenderThread:8253 [sender.py:send_request():248] send_request: stop_status
2022-01-14 21:32:16,649 INFO    Thread-8  :8253 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/output.log
2022-01-14 21:32:16,893 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 21:32:16,893 DEBUG   SenderThread:8253 [sender.py:send():234] send: telemetry
2022-01-14 21:32:16,893 DEBUG   SenderThread:8253 [sender.py:send():234] send: exit
2022-01-14 21:32:16,893 INFO    SenderThread:8253 [sender.py:send_exit():366] handling exit code: 1
2022-01-14 21:32:16,894 INFO    SenderThread:8253 [sender.py:send_exit():368] handling runtime: 200
2022-01-14 21:32:16,894 INFO    SenderThread:8253 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-14 21:32:16,894 INFO    SenderThread:8253 [sender.py:send_exit():374] send defer
2022-01-14 21:32:16,894 DEBUG   SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 21:32:16,895 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer
2022-01-14 21:32:16,895 INFO    HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 0
2022-01-14 21:32:16,895 DEBUG   SenderThread:8253 [sender.py:send_request():248] send_request: defer
2022-01-14 21:32:16,895 INFO    SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 0
2022-01-14 21:32:16,895 INFO    SenderThread:8253 [sender.py:transition_state():387] send defer: 1
2022-01-14 21:32:16,896 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer
2022-01-14 21:32:16,896 INFO    HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 1
2022-01-14 21:32:16,941 DEBUG   SenderThread:8253 [sender.py:send_request():248] send_request: defer
2022-01-14 21:32:16,941 INFO    SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 1
2022-01-14 21:32:16,941 INFO    SenderThread:8253 [sender.py:transition_state():387] send defer: 2
2022-01-14 21:32:16,941 DEBUG   SenderThread:8253 [sender.py:send():234] send: stats
2022-01-14 21:32:16,942 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer
2022-01-14 21:32:16,942 INFO    HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 2
2022-01-14 21:32:16,942 DEBUG   SenderThread:8253 [sender.py:send_request():248] send_request: defer
2022-01-14 21:32:16,942 INFO    SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 2
2022-01-14 21:32:16,942 INFO    SenderThread:8253 [sender.py:transition_state():387] send defer: 3
2022-01-14 21:32:16,942 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer
2022-01-14 21:32:16,942 INFO    HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 3
2022-01-14 21:32:16,943 DEBUG   SenderThread:8253 [sender.py:send():234] send: summary
2022-01-14 21:32:16,943 INFO    SenderThread:8253 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-14 21:32:16,943 DEBUG   SenderThread:8253 [sender.py:send_request():248] send_request: defer
2022-01-14 21:32:16,943 INFO    SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 3
2022-01-14 21:32:16,943 INFO    SenderThread:8253 [sender.py:transition_state():387] send defer: 4
2022-01-14 21:32:16,943 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer
2022-01-14 21:32:16,943 INFO    HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 4
2022-01-14 21:32:16,944 DEBUG   SenderThread:8253 [sender.py:send_request():248] send_request: defer
2022-01-14 21:32:16,944 INFO    SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 4
2022-01-14 21:32:16,997 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 21:32:17,650 INFO    Thread-8  :8253 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/output.log
2022-01-14 21:32:17,650 INFO    Thread-8  :8253 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/wandb-summary.json
2022-01-14 21:32:17,685 INFO    SenderThread:8253 [sender.py:transition_state():387] send defer: 5
2022-01-14 21:32:17,686 DEBUG   SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 21:32:17,686 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer
2022-01-14 21:32:17,686 INFO    HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 5
2022-01-14 21:32:17,686 DEBUG   SenderThread:8253 [sender.py:send_request():248] send_request: defer
2022-01-14 21:32:17,686 INFO    SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 5
2022-01-14 21:32:17,687 INFO    SenderThread:8253 [dir_watcher.py:finish():283] shutting down directory watcher
2022-01-14 21:32:17,787 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 21:32:18,650 INFO    Thread-8  :8253 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/config.yaml
2022-01-14 21:32:18,651 INFO    SenderThread:8253 [dir_watcher.py:finish():313] scan: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files
2022-01-14 21:32:18,651 INFO    SenderThread:8253 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/config.yaml config.yaml
2022-01-14 21:32:18,651 INFO    SenderThread:8253 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/diff.patch diff.patch
2022-01-14 21:32:18,651 INFO    SenderThread:8253 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/requirements.txt requirements.txt
2022-01-14 21:32:18,652 INFO    SenderThread:8253 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/output.log output.log
2022-01-14 21:32:18,652 INFO    SenderThread:8253 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/wandb-summary.json wandb-summary.json
2022-01-14 21:32:18,652 INFO    SenderThread:8253 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/wandb-metadata.json wandb-metadata.json
2022-01-14 21:32:18,656 INFO    SenderThread:8253 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/code/run_mlm_flax.py code/run_mlm_flax.py
2022-01-14 21:32:18,656 INFO    SenderThread:8253 [sender.py:transition_state():387] send defer: 6
2022-01-14 21:32:18,656 DEBUG   SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 21:32:18,657 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer
2022-01-14 21:32:18,657 INFO    HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 6
2022-01-14 21:32:18,662 DEBUG   SenderThread:8253 [sender.py:send_request():248] send_request: defer
2022-01-14 21:32:18,665 INFO    SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 6
2022-01-14 21:32:18,665 INFO    SenderThread:8253 [file_pusher.py:finish():177] shutting down file pusher
2022-01-14 21:32:18,757 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 21:32:18,758 DEBUG   SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 21:32:18,859 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 21:32:18,860 DEBUG   SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 21:32:18,961 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 21:32:18,962 DEBUG   SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 21:32:19,063 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 21:32:19,064 DEBUG   SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 21:32:19,139 INFO    Thread-15 :8253 [upload_job.py:push():137] Uploaded file /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/output.log
2022-01-14 21:32:19,148 INFO    Thread-14 :8253 [upload_job.py:push():137] Uploaded file /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/requirements.txt
2022-01-14 21:32:19,165 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 21:32:19,165 DEBUG   SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 21:32:19,171 INFO    Thread-13 :8253 [upload_job.py:push():137] Uploaded file /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/config.yaml
2022-01-14 21:32:19,267 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 21:32:19,267 DEBUG   SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 21:32:19,288 INFO    Thread-16 :8253 [upload_job.py:push():137] Uploaded file /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/wandb-summary.json
2022-01-14 21:32:19,370 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 21:32:19,370 DEBUG   SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 21:32:19,472 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 21:32:19,472 DEBUG   SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 21:32:19,489 INFO    Thread-7  :8253 [sender.py:transition_state():387] send defer: 7
2022-01-14 21:32:19,489 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer
2022-01-14 21:32:19,490 INFO    HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 7
2022-01-14 21:32:19,490 DEBUG   SenderThread:8253 [sender.py:send_request():248] send_request: defer
2022-01-14 21:32:19,490 INFO    SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 7
2022-01-14 21:32:19,573 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 21:32:19,915 INFO    SenderThread:8253 [sender.py:transition_state():387] send defer: 8
2022-01-14 21:32:19,916 DEBUG   SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 21:32:19,916 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer
2022-01-14 21:32:19,916 INFO    HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 8
2022-01-14 21:32:19,917 DEBUG   SenderThread:8253 [sender.py:send_request():248] send_request: defer
2022-01-14 21:32:19,917 INFO    SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 8
2022-01-14 21:32:19,917 INFO    SenderThread:8253 [sender.py:transition_state():387] send defer: 9
2022-01-14 21:32:19,917 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer
2022-01-14 21:32:19,917 INFO    HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 9
2022-01-14 21:32:19,918 DEBUG   SenderThread:8253 [sender.py:send():234] send: final
2022-01-14 21:32:19,918 DEBUG   SenderThread:8253 [sender.py:send():234] send: footer
2022-01-14 21:32:19,918 DEBUG   SenderThread:8253 [sender.py:send_request():248] send_request: defer
2022-01-14 21:32:19,918 INFO    SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 9
2022-01-14 21:32:20,017 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 21:32:20,018 DEBUG   SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 21:32:20,018 INFO    SenderThread:8253 [file_pusher.py:join():182] waiting for file pusher
2022-01-14 21:32:20,278 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: get_summary
2022-01-14 21:32:20,278 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: sampled_history
2022-01-14 21:32:20,279 DEBUG   HandlerThread:8253 [handler.py:handle_request():130] handle_request: shutdown
2022-01-14 21:32:20,279 INFO    HandlerThread:8253 [handler.py:finish():731] shutting down handler
2022-01-14 21:32:20,918 INFO    WriterThread:8253 [datastore.py:close():281] close: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/run-32qdb4k5.wandb
2022-01-14 21:32:21,277 INFO    SenderThread:8253 [sender.py:finish():1070] shutting down sender
2022-01-14 21:32:21,277 INFO    SenderThread:8253 [file_pusher.py:finish():177] shutting down file pusher
2022-01-14 21:32:21,277 INFO    SenderThread:8253 [file_pusher.py:join():182] waiting for file pusher
2022-01-14 21:32:21,279 INFO    MainThread:8253 [internal.py:handle_exit():77] Internal process exited