File size: 26,850 Bytes
9915204 731244f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 |
2021-07-14 22:58:21,337 INFO MainThread:601574 [internal.py:wandb_internal():88] W&B internal server running at pid: 601574, started at: 2021-07-14 22:58:21.336704
2021-07-14 22:58:21,339 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: check_version
2021-07-14 22:58:21,339 INFO WriterThread:601574 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/run-1dpoijkp.wandb
2021-07-14 22:58:21,340 DEBUG SenderThread:601574 [sender.py:send():179] send: header
2021-07-14 22:58:21,340 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: check_version
2021-07-14 22:58:21,377 DEBUG SenderThread:601574 [sender.py:send():179] send: run
2021-07-14 22:58:21,602 INFO SenderThread:601574 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files
2021-07-14 22:58:21,602 INFO SenderThread:601574 [sender.py:_start_run_threads():716] run started: 1dpoijkp with start time 1626303500
2021-07-14 22:58:21,602 DEBUG SenderThread:601574 [sender.py:send():179] send: summary
2021-07-14 22:58:21,603 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: run_start
2021-07-14 22:58:21,603 INFO SenderThread:601574 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 22:58:22,605 INFO Thread-8 :601574 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/wandb-summary.json
2021-07-14 22:58:22,701 DEBUG HandlerThread:601574 [meta.py:__init__():39] meta init
2021-07-14 22:58:22,701 DEBUG HandlerThread:601574 [meta.py:__init__():53] meta init done
2021-07-14 22:58:22,701 DEBUG HandlerThread:601574 [meta.py:probe():210] probe
2021-07-14 22:58:22,702 DEBUG HandlerThread:601574 [meta.py:_setup_git():200] setup git
2021-07-14 22:58:22,732 DEBUG HandlerThread:601574 [meta.py:_setup_git():207] setup git done
2021-07-14 22:58:22,732 DEBUG HandlerThread:601574 [meta.py:_save_pip():57] save pip
2021-07-14 22:58:22,732 DEBUG HandlerThread:601574 [meta.py:_save_pip():71] save pip done
2021-07-14 22:58:22,733 DEBUG HandlerThread:601574 [meta.py:probe():252] probe done
2021-07-14 22:58:22,736 DEBUG SenderThread:601574 [sender.py:send():179] send: files
2021-07-14 22:58:22,736 INFO SenderThread:601574 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-14 22:58:22,742 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:58:22,742 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:58:22,871 DEBUG SenderThread:601574 [sender.py:send():179] send: config
2021-07-14 22:58:22,872 DEBUG SenderThread:601574 [sender.py:send():179] send: config
2021-07-14 22:58:22,872 DEBUG SenderThread:601574 [sender.py:send():179] send: config
2021-07-14 22:58:23,214 INFO Thread-11 :601574 [upload_job.py:push():137] Uploaded file /tmp/tmpg5fs3m8gwandb/35jlequ6-wandb-metadata.json
2021-07-14 22:58:23,603 INFO Thread-8 :601574 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/output.log
2021-07-14 22:58:23,604 INFO Thread-8 :601574 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/requirements.txt
2021-07-14 22:58:23,604 INFO Thread-8 :601574 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/wandb-metadata.json
2021-07-14 22:58:37,609 INFO Thread-8 :601574 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/output.log
2021-07-14 22:58:37,873 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:58:37,874 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:58:50,784 DEBUG SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 22:58:52,614 INFO Thread-8 :601574 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/config.yaml
2021-07-14 22:58:53,006 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:58:53,006 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:59:08,141 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:59:08,141 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:59:20,866 DEBUG SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 22:59:23,276 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:59:23,277 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:59:38,415 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:59:38,415 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 22:59:50,943 DEBUG SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 22:59:53,547 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 22:59:53,547 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:00:08,680 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:00:08,680 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:00:21,020 DEBUG SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 23:00:23,810 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:00:23,811 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:00:38,944 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:00:38,945 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:00:51,098 DEBUG SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 23:00:54,080 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:00:54,080 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:01:09,212 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:01:09,212 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:01:21,176 DEBUG SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 23:01:24,345 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:01:24,346 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:01:39,477 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:01:39,478 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:01:51,254 DEBUG SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 23:01:54,612 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:01:54,612 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:02:09,744 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:02:09,744 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:02:21,332 DEBUG SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 23:02:24,932 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:02:24,932 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:02:40,066 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:02:40,067 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:02:51,409 DEBUG SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 23:02:55,209 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:02:55,209 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:03:10,341 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:03:10,341 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:03:21,484 DEBUG SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 23:03:25,483 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:03:25,483 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:03:40,615 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:03:40,615 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:03:51,547 DEBUG SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 23:03:56,377 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:03:56,377 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:04:11,520 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:04:11,520 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:04:21,610 DEBUG SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 23:04:26,652 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:04:26,652 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:04:41,781 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:04:41,781 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:04:51,676 DEBUG SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 23:04:56,913 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:04:56,913 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:05:13,026 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:05:13,027 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:05:21,744 DEBUG SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 23:05:28,237 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:05:28,238 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:05:44,049 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:05:44,049 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:05:51,809 DEBUG SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 23:05:59,179 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:05:59,180 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:06:14,311 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:06:14,311 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:06:21,764 INFO Thread-8 :601574 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/output.log
2021-07-14 23:06:21,877 DEBUG SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 23:06:29,456 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:06:29,457 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:06:44,749 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:06:44,749 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:06:51,950 DEBUG SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 23:06:59,941 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:06:59,941 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:07:15,073 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:07:15,077 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:07:22,025 DEBUG SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 23:07:30,221 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:07:30,221 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:07:45,363 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 23:07:45,364 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: stop_status
2021-07-14 23:07:50,438 DEBUG SenderThread:601574 [sender.py:send():179] send: telemetry
2021-07-14 23:07:50,439 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:07:50,439 DEBUG SenderThread:601574 [sender.py:send():179] send: exit
2021-07-14 23:07:50,439 INFO SenderThread:601574 [sender.py:send_exit():287] handling exit code: 1
2021-07-14 23:07:50,440 INFO SenderThread:601574 [sender.py:send_exit():295] send defer
2021-07-14 23:07:50,440 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:07:50,441 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:07:50,441 INFO HandlerThread:601574 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-14 23:07:50,441 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: defer
2021-07-14 23:07:50,441 INFO SenderThread:601574 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-14 23:07:50,441 INFO SenderThread:601574 [sender.py:transition_state():308] send defer: 1
2021-07-14 23:07:50,442 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:07:50,442 INFO HandlerThread:601574 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-14 23:07:50,497 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: defer
2021-07-14 23:07:50,497 INFO SenderThread:601574 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-14 23:07:50,497 INFO SenderThread:601574 [sender.py:transition_state():308] send defer: 2
2021-07-14 23:07:50,497 DEBUG SenderThread:601574 [sender.py:send():179] send: stats
2021-07-14 23:07:50,498 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:07:50,498 INFO HandlerThread:601574 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-14 23:07:50,498 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: defer
2021-07-14 23:07:50,498 INFO SenderThread:601574 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-14 23:07:50,498 INFO SenderThread:601574 [sender.py:transition_state():308] send defer: 3
2021-07-14 23:07:50,499 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:07:50,499 INFO HandlerThread:601574 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-14 23:07:50,499 DEBUG SenderThread:601574 [sender.py:send():179] send: summary
2021-07-14 23:07:50,499 INFO SenderThread:601574 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 23:07:50,499 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: defer
2021-07-14 23:07:50,499 INFO SenderThread:601574 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-14 23:07:50,499 INFO SenderThread:601574 [sender.py:transition_state():308] send defer: 4
2021-07-14 23:07:50,500 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:07:50,500 INFO HandlerThread:601574 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-14 23:07:50,500 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: defer
2021-07-14 23:07:50,500 INFO SenderThread:601574 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-14 23:07:50,543 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:07:50,746 INFO SenderThread:601574 [sender.py:transition_state():308] send defer: 5
2021-07-14 23:07:50,746 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:07:50,747 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:07:50,747 INFO HandlerThread:601574 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-14 23:07:50,747 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: defer
2021-07-14 23:07:50,747 INFO SenderThread:601574 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-14 23:07:50,747 INFO SenderThread:601574 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-14 23:07:50,787 INFO SenderThread:601574 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/output.log
2021-07-14 23:07:50,787 INFO SenderThread:601574 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/config.yaml
2021-07-14 23:07:50,787 INFO SenderThread:601574 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/wandb-summary.json
2021-07-14 23:07:50,788 INFO SenderThread:601574 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files
2021-07-14 23:07:50,788 INFO SenderThread:601574 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/requirements.txt requirements.txt
2021-07-14 23:07:50,788 INFO SenderThread:601574 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/output.log output.log
2021-07-14 23:07:50,788 INFO SenderThread:601574 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/wandb-metadata.json wandb-metadata.json
2021-07-14 23:07:50,788 INFO SenderThread:601574 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/config.yaml config.yaml
2021-07-14 23:07:50,789 INFO SenderThread:601574 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/wandb-summary.json wandb-summary.json
2021-07-14 23:07:50,792 INFO SenderThread:601574 [sender.py:transition_state():308] send defer: 6
2021-07-14 23:07:50,792 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:07:50,792 INFO HandlerThread:601574 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-14 23:07:50,795 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: defer
2021-07-14 23:07:50,795 INFO SenderThread:601574 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-14 23:07:50,795 INFO SenderThread:601574 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 23:07:50,848 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:07:50,849 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:07:50,951 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:07:50,951 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:07:51,053 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:07:51,053 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:07:51,155 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:07:51,155 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:07:51,257 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:07:51,257 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:07:51,284 INFO Thread-12 :601574 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/requirements.txt
2021-07-14 23:07:51,311 INFO Thread-14 :601574 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/config.yaml
2021-07-14 23:07:51,322 INFO Thread-13 :601574 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/output.log
2021-07-14 23:07:51,359 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:07:51,359 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:07:51,461 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:07:51,461 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:07:51,466 INFO Thread-15 :601574 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/files/wandb-summary.json
2021-07-14 23:07:51,563 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:07:51,563 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:07:51,665 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:07:51,665 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:07:51,666 INFO Thread-7 :601574 [sender.py:transition_state():308] send defer: 7
2021-07-14 23:07:51,667 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:07:51,667 INFO HandlerThread:601574 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-14 23:07:51,667 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: defer
2021-07-14 23:07:51,667 INFO SenderThread:601574 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-14 23:07:51,766 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:07:51,941 INFO SenderThread:601574 [sender.py:transition_state():308] send defer: 8
2021-07-14 23:07:51,941 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:07:51,942 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:07:51,942 INFO HandlerThread:601574 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-14 23:07:51,942 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: defer
2021-07-14 23:07:51,942 INFO SenderThread:601574 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-14 23:07:51,942 INFO SenderThread:601574 [sender.py:transition_state():308] send defer: 9
2021-07-14 23:07:51,942 DEBUG SenderThread:601574 [sender.py:send():179] send: final
2021-07-14 23:07:51,942 DEBUG SenderThread:601574 [sender.py:send():179] send: footer
2021-07-14 23:07:51,943 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: defer
2021-07-14 23:07:51,943 INFO HandlerThread:601574 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-14 23:07:51,943 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: defer
2021-07-14 23:07:51,943 INFO SenderThread:601574 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-14 23:07:52,043 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 23:07:52,043 DEBUG SenderThread:601574 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 23:07:52,044 INFO SenderThread:601574 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 23:07:52,045 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: get_summary
2021-07-14 23:07:52,045 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-14 23:07:52,046 DEBUG HandlerThread:601574 [handler.py:handle_request():124] handle_request: shutdown
2021-07-14 23:07:52,046 INFO HandlerThread:601574 [handler.py:finish():638] shutting down handler
2021-07-14 23:07:52,943 INFO WriterThread:601574 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_225820-1dpoijkp/run-1dpoijkp.wandb
2021-07-14 23:07:53,044 INFO SenderThread:601574 [sender.py:finish():945] shutting down sender
2021-07-14 23:07:53,044 INFO SenderThread:601574 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 23:07:53,044 INFO SenderThread:601574 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 23:07:53,047 INFO MainThread:601574 [internal.py:handle_exit():78] Internal process exited
|