File size: 27,099 Bytes
f291f93 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 |
2021-07-14 17:55:20,938 INFO MainThread:545287 [internal.py:wandb_internal():88] W&B internal server running at pid: 545287, started at: 2021-07-14 17:55:20.938091
2021-07-14 17:55:20,940 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: check_version
2021-07-14 17:55:20,941 INFO WriterThread:545287 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/run-t843i2it.wandb
2021-07-14 17:55:20,942 DEBUG SenderThread:545287 [sender.py:send():179] send: header
2021-07-14 17:55:20,942 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: check_version
2021-07-14 17:55:20,985 DEBUG SenderThread:545287 [sender.py:send():179] send: run
2021-07-14 17:55:21,157 INFO SenderThread:545287 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files
2021-07-14 17:55:21,158 INFO SenderThread:545287 [sender.py:_start_run_threads():716] run started: t843i2it with start time 1626285318
2021-07-14 17:55:21,158 DEBUG SenderThread:545287 [sender.py:send():179] send: summary
2021-07-14 17:55:21,158 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: run_start
2021-07-14 17:55:21,159 INFO SenderThread:545287 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 17:55:24,792 INFO Thread-8 :545287 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/wandb-summary.json
2021-07-14 17:55:26,883 INFO Thread-8 :545287 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/output.log
2021-07-14 17:55:32,372 DEBUG HandlerThread:545287 [meta.py:__init__():39] meta init
2021-07-14 17:55:32,372 DEBUG HandlerThread:545287 [meta.py:__init__():53] meta init done
2021-07-14 17:55:32,372 DEBUG HandlerThread:545287 [meta.py:probe():210] probe
2021-07-14 17:55:32,373 DEBUG HandlerThread:545287 [meta.py:_setup_git():200] setup git
2021-07-14 17:55:32,402 DEBUG HandlerThread:545287 [meta.py:_setup_git():207] setup git done
2021-07-14 17:55:32,402 DEBUG HandlerThread:545287 [meta.py:_save_pip():57] save pip
2021-07-14 17:55:32,403 DEBUG HandlerThread:545287 [meta.py:_save_pip():71] save pip done
2021-07-14 17:55:32,403 DEBUG HandlerThread:545287 [meta.py:probe():252] probe done
2021-07-14 17:55:32,406 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 17:55:32,406 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 17:55:32,544 DEBUG SenderThread:545287 [sender.py:send():179] send: config
2021-07-14 17:55:32,545 DEBUG SenderThread:545287 [sender.py:send():179] send: config
2021-07-14 17:55:32,545 DEBUG SenderThread:545287 [sender.py:send():179] send: config
2021-07-14 17:55:32,545 DEBUG SenderThread:545287 [sender.py:send():179] send: files
2021-07-14 17:55:32,545 INFO SenderThread:545287 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-14 17:55:32,952 INFO Thread-8 :545287 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/wandb-metadata.json
2021-07-14 17:55:32,953 INFO Thread-8 :545287 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/requirements.txt
2021-07-14 17:55:33,181 INFO Thread-11 :545287 [upload_job.py:push():137] Uploaded file /tmp/tmpldoi9p0fwandb/1bn7qo44-wandb-metadata.json
2021-07-14 17:55:42,956 INFO Thread-8 :545287 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/output.log
2021-07-14 17:55:44,957 INFO Thread-8 :545287 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/output.log
2021-07-14 17:55:47,548 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 17:55:47,549 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 17:55:51,960 INFO Thread-8 :545287 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/config.yaml
2021-07-14 17:56:00,458 DEBUG SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 17:56:02,681 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 17:56:02,681 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 17:56:17,816 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 17:56:17,816 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 17:56:30,528 DEBUG SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 17:56:32,958 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 17:56:32,959 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 17:56:48,094 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 17:56:48,094 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 17:57:00,620 DEBUG SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 17:57:03,228 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 17:57:03,229 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 17:57:18,364 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 17:57:18,364 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 17:57:30,693 DEBUG SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 17:57:33,502 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 17:57:33,502 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 17:57:48,640 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 17:57:48,640 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 17:58:00,768 DEBUG SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 17:58:03,776 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 17:58:03,776 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 17:58:18,928 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 17:58:18,928 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 17:58:30,841 DEBUG SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 17:58:34,063 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 17:58:34,063 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 17:58:49,196 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 17:58:49,196 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 17:59:00,919 DEBUG SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 17:59:04,329 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 17:59:04,330 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 17:59:19,466 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 17:59:19,467 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 17:59:30,993 DEBUG SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 17:59:34,601 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 17:59:34,601 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 17:59:49,736 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 17:59:49,736 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:00:01,070 DEBUG SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 18:00:04,866 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:00:04,867 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:00:20,000 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:00:20,001 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:00:31,138 DEBUG SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 18:00:35,144 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:00:35,144 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:00:50,279 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:00:50,280 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:01:01,215 DEBUG SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 18:01:05,414 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:01:05,414 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:01:20,564 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:01:20,565 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:01:31,294 DEBUG SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 18:01:35,699 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:01:35,699 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:01:50,829 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:01:50,830 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:02:01,371 DEBUG SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 18:02:05,966 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:02:05,967 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:02:21,099 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:02:21,099 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:02:31,448 DEBUG SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 18:02:36,234 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:02:36,235 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:02:51,370 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:02:51,370 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:03:01,521 DEBUG SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 18:03:06,821 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:03:06,821 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:03:22,036 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:03:22,036 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:03:31,600 DEBUG SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 18:03:33,147 INFO Thread-8 :545287 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/output.log
2021-07-14 18:03:37,189 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:03:37,190 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:03:52,336 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:03:52,337 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:04:01,678 DEBUG SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 18:04:07,470 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:04:07,470 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:04:22,603 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:04:22,604 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:04:31,760 DEBUG SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 18:04:37,735 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:04:37,735 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:04:52,873 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:04:52,873 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:05:01,843 DEBUG SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 18:05:08,009 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: stop_status
2021-07-14 18:05:08,010 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: stop_status
2021-07-14 18:05:21,189 INFO Thread-8 :545287 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/output.log
2021-07-14 18:05:22,751 DEBUG SenderThread:545287 [sender.py:send():179] send: telemetry
2021-07-14 18:05:22,752 DEBUG SenderThread:545287 [sender.py:send():179] send: exit
2021-07-14 18:05:22,752 INFO SenderThread:545287 [sender.py:send_exit():287] handling exit code: 1
2021-07-14 18:05:22,753 INFO SenderThread:545287 [sender.py:send_exit():295] send defer
2021-07-14 18:05:22,753 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:05:22,754 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:05:22,754 INFO HandlerThread:545287 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-14 18:05:22,754 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:05:22,754 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: defer
2021-07-14 18:05:22,754 INFO SenderThread:545287 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-14 18:05:22,754 INFO SenderThread:545287 [sender.py:transition_state():308] send defer: 1
2021-07-14 18:05:22,755 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:05:22,755 INFO HandlerThread:545287 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-14 18:05:22,795 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: defer
2021-07-14 18:05:22,795 INFO SenderThread:545287 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-14 18:05:22,795 INFO SenderThread:545287 [sender.py:transition_state():308] send defer: 2
2021-07-14 18:05:22,796 DEBUG SenderThread:545287 [sender.py:send():179] send: stats
2021-07-14 18:05:22,796 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:05:22,796 INFO HandlerThread:545287 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-14 18:05:22,796 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: defer
2021-07-14 18:05:22,797 INFO SenderThread:545287 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-14 18:05:22,797 INFO SenderThread:545287 [sender.py:transition_state():308] send defer: 3
2021-07-14 18:05:22,797 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:05:22,797 INFO HandlerThread:545287 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-14 18:05:22,797 DEBUG SenderThread:545287 [sender.py:send():179] send: summary
2021-07-14 18:05:22,797 INFO SenderThread:545287 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-14 18:05:22,798 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: defer
2021-07-14 18:05:22,798 INFO SenderThread:545287 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-14 18:05:22,798 INFO SenderThread:545287 [sender.py:transition_state():308] send defer: 4
2021-07-14 18:05:22,798 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:05:22,798 INFO HandlerThread:545287 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-14 18:05:22,798 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: defer
2021-07-14 18:05:22,798 INFO SenderThread:545287 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-14 18:05:22,856 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:05:22,985 INFO SenderThread:545287 [sender.py:transition_state():308] send defer: 5
2021-07-14 18:05:22,985 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:05:22,986 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:05:22,986 INFO HandlerThread:545287 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-14 18:05:22,986 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: defer
2021-07-14 18:05:22,986 INFO SenderThread:545287 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-14 18:05:22,986 INFO SenderThread:545287 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-14 18:05:23,087 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:05:23,190 INFO Thread-8 :545287 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/output.log
2021-07-14 18:05:23,190 INFO SenderThread:545287 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/wandb-summary.json
2021-07-14 18:05:23,190 INFO SenderThread:545287 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/config.yaml
2021-07-14 18:05:23,191 INFO SenderThread:545287 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files
2021-07-14 18:05:23,191 INFO SenderThread:545287 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/requirements.txt requirements.txt
2021-07-14 18:05:23,191 INFO SenderThread:545287 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/output.log output.log
2021-07-14 18:05:23,195 INFO SenderThread:545287 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/wandb-metadata.json wandb-metadata.json
2021-07-14 18:05:23,195 INFO SenderThread:545287 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/config.yaml config.yaml
2021-07-14 18:05:23,195 INFO SenderThread:545287 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/wandb-summary.json wandb-summary.json
2021-07-14 18:05:23,195 INFO SenderThread:545287 [sender.py:transition_state():308] send defer: 6
2021-07-14 18:05:23,195 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:05:23,196 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:05:23,196 INFO HandlerThread:545287 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-14 18:05:23,197 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: defer
2021-07-14 18:05:23,197 INFO SenderThread:545287 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-14 18:05:23,197 INFO SenderThread:545287 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 18:05:23,299 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:05:23,299 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:05:23,401 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:05:23,401 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:05:23,503 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:05:23,503 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:05:23,605 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:05:23,605 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:05:23,648 INFO Thread-12 :545287 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/requirements.txt
2021-07-14 18:05:23,655 INFO Thread-13 :545287 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/output.log
2021-07-14 18:05:23,658 INFO Thread-15 :545287 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/wandb-summary.json
2021-07-14 18:05:23,661 INFO Thread-14 :545287 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/files/config.yaml
2021-07-14 18:05:23,708 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:05:23,708 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:05:23,810 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:05:23,810 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:05:23,861 INFO Thread-7 :545287 [sender.py:transition_state():308] send defer: 7
2021-07-14 18:05:23,862 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:05:23,862 INFO HandlerThread:545287 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-14 18:05:23,862 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: defer
2021-07-14 18:05:23,862 INFO SenderThread:545287 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-14 18:05:23,912 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:05:24,152 INFO SenderThread:545287 [sender.py:transition_state():308] send defer: 8
2021-07-14 18:05:24,152 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:05:24,153 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:05:24,153 INFO HandlerThread:545287 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-14 18:05:24,154 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: defer
2021-07-14 18:05:24,154 INFO SenderThread:545287 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-14 18:05:24,154 INFO SenderThread:545287 [sender.py:transition_state():308] send defer: 9
2021-07-14 18:05:24,154 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: defer
2021-07-14 18:05:24,155 INFO HandlerThread:545287 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-14 18:05:24,155 DEBUG SenderThread:545287 [sender.py:send():179] send: final
2021-07-14 18:05:24,155 DEBUG SenderThread:545287 [sender.py:send():179] send: footer
2021-07-14 18:05:24,155 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: defer
2021-07-14 18:05:24,155 INFO SenderThread:545287 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-14 18:05:24,255 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-14 18:05:24,255 DEBUG SenderThread:545287 [sender.py:send_request():193] send_request: poll_exit
2021-07-14 18:05:24,255 INFO SenderThread:545287 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 18:05:24,257 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: get_summary
2021-07-14 18:05:24,257 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-14 18:05:24,258 DEBUG HandlerThread:545287 [handler.py:handle_request():124] handle_request: shutdown
2021-07-14 18:05:24,258 INFO HandlerThread:545287 [handler.py:finish():638] shutting down handler
2021-07-14 18:05:25,155 INFO WriterThread:545287 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210714_175518-t843i2it/run-t843i2it.wandb
2021-07-14 18:05:25,256 INFO SenderThread:545287 [sender.py:finish():945] shutting down sender
2021-07-14 18:05:25,256 INFO SenderThread:545287 [file_pusher.py:finish():177] shutting down file pusher
2021-07-14 18:05:25,256 INFO SenderThread:545287 [file_pusher.py:join():182] waiting for file pusher
2021-07-14 18:05:25,275 INFO MainThread:545287 [internal.py:handle_exit():78] Internal process exited
|