File size: 25,619 Bytes
731244f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 |
2021-07-15 00:07:58,272 INFO MainThread:615600 [internal.py:wandb_internal():88] W&B internal server running at pid: 615600, started at: 2021-07-15 00:07:58.272128
2021-07-15 00:07:58,274 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: check_version
2021-07-15 00:07:58,274 INFO WriterThread:615600 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210715_000757-1ymp5lov/run-1ymp5lov.wandb
2021-07-15 00:07:58,275 DEBUG SenderThread:615600 [sender.py:send():179] send: header
2021-07-15 00:07:58,275 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: check_version
2021-07-15 00:07:58,315 DEBUG SenderThread:615600 [sender.py:send():179] send: run
2021-07-15 00:07:58,483 INFO SenderThread:615600 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210715_000757-1ymp5lov/files
2021-07-15 00:07:58,483 INFO SenderThread:615600 [sender.py:_start_run_threads():716] run started: 1ymp5lov with start time 1626307677
2021-07-15 00:07:58,484 DEBUG SenderThread:615600 [sender.py:send():179] send: summary
2021-07-15 00:07:58,484 INFO SenderThread:615600 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-15 00:07:58,485 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: run_start
2021-07-15 00:07:59,486 INFO Thread-8 :615600 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_000757-1ymp5lov/files/wandb-summary.json
2021-07-15 00:07:59,582 DEBUG HandlerThread:615600 [meta.py:__init__():39] meta init
2021-07-15 00:07:59,582 DEBUG HandlerThread:615600 [meta.py:__init__():53] meta init done
2021-07-15 00:07:59,582 DEBUG HandlerThread:615600 [meta.py:probe():210] probe
2021-07-15 00:07:59,583 DEBUG HandlerThread:615600 [meta.py:_setup_git():200] setup git
2021-07-15 00:07:59,609 DEBUG HandlerThread:615600 [meta.py:_setup_git():207] setup git done
2021-07-15 00:07:59,610 DEBUG HandlerThread:615600 [meta.py:_save_pip():57] save pip
2021-07-15 00:07:59,610 DEBUG HandlerThread:615600 [meta.py:_save_pip():71] save pip done
2021-07-15 00:07:59,610 DEBUG HandlerThread:615600 [meta.py:probe():252] probe done
2021-07-15 00:07:59,613 DEBUG SenderThread:615600 [sender.py:send():179] send: files
2021-07-15 00:07:59,614 INFO SenderThread:615600 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-15 00:07:59,622 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:07:59,622 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:07:59,752 DEBUG SenderThread:615600 [sender.py:send():179] send: config
2021-07-15 00:07:59,752 DEBUG SenderThread:615600 [sender.py:send():179] send: config
2021-07-15 00:07:59,752 DEBUG SenderThread:615600 [sender.py:send():179] send: config
2021-07-15 00:08:00,093 INFO Thread-11 :615600 [upload_job.py:push():137] Uploaded file /tmp/tmpqbmmx6zswandb/3r9qdffv-wandb-metadata.json
2021-07-15 00:08:00,485 INFO Thread-8 :615600 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_000757-1ymp5lov/files/requirements.txt
2021-07-15 00:08:00,485 INFO Thread-8 :615600 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_000757-1ymp5lov/files/wandb-metadata.json
2021-07-15 00:08:00,486 INFO Thread-8 :615600 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_000757-1ymp5lov/files/output.log
2021-07-15 00:08:14,491 INFO Thread-8 :615600 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_000757-1ymp5lov/files/output.log
2021-07-15 00:08:14,754 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:08:14,754 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:08:27,665 DEBUG SenderThread:615600 [sender.py:send():179] send: stats
2021-07-15 00:08:29,497 INFO Thread-8 :615600 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_000757-1ymp5lov/files/config.yaml
2021-07-15 00:08:29,886 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:08:29,887 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:08:45,017 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:08:45,018 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:08:57,741 DEBUG SenderThread:615600 [sender.py:send():179] send: stats
2021-07-15 00:09:00,151 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:09:00,151 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:09:15,283 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:09:15,283 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:09:27,808 DEBUG SenderThread:615600 [sender.py:send():179] send: stats
2021-07-15 00:09:30,415 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:09:30,415 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:09:45,546 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:09:45,547 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:09:57,861 DEBUG SenderThread:615600 [sender.py:send():179] send: stats
2021-07-15 00:10:00,678 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:10:00,679 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:10:15,809 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:10:15,810 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:10:27,924 DEBUG SenderThread:615600 [sender.py:send():179] send: stats
2021-07-15 00:10:30,942 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:10:30,942 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:10:46,074 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:10:46,074 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:10:57,996 DEBUG SenderThread:615600 [sender.py:send():179] send: stats
2021-07-15 00:11:01,208 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:11:01,208 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:11:16,344 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:11:16,344 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:11:28,070 DEBUG SenderThread:615600 [sender.py:send():179] send: stats
2021-07-15 00:11:31,476 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:11:31,476 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:11:46,611 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:11:46,611 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:11:58,144 DEBUG SenderThread:615600 [sender.py:send():179] send: stats
2021-07-15 00:12:01,748 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:12:01,749 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:12:16,884 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:12:16,884 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:12:28,209 DEBUG SenderThread:615600 [sender.py:send():179] send: stats
2021-07-15 00:12:32,014 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:12:32,014 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:12:47,146 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:12:47,147 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:12:58,278 DEBUG SenderThread:615600 [sender.py:send():179] send: stats
2021-07-15 00:13:02,280 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:13:02,280 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:13:17,412 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:13:17,413 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:13:28,350 DEBUG SenderThread:615600 [sender.py:send():179] send: stats
2021-07-15 00:13:32,544 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:13:32,544 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:13:47,679 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:13:47,679 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:13:58,420 DEBUG SenderThread:615600 [sender.py:send():179] send: stats
2021-07-15 00:14:02,813 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:14:02,814 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:14:17,958 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:14:17,959 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:14:28,489 DEBUG SenderThread:615600 [sender.py:send():179] send: stats
2021-07-15 00:14:33,090 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:14:33,091 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:14:48,223 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:14:48,224 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:14:58,559 DEBUG SenderThread:615600 [sender.py:send():179] send: stats
2021-07-15 00:15:03,354 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:15:03,355 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:15:16,673 INFO Thread-8 :615600 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_000757-1ymp5lov/files/output.log
2021-07-15 00:15:18,488 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:15:18,488 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:15:28,630 DEBUG SenderThread:615600 [sender.py:send():179] send: stats
2021-07-15 00:15:33,644 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:15:33,645 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:15:48,779 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:15:48,780 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:15:58,711 DEBUG SenderThread:615600 [sender.py:send():179] send: stats
2021-07-15 00:16:03,911 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:16:03,912 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:16:19,047 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:16:19,047 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:16:28,794 DEBUG SenderThread:615600 [sender.py:send():179] send: stats
2021-07-15 00:16:34,182 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:16:34,183 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:16:38,710 INFO Thread-8 :615600 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_000757-1ymp5lov/files/output.log
2021-07-15 00:16:39,455 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:16:39,455 DEBUG SenderThread:615600 [sender.py:send():179] send: telemetry
2021-07-15 00:16:39,456 DEBUG SenderThread:615600 [sender.py:send():179] send: exit
2021-07-15 00:16:39,456 INFO SenderThread:615600 [sender.py:send_exit():287] handling exit code: 1
2021-07-15 00:16:39,457 INFO SenderThread:615600 [sender.py:send_exit():295] send defer
2021-07-15 00:16:39,458 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:16:39,458 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:16:39,459 INFO HandlerThread:615600 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-15 00:16:39,459 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: defer
2021-07-15 00:16:39,459 INFO SenderThread:615600 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-15 00:16:39,459 INFO SenderThread:615600 [sender.py:transition_state():308] send defer: 1
2021-07-15 00:16:39,459 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:16:39,459 INFO HandlerThread:615600 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-15 00:16:39,520 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: defer
2021-07-15 00:16:39,520 INFO SenderThread:615600 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-15 00:16:39,520 INFO SenderThread:615600 [sender.py:transition_state():308] send defer: 2
2021-07-15 00:16:39,520 DEBUG SenderThread:615600 [sender.py:send():179] send: stats
2021-07-15 00:16:39,521 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:16:39,521 INFO HandlerThread:615600 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-15 00:16:39,521 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: defer
2021-07-15 00:16:39,521 INFO SenderThread:615600 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-15 00:16:39,521 INFO SenderThread:615600 [sender.py:transition_state():308] send defer: 3
2021-07-15 00:16:39,522 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:16:39,522 INFO HandlerThread:615600 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-15 00:16:39,522 DEBUG SenderThread:615600 [sender.py:send():179] send: summary
2021-07-15 00:16:39,523 INFO SenderThread:615600 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-15 00:16:39,523 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: defer
2021-07-15 00:16:39,523 INFO SenderThread:615600 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-15 00:16:39,523 INFO SenderThread:615600 [sender.py:transition_state():308] send defer: 4
2021-07-15 00:16:39,524 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:16:39,524 INFO HandlerThread:615600 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-15 00:16:39,524 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: defer
2021-07-15 00:16:39,524 INFO SenderThread:615600 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-15 00:16:39,560 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:16:39,701 INFO SenderThread:615600 [sender.py:transition_state():308] send defer: 5
2021-07-15 00:16:39,701 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:16:39,702 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:16:39,702 INFO HandlerThread:615600 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-15 00:16:39,702 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: defer
2021-07-15 00:16:39,702 INFO SenderThread:615600 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-15 00:16:39,702 INFO SenderThread:615600 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-15 00:16:39,710 INFO Thread-8 :615600 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_000757-1ymp5lov/files/config.yaml
2021-07-15 00:16:39,711 INFO SenderThread:615600 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_000757-1ymp5lov/files/wandb-summary.json
2021-07-15 00:16:39,711 INFO SenderThread:615600 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_000757-1ymp5lov/files/output.log
2021-07-15 00:16:39,711 INFO SenderThread:615600 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210715_000757-1ymp5lov/files
2021-07-15 00:16:39,711 INFO SenderThread:615600 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_000757-1ymp5lov/files/requirements.txt requirements.txt
2021-07-15 00:16:39,711 INFO SenderThread:615600 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_000757-1ymp5lov/files/output.log output.log
2021-07-15 00:16:39,711 INFO SenderThread:615600 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_000757-1ymp5lov/files/wandb-metadata.json wandb-metadata.json
2021-07-15 00:16:39,712 INFO SenderThread:615600 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_000757-1ymp5lov/files/config.yaml config.yaml
2021-07-15 00:16:39,712 INFO SenderThread:615600 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_000757-1ymp5lov/files/wandb-summary.json wandb-summary.json
2021-07-15 00:16:39,712 INFO SenderThread:615600 [sender.py:transition_state():308] send defer: 6
2021-07-15 00:16:39,716 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:16:39,716 INFO HandlerThread:615600 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-15 00:16:39,719 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: defer
2021-07-15 00:16:39,719 INFO SenderThread:615600 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-15 00:16:39,719 INFO SenderThread:615600 [file_pusher.py:finish():177] shutting down file pusher
2021-07-15 00:16:39,803 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:16:39,803 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:16:39,905 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:16:39,905 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:16:40,007 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:16:40,007 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:16:40,109 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:16:40,109 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:16:40,166 INFO Thread-14 :615600 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_000757-1ymp5lov/files/config.yaml
2021-07-15 00:16:40,175 INFO Thread-12 :615600 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_000757-1ymp5lov/files/requirements.txt
2021-07-15 00:16:40,189 INFO Thread-15 :615600 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_000757-1ymp5lov/files/wandb-summary.json
2021-07-15 00:16:40,193 INFO Thread-13 :615600 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_000757-1ymp5lov/files/output.log
2021-07-15 00:16:40,211 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:16:40,211 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:16:40,313 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:16:40,313 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:16:40,394 INFO Thread-7 :615600 [sender.py:transition_state():308] send defer: 7
2021-07-15 00:16:40,394 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:16:40,394 INFO HandlerThread:615600 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-15 00:16:40,394 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: defer
2021-07-15 00:16:40,395 INFO SenderThread:615600 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-15 00:16:40,415 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:16:40,671 INFO SenderThread:615600 [sender.py:transition_state():308] send defer: 8
2021-07-15 00:16:40,671 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:16:40,671 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:16:40,671 INFO HandlerThread:615600 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-15 00:16:40,672 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: defer
2021-07-15 00:16:40,672 INFO SenderThread:615600 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-15 00:16:40,672 INFO SenderThread:615600 [sender.py:transition_state():308] send defer: 9
2021-07-15 00:16:40,672 DEBUG SenderThread:615600 [sender.py:send():179] send: final
2021-07-15 00:16:40,672 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:16:40,673 INFO HandlerThread:615600 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-15 00:16:40,673 DEBUG SenderThread:615600 [sender.py:send():179] send: footer
2021-07-15 00:16:40,673 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: defer
2021-07-15 00:16:40,673 INFO SenderThread:615600 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-15 00:16:40,772 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:16:40,773 DEBUG SenderThread:615600 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:16:40,773 INFO SenderThread:615600 [file_pusher.py:join():182] waiting for file pusher
2021-07-15 00:16:40,774 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: get_summary
2021-07-15 00:16:40,775 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-15 00:16:40,775 DEBUG HandlerThread:615600 [handler.py:handle_request():124] handle_request: shutdown
2021-07-15 00:16:40,776 INFO HandlerThread:615600 [handler.py:finish():638] shutting down handler
2021-07-15 00:16:41,673 INFO WriterThread:615600 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210715_000757-1ymp5lov/run-1ymp5lov.wandb
2021-07-15 00:16:41,773 INFO SenderThread:615600 [sender.py:finish():945] shutting down sender
2021-07-15 00:16:41,774 INFO SenderThread:615600 [file_pusher.py:finish():177] shutting down file pusher
2021-07-15 00:16:41,774 INFO SenderThread:615600 [file_pusher.py:join():182] waiting for file pusher
2021-07-15 00:16:41,776 INFO MainThread:615600 [internal.py:handle_exit():78] Internal process exited
|