File size: 25,524 Bytes
731244f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 |
2021-07-15 00:17:36,936 INFO MainThread:617849 [internal.py:wandb_internal():88] W&B internal server running at pid: 617849, started at: 2021-07-15 00:17:36.936178
2021-07-15 00:17:36,938 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: check_version
2021-07-15 00:17:36,939 DEBUG SenderThread:617849 [sender.py:send():179] send: header
2021-07-15 00:17:36,939 INFO WriterThread:617849 [datastore.py:open_for_write():80] open: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/run-u185degl.wandb
2021-07-15 00:17:36,939 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: check_version
2021-07-15 00:17:36,977 DEBUG SenderThread:617849 [sender.py:send():179] send: run
2021-07-15 00:17:37,151 INFO SenderThread:617849 [dir_watcher.py:__init__():168] watching files in: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files
2021-07-15 00:17:37,151 INFO SenderThread:617849 [sender.py:_start_run_threads():716] run started: u185degl with start time 1626308256
2021-07-15 00:17:37,151 DEBUG SenderThread:617849 [sender.py:send():179] send: summary
2021-07-15 00:17:37,151 INFO SenderThread:617849 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-15 00:17:37,151 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: run_start
2021-07-15 00:17:38,153 INFO Thread-8 :617849 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/wandb-summary.json
2021-07-15 00:17:38,283 DEBUG HandlerThread:617849 [meta.py:__init__():39] meta init
2021-07-15 00:17:38,284 DEBUG HandlerThread:617849 [meta.py:__init__():53] meta init done
2021-07-15 00:17:38,284 DEBUG HandlerThread:617849 [meta.py:probe():210] probe
2021-07-15 00:17:38,285 DEBUG HandlerThread:617849 [meta.py:_setup_git():200] setup git
2021-07-15 00:17:38,315 DEBUG HandlerThread:617849 [meta.py:_setup_git():207] setup git done
2021-07-15 00:17:38,315 DEBUG HandlerThread:617849 [meta.py:_save_pip():57] save pip
2021-07-15 00:17:38,316 DEBUG HandlerThread:617849 [meta.py:_save_pip():71] save pip done
2021-07-15 00:17:38,316 DEBUG HandlerThread:617849 [meta.py:probe():252] probe done
2021-07-15 00:17:38,319 DEBUG SenderThread:617849 [sender.py:send():179] send: files
2021-07-15 00:17:38,319 INFO SenderThread:617849 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-15 00:17:38,325 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:17:38,325 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:17:38,454 DEBUG SenderThread:617849 [sender.py:send():179] send: config
2021-07-15 00:17:38,454 DEBUG SenderThread:617849 [sender.py:send():179] send: config
2021-07-15 00:17:38,454 DEBUG SenderThread:617849 [sender.py:send():179] send: config
2021-07-15 00:17:38,776 INFO Thread-11 :617849 [upload_job.py:push():137] Uploaded file /tmp/tmpxo1vs5f9wandb/odzm0bpa-wandb-metadata.json
2021-07-15 00:17:39,152 INFO Thread-8 :617849 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/requirements.txt
2021-07-15 00:17:39,152 INFO Thread-8 :617849 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/wandb-metadata.json
2021-07-15 00:17:39,152 INFO Thread-8 :617849 [dir_watcher.py:_on_file_created():216] file/dir created: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/output.log
2021-07-15 00:17:53,157 INFO Thread-8 :617849 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/output.log
2021-07-15 00:17:53,456 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:17:53,456 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:18:06,368 DEBUG SenderThread:617849 [sender.py:send():179] send: stats
2021-07-15 00:18:08,163 INFO Thread-8 :617849 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/config.yaml
2021-07-15 00:18:08,591 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:18:08,592 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:18:23,724 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:18:23,725 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:18:36,448 DEBUG SenderThread:617849 [sender.py:send():179] send: stats
2021-07-15 00:18:38,860 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:18:38,860 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:18:53,994 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:18:53,994 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:19:06,524 DEBUG SenderThread:617849 [sender.py:send():179] send: stats
2021-07-15 00:19:09,126 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:19:09,127 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:19:24,259 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:19:24,260 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:19:36,600 DEBUG SenderThread:617849 [sender.py:send():179] send: stats
2021-07-15 00:19:39,407 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:19:39,407 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:19:54,540 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:19:54,540 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:20:06,663 DEBUG SenderThread:617849 [sender.py:send():179] send: stats
2021-07-15 00:20:09,672 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:20:09,672 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:20:24,807 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:20:24,807 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:20:36,726 DEBUG SenderThread:617849 [sender.py:send():179] send: stats
2021-07-15 00:20:39,938 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:20:39,938 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:20:55,070 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:20:55,070 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:21:06,791 DEBUG SenderThread:617849 [sender.py:send():179] send: stats
2021-07-15 00:21:10,200 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:21:10,200 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:21:25,331 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:21:25,331 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:21:36,863 DEBUG SenderThread:617849 [sender.py:send():179] send: stats
2021-07-15 00:21:40,462 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:21:40,462 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:21:55,592 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:21:55,593 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:22:06,940 DEBUG SenderThread:617849 [sender.py:send():179] send: stats
2021-07-15 00:22:10,734 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:22:10,735 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:22:25,867 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:22:25,867 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:22:37,017 DEBUG SenderThread:617849 [sender.py:send():179] send: stats
2021-07-15 00:22:40,998 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:22:40,998 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:22:56,126 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:22:56,126 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:23:07,091 DEBUG SenderThread:617849 [sender.py:send():179] send: stats
2021-07-15 00:23:11,257 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:23:11,257 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:23:26,389 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:23:26,389 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:23:37,156 DEBUG SenderThread:617849 [sender.py:send():179] send: stats
2021-07-15 00:23:41,522 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:23:41,522 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:23:56,651 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:23:56,652 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:24:07,226 DEBUG SenderThread:617849 [sender.py:send():179] send: stats
2021-07-15 00:24:11,783 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:24:11,783 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:24:26,932 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:24:26,932 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:24:27,307 INFO Thread-8 :617849 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/output.log
2021-07-15 00:24:37,304 DEBUG SenderThread:617849 [sender.py:send():179] send: stats
2021-07-15 00:24:42,077 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:24:42,078 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:24:57,226 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:24:57,227 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:25:07,379 DEBUG SenderThread:617849 [sender.py:send():179] send: stats
2021-07-15 00:25:12,357 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:25:12,357 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:25:27,488 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:25:27,489 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:25:37,452 DEBUG SenderThread:617849 [sender.py:send():179] send: stats
2021-07-15 00:25:42,619 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: stop_status
2021-07-15 00:25:42,619 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: stop_status
2021-07-15 00:25:54,343 INFO Thread-8 :617849 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/output.log
2021-07-15 00:25:54,412 DEBUG SenderThread:617849 [sender.py:send():179] send: telemetry
2021-07-15 00:25:54,412 DEBUG SenderThread:617849 [sender.py:send():179] send: exit
2021-07-15 00:25:54,412 INFO SenderThread:617849 [sender.py:send_exit():287] handling exit code: 1
2021-07-15 00:25:54,412 INFO SenderThread:617849 [sender.py:send_exit():295] send defer
2021-07-15 00:25:54,413 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:25:54,413 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:25:54,413 INFO HandlerThread:617849 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-15 00:25:54,413 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:25:54,413 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: defer
2021-07-15 00:25:54,414 INFO SenderThread:617849 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-15 00:25:54,414 INFO SenderThread:617849 [sender.py:transition_state():308] send defer: 1
2021-07-15 00:25:54,414 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:25:54,414 INFO HandlerThread:617849 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-15 00:25:54,493 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: defer
2021-07-15 00:25:54,493 INFO SenderThread:617849 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-15 00:25:54,493 INFO SenderThread:617849 [sender.py:transition_state():308] send defer: 2
2021-07-15 00:25:54,493 DEBUG SenderThread:617849 [sender.py:send():179] send: stats
2021-07-15 00:25:54,494 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:25:54,494 INFO HandlerThread:617849 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-15 00:25:54,494 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: defer
2021-07-15 00:25:54,494 INFO SenderThread:617849 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-15 00:25:54,494 INFO SenderThread:617849 [sender.py:transition_state():308] send defer: 3
2021-07-15 00:25:54,494 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:25:54,494 INFO HandlerThread:617849 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-15 00:25:54,494 DEBUG SenderThread:617849 [sender.py:send():179] send: summary
2021-07-15 00:25:54,495 INFO SenderThread:617849 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-15 00:25:54,495 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: defer
2021-07-15 00:25:54,495 INFO SenderThread:617849 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-15 00:25:54,495 INFO SenderThread:617849 [sender.py:transition_state():308] send defer: 4
2021-07-15 00:25:54,495 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:25:54,495 INFO HandlerThread:617849 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-15 00:25:54,495 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: defer
2021-07-15 00:25:54,495 INFO SenderThread:617849 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-15 00:25:54,516 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:25:54,687 INFO SenderThread:617849 [sender.py:transition_state():308] send defer: 5
2021-07-15 00:25:54,688 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:25:54,688 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:25:54,688 INFO HandlerThread:617849 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-15 00:25:54,688 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: defer
2021-07-15 00:25:54,688 INFO SenderThread:617849 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-15 00:25:54,688 INFO SenderThread:617849 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-15 00:25:54,790 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:25:55,343 INFO Thread-8 :617849 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/output.log
2021-07-15 00:25:55,344 INFO SenderThread:617849 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/config.yaml
2021-07-15 00:25:55,344 INFO SenderThread:617849 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/wandb-summary.json
2021-07-15 00:25:55,344 INFO SenderThread:617849 [dir_watcher.py:finish():312] scan: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files
2021-07-15 00:25:55,344 INFO SenderThread:617849 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/requirements.txt requirements.txt
2021-07-15 00:25:55,344 INFO SenderThread:617849 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/output.log output.log
2021-07-15 00:25:55,344 INFO SenderThread:617849 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/wandb-metadata.json wandb-metadata.json
2021-07-15 00:25:55,344 INFO SenderThread:617849 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/config.yaml config.yaml
2021-07-15 00:25:55,345 INFO SenderThread:617849 [dir_watcher.py:finish():318] scan save: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/wandb-summary.json wandb-summary.json
2021-07-15 00:25:55,345 INFO SenderThread:617849 [sender.py:transition_state():308] send defer: 6
2021-07-15 00:25:55,345 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:25:55,346 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:25:55,346 INFO HandlerThread:617849 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-15 00:25:55,349 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: defer
2021-07-15 00:25:55,349 INFO SenderThread:617849 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-15 00:25:55,349 INFO SenderThread:617849 [file_pusher.py:finish():177] shutting down file pusher
2021-07-15 00:25:55,447 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:25:55,448 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:25:55,550 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:25:55,550 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:25:55,652 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:25:55,652 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:25:55,754 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:25:55,754 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:25:55,810 INFO Thread-13 :617849 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/output.log
2021-07-15 00:25:55,821 INFO Thread-15 :617849 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/wandb-summary.json
2021-07-15 00:25:55,827 INFO Thread-12 :617849 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/requirements.txt
2021-07-15 00:25:55,856 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:25:55,856 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:25:55,869 INFO Thread-14 :617849 [upload_job.py:push():137] Uploaded file /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/files/config.yaml
2021-07-15 00:25:55,958 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:25:55,958 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:25:56,060 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:25:56,060 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:25:56,069 INFO Thread-7 :617849 [sender.py:transition_state():308] send defer: 7
2021-07-15 00:25:56,070 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:25:56,070 INFO HandlerThread:617849 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-15 00:25:56,070 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: defer
2021-07-15 00:25:56,070 INFO SenderThread:617849 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-15 00:25:56,162 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:25:56,354 INFO SenderThread:617849 [sender.py:transition_state():308] send defer: 8
2021-07-15 00:25:56,354 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:25:56,355 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:25:56,355 INFO HandlerThread:617849 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-15 00:25:56,355 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: defer
2021-07-15 00:25:56,355 INFO SenderThread:617849 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-15 00:25:56,355 INFO SenderThread:617849 [sender.py:transition_state():308] send defer: 9
2021-07-15 00:25:56,356 DEBUG SenderThread:617849 [sender.py:send():179] send: final
2021-07-15 00:25:56,356 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: defer
2021-07-15 00:25:56,356 DEBUG SenderThread:617849 [sender.py:send():179] send: footer
2021-07-15 00:25:56,356 INFO HandlerThread:617849 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-15 00:25:56,356 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: defer
2021-07-15 00:25:56,356 INFO SenderThread:617849 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-15 00:25:56,456 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-15 00:25:56,456 DEBUG SenderThread:617849 [sender.py:send_request():193] send_request: poll_exit
2021-07-15 00:25:56,456 INFO SenderThread:617849 [file_pusher.py:join():182] waiting for file pusher
2021-07-15 00:25:56,458 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: get_summary
2021-07-15 00:25:56,458 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-15 00:25:56,459 DEBUG HandlerThread:617849 [handler.py:handle_request():124] handle_request: shutdown
2021-07-15 00:25:56,459 INFO HandlerThread:617849 [handler.py:finish():638] shutting down handler
2021-07-15 00:25:57,356 INFO WriterThread:617849 [datastore.py:close():288] close: /home/dat/pino-roberta-base/wandb/run-20210715_001736-u185degl/run-u185degl.wandb
2021-07-15 00:25:57,457 INFO SenderThread:617849 [sender.py:finish():945] shutting down sender
2021-07-15 00:25:57,457 INFO SenderThread:617849 [file_pusher.py:finish():177] shutting down file pusher
2021-07-15 00:25:57,457 INFO SenderThread:617849 [file_pusher.py:join():182] waiting for file pusher
2021-07-15 00:25:57,462 INFO MainThread:617849 [internal.py:handle_exit():78] Internal process exited
|