File size: 22,077 Bytes
9c3de9e |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 |
2022-01-14 21:28:56,265 INFO MainThread:8253 [internal.py:wandb_internal():87] W&B internal server running at pid: 8253, started at: 2022-01-14 21:28:56.265129
2022-01-14 21:28:56,268 DEBUG SenderThread:8253 [sender.py:send():234] send: header
2022-01-14 21:28:56,268 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: check_version
2022-01-14 21:28:56,268 INFO WriterThread:8253 [datastore.py:open_for_write():77] open: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/run-32qdb4k5.wandb
2022-01-14 21:28:56,268 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: check_version
2022-01-14 21:28:56,352 DEBUG SenderThread:8253 [sender.py:send():234] send: run
2022-01-14 21:28:56,515 INFO SenderThread:8253 [dir_watcher.py:__init__():169] watching files in: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files
2022-01-14 21:28:56,515 INFO SenderThread:8253 [sender.py:_start_run_threads():804] run started: 32qdb4k5 with start time 1642195735
2022-01-14 21:28:56,515 DEBUG SenderThread:8253 [sender.py:send():234] send: summary
2022-01-14 21:28:56,515 INFO SenderThread:8253 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-14 21:28:56,515 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: run_start
2022-01-14 21:28:57,561 INFO Thread-8 :8253 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/wandb-summary.json
2022-01-14 21:28:58,974 DEBUG HandlerThread:8253 [meta.py:__init__():40] meta init
2022-01-14 21:28:58,974 DEBUG HandlerThread:8253 [meta.py:__init__():54] meta init done
2022-01-14 21:28:58,974 DEBUG HandlerThread:8253 [meta.py:probe():214] probe
2022-01-14 21:28:58,975 DEBUG HandlerThread:8253 [meta.py:_setup_git():204] setup git
2022-01-14 21:28:59,006 DEBUG HandlerThread:8253 [meta.py:_setup_git():211] setup git done
2022-01-14 21:28:59,006 DEBUG HandlerThread:8253 [meta.py:_save_code():92] save code
2022-01-14 21:28:59,018 DEBUG HandlerThread:8253 [meta.py:_save_code():113] save code done
2022-01-14 21:28:59,018 DEBUG HandlerThread:8253 [meta.py:_save_patches():130] save patches
2022-01-14 21:28:59,561 INFO Thread-8 :8253 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/code/run_mlm_flax.py
2022-01-14 21:28:59,562 INFO Thread-8 :8253 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/code
2022-01-14 21:29:01,562 INFO Thread-8 :8253 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/output.log
2022-01-14 21:29:03,563 INFO Thread-8 :8253 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/output.log
2022-01-14 21:29:03,563 INFO Thread-8 :8253 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/diff.patch
2022-01-14 21:29:08,274 ERROR HandlerThread:8253 [meta.py:_save_patches():171] Error generating diff: Command '['git', 'diff', '--submodule=diff', 'HEAD']' timed out after 5 seconds
2022-01-14 21:29:08,274 DEBUG HandlerThread:8253 [meta.py:_save_patches():172] save patches done
2022-01-14 21:29:08,274 DEBUG HandlerThread:8253 [meta.py:_save_pip():58] save pip
2022-01-14 21:29:08,275 DEBUG HandlerThread:8253 [meta.py:_save_pip():72] save pip done
2022-01-14 21:29:08,275 DEBUG HandlerThread:8253 [meta.py:probe():252] probe done
2022-01-14 21:29:08,283 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 21:29:08,284 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status
2022-01-14 21:29:08,566 DEBUG SenderThread:8253 [sender.py:send():234] send: config
2022-01-14 21:29:08,566 DEBUG SenderThread:8253 [sender.py:send():234] send: config
2022-01-14 21:29:08,566 DEBUG SenderThread:8253 [sender.py:send():234] send: config
2022-01-14 21:29:08,567 DEBUG SenderThread:8253 [sender.py:send():234] send: files
2022-01-14 21:29:08,567 INFO SenderThread:8253 [sender.py:_save_file():939] saving file wandb-metadata.json with policy now
2022-01-14 21:29:08,567 INFO SenderThread:8253 [sender.py:_save_file():939] saving file code/run_mlm_flax.py with policy now
2022-01-14 21:29:08,571 INFO Thread-8 :8253 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/requirements.txt
2022-01-14 21:29:08,571 INFO Thread-8 :8253 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/wandb-metadata.json
2022-01-14 21:29:09,032 INFO Thread-12 :8253 [upload_job.py:push():137] Uploaded file /tmp/tmpdg54qv_0wandb/w1tibuxq-code/run_mlm_flax.py
2022-01-14 21:29:09,069 INFO Thread-11 :8253 [upload_job.py:push():137] Uploaded file /tmp/tmpdg54qv_0wandb/35h4ryp5-wandb-metadata.json
2022-01-14 21:29:09,571 INFO Thread-8 :8253 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/output.log
2022-01-14 21:29:21,520 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 21:29:21,521 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status
2022-01-14 21:29:27,049 DEBUG SenderThread:8253 [sender.py:send():234] send: stats
2022-01-14 21:29:27,579 INFO Thread-8 :8253 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/config.yaml
2022-01-14 21:29:36,656 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 21:29:36,657 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status
2022-01-14 21:29:51,794 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 21:29:51,795 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status
2022-01-14 21:29:57,119 DEBUG SenderThread:8253 [sender.py:send():234] send: stats
2022-01-14 21:29:58,591 INFO Thread-8 :8253 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/output.log
2022-01-14 21:30:06,945 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 21:30:06,945 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status
2022-01-14 21:30:22,126 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 21:30:22,127 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status
2022-01-14 21:30:27,189 DEBUG SenderThread:8253 [sender.py:send():234] send: stats
2022-01-14 21:30:37,330 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 21:30:37,330 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status
2022-01-14 21:30:52,532 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 21:30:52,532 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status
2022-01-14 21:30:57,257 DEBUG SenderThread:8253 [sender.py:send():234] send: stats
2022-01-14 21:31:07,691 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 21:31:07,692 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status
2022-01-14 21:31:22,944 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 21:31:22,945 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status
2022-01-14 21:31:27,323 DEBUG SenderThread:8253 [sender.py:send():234] send: stats
2022-01-14 21:31:38,085 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 21:31:38,086 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status
2022-01-14 21:31:53,231 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 21:31:53,231 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status
2022-01-14 21:31:57,395 DEBUG SenderThread:8253 [sender.py:send():234] send: stats
2022-01-14 21:32:08,366 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 21:32:08,367 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: stop_status
2022-01-14 21:32:16,649 INFO Thread-8 :8253 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/output.log
2022-01-14 21:32:16,893 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 21:32:16,893 DEBUG SenderThread:8253 [sender.py:send():234] send: telemetry
2022-01-14 21:32:16,893 DEBUG SenderThread:8253 [sender.py:send():234] send: exit
2022-01-14 21:32:16,893 INFO SenderThread:8253 [sender.py:send_exit():366] handling exit code: 1
2022-01-14 21:32:16,894 INFO SenderThread:8253 [sender.py:send_exit():368] handling runtime: 200
2022-01-14 21:32:16,894 INFO SenderThread:8253 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-14 21:32:16,894 INFO SenderThread:8253 [sender.py:send_exit():374] send defer
2022-01-14 21:32:16,894 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 21:32:16,895 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer
2022-01-14 21:32:16,895 INFO HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 0
2022-01-14 21:32:16,895 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: defer
2022-01-14 21:32:16,895 INFO SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 0
2022-01-14 21:32:16,895 INFO SenderThread:8253 [sender.py:transition_state():387] send defer: 1
2022-01-14 21:32:16,896 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer
2022-01-14 21:32:16,896 INFO HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 1
2022-01-14 21:32:16,941 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: defer
2022-01-14 21:32:16,941 INFO SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 1
2022-01-14 21:32:16,941 INFO SenderThread:8253 [sender.py:transition_state():387] send defer: 2
2022-01-14 21:32:16,941 DEBUG SenderThread:8253 [sender.py:send():234] send: stats
2022-01-14 21:32:16,942 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer
2022-01-14 21:32:16,942 INFO HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 2
2022-01-14 21:32:16,942 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: defer
2022-01-14 21:32:16,942 INFO SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 2
2022-01-14 21:32:16,942 INFO SenderThread:8253 [sender.py:transition_state():387] send defer: 3
2022-01-14 21:32:16,942 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer
2022-01-14 21:32:16,942 INFO HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 3
2022-01-14 21:32:16,943 DEBUG SenderThread:8253 [sender.py:send():234] send: summary
2022-01-14 21:32:16,943 INFO SenderThread:8253 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-14 21:32:16,943 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: defer
2022-01-14 21:32:16,943 INFO SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 3
2022-01-14 21:32:16,943 INFO SenderThread:8253 [sender.py:transition_state():387] send defer: 4
2022-01-14 21:32:16,943 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer
2022-01-14 21:32:16,943 INFO HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 4
2022-01-14 21:32:16,944 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: defer
2022-01-14 21:32:16,944 INFO SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 4
2022-01-14 21:32:16,997 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 21:32:17,650 INFO Thread-8 :8253 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/output.log
2022-01-14 21:32:17,650 INFO Thread-8 :8253 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/wandb-summary.json
2022-01-14 21:32:17,685 INFO SenderThread:8253 [sender.py:transition_state():387] send defer: 5
2022-01-14 21:32:17,686 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 21:32:17,686 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer
2022-01-14 21:32:17,686 INFO HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 5
2022-01-14 21:32:17,686 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: defer
2022-01-14 21:32:17,686 INFO SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 5
2022-01-14 21:32:17,687 INFO SenderThread:8253 [dir_watcher.py:finish():283] shutting down directory watcher
2022-01-14 21:32:17,787 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 21:32:18,650 INFO Thread-8 :8253 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/config.yaml
2022-01-14 21:32:18,651 INFO SenderThread:8253 [dir_watcher.py:finish():313] scan: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files
2022-01-14 21:32:18,651 INFO SenderThread:8253 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/config.yaml config.yaml
2022-01-14 21:32:18,651 INFO SenderThread:8253 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/diff.patch diff.patch
2022-01-14 21:32:18,651 INFO SenderThread:8253 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/requirements.txt requirements.txt
2022-01-14 21:32:18,652 INFO SenderThread:8253 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/output.log output.log
2022-01-14 21:32:18,652 INFO SenderThread:8253 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/wandb-summary.json wandb-summary.json
2022-01-14 21:32:18,652 INFO SenderThread:8253 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/wandb-metadata.json wandb-metadata.json
2022-01-14 21:32:18,656 INFO SenderThread:8253 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/code/run_mlm_flax.py code/run_mlm_flax.py
2022-01-14 21:32:18,656 INFO SenderThread:8253 [sender.py:transition_state():387] send defer: 6
2022-01-14 21:32:18,656 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 21:32:18,657 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer
2022-01-14 21:32:18,657 INFO HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 6
2022-01-14 21:32:18,662 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: defer
2022-01-14 21:32:18,665 INFO SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 6
2022-01-14 21:32:18,665 INFO SenderThread:8253 [file_pusher.py:finish():177] shutting down file pusher
2022-01-14 21:32:18,757 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 21:32:18,758 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 21:32:18,859 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 21:32:18,860 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 21:32:18,961 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 21:32:18,962 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 21:32:19,063 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 21:32:19,064 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 21:32:19,139 INFO Thread-15 :8253 [upload_job.py:push():137] Uploaded file /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/output.log
2022-01-14 21:32:19,148 INFO Thread-14 :8253 [upload_job.py:push():137] Uploaded file /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/requirements.txt
2022-01-14 21:32:19,165 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 21:32:19,165 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 21:32:19,171 INFO Thread-13 :8253 [upload_job.py:push():137] Uploaded file /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/config.yaml
2022-01-14 21:32:19,267 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 21:32:19,267 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 21:32:19,288 INFO Thread-16 :8253 [upload_job.py:push():137] Uploaded file /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/files/wandb-summary.json
2022-01-14 21:32:19,370 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 21:32:19,370 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 21:32:19,472 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 21:32:19,472 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 21:32:19,489 INFO Thread-7 :8253 [sender.py:transition_state():387] send defer: 7
2022-01-14 21:32:19,489 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer
2022-01-14 21:32:19,490 INFO HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 7
2022-01-14 21:32:19,490 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: defer
2022-01-14 21:32:19,490 INFO SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 7
2022-01-14 21:32:19,573 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 21:32:19,915 INFO SenderThread:8253 [sender.py:transition_state():387] send defer: 8
2022-01-14 21:32:19,916 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 21:32:19,916 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer
2022-01-14 21:32:19,916 INFO HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 8
2022-01-14 21:32:19,917 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: defer
2022-01-14 21:32:19,917 INFO SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 8
2022-01-14 21:32:19,917 INFO SenderThread:8253 [sender.py:transition_state():387] send defer: 9
2022-01-14 21:32:19,917 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: defer
2022-01-14 21:32:19,917 INFO HandlerThread:8253 [handler.py:handle_request_defer():147] handle defer: 9
2022-01-14 21:32:19,918 DEBUG SenderThread:8253 [sender.py:send():234] send: final
2022-01-14 21:32:19,918 DEBUG SenderThread:8253 [sender.py:send():234] send: footer
2022-01-14 21:32:19,918 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: defer
2022-01-14 21:32:19,918 INFO SenderThread:8253 [sender.py:send_request_defer():383] handle sender defer: 9
2022-01-14 21:32:20,017 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 21:32:20,018 DEBUG SenderThread:8253 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 21:32:20,018 INFO SenderThread:8253 [file_pusher.py:join():182] waiting for file pusher
2022-01-14 21:32:20,278 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: get_summary
2022-01-14 21:32:20,278 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: sampled_history
2022-01-14 21:32:20,279 DEBUG HandlerThread:8253 [handler.py:handle_request():130] handle_request: shutdown
2022-01-14 21:32:20,279 INFO HandlerThread:8253 [handler.py:finish():731] shutting down handler
2022-01-14 21:32:20,918 INFO WriterThread:8253 [datastore.py:close():281] close: /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/run-32qdb4k5.wandb
2022-01-14 21:32:21,277 INFO SenderThread:8253 [sender.py:finish():1070] shutting down sender
2022-01-14 21:32:21,277 INFO SenderThread:8253 [file_pusher.py:finish():177] shutting down file pusher
2022-01-14 21:32:21,277 INFO SenderThread:8253 [file_pusher.py:join():182] waiting for file pusher
2022-01-14 21:32:21,279 INFO MainThread:8253 [internal.py:handle_exit():77] Internal process exited
|