File size: 48,934 Bytes
872c781 492601d f5d3959 c555954 e5892f9 5909771 bfedc8c b555d05 df53f6a 9b94734 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 |
2021-07-09 14:14:46,066 INFO MainThread:246776 [internal.py:wandb_internal():88] W&B internal server running at pid: 246776, started at: 2021-07-09 14:14:46.066421
2021-07-09 14:14:46,069 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: check_version
2021-07-09 14:14:46,069 INFO WriterThread:246776 [datastore.py:open_for_write():80] open: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/run-2k8cnty2.wandb
2021-07-09 14:14:46,070 DEBUG SenderThread:246776 [sender.py:send():179] send: header
2021-07-09 14:14:46,070 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: check_version
2021-07-09 14:14:46,106 DEBUG SenderThread:246776 [sender.py:send():179] send: run
2021-07-09 14:14:46,291 INFO SenderThread:246776 [dir_watcher.py:__init__():168] watching files in: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files
2021-07-09 14:14:46,291 INFO SenderThread:246776 [sender.py:_start_run_threads():716] run started: 2k8cnty2 with start time 1625840085
2021-07-09 14:14:46,291 DEBUG SenderThread:246776 [sender.py:send():179] send: summary
2021-07-09 14:14:46,291 INFO SenderThread:246776 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 14:14:46,292 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: run_start
2021-07-09 14:14:47,296 INFO Thread-8 :246776 [dir_watcher.py:_on_file_created():216] file/dir created: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/wandb-summary.json
2021-07-09 14:14:47,469 DEBUG HandlerThread:246776 [meta.py:__init__():39] meta init
2021-07-09 14:14:47,469 DEBUG HandlerThread:246776 [meta.py:__init__():53] meta init done
2021-07-09 14:14:47,469 DEBUG HandlerThread:246776 [meta.py:probe():210] probe
2021-07-09 14:14:47,470 DEBUG HandlerThread:246776 [meta.py:_setup_git():200] setup git
2021-07-09 14:14:47,497 DEBUG HandlerThread:246776 [meta.py:_setup_git():207] setup git done
2021-07-09 14:14:47,498 DEBUG HandlerThread:246776 [meta.py:_save_pip():57] save pip
2021-07-09 14:14:47,498 DEBUG HandlerThread:246776 [meta.py:_save_pip():71] save pip done
2021-07-09 14:14:47,498 DEBUG HandlerThread:246776 [meta.py:_save_conda():78] save conda
2021-07-09 14:14:48,296 INFO Thread-8 :246776 [dir_watcher.py:_on_file_created():216] file/dir created: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/conda-environment.yaml
2021-07-09 14:14:48,296 INFO Thread-8 :246776 [dir_watcher.py:_on_file_created():216] file/dir created: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/requirements.txt
2021-07-09 14:14:49,937 DEBUG HandlerThread:246776 [meta.py:_save_conda():86] save conda done
2021-07-09 14:14:49,937 DEBUG HandlerThread:246776 [meta.py:probe():252] probe done
2021-07-09 14:14:49,940 DEBUG SenderThread:246776 [sender.py:send():179] send: files
2021-07-09 14:14:49,940 INFO SenderThread:246776 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-09 14:14:49,947 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:14:49,947 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:14:50,072 DEBUG SenderThread:246776 [sender.py:send():179] send: config
2021-07-09 14:14:50,073 DEBUG SenderThread:246776 [sender.py:send():179] send: config
2021-07-09 14:14:50,073 DEBUG SenderThread:246776 [sender.py:send():179] send: config
2021-07-09 14:14:50,073 DEBUG SenderThread:246776 [sender.py:send():179] send: config
2021-07-09 14:14:50,296 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/conda-environment.yaml
2021-07-09 14:14:50,297 INFO Thread-8 :246776 [dir_watcher.py:_on_file_created():216] file/dir created: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:14:50,297 INFO Thread-8 :246776 [dir_watcher.py:_on_file_created():216] file/dir created: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/wandb-metadata.json
2021-07-09 14:14:50,380 INFO Thread-11 :246776 [upload_job.py:push():137] Uploaded file /tmp/tmp4f0a6i2mwandb/2kbqshuw-wandb-metadata.json
2021-07-09 14:14:52,297 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:14:54,298 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:15:02,301 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:15:04,302 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:15:05,089 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:15:05,089 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:15:08,303 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:15:10,304 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:15:15,553 DEBUG SenderThread:246776 [sender.py:send():179] send: stats
2021-07-09 14:15:17,307 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/config.yaml
2021-07-09 14:15:19,308 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:15:20,261 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:15:20,261 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:15:21,309 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:15:27,311 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:15:27,425 INFO HandlerThread:246776 [handler.py:handle_tbrecord():547] handling tbrecord: tbrecord {
log_dir: "/home/cahya/Work/flax-community/gpt2-medium-indonesian"
save: true
}
2021-07-09 14:15:27,426 DEBUG HandlerThread:246776 [config_util.py:dict_from_config_file():101] no default config file found in config-defaults.yaml
2021-07-09 14:15:27,434 DEBUG SenderThread:246776 [sender.py:send():179] send: tbrecord
2021-07-09 14:15:27,435 DEBUG SenderThread:246776 [sender.py:send():179] send: files
2021-07-09 14:15:27,435 INFO SenderThread:246776 [sender.py:_save_file():841] saving file events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2 with policy live
2021-07-09 14:15:27,879 INFO Thread-14 :246776 [upload_job.py:push():137] Uploaded file /tmp/tmp4f0a6i2mwandb/2a4g7r5i-events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2
2021-07-09 14:15:28,312 INFO Thread-8 :246776 [dir_watcher.py:_on_file_created():216] file/dir created: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2
2021-07-09 14:15:29,312 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:15:31,313 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:15:35,424 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:15:35,424 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:15:45,633 DEBUG SenderThread:246776 [sender.py:send():179] send: stats
2021-07-09 14:15:50,786 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:15:50,786 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:16:05,918 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:16:05,918 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:16:15,713 DEBUG SenderThread:246776 [sender.py:send():179] send: stats
2021-07-09 14:16:21,055 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:16:21,056 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:16:36,187 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:16:36,187 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:16:45,342 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:16:45,792 DEBUG SenderThread:246776 [sender.py:send():179] send: stats
2021-07-09 14:16:47,342 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:16:49,343 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:16:51,317 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:16:51,317 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:16:51,344 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:17:05,350 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:17:06,350 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2
2021-07-09 14:17:06,451 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:17:06,451 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:17:06,818 INFO Thread-15 :246776 [upload_job.py:push():137] Uploaded file /tmp/tmp4f0a6i2mwandb/5degf70d-events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2
2021-07-09 14:17:07,350 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:17:12,353 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:17:14,354 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:17:15,871 DEBUG SenderThread:246776 [sender.py:send():179] send: stats
2021-07-09 14:17:16,355 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:17:21,580 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:17:21,580 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:17:28,359 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:17:30,360 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:17:32,361 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:17:36,712 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:17:36,712 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:17:38,363 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:17:38,663 DEBUG SenderThread:246776 [sender.py:send():179] send: history
2021-07-09 14:17:38,663 DEBUG SenderThread:246776 [sender.py:send():179] send: summary
2021-07-09 14:17:38,663 INFO SenderThread:246776 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 14:17:39,364 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2
2021-07-09 14:17:39,364 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/wandb-summary.json
2021-07-09 14:17:40,364 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:17:45,949 DEBUG SenderThread:246776 [sender.py:send():179] send: stats
2021-07-09 14:17:46,366 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:17:51,907 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:17:51,907 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:18:02,372 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:18:07,040 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:18:07,040 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:18:16,028 DEBUG SenderThread:246776 [sender.py:send():179] send: stats
2021-07-09 14:18:22,169 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:18:22,169 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:18:37,300 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:18:37,300 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:18:46,105 DEBUG SenderThread:246776 [sender.py:send():179] send: stats
2021-07-09 14:18:52,428 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:18:52,428 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:19:04,397 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:19:06,398 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:19:07,560 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:19:07,560 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:19:08,398 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:19:10,399 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:19:14,401 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:19:15,401 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2
2021-07-09 14:19:15,821 DEBUG SenderThread:246776 [sender.py:send():179] send: history
2021-07-09 14:19:15,821 DEBUG SenderThread:246776 [sender.py:send():179] send: summary
2021-07-09 14:19:15,821 INFO SenderThread:246776 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 14:19:15,892 INFO Thread-16 :246776 [upload_job.py:push():137] Uploaded file /tmp/tmp4f0a6i2mwandb/2a28xt0m-events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2
2021-07-09 14:19:16,180 DEBUG SenderThread:246776 [sender.py:send():179] send: stats
2021-07-09 14:19:16,402 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:19:16,402 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/wandb-summary.json
2021-07-09 14:19:22,689 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:19:22,690 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:19:23,405 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:19:35,410 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:19:37,819 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:19:37,819 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:19:46,259 DEBUG SenderThread:246776 [sender.py:send():179] send: stats
2021-07-09 14:19:52,949 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:19:52,949 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:20:08,078 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:20:08,078 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:20:16,335 DEBUG SenderThread:246776 [sender.py:send():179] send: stats
2021-07-09 14:20:23,209 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:20:23,209 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:20:27,430 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:20:29,431 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:20:31,432 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:20:33,433 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:20:37,434 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:20:37,965 DEBUG SenderThread:246776 [sender.py:send():179] send: history
2021-07-09 14:20:37,965 DEBUG SenderThread:246776 [sender.py:send():179] send: summary
2021-07-09 14:20:37,966 INFO SenderThread:246776 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 14:20:38,340 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:20:38,340 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:20:38,435 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2
2021-07-09 14:20:38,435 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/wandb-summary.json
2021-07-09 14:20:38,913 INFO Thread-17 :246776 [upload_job.py:push():137] Uploaded file /tmp/tmp4f0a6i2mwandb/i981d3j1-events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2
2021-07-09 14:20:39,435 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:20:43,437 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:20:46,411 DEBUG SenderThread:246776 [sender.py:send():179] send: stats
2021-07-09 14:20:53,491 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:20:53,491 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:20:57,442 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:21:08,625 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:21:08,625 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:21:16,486 DEBUG SenderThread:246776 [sender.py:send():179] send: stats
2021-07-09 14:21:23,754 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:21:23,754 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:21:38,887 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:21:38,888 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:21:46,566 DEBUG SenderThread:246776 [sender.py:send():179] send: stats
2021-07-09 14:21:54,019 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:21:54,020 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:21:56,465 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:21:58,465 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:22:00,466 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:22:02,467 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:22:08,113 DEBUG SenderThread:246776 [sender.py:send():179] send: history
2021-07-09 14:22:08,114 DEBUG SenderThread:246776 [sender.py:send():179] send: summary
2021-07-09 14:22:08,114 INFO SenderThread:246776 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 14:22:08,469 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:22:08,470 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2
2021-07-09 14:22:08,470 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/wandb-summary.json
2021-07-09 14:22:09,013 INFO Thread-18 :246776 [upload_job.py:push():137] Uploaded file /tmp/tmp4f0a6i2mwandb/otuqhtyr-events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2
2021-07-09 14:22:09,151 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:22:09,151 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:22:09,470 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:22:10,470 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:22:16,473 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:22:16,645 DEBUG SenderThread:246776 [sender.py:send():179] send: stats
2021-07-09 14:22:24,304 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:22:24,304 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:22:28,478 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:22:39,446 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:22:39,447 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:22:46,723 DEBUG SenderThread:246776 [sender.py:send():179] send: stats
2021-07-09 14:22:54,575 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:22:54,575 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:23:09,708 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:23:09,708 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:23:16,801 DEBUG SenderThread:246776 [sender.py:send():179] send: stats
2021-07-09 14:23:20,500 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:23:22,501 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:23:24,502 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:23:24,837 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:23:24,838 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:23:26,503 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:23:30,505 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:23:31,259 DEBUG SenderThread:246776 [sender.py:send():179] send: history
2021-07-09 14:23:31,259 DEBUG SenderThread:246776 [sender.py:send():179] send: summary
2021-07-09 14:23:31,260 INFO SenderThread:246776 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 14:23:31,505 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2
2021-07-09 14:23:31,506 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/wandb-summary.json
2021-07-09 14:23:32,506 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:23:39,978 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:23:39,979 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:23:46,879 DEBUG SenderThread:246776 [sender.py:send():179] send: stats
2021-07-09 14:23:50,514 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:23:55,122 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:23:55,122 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:24:10,254 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:24:10,254 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:24:16,957 DEBUG SenderThread:246776 [sender.py:send():179] send: stats
2021-07-09 14:24:25,385 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:24:25,385 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:24:40,609 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:24:40,609 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:24:42,535 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:24:44,536 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:24:46,536 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:24:47,034 DEBUG SenderThread:246776 [sender.py:send():179] send: stats
2021-07-09 14:24:52,539 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:24:53,395 DEBUG SenderThread:246776 [sender.py:send():179] send: history
2021-07-09 14:24:53,395 DEBUG SenderThread:246776 [sender.py:send():179] send: summary
2021-07-09 14:24:53,395 INFO SenderThread:246776 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 14:24:53,539 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2
2021-07-09 14:24:53,540 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/wandb-summary.json
2021-07-09 14:24:54,013 INFO Thread-19 :246776 [upload_job.py:push():137] Uploaded file /tmp/tmp4f0a6i2mwandb/72n81odb-events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2
2021-07-09 14:24:54,540 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:24:55,540 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:24:55,836 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:24:55,836 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:25:10,974 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:25:10,974 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:25:13,548 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:25:17,112 DEBUG SenderThread:246776 [sender.py:send():179] send: stats
2021-07-09 14:25:26,143 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:25:26,144 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:25:41,276 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:25:41,276 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:25:47,189 DEBUG SenderThread:246776 [sender.py:send():179] send: stats
2021-07-09 14:25:56,420 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:25:56,420 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:26:11,570 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:26:11,570 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:26:13,573 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:26:15,574 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:26:17,268 DEBUG SenderThread:246776 [sender.py:send():179] send: stats
2021-07-09 14:26:17,575 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:26:23,578 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:26:24,547 DEBUG SenderThread:246776 [sender.py:send():179] send: history
2021-07-09 14:26:24,547 DEBUG SenderThread:246776 [sender.py:send():179] send: summary
2021-07-09 14:26:24,548 INFO SenderThread:246776 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 14:26:24,578 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2
2021-07-09 14:26:24,578 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/wandb-summary.json
2021-07-09 14:26:25,579 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:26:26,793 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:26:26,794 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:26:42,104 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:26:42,104 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:26:44,587 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:26:47,351 DEBUG SenderThread:246776 [sender.py:send():179] send: stats
2021-07-09 14:26:57,242 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:26:57,242 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:27:12,371 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:27:12,372 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:27:17,427 DEBUG SenderThread:246776 [sender.py:send():179] send: stats
2021-07-09 14:27:27,503 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:27:27,504 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:27:36,610 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:27:38,611 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:27:40,612 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:27:42,613 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:27:42,635 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:27:42,636 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:27:47,505 DEBUG SenderThread:246776 [sender.py:send():179] send: stats
2021-07-09 14:27:47,615 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2
2021-07-09 14:27:47,684 DEBUG SenderThread:246776 [sender.py:send():179] send: history
2021-07-09 14:27:47,684 DEBUG SenderThread:246776 [sender.py:send():179] send: summary
2021-07-09 14:27:47,684 INFO SenderThread:246776 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 14:27:48,086 INFO Thread-20 :246776 [upload_job.py:push():137] Uploaded file /tmp/tmp4f0a6i2mwandb/196d2q98-events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2
2021-07-09 14:27:48,616 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:27:48,616 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/wandb-summary.json
2021-07-09 14:27:50,617 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:27:57,771 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:27:57,771 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:28:06,624 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:28:12,907 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:28:12,907 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:28:17,583 DEBUG SenderThread:246776 [sender.py:send():179] send: stats
2021-07-09 14:28:28,040 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:28:28,040 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:28:43,180 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:28:43,181 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:28:47,661 DEBUG SenderThread:246776 [sender.py:send():179] send: stats
2021-07-09 14:28:54,643 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:28:56,644 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:28:58,315 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:28:58,315 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:28:58,645 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:29:00,646 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:29:06,648 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:29:06,649 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2
2021-07-09 14:29:06,812 DEBUG SenderThread:246776 [sender.py:send():179] send: history
2021-07-09 14:29:06,812 DEBUG SenderThread:246776 [sender.py:send():179] send: summary
2021-07-09 14:29:06,813 INFO SenderThread:246776 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 14:29:07,649 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:29:07,649 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/wandb-summary.json
2021-07-09 14:29:08,649 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:29:13,446 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:29:13,446 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
2021-07-09 14:29:17,738 DEBUG SenderThread:246776 [sender.py:send():179] send: stats
2021-07-09 14:29:24,657 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log
2021-07-09 14:29:28,577 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 14:29:28,578 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status
|