2021-07-09 14:14:46,066 INFO MainThread:246776 [internal.py:wandb_internal():88] W&B internal server running at pid: 246776, started at: 2021-07-09 14:14:46.066421 2021-07-09 14:14:46,069 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: check_version 2021-07-09 14:14:46,069 INFO WriterThread:246776 [datastore.py:open_for_write():80] open: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/run-2k8cnty2.wandb 2021-07-09 14:14:46,070 DEBUG SenderThread:246776 [sender.py:send():179] send: header 2021-07-09 14:14:46,070 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: check_version 2021-07-09 14:14:46,106 DEBUG SenderThread:246776 [sender.py:send():179] send: run 2021-07-09 14:14:46,291 INFO SenderThread:246776 [dir_watcher.py:__init__():168] watching files in: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files 2021-07-09 14:14:46,291 INFO SenderThread:246776 [sender.py:_start_run_threads():716] run started: 2k8cnty2 with start time 1625840085 2021-07-09 14:14:46,291 DEBUG SenderThread:246776 [sender.py:send():179] send: summary 2021-07-09 14:14:46,291 INFO SenderThread:246776 [sender.py:_save_file():841] saving file wandb-summary.json with policy end 2021-07-09 14:14:46,292 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: run_start 2021-07-09 14:14:47,296 INFO Thread-8 :246776 [dir_watcher.py:_on_file_created():216] file/dir created: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/wandb-summary.json 2021-07-09 14:14:47,469 DEBUG HandlerThread:246776 [meta.py:__init__():39] meta init 2021-07-09 14:14:47,469 DEBUG HandlerThread:246776 [meta.py:__init__():53] meta init done 2021-07-09 14:14:47,469 DEBUG HandlerThread:246776 [meta.py:probe():210] probe 2021-07-09 14:14:47,470 DEBUG HandlerThread:246776 [meta.py:_setup_git():200] setup git 2021-07-09 14:14:47,497 DEBUG HandlerThread:246776 [meta.py:_setup_git():207] setup git done 2021-07-09 14:14:47,498 DEBUG HandlerThread:246776 [meta.py:_save_pip():57] save pip 2021-07-09 14:14:47,498 DEBUG HandlerThread:246776 [meta.py:_save_pip():71] save pip done 2021-07-09 14:14:47,498 DEBUG HandlerThread:246776 [meta.py:_save_conda():78] save conda 2021-07-09 14:14:48,296 INFO Thread-8 :246776 [dir_watcher.py:_on_file_created():216] file/dir created: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/conda-environment.yaml 2021-07-09 14:14:48,296 INFO Thread-8 :246776 [dir_watcher.py:_on_file_created():216] file/dir created: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/requirements.txt 2021-07-09 14:14:49,937 DEBUG HandlerThread:246776 [meta.py:_save_conda():86] save conda done 2021-07-09 14:14:49,937 DEBUG HandlerThread:246776 [meta.py:probe():252] probe done 2021-07-09 14:14:49,940 DEBUG SenderThread:246776 [sender.py:send():179] send: files 2021-07-09 14:14:49,940 INFO SenderThread:246776 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now 2021-07-09 14:14:49,947 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:14:49,947 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:14:50,072 DEBUG SenderThread:246776 [sender.py:send():179] send: config 2021-07-09 14:14:50,073 DEBUG SenderThread:246776 [sender.py:send():179] send: config 2021-07-09 14:14:50,073 DEBUG SenderThread:246776 [sender.py:send():179] send: config 2021-07-09 14:14:50,073 DEBUG SenderThread:246776 [sender.py:send():179] send: config 2021-07-09 14:14:50,296 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/conda-environment.yaml 2021-07-09 14:14:50,297 INFO Thread-8 :246776 [dir_watcher.py:_on_file_created():216] file/dir created: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:14:50,297 INFO Thread-8 :246776 [dir_watcher.py:_on_file_created():216] file/dir created: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/wandb-metadata.json 2021-07-09 14:14:50,380 INFO Thread-11 :246776 [upload_job.py:push():137] Uploaded file /tmp/tmp4f0a6i2mwandb/2kbqshuw-wandb-metadata.json 2021-07-09 14:14:52,297 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:14:54,298 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:15:02,301 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:15:04,302 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:15:05,089 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:15:05,089 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:15:08,303 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:15:10,304 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:15:15,553 DEBUG SenderThread:246776 [sender.py:send():179] send: stats 2021-07-09 14:15:17,307 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/config.yaml 2021-07-09 14:15:19,308 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:15:20,261 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:15:20,261 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:15:21,309 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:15:27,311 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:15:27,425 INFO HandlerThread:246776 [handler.py:handle_tbrecord():547] handling tbrecord: tbrecord { log_dir: "/home/cahya/Work/flax-community/gpt2-medium-indonesian" save: true } 2021-07-09 14:15:27,426 DEBUG HandlerThread:246776 [config_util.py:dict_from_config_file():101] no default config file found in config-defaults.yaml 2021-07-09 14:15:27,434 DEBUG SenderThread:246776 [sender.py:send():179] send: tbrecord 2021-07-09 14:15:27,435 DEBUG SenderThread:246776 [sender.py:send():179] send: files 2021-07-09 14:15:27,435 INFO SenderThread:246776 [sender.py:_save_file():841] saving file events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2 with policy live 2021-07-09 14:15:27,879 INFO Thread-14 :246776 [upload_job.py:push():137] Uploaded file /tmp/tmp4f0a6i2mwandb/2a4g7r5i-events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2 2021-07-09 14:15:28,312 INFO Thread-8 :246776 [dir_watcher.py:_on_file_created():216] file/dir created: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2 2021-07-09 14:15:29,312 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:15:31,313 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:15:35,424 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:15:35,424 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:15:45,633 DEBUG SenderThread:246776 [sender.py:send():179] send: stats 2021-07-09 14:15:50,786 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:15:50,786 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:16:05,918 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:16:05,918 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:16:15,713 DEBUG SenderThread:246776 [sender.py:send():179] send: stats 2021-07-09 14:16:21,055 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:16:21,056 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:16:36,187 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:16:36,187 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:16:45,342 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:16:45,792 DEBUG SenderThread:246776 [sender.py:send():179] send: stats 2021-07-09 14:16:47,342 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:16:49,343 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:16:51,317 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:16:51,317 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:16:51,344 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:17:05,350 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:17:06,350 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2 2021-07-09 14:17:06,451 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:17:06,451 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:17:06,818 INFO Thread-15 :246776 [upload_job.py:push():137] Uploaded file /tmp/tmp4f0a6i2mwandb/5degf70d-events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2 2021-07-09 14:17:07,350 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:17:12,353 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:17:14,354 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:17:15,871 DEBUG SenderThread:246776 [sender.py:send():179] send: stats 2021-07-09 14:17:16,355 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:17:21,580 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:17:21,580 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:17:28,359 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:17:30,360 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:17:32,361 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:17:36,712 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:17:36,712 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:17:38,363 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:17:38,663 DEBUG SenderThread:246776 [sender.py:send():179] send: history 2021-07-09 14:17:38,663 DEBUG SenderThread:246776 [sender.py:send():179] send: summary 2021-07-09 14:17:38,663 INFO SenderThread:246776 [sender.py:_save_file():841] saving file wandb-summary.json with policy end 2021-07-09 14:17:39,364 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2 2021-07-09 14:17:39,364 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/wandb-summary.json 2021-07-09 14:17:40,364 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:17:45,949 DEBUG SenderThread:246776 [sender.py:send():179] send: stats 2021-07-09 14:17:46,366 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:17:51,907 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:17:51,907 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:18:02,372 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:18:07,040 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:18:07,040 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:18:16,028 DEBUG SenderThread:246776 [sender.py:send():179] send: stats 2021-07-09 14:18:22,169 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:18:22,169 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:18:37,300 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:18:37,300 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:18:46,105 DEBUG SenderThread:246776 [sender.py:send():179] send: stats 2021-07-09 14:18:52,428 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:18:52,428 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:19:04,397 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:19:06,398 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:19:07,560 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:19:07,560 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:19:08,398 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:19:10,399 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:19:14,401 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:19:15,401 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2 2021-07-09 14:19:15,821 DEBUG SenderThread:246776 [sender.py:send():179] send: history 2021-07-09 14:19:15,821 DEBUG SenderThread:246776 [sender.py:send():179] send: summary 2021-07-09 14:19:15,821 INFO SenderThread:246776 [sender.py:_save_file():841] saving file wandb-summary.json with policy end 2021-07-09 14:19:15,892 INFO Thread-16 :246776 [upload_job.py:push():137] Uploaded file /tmp/tmp4f0a6i2mwandb/2a28xt0m-events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2 2021-07-09 14:19:16,180 DEBUG SenderThread:246776 [sender.py:send():179] send: stats 2021-07-09 14:19:16,402 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:19:16,402 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/wandb-summary.json 2021-07-09 14:19:22,689 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:19:22,690 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:19:23,405 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:19:35,410 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:19:37,819 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:19:37,819 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:19:46,259 DEBUG SenderThread:246776 [sender.py:send():179] send: stats 2021-07-09 14:19:52,949 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:19:52,949 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:20:08,078 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:20:08,078 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:20:16,335 DEBUG SenderThread:246776 [sender.py:send():179] send: stats 2021-07-09 14:20:23,209 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:20:23,209 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:20:27,430 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:20:29,431 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:20:31,432 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:20:33,433 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:20:37,434 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:20:37,965 DEBUG SenderThread:246776 [sender.py:send():179] send: history 2021-07-09 14:20:37,965 DEBUG SenderThread:246776 [sender.py:send():179] send: summary 2021-07-09 14:20:37,966 INFO SenderThread:246776 [sender.py:_save_file():841] saving file wandb-summary.json with policy end 2021-07-09 14:20:38,340 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:20:38,340 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:20:38,435 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2 2021-07-09 14:20:38,435 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/wandb-summary.json 2021-07-09 14:20:38,913 INFO Thread-17 :246776 [upload_job.py:push():137] Uploaded file /tmp/tmp4f0a6i2mwandb/i981d3j1-events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2 2021-07-09 14:20:39,435 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:20:43,437 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:20:46,411 DEBUG SenderThread:246776 [sender.py:send():179] send: stats 2021-07-09 14:20:53,491 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:20:53,491 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:20:57,442 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:21:08,625 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:21:08,625 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:21:16,486 DEBUG SenderThread:246776 [sender.py:send():179] send: stats 2021-07-09 14:21:23,754 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:21:23,754 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:21:38,887 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:21:38,888 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:21:46,566 DEBUG SenderThread:246776 [sender.py:send():179] send: stats 2021-07-09 14:21:54,019 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:21:54,020 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:21:56,465 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:21:58,465 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:22:00,466 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:22:02,467 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:22:08,113 DEBUG SenderThread:246776 [sender.py:send():179] send: history 2021-07-09 14:22:08,114 DEBUG SenderThread:246776 [sender.py:send():179] send: summary 2021-07-09 14:22:08,114 INFO SenderThread:246776 [sender.py:_save_file():841] saving file wandb-summary.json with policy end 2021-07-09 14:22:08,469 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:22:08,470 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2 2021-07-09 14:22:08,470 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/wandb-summary.json 2021-07-09 14:22:09,013 INFO Thread-18 :246776 [upload_job.py:push():137] Uploaded file /tmp/tmp4f0a6i2mwandb/otuqhtyr-events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2 2021-07-09 14:22:09,151 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:22:09,151 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:22:09,470 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:22:10,470 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:22:16,473 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:22:16,645 DEBUG SenderThread:246776 [sender.py:send():179] send: stats 2021-07-09 14:22:24,304 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:22:24,304 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:22:28,478 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:22:39,446 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:22:39,447 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:22:46,723 DEBUG SenderThread:246776 [sender.py:send():179] send: stats 2021-07-09 14:22:54,575 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:22:54,575 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:23:09,708 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:23:09,708 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:23:16,801 DEBUG SenderThread:246776 [sender.py:send():179] send: stats 2021-07-09 14:23:20,500 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:23:22,501 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:23:24,502 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:23:24,837 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:23:24,838 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:23:26,503 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:23:30,505 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:23:31,259 DEBUG SenderThread:246776 [sender.py:send():179] send: history 2021-07-09 14:23:31,259 DEBUG SenderThread:246776 [sender.py:send():179] send: summary 2021-07-09 14:23:31,260 INFO SenderThread:246776 [sender.py:_save_file():841] saving file wandb-summary.json with policy end 2021-07-09 14:23:31,505 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2 2021-07-09 14:23:31,506 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/wandb-summary.json 2021-07-09 14:23:32,506 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:23:39,978 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:23:39,979 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:23:46,879 DEBUG SenderThread:246776 [sender.py:send():179] send: stats 2021-07-09 14:23:50,514 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:23:55,122 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:23:55,122 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:24:10,254 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:24:10,254 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:24:16,957 DEBUG SenderThread:246776 [sender.py:send():179] send: stats 2021-07-09 14:24:25,385 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:24:25,385 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:24:40,609 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:24:40,609 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:24:42,535 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:24:44,536 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:24:46,536 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:24:47,034 DEBUG SenderThread:246776 [sender.py:send():179] send: stats 2021-07-09 14:24:52,539 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:24:53,395 DEBUG SenderThread:246776 [sender.py:send():179] send: history 2021-07-09 14:24:53,395 DEBUG SenderThread:246776 [sender.py:send():179] send: summary 2021-07-09 14:24:53,395 INFO SenderThread:246776 [sender.py:_save_file():841] saving file wandb-summary.json with policy end 2021-07-09 14:24:53,539 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2 2021-07-09 14:24:53,540 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/wandb-summary.json 2021-07-09 14:24:54,013 INFO Thread-19 :246776 [upload_job.py:push():137] Uploaded file /tmp/tmp4f0a6i2mwandb/72n81odb-events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2 2021-07-09 14:24:54,540 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:24:55,540 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:24:55,836 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:24:55,836 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:25:10,974 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:25:10,974 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:25:13,548 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:25:17,112 DEBUG SenderThread:246776 [sender.py:send():179] send: stats 2021-07-09 14:25:26,143 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:25:26,144 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:25:41,276 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:25:41,276 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:25:47,189 DEBUG SenderThread:246776 [sender.py:send():179] send: stats 2021-07-09 14:25:56,420 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:25:56,420 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:26:11,570 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:26:11,570 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:26:13,573 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:26:15,574 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:26:17,268 DEBUG SenderThread:246776 [sender.py:send():179] send: stats 2021-07-09 14:26:17,575 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:26:23,578 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:26:24,547 DEBUG SenderThread:246776 [sender.py:send():179] send: history 2021-07-09 14:26:24,547 DEBUG SenderThread:246776 [sender.py:send():179] send: summary 2021-07-09 14:26:24,548 INFO SenderThread:246776 [sender.py:_save_file():841] saving file wandb-summary.json with policy end 2021-07-09 14:26:24,578 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2 2021-07-09 14:26:24,578 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/wandb-summary.json 2021-07-09 14:26:25,579 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:26:26,793 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:26:26,794 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:26:42,104 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:26:42,104 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:26:44,587 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:26:47,351 DEBUG SenderThread:246776 [sender.py:send():179] send: stats 2021-07-09 14:26:57,242 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:26:57,242 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:27:12,371 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:27:12,372 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:27:17,427 DEBUG SenderThread:246776 [sender.py:send():179] send: stats 2021-07-09 14:27:27,503 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:27:27,504 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:27:36,610 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:27:38,611 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:27:40,612 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:27:42,613 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:27:42,635 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:27:42,636 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:27:47,505 DEBUG SenderThread:246776 [sender.py:send():179] send: stats 2021-07-09 14:27:47,615 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2 2021-07-09 14:27:47,684 DEBUG SenderThread:246776 [sender.py:send():179] send: history 2021-07-09 14:27:47,684 DEBUG SenderThread:246776 [sender.py:send():179] send: summary 2021-07-09 14:27:47,684 INFO SenderThread:246776 [sender.py:_save_file():841] saving file wandb-summary.json with policy end 2021-07-09 14:27:48,086 INFO Thread-20 :246776 [upload_job.py:push():137] Uploaded file /tmp/tmp4f0a6i2mwandb/196d2q98-events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2 2021-07-09 14:27:48,616 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:27:48,616 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/wandb-summary.json 2021-07-09 14:27:50,617 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:27:57,771 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:27:57,771 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:28:06,624 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:28:12,907 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:28:12,907 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:28:17,583 DEBUG SenderThread:246776 [sender.py:send():179] send: stats 2021-07-09 14:28:28,040 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:28:28,040 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:28:43,180 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:28:43,181 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:28:47,661 DEBUG SenderThread:246776 [sender.py:send():179] send: stats 2021-07-09 14:28:54,643 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:28:56,644 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:28:58,315 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:28:58,315 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:28:58,645 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:29:00,646 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:29:06,648 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:29:06,649 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/events.out.tfevents.1625840127.t1v-n-528d9406-w-0.245719.3.v2 2021-07-09 14:29:06,812 DEBUG SenderThread:246776 [sender.py:send():179] send: history 2021-07-09 14:29:06,812 DEBUG SenderThread:246776 [sender.py:send():179] send: summary 2021-07-09 14:29:06,813 INFO SenderThread:246776 [sender.py:_save_file():841] saving file wandb-summary.json with policy end 2021-07-09 14:29:07,649 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:29:07,649 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/wandb-summary.json 2021-07-09 14:29:08,649 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:29:13,446 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:29:13,446 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status 2021-07-09 14:29:17,738 DEBUG SenderThread:246776 [sender.py:send():179] send: stats 2021-07-09 14:29:24,657 INFO Thread-8 :246776 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/files/output.log 2021-07-09 14:29:28,577 DEBUG HandlerThread:246776 [handler.py:handle_request():124] handle_request: stop_status 2021-07-09 14:29:28,578 DEBUG SenderThread:246776 [sender.py:send_request():193] send_request: stop_status