logs/20231117083158/wandb/debug-internal.log

2023-11-17 08:33:59,189 INFO    StreamThr :5676 [internal.py:wandb_internal():86] W&B internal server running at pid: 5676, started at: 2023-11-17 08:33:59.188541
2023-11-17 08:33:59,191 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status
2023-11-17 08:33:59,197 INFO    WriterThread:5676 [datastore.py:open_for_write():85] open: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/run-f858xji5.wandb
2023-11-17 08:33:59,197 DEBUG   SenderThread:5676 [sender.py:send():379] send: header
2023-11-17 08:33:59,217 DEBUG   SenderThread:5676 [sender.py:send():379] send: run
2023-11-17 08:33:59,506 INFO    SenderThread:5676 [dir_watcher.py:__init__():211] watching files in: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files
2023-11-17 08:33:59,507 INFO    SenderThread:5676 [sender.py:_start_run_threads():1122] run started: f858xji5 with start time 1700210039.194491
2023-11-17 08:33:59,507 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:33:59,507 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:33:59,512 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: check_version
2023-11-17 08:33:59,512 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: check_version
2023-11-17 08:33:59,704 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: run_start
2023-11-17 08:33:59,713 DEBUG   HandlerThread:5676 [system_info.py:__init__():31] System info init
2023-11-17 08:33:59,713 DEBUG   HandlerThread:5676 [system_info.py:__init__():46] System info init done
2023-11-17 08:33:59,713 INFO    HandlerThread:5676 [system_monitor.py:start():181] Starting system monitor
2023-11-17 08:33:59,713 INFO    SystemMonitor:5676 [system_monitor.py:_start():145] Starting system asset monitoring threads
2023-11-17 08:33:59,714 INFO    HandlerThread:5676 [system_monitor.py:probe():201] Collecting system info
2023-11-17 08:33:59,715 INFO    SystemMonitor:5676 [interfaces.py:start():190] Started cpu monitoring
2023-11-17 08:33:59,716 INFO    SystemMonitor:5676 [interfaces.py:start():190] Started disk monitoring
2023-11-17 08:33:59,717 INFO    SystemMonitor:5676 [interfaces.py:start():190] Started gpu monitoring
2023-11-17 08:33:59,718 INFO    SystemMonitor:5676 [interfaces.py:start():190] Started memory monitoring
2023-11-17 08:33:59,718 INFO    SystemMonitor:5676 [interfaces.py:start():190] Started network monitoring
2023-11-17 08:33:59,722 DEBUG   HandlerThread:5676 [system_info.py:probe():195] Probing system
2023-11-17 08:33:59,727 DEBUG   HandlerThread:5676 [system_info.py:_probe_git():180] Probing git
2023-11-17 08:33:59,740 DEBUG   HandlerThread:5676 [system_info.py:_probe_git():188] Probing git done
2023-11-17 08:33:59,740 DEBUG   HandlerThread:5676 [system_info.py:probe():240] Probing system done
2023-11-17 08:33:59,740 DEBUG   HandlerThread:5676 [system_monitor.py:probe():210] {'os': 'Linux-5.15.120+-x86_64-with-glibc2.35', 'python': '3.10.12', 'heartbeatAt': '2023-11-17T08:33:59.722363', 'startedAt': '2023-11-17T08:33:59.181100', 'docker': None, 'cuda': None, 'args': ('--sample_prompts=/content/LoRA/config/sample_prompt.toml', '--config_file=/content/LoRA/config/config_file.toml', '--wandb_api_key=540d5e6b5446a44f2f5c8269e0da63b4c7ad408c'), 'state': 'running', 'program': '/content/kohya-trainer/sdxl_train_network.py', 'codePath': 'sdxl_train_network.py', 'git': {'remote': 'https://github.com/qaneel/kohya-trainer', 'commit': '05811296f6dc987f67f194689e106a326017b9d4'}, 'email': None, 'root': '/content/kohya-trainer', 'host': '3af72b0d0270', 'username': 'root', 'executable': '/usr/bin/python3', 'cpu_count': 1, 'cpu_count_logical': 2, 'cpu_freq': {'current': 2299.998, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2299.998, 'min': 0.0, 'max': 0.0}, {'current': 2299.998, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 78.1898422241211, 'used': 41.74154281616211}, 'gpu': 'Tesla T4', 'gpu_count': 1, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 12.6783447265625}}
2023-11-17 08:33:59,740 INFO    HandlerThread:5676 [system_monitor.py:probe():211] Finished collecting system info
2023-11-17 08:33:59,740 INFO    HandlerThread:5676 [system_monitor.py:probe():214] Publishing system info
2023-11-17 08:33:59,740 DEBUG   HandlerThread:5676 [system_info.py:_save_pip():51] Saving list of pip packages installed into the current environment
2023-11-17 08:33:59,741 DEBUG   HandlerThread:5676 [system_info.py:_save_pip():67] Saving pip packages done
2023-11-17 08:33:59,755 INFO    HandlerThread:5676 [system_monitor.py:probe():216] Finished publishing system info
2023-11-17 08:33:59,768 DEBUG   SenderThread:5676 [sender.py:send():379] send: files
2023-11-17 08:33:59,768 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-metadata.json with policy now
2023-11-17 08:33:59,776 DEBUG   SenderThread:5676 [sender.py:send():379] send: telemetry
2023-11-17 08:33:59,816 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:33:59,817 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:34:00,349 INFO    wandb-upload_0:5676 [upload_job.py:push():131] Uploaded file /tmp/tmp5kii06quwandb/24x79thl-wandb-metadata.json
2023-11-17 08:34:00,508 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_created():272] file/dir created: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/requirements.txt
2023-11-17 08:34:00,508 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_created():272] file/dir created: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-metadata.json
2023-11-17 08:34:00,509 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_created():272] file/dir created: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:34:00,509 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_created():272] file/dir created: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/output.log
2023-11-17 08:34:02,508 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/output.log
2023-11-17 08:34:04,249 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:34:09,250 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:34:14,251 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:34:14,775 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:34:14,776 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:34:15,117 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:34:19,835 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:34:21,204 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:34:21,206 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:34:21,206 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:34:21,209 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:34:21,515 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:34:25,210 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:34:26,805 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:34:26,806 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:34:26,806 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:34:26,809 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:34:27,518 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:34:29,774 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:34:29,775 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:34:30,895 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:34:31,519 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/config.yaml
2023-11-17 08:34:34,164 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:34:34,166 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:34:34,166 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:34:34,168 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:34:34,521 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:34:36,169 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:34:41,170 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:34:41,266 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:34:41,268 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:34:41,268 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:34:41,270 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:34:41,523 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:34:43,193 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:34:43,195 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:34:43,195 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:34:43,195 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:34:43,524 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:34:44,899 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:34:44,900 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:34:46,942 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:34:49,334 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:34:49,335 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:34:49,335 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:34:49,338 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:34:49,527 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:34:52,338 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:34:57,339 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:34:57,438 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:34:57,439 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:34:57,440 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:34:57,442 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:34:57,530 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:34:59,719 DEBUG   SystemMonitor:5676 [system_monitor.py:_start():159] Starting system metrics aggregation loop
2023-11-17 08:34:59,720 DEBUG   SenderThread:5676 [sender.py:send():379] send: stats
2023-11-17 08:34:59,900 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:34:59,900 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:35:03,059 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:35:03,726 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:35:03,727 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:35:03,728 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:35:03,728 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:35:04,532 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:35:08,729 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:35:10,018 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:35:10,019 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:35:10,019 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:35:10,022 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:35:10,535 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:35:14,023 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:35:14,900 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:35:14,900 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:35:18,823 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:35:18,824 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:35:18,825 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:35:18,825 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:35:19,539 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:35:19,826 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:35:24,827 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:35:25,174 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:35:25,175 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:35:25,176 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:35:25,178 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:35:25,541 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:35:29,721 DEBUG   SenderThread:5676 [sender.py:send():379] send: stats
2023-11-17 08:35:29,899 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:35:29,900 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:35:29,934 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:35:31,568 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:35:31,569 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:35:31,570 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:35:31,570 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:35:32,544 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:35:35,571 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:35:38,233 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:35:38,235 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:35:38,235 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:35:38,238 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:35:38,547 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:35:41,239 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:35:44,900 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:35:44,900 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:35:45,128 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:35:45,129 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:35:45,130 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:35:45,130 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:35:45,549 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:35:47,131 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:35:51,509 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:35:51,510 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:35:51,511 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:35:51,513 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:35:51,552 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:35:52,514 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:35:57,515 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:35:58,081 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:35:58,082 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:35:58,082 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:35:58,083 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:35:58,554 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:35:59,722 DEBUG   SenderThread:5676 [sender.py:send():379] send: stats
2023-11-17 08:36:00,353 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:36:00,354 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:36:03,433 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:36:06,396 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:36:06,397 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:36:06,398 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:36:06,400 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:36:06,557 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:36:09,401 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:36:11,006 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:36:11,007 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:36:11,007 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:36:11,008 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:36:11,559 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:36:15,009 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:36:15,353 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:36:15,354 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:36:17,319 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:36:17,320 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:36:17,320 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:36:17,323 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:36:17,562 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:36:20,323 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:36:23,837 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:36:23,838 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:36:23,838 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:36:23,839 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:36:24,564 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:36:25,839 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:36:28,600 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:36:28,602 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:36:28,602 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:36:28,604 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:36:29,566 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:36:29,723 DEBUG   SenderThread:5676 [sender.py:send():379] send: stats
2023-11-17 08:36:30,584 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:36:30,585 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:36:31,637 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:36:35,533 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:36:35,534 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:36:35,535 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:36:35,535 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:36:35,568 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:36:37,536 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:36:42,537 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:36:42,644 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:36:42,645 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:36:42,645 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:36:42,647 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:36:43,571 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:36:45,585 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:36:45,586 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:36:47,650 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:36:50,041 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:36:50,043 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:36:50,043 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:36:50,043 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:36:50,574 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:36:53,044 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:36:57,064 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:36:57,066 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:36:57,066 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:36:57,068 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:36:57,585 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:36:58,069 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:36:59,723 DEBUG   SenderThread:5676 [sender.py:send():379] send: stats
2023-11-17 08:37:00,585 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:37:00,586 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:37:03,629 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:37:03,630 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:37:03,630 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:37:03,631 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:37:03,631 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:37:04,588 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:37:08,644 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:37:10,398 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:37:10,399 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:37:10,399 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:37:10,402 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:37:10,590 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:37:14,403 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:37:15,585 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:37:15,587 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:37:17,475 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:37:17,477 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:37:17,477 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:37:17,478 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:37:17,593 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:37:19,478 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:37:23,807 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:37:23,808 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:37:23,808 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:37:23,811 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:37:24,595 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:37:24,811 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:37:29,724 DEBUG   SenderThread:5676 [sender.py:send():379] send: stats
2023-11-17 08:37:30,585 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:37:30,587 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:37:30,587 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:37:31,197 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:37:31,198 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:37:31,198 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:37:31,199 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:37:31,598 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:37:36,200 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:37:37,531 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:37:37,532 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:37:37,533 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:37:37,535 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:37:37,601 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:37:41,536 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:37:45,483 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:37:45,484 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:37:45,485 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:37:45,485 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:37:45,586 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:37:45,586 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:37:45,604 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:37:46,644 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:37:51,645 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:37:53,283 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:37:53,284 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:37:53,285 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:37:53,287 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:37:53,607 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:37:56,421 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:37:56,422 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:37:56,422 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:37:56,423 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:37:56,608 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:37:57,424 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:37:59,725 DEBUG   SenderThread:5676 [sender.py:send():379] send: stats
2023-11-17 08:38:00,585 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:38:00,586 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:38:02,711 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:38:03,398 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:38:03,399 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:38:03,399 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:38:03,400 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:38:03,611 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:38:08,401 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:38:09,929 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:38:09,930 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:38:09,931 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:38:09,931 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:38:10,613 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:38:13,933 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:38:15,586 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:38:15,588 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:38:16,655 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:38:16,656 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:38:16,657 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:38:16,659 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:38:17,616 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:38:19,660 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:38:21,011 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:38:21,012 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:38:21,013 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:38:21,013 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:38:21,617 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:38:25,017 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:38:27,363 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:38:27,364 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:38:27,365 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:38:27,367 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:38:27,620 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:38:29,726 DEBUG   SenderThread:5676 [sender.py:send():379] send: stats
2023-11-17 08:38:30,586 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:38:30,587 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:38:30,626 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:38:33,929 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:38:33,930 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:38:33,930 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:38:33,931 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:38:34,622 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:38:35,932 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:38:40,662 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:38:40,663 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:38:40,663 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:38:40,664 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:38:41,625 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:38:41,664 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:38:45,586 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:38:45,587 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:38:47,631 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:38:47,706 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:38:47,707 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:38:47,708 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:38:47,708 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:38:48,628 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:38:50,522 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:38:50,524 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:38:50,524 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:38:50,525 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:38:50,629 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:38:53,526 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:38:57,084 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:38:57,086 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:38:57,086 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:38:57,088 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:38:57,632 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:38:59,089 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:38:59,727 DEBUG   SenderThread:5676 [sender.py:send():379] send: stats
2023-11-17 08:39:00,586 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:39:00,587 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:39:03,805 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:39:03,806 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:39:03,807 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:39:03,807 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:39:04,635 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:39:04,808 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:39:09,809 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:39:10,844 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:39:10,846 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:39:10,846 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:39:10,846 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:39:11,637 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:39:14,847 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:39:15,586 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:39:15,587 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:39:18,495 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:39:18,496 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:39:18,496 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:39:18,496 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:39:18,640 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:39:20,497 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:39:25,197 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:39:25,198 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:39:25,198 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:39:25,199 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:39:25,643 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:39:26,200 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:39:29,727 DEBUG   SenderThread:5676 [sender.py:send():379] send: stats
2023-11-17 08:39:30,586 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:39:30,587 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:39:31,576 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:39:31,577 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:39:31,577 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:39:31,578 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:39:31,578 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:39:31,645 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:39:36,579 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:39:38,079 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:39:38,080 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:39:38,080 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:39:38,081 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:39:38,647 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:39:42,082 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:39:45,488 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:39:45,489 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:39:45,490 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:39:45,490 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:39:45,587 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:39:45,587 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:39:45,650 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:39:47,669 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:39:52,670 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:39:53,424 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:39:53,426 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:39:53,426 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:39:53,426 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:39:53,664 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:39:58,214 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:39:58,217 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:39:58,218 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:39:58,218 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:39:58,218 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:39:58,666 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:39:59,728 DEBUG   SenderThread:5676 [sender.py:send():379] send: stats
2023-11-17 08:40:00,587 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:40:00,587 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:40:03,716 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:40:06,535 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:40:06,537 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:40:06,537 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:40:06,538 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:40:06,668 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:40:09,539 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:40:12,914 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:40:12,915 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:40:12,915 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:40:12,916 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:40:13,671 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:40:14,916 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:40:15,587 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:40:15,587 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:40:20,700 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:40:21,554 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:40:21,555 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:40:21,555 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:40:21,555 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:40:21,674 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:40:26,557 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:40:28,596 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:40:28,597 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:40:28,597 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:40:28,598 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:40:28,676 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:40:29,729 DEBUG   SenderThread:5676 [sender.py:send():379] send: stats
2023-11-17 08:40:30,600 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:40:30,600 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:40:31,719 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:40:35,624 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:40:35,625 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:40:35,625 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:40:35,626 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:40:35,680 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:40:37,627 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:40:42,628 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:40:42,839 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:40:42,840 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:40:42,840 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:40:42,843 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:40:43,683 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:40:45,600 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:40:45,601 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:40:45,718 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:40:45,737 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:40:45,738 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:40:45,742 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:40:46,684 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:40:47,743 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:40:52,458 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:40:52,459 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:40:52,460 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:40:52,460 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:40:52,686 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:40:53,461 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:40:58,462 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:40:59,634 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:40:59,635 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:40:59,635 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:40:59,636 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:40:59,689 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:40:59,729 DEBUG   SenderThread:5676 [sender.py:send():379] send: stats
2023-11-17 08:41:00,601 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:41:00,601 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:41:03,681 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:41:06,288 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:41:06,289 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:41:06,290 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:41:06,290 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:41:06,691 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:41:09,291 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:41:13,723 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:41:13,724 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:41:13,724 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:41:13,725 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:41:14,694 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:41:14,726 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:41:15,601 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:41:15,601 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:41:20,117 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:41:20,118 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:41:20,118 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:41:20,119 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:41:20,119 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:41:20,696 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:41:25,120 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:41:26,470 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:41:26,471 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:41:26,471 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:41:26,472 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:41:26,699 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:41:29,730 DEBUG   SenderThread:5676 [sender.py:send():379] send: stats
2023-11-17 08:41:30,601 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:41:30,601 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:41:30,602 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:41:34,765 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:41:34,766 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:41:34,767 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:41:34,769 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:41:35,702 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:41:35,769 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:41:40,770 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:41:41,147 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:41:41,148 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:41:41,148 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:41:41,149 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:41:41,704 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:41:45,601 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:41:45,601 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:41:46,687 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:41:47,110 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:41:47,111 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:41:47,112 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:41:47,112 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:41:47,707 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:41:52,113 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:41:53,652 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:41:53,653 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:41:53,653 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:41:53,653 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:41:53,709 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:41:57,655 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:41:59,731 DEBUG   SenderThread:5676 [sender.py:send():379] send: stats
2023-11-17 08:42:00,602 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:42:00,602 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:42:01,024 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:42:01,025 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:42:01,026 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:42:01,026 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:42:01,712 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:42:03,027 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:42:07,450 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:42:07,451 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:42:07,451 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:42:07,452 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:42:07,715 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:42:08,457 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:42:13,458 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:42:13,974 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:42:13,976 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:42:13,976 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:42:13,976 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:42:14,717 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:42:15,968 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:42:15,968 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:42:19,059 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:42:21,348 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:42:21,349 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:42:21,349 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:42:21,350 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:42:21,720 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:42:24,350 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:42:29,351 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:42:29,732 DEBUG   SenderThread:5676 [sender.py:send():379] send: stats
2023-11-17 08:42:29,864 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:42:29,865 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:42:29,866 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:42:29,866 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:42:30,723 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:42:30,968 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:42:30,969 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:42:35,018 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:42:38,032 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:42:38,033 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:42:38,034 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:42:38,034 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:42:38,726 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:42:40,035 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:42:45,036 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:42:45,430 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:42:45,431 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:42:45,431 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:42:45,431 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:42:45,729 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:42:45,969 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:42:45,970 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:42:50,107 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:42:51,807 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:42:51,808 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:42:51,809 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:42:51,809 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:42:52,732 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:42:55,810 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:42:58,318 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:42:58,319 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:42:58,320 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:42:58,320 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:42:58,734 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:42:59,733 DEBUG   SenderThread:5676 [sender.py:send():379] send: stats
2023-11-17 08:43:00,968 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:43:00,969 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:43:00,970 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:43:05,708 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:43:05,709 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:43:05,709 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:43:05,710 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:43:05,744 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:43:06,710 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:43:10,829 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:43:10,830 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:43:10,831 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:43:10,831 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:43:11,747 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:43:11,832 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:43:15,969 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:43:15,970 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:43:17,008 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:43:17,158 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:43:17,159 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:43:17,159 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:43:17,160 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:43:17,750 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:43:22,161 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:43:24,520 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:43:24,521 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:43:24,521 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:43:24,522 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:43:24,752 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:43:27,523 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:43:29,733 DEBUG   SenderThread:5676 [sender.py:send():379] send: stats
2023-11-17 08:43:30,970 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:43:30,970 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:43:31,554 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:43:31,556 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:43:31,556 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:43:31,556 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:43:31,755 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:43:32,557 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:43:37,558 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:43:37,890 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:43:37,892 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:43:37,893 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:43:37,893 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:43:37,893 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history
2023-11-17 08:43:37,894 WARNING HandlerThread:5676 [handler.py:handle_request_partial_history():553] Step 1 < 86. Dropping entry: {'loss/epoch': 0.11442129172352251, '_timestamp': 1700210617.890885}.
2023-11-17 08:43:38,758 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:43:39,758 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/output.log
2023-11-17 08:43:41,759 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/output.log
2023-11-17 08:43:42,759 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/output.log
2023-11-17 08:43:42,894 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:43:44,760 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/output.log
2023-11-17 08:43:45,970 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status
2023-11-17 08:43:45,970 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: stop_status
2023-11-17 08:43:46,761 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/output.log
2023-11-17 08:43:48,106 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:43:48,762 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/output.log
2023-11-17 08:43:50,762 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/output.log
2023-11-17 08:43:51,526 DEBUG   SenderThread:5676 [sender.py:send():379] send: exit
2023-11-17 08:43:51,526 INFO    SenderThread:5676 [sender.py:send_exit():584] handling exit code: 1
2023-11-17 08:43:51,526 INFO    SenderThread:5676 [sender.py:send_exit():586] handling runtime: 591
2023-11-17 08:43:51,526 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:43:51,527 INFO    SenderThread:5676 [sender.py:send_exit():592] send defer
2023-11-17 08:43:51,527 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: defer
2023-11-17 08:43:51,527 INFO    HandlerThread:5676 [handler.py:handle_request_defer():170] handle defer: 0
2023-11-17 08:43:51,527 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: defer
2023-11-17 08:43:51,527 INFO    SenderThread:5676 [sender.py:send_request_defer():608] handle sender defer: 0
2023-11-17 08:43:51,527 INFO    SenderThread:5676 [sender.py:transition_state():612] send defer: 1
2023-11-17 08:43:51,527 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: defer
2023-11-17 08:43:51,527 INFO    HandlerThread:5676 [handler.py:handle_request_defer():170] handle defer: 1
2023-11-17 08:43:51,528 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: defer
2023-11-17 08:43:51,528 INFO    SenderThread:5676 [sender.py:send_request_defer():608] handle sender defer: 1
2023-11-17 08:43:51,528 INFO    SenderThread:5676 [sender.py:transition_state():612] send defer: 2
2023-11-17 08:43:51,528 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: defer
2023-11-17 08:43:51,528 INFO    HandlerThread:5676 [handler.py:handle_request_defer():170] handle defer: 2
2023-11-17 08:43:51,528 INFO    HandlerThread:5676 [system_monitor.py:finish():190] Stopping system monitor
2023-11-17 08:43:51,528 DEBUG   SystemMonitor:5676 [system_monitor.py:_start():166] Finished system metrics aggregation loop
2023-11-17 08:43:51,528 DEBUG   SystemMonitor:5676 [system_monitor.py:_start():170] Publishing last batch of metrics
2023-11-17 08:43:51,529 INFO    HandlerThread:5676 [interfaces.py:finish():202] Joined cpu monitor
2023-11-17 08:43:51,529 INFO    HandlerThread:5676 [interfaces.py:finish():202] Joined disk monitor
2023-11-17 08:43:51,538 INFO    HandlerThread:5676 [interfaces.py:finish():202] Joined gpu monitor
2023-11-17 08:43:51,538 INFO    HandlerThread:5676 [interfaces.py:finish():202] Joined memory monitor
2023-11-17 08:43:51,538 INFO    HandlerThread:5676 [interfaces.py:finish():202] Joined network monitor
2023-11-17 08:43:51,539 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: defer
2023-11-17 08:43:51,539 INFO    SenderThread:5676 [sender.py:send_request_defer():608] handle sender defer: 2
2023-11-17 08:43:51,539 INFO    SenderThread:5676 [sender.py:transition_state():612] send defer: 3
2023-11-17 08:43:51,539 DEBUG   SenderThread:5676 [sender.py:send():379] send: stats
2023-11-17 08:43:51,539 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: defer
2023-11-17 08:43:51,540 INFO    HandlerThread:5676 [handler.py:handle_request_defer():170] handle defer: 3
2023-11-17 08:43:51,540 DEBUG   SenderThread:5676 [sender.py:send():379] send: history
2023-11-17 08:43:51,541 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: summary_record
2023-11-17 08:43:51,541 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:43:51,541 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: defer
2023-11-17 08:43:51,541 INFO    SenderThread:5676 [sender.py:send_request_defer():608] handle sender defer: 3
2023-11-17 08:43:51,541 INFO    SenderThread:5676 [sender.py:transition_state():612] send defer: 4
2023-11-17 08:43:51,542 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: defer
2023-11-17 08:43:51,542 INFO    HandlerThread:5676 [handler.py:handle_request_defer():170] handle defer: 4
2023-11-17 08:43:51,542 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: defer
2023-11-17 08:43:51,542 INFO    SenderThread:5676 [sender.py:send_request_defer():608] handle sender defer: 4
2023-11-17 08:43:51,542 INFO    SenderThread:5676 [sender.py:transition_state():612] send defer: 5
2023-11-17 08:43:51,542 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: defer
2023-11-17 08:43:51,542 INFO    HandlerThread:5676 [handler.py:handle_request_defer():170] handle defer: 5
2023-11-17 08:43:51,543 DEBUG   SenderThread:5676 [sender.py:send():379] send: summary
2023-11-17 08:43:51,543 INFO    SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end
2023-11-17 08:43:51,543 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: defer
2023-11-17 08:43:51,543 INFO    SenderThread:5676 [sender.py:send_request_defer():608] handle sender defer: 5
2023-11-17 08:43:51,543 INFO    SenderThread:5676 [sender.py:transition_state():612] send defer: 6
2023-11-17 08:43:51,543 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: defer
2023-11-17 08:43:51,543 INFO    HandlerThread:5676 [handler.py:handle_request_defer():170] handle defer: 6
2023-11-17 08:43:51,544 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: defer
2023-11-17 08:43:51,544 INFO    SenderThread:5676 [sender.py:send_request_defer():608] handle sender defer: 6
2023-11-17 08:43:51,544 INFO    SenderThread:5676 [sender.py:transition_state():612] send defer: 7
2023-11-17 08:43:51,544 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report
2023-11-17 08:43:51,544 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: defer
2023-11-17 08:43:51,544 INFO    HandlerThread:5676 [handler.py:handle_request_defer():170] handle defer: 7
2023-11-17 08:43:51,545 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: defer
2023-11-17 08:43:51,545 INFO    SenderThread:5676 [sender.py:send_request_defer():608] handle sender defer: 7
2023-11-17 08:43:51,763 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:43:52,526 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: poll_exit
2023-11-17 08:43:52,764 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/output.log
2023-11-17 08:43:54,657 INFO    SenderThread:5676 [sender.py:transition_state():612] send defer: 8
2023-11-17 08:43:54,657 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: poll_exit
2023-11-17 08:43:54,657 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: defer
2023-11-17 08:43:54,657 INFO    HandlerThread:5676 [handler.py:handle_request_defer():170] handle defer: 8
2023-11-17 08:43:54,658 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: defer
2023-11-17 08:43:54,658 INFO    SenderThread:5676 [sender.py:send_request_defer():608] handle sender defer: 8
2023-11-17 08:43:54,658 INFO    SenderThread:5676 [job_builder.py:build():280] Attempting to build job artifact
2023-11-17 08:43:54,658 INFO    SenderThread:5676 [job_builder.py:_get_source_type():389] is repo sourced job
2023-11-17 08:43:54,665 INFO    SenderThread:5676 [job_builder.py:build():363] adding wandb-job metadata file
2023-11-17 08:43:54,671 INFO    SenderThread:5676 [sender.py:transition_state():612] send defer: 9
2023-11-17 08:43:54,671 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: defer
2023-11-17 08:43:54,671 INFO    HandlerThread:5676 [handler.py:handle_request_defer():170] handle defer: 9
2023-11-17 08:43:54,672 DEBUG   SenderThread:5676 [sender.py:send():379] send: artifact
2023-11-17 08:43:54,764 INFO    Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/output.log
2023-11-17 08:43:55,177 INFO    wandb-upload_0:5676 [upload_job.py:push():86] Skipped uploading /root/.local/share/wandb/artifacts/staging/tmpaxl62r1n
2023-11-17 08:43:55,516 INFO    wandb-upload_1:5676 [upload_job.py:push():89] Uploaded file /root/.local/share/wandb/artifacts/staging/tmpbhp9625f
2023-11-17 08:43:56,385 INFO    SenderThread:5676 [sender.py:send_artifact():1476] sent artifact job-https___github.com_qaneel_kohya-trainer_sdxl_train_network.py - {'id': 'QXJ0aWZhY3Q6NjQxMzE0Njcx', 'digest': 'd545101fca5d4391fee1f3b40d5981eb', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjExNjM5NTUzOQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NjQwNjQ0OTYz', 'versionIndex': 0}}, 'version': 'latest'}
2023-11-17 08:43:56,385 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: defer
2023-11-17 08:43:56,385 INFO    SenderThread:5676 [sender.py:send_request_defer():608] handle sender defer: 9
2023-11-17 08:43:56,386 INFO    SenderThread:5676 [dir_watcher.py:finish():359] shutting down directory watcher
2023-11-17 08:43:56,765 INFO    SenderThread:5676 [dir_watcher.py:finish():389] scan: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files
2023-11-17 08:43:56,765 INFO    SenderThread:5676 [dir_watcher.py:finish():403] scan save: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/config.yaml config.yaml
2023-11-17 08:43:56,766 INFO    SenderThread:5676 [dir_watcher.py:finish():403] scan save: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/output.log output.log
2023-11-17 08:43:56,768 INFO    SenderThread:5676 [dir_watcher.py:finish():403] scan save: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json wandb-summary.json
2023-11-17 08:43:56,769 INFO    SenderThread:5676 [dir_watcher.py:finish():403] scan save: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-metadata.json wandb-metadata.json
2023-11-17 08:43:56,769 INFO    SenderThread:5676 [dir_watcher.py:finish():403] scan save: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/requirements.txt requirements.txt
2023-11-17 08:43:56,774 INFO    SenderThread:5676 [sender.py:transition_state():612] send defer: 10
2023-11-17 08:43:56,774 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: defer
2023-11-17 08:43:56,774 INFO    HandlerThread:5676 [handler.py:handle_request_defer():170] handle defer: 10
2023-11-17 08:43:56,777 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: defer
2023-11-17 08:43:56,778 INFO    SenderThread:5676 [sender.py:send_request_defer():608] handle sender defer: 10
2023-11-17 08:43:56,778 INFO    SenderThread:5676 [file_pusher.py:finish():159] shutting down file pusher
2023-11-17 08:43:56,983 INFO    wandb-upload_2:5676 [upload_job.py:push():131] Uploaded file /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/output.log
2023-11-17 08:43:57,302 INFO    wandb-upload_0:5676 [upload_job.py:push():131] Uploaded file /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/config.yaml
2023-11-17 08:43:57,303 INFO    wandb-upload_1:5676 [upload_job.py:push():131] Uploaded file /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json
2023-11-17 08:43:57,379 INFO    wandb-upload_3:5676 [upload_job.py:push():131] Uploaded file /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/requirements.txt
2023-11-17 08:43:57,527 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: keepalive
2023-11-17 08:43:57,579 INFO    Thread-11 (_thread_body):5676 [sender.py:transition_state():612] send defer: 11
2023-11-17 08:43:57,580 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: defer
2023-11-17 08:43:57,580 INFO    HandlerThread:5676 [handler.py:handle_request_defer():170] handle defer: 11
2023-11-17 08:43:57,580 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: defer
2023-11-17 08:43:57,581 INFO    SenderThread:5676 [sender.py:send_request_defer():608] handle sender defer: 11
2023-11-17 08:43:57,581 INFO    SenderThread:5676 [file_pusher.py:join():164] waiting for file pusher
2023-11-17 08:43:57,581 INFO    SenderThread:5676 [sender.py:transition_state():612] send defer: 12
2023-11-17 08:43:57,581 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: defer
2023-11-17 08:43:57,581 INFO    HandlerThread:5676 [handler.py:handle_request_defer():170] handle defer: 12
2023-11-17 08:43:57,581 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: defer
2023-11-17 08:43:57,581 INFO    SenderThread:5676 [sender.py:send_request_defer():608] handle sender defer: 12
2023-11-17 08:43:57,810 INFO    SenderThread:5676 [sender.py:transition_state():612] send defer: 13
2023-11-17 08:43:57,811 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: defer
2023-11-17 08:43:57,811 INFO    HandlerThread:5676 [handler.py:handle_request_defer():170] handle defer: 13
2023-11-17 08:43:57,811 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: defer
2023-11-17 08:43:57,811 INFO    SenderThread:5676 [sender.py:send_request_defer():608] handle sender defer: 13
2023-11-17 08:43:57,811 INFO    SenderThread:5676 [sender.py:transition_state():612] send defer: 14
2023-11-17 08:43:57,812 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: defer
2023-11-17 08:43:57,812 INFO    HandlerThread:5676 [handler.py:handle_request_defer():170] handle defer: 14
2023-11-17 08:43:57,812 DEBUG   SenderThread:5676 [sender.py:send():379] send: final
2023-11-17 08:43:57,812 DEBUG   SenderThread:5676 [sender.py:send():379] send: footer
2023-11-17 08:43:57,813 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: defer
2023-11-17 08:43:57,813 INFO    SenderThread:5676 [sender.py:send_request_defer():608] handle sender defer: 14
2023-11-17 08:43:57,813 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: poll_exit
2023-11-17 08:43:57,813 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: poll_exit
2023-11-17 08:43:57,814 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: server_info
2023-11-17 08:43:57,814 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: server_info
2023-11-17 08:43:57,816 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: get_summary
2023-11-17 08:43:57,817 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: sampled_history
2023-11-17 08:43:57,818 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: job_info
2023-11-17 08:43:57,837 DEBUG   SenderThread:5676 [sender.py:send_request():406] send_request: job_info
2023-11-17 08:43:57,838 INFO    MainThread:5676 [wandb_run.py:_footer_history_summary_info():3475] rendering history
2023-11-17 08:43:57,838 INFO    MainThread:5676 [wandb_run.py:_footer_history_summary_info():3507] rendering summary
2023-11-17 08:43:57,838 INFO    MainThread:5676 [wandb_run.py:_footer_sync_info():3434] logging synced files
2023-11-17 08:43:57,839 DEBUG   HandlerThread:5676 [handler.py:handle_request():144] handle_request: shutdown
2023-11-17 08:43:57,839 INFO    HandlerThread:5676 [handler.py:finish():857] shutting down handler
2023-11-17 08:43:58,818 INFO    WriterThread:5676 [datastore.py:close():294] close: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/run-f858xji5.wandb
2023-11-17 08:43:58,838 INFO    SenderThread:5676 [sender.py:finish():1552] shutting down sender
2023-11-17 08:43:58,838 INFO    SenderThread:5676 [file_pusher.py:finish():159] shutting down file pusher
2023-11-17 08:43:58,838 INFO    SenderThread:5676 [file_pusher.py:join():164] waiting for file pusher