|
2023-11-17 08:33:59,189 INFO StreamThr :5676 [internal.py:wandb_internal():86] W&B internal server running at pid: 5676, started at: 2023-11-17 08:33:59.188541 |
|
2023-11-17 08:33:59,191 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status |
|
2023-11-17 08:33:59,197 INFO WriterThread:5676 [datastore.py:open_for_write():85] open: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/run-f858xji5.wandb |
|
2023-11-17 08:33:59,197 DEBUG SenderThread:5676 [sender.py:send():379] send: header |
|
2023-11-17 08:33:59,217 DEBUG SenderThread:5676 [sender.py:send():379] send: run |
|
2023-11-17 08:33:59,506 INFO SenderThread:5676 [dir_watcher.py:__init__():211] watching files in: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files |
|
2023-11-17 08:33:59,507 INFO SenderThread:5676 [sender.py:_start_run_threads():1122] run started: f858xji5 with start time 1700210039.194491 |
|
2023-11-17 08:33:59,507 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:33:59,507 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:33:59,512 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: check_version |
|
2023-11-17 08:33:59,512 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: check_version |
|
2023-11-17 08:33:59,704 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: run_start |
|
2023-11-17 08:33:59,713 DEBUG HandlerThread:5676 [system_info.py:__init__():31] System info init |
|
2023-11-17 08:33:59,713 DEBUG HandlerThread:5676 [system_info.py:__init__():46] System info init done |
|
2023-11-17 08:33:59,713 INFO HandlerThread:5676 [system_monitor.py:start():181] Starting system monitor |
|
2023-11-17 08:33:59,713 INFO SystemMonitor:5676 [system_monitor.py:_start():145] Starting system asset monitoring threads |
|
2023-11-17 08:33:59,714 INFO HandlerThread:5676 [system_monitor.py:probe():201] Collecting system info |
|
2023-11-17 08:33:59,715 INFO SystemMonitor:5676 [interfaces.py:start():190] Started cpu monitoring |
|
2023-11-17 08:33:59,716 INFO SystemMonitor:5676 [interfaces.py:start():190] Started disk monitoring |
|
2023-11-17 08:33:59,717 INFO SystemMonitor:5676 [interfaces.py:start():190] Started gpu monitoring |
|
2023-11-17 08:33:59,718 INFO SystemMonitor:5676 [interfaces.py:start():190] Started memory monitoring |
|
2023-11-17 08:33:59,718 INFO SystemMonitor:5676 [interfaces.py:start():190] Started network monitoring |
|
2023-11-17 08:33:59,722 DEBUG HandlerThread:5676 [system_info.py:probe():195] Probing system |
|
2023-11-17 08:33:59,727 DEBUG HandlerThread:5676 [system_info.py:_probe_git():180] Probing git |
|
2023-11-17 08:33:59,740 DEBUG HandlerThread:5676 [system_info.py:_probe_git():188] Probing git done |
|
2023-11-17 08:33:59,740 DEBUG HandlerThread:5676 [system_info.py:probe():240] Probing system done |
|
2023-11-17 08:33:59,740 DEBUG HandlerThread:5676 [system_monitor.py:probe():210] {'os': 'Linux-5.15.120+-x86_64-with-glibc2.35', 'python': '3.10.12', 'heartbeatAt': '2023-11-17T08:33:59.722363', 'startedAt': '2023-11-17T08:33:59.181100', 'docker': None, 'cuda': None, 'args': ('--sample_prompts=/content/LoRA/config/sample_prompt.toml', '--config_file=/content/LoRA/config/config_file.toml', '--wandb_api_key=540d5e6b5446a44f2f5c8269e0da63b4c7ad408c'), 'state': 'running', 'program': '/content/kohya-trainer/sdxl_train_network.py', 'codePath': 'sdxl_train_network.py', 'git': {'remote': 'https://github.com/qaneel/kohya-trainer', 'commit': '05811296f6dc987f67f194689e106a326017b9d4'}, 'email': None, 'root': '/content/kohya-trainer', 'host': '3af72b0d0270', 'username': 'root', 'executable': '/usr/bin/python3', 'cpu_count': 1, 'cpu_count_logical': 2, 'cpu_freq': {'current': 2299.998, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2299.998, 'min': 0.0, 'max': 0.0}, {'current': 2299.998, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 78.1898422241211, 'used': 41.74154281616211}, 'gpu': 'Tesla T4', 'gpu_count': 1, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 12.6783447265625}} |
|
2023-11-17 08:33:59,740 INFO HandlerThread:5676 [system_monitor.py:probe():211] Finished collecting system info |
|
2023-11-17 08:33:59,740 INFO HandlerThread:5676 [system_monitor.py:probe():214] Publishing system info |
|
2023-11-17 08:33:59,740 DEBUG HandlerThread:5676 [system_info.py:_save_pip():51] Saving list of pip packages installed into the current environment |
|
2023-11-17 08:33:59,741 DEBUG HandlerThread:5676 [system_info.py:_save_pip():67] Saving pip packages done |
|
2023-11-17 08:33:59,755 INFO HandlerThread:5676 [system_monitor.py:probe():216] Finished publishing system info |
|
2023-11-17 08:33:59,768 DEBUG SenderThread:5676 [sender.py:send():379] send: files |
|
2023-11-17 08:33:59,768 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-metadata.json with policy now |
|
2023-11-17 08:33:59,776 DEBUG SenderThread:5676 [sender.py:send():379] send: telemetry |
|
2023-11-17 08:33:59,816 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:33:59,817 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:34:00,349 INFO wandb-upload_0:5676 [upload_job.py:push():131] Uploaded file /tmp/tmp5kii06quwandb/24x79thl-wandb-metadata.json |
|
2023-11-17 08:34:00,508 INFO Thread-12 :5676 [dir_watcher.py:_on_file_created():272] file/dir created: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/requirements.txt |
|
2023-11-17 08:34:00,508 INFO Thread-12 :5676 [dir_watcher.py:_on_file_created():272] file/dir created: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-metadata.json |
|
2023-11-17 08:34:00,509 INFO Thread-12 :5676 [dir_watcher.py:_on_file_created():272] file/dir created: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:34:00,509 INFO Thread-12 :5676 [dir_watcher.py:_on_file_created():272] file/dir created: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/output.log |
|
2023-11-17 08:34:02,508 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/output.log |
|
2023-11-17 08:34:04,249 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:34:09,250 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:34:14,251 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:34:14,775 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:34:14,776 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:34:15,117 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:34:19,835 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:34:21,204 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:34:21,206 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:34:21,206 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:34:21,209 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:34:21,515 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:34:25,210 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:34:26,805 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:34:26,806 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:34:26,806 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:34:26,809 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:34:27,518 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:34:29,774 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:34:29,775 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:34:30,895 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:34:31,519 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/config.yaml |
|
2023-11-17 08:34:34,164 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:34:34,166 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:34:34,166 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:34:34,168 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:34:34,521 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:34:36,169 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:34:41,170 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:34:41,266 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:34:41,268 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:34:41,268 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:34:41,270 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:34:41,523 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:34:43,193 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:34:43,195 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:34:43,195 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:34:43,195 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:34:43,524 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:34:44,899 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:34:44,900 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:34:46,942 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:34:49,334 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:34:49,335 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:34:49,335 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:34:49,338 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:34:49,527 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:34:52,338 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:34:57,339 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:34:57,438 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:34:57,439 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:34:57,440 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:34:57,442 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:34:57,530 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:34:59,719 DEBUG SystemMonitor:5676 [system_monitor.py:_start():159] Starting system metrics aggregation loop |
|
2023-11-17 08:34:59,720 DEBUG SenderThread:5676 [sender.py:send():379] send: stats |
|
2023-11-17 08:34:59,900 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:34:59,900 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:35:03,059 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:35:03,726 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:35:03,727 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:35:03,728 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:35:03,728 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:35:04,532 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:35:08,729 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:35:10,018 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:35:10,019 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:35:10,019 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:35:10,022 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:35:10,535 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:35:14,023 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:35:14,900 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:35:14,900 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:35:18,823 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:35:18,824 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:35:18,825 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:35:18,825 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:35:19,539 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:35:19,826 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:35:24,827 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:35:25,174 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:35:25,175 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:35:25,176 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:35:25,178 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:35:25,541 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:35:29,721 DEBUG SenderThread:5676 [sender.py:send():379] send: stats |
|
2023-11-17 08:35:29,899 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:35:29,900 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:35:29,934 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:35:31,568 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:35:31,569 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:35:31,570 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:35:31,570 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:35:32,544 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:35:35,571 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:35:38,233 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:35:38,235 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:35:38,235 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:35:38,238 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:35:38,547 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:35:41,239 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:35:44,900 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:35:44,900 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:35:45,128 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:35:45,129 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:35:45,130 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:35:45,130 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:35:45,549 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:35:47,131 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:35:51,509 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:35:51,510 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:35:51,511 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:35:51,513 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:35:51,552 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:35:52,514 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:35:57,515 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:35:58,081 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:35:58,082 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:35:58,082 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:35:58,083 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:35:58,554 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:35:59,722 DEBUG SenderThread:5676 [sender.py:send():379] send: stats |
|
2023-11-17 08:36:00,353 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:36:00,354 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:36:03,433 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:36:06,396 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:36:06,397 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:36:06,398 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:36:06,400 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:36:06,557 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:36:09,401 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:36:11,006 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:36:11,007 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:36:11,007 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:36:11,008 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:36:11,559 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:36:15,009 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:36:15,353 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:36:15,354 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:36:17,319 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:36:17,320 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:36:17,320 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:36:17,323 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:36:17,562 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:36:20,323 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:36:23,837 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:36:23,838 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:36:23,838 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:36:23,839 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:36:24,564 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:36:25,839 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:36:28,600 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:36:28,602 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:36:28,602 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:36:28,604 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:36:29,566 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:36:29,723 DEBUG SenderThread:5676 [sender.py:send():379] send: stats |
|
2023-11-17 08:36:30,584 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:36:30,585 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:36:31,637 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:36:35,533 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:36:35,534 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:36:35,535 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:36:35,535 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:36:35,568 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:36:37,536 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:36:42,537 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:36:42,644 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:36:42,645 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:36:42,645 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:36:42,647 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:36:43,571 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:36:45,585 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:36:45,586 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:36:47,650 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:36:50,041 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:36:50,043 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:36:50,043 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:36:50,043 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:36:50,574 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:36:53,044 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:36:57,064 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:36:57,066 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:36:57,066 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:36:57,068 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:36:57,585 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:36:58,069 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:36:59,723 DEBUG SenderThread:5676 [sender.py:send():379] send: stats |
|
2023-11-17 08:37:00,585 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:37:00,586 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:37:03,629 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:37:03,630 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:37:03,630 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:37:03,631 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:37:03,631 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:37:04,588 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:37:08,644 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:37:10,398 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:37:10,399 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:37:10,399 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:37:10,402 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:37:10,590 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:37:14,403 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:37:15,585 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:37:15,587 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:37:17,475 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:37:17,477 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:37:17,477 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:37:17,478 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:37:17,593 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:37:19,478 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:37:23,807 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:37:23,808 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:37:23,808 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:37:23,811 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:37:24,595 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:37:24,811 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:37:29,724 DEBUG SenderThread:5676 [sender.py:send():379] send: stats |
|
2023-11-17 08:37:30,585 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:37:30,587 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:37:30,587 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:37:31,197 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:37:31,198 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:37:31,198 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:37:31,199 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:37:31,598 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:37:36,200 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:37:37,531 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:37:37,532 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:37:37,533 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:37:37,535 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:37:37,601 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:37:41,536 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:37:45,483 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:37:45,484 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:37:45,485 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:37:45,485 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:37:45,586 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:37:45,586 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:37:45,604 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:37:46,644 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:37:51,645 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:37:53,283 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:37:53,284 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:37:53,285 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:37:53,287 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:37:53,607 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:37:56,421 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:37:56,422 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:37:56,422 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:37:56,423 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:37:56,608 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:37:57,424 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:37:59,725 DEBUG SenderThread:5676 [sender.py:send():379] send: stats |
|
2023-11-17 08:38:00,585 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:38:00,586 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:38:02,711 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:38:03,398 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:38:03,399 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:38:03,399 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:38:03,400 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:38:03,611 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:38:08,401 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:38:09,929 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:38:09,930 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:38:09,931 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:38:09,931 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:38:10,613 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:38:13,933 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:38:15,586 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:38:15,588 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:38:16,655 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:38:16,656 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:38:16,657 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:38:16,659 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:38:17,616 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:38:19,660 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:38:21,011 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:38:21,012 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:38:21,013 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:38:21,013 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:38:21,617 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:38:25,017 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:38:27,363 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:38:27,364 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:38:27,365 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:38:27,367 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:38:27,620 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:38:29,726 DEBUG SenderThread:5676 [sender.py:send():379] send: stats |
|
2023-11-17 08:38:30,586 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:38:30,587 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:38:30,626 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:38:33,929 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:38:33,930 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:38:33,930 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:38:33,931 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:38:34,622 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:38:35,932 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:38:40,662 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:38:40,663 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:38:40,663 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:38:40,664 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:38:41,625 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:38:41,664 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:38:45,586 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:38:45,587 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:38:47,631 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:38:47,706 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:38:47,707 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:38:47,708 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:38:47,708 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:38:48,628 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:38:50,522 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:38:50,524 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:38:50,524 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:38:50,525 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:38:50,629 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:38:53,526 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:38:57,084 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:38:57,086 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:38:57,086 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:38:57,088 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:38:57,632 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:38:59,089 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:38:59,727 DEBUG SenderThread:5676 [sender.py:send():379] send: stats |
|
2023-11-17 08:39:00,586 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:39:00,587 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:39:03,805 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:39:03,806 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:39:03,807 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:39:03,807 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:39:04,635 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:39:04,808 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:39:09,809 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:39:10,844 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:39:10,846 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:39:10,846 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:39:10,846 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:39:11,637 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:39:14,847 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:39:15,586 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:39:15,587 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:39:18,495 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:39:18,496 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:39:18,496 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:39:18,496 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:39:18,640 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:39:20,497 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:39:25,197 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:39:25,198 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:39:25,198 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:39:25,199 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:39:25,643 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:39:26,200 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:39:29,727 DEBUG SenderThread:5676 [sender.py:send():379] send: stats |
|
2023-11-17 08:39:30,586 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:39:30,587 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:39:31,576 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:39:31,577 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:39:31,577 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:39:31,578 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:39:31,578 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:39:31,645 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:39:36,579 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:39:38,079 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:39:38,080 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:39:38,080 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:39:38,081 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:39:38,647 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:39:42,082 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:39:45,488 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:39:45,489 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:39:45,490 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:39:45,490 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:39:45,587 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:39:45,587 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:39:45,650 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:39:47,669 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:39:52,670 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:39:53,424 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:39:53,426 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:39:53,426 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:39:53,426 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:39:53,664 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:39:58,214 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:39:58,217 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:39:58,218 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:39:58,218 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:39:58,218 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:39:58,666 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:39:59,728 DEBUG SenderThread:5676 [sender.py:send():379] send: stats |
|
2023-11-17 08:40:00,587 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:40:00,587 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:40:03,716 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:40:06,535 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:40:06,537 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:40:06,537 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:40:06,538 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:40:06,668 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:40:09,539 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:40:12,914 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:40:12,915 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:40:12,915 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:40:12,916 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:40:13,671 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:40:14,916 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:40:15,587 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:40:15,587 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:40:20,700 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:40:21,554 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:40:21,555 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:40:21,555 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:40:21,555 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:40:21,674 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:40:26,557 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:40:28,596 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:40:28,597 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:40:28,597 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:40:28,598 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:40:28,676 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:40:29,729 DEBUG SenderThread:5676 [sender.py:send():379] send: stats |
|
2023-11-17 08:40:30,600 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:40:30,600 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:40:31,719 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:40:35,624 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:40:35,625 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:40:35,625 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:40:35,626 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:40:35,680 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:40:37,627 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:40:42,628 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:40:42,839 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:40:42,840 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:40:42,840 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:40:42,843 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:40:43,683 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:40:45,600 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:40:45,601 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:40:45,718 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:40:45,737 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:40:45,738 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:40:45,742 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:40:46,684 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:40:47,743 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:40:52,458 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:40:52,459 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:40:52,460 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:40:52,460 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:40:52,686 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:40:53,461 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:40:58,462 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:40:59,634 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:40:59,635 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:40:59,635 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:40:59,636 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:40:59,689 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:40:59,729 DEBUG SenderThread:5676 [sender.py:send():379] send: stats |
|
2023-11-17 08:41:00,601 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:41:00,601 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:41:03,681 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:41:06,288 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:41:06,289 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:41:06,290 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:41:06,290 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:41:06,691 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:41:09,291 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:41:13,723 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:41:13,724 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:41:13,724 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:41:13,725 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:41:14,694 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:41:14,726 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:41:15,601 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:41:15,601 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:41:20,117 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:41:20,118 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:41:20,118 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:41:20,119 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:41:20,119 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:41:20,696 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:41:25,120 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:41:26,470 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:41:26,471 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:41:26,471 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:41:26,472 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:41:26,699 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:41:29,730 DEBUG SenderThread:5676 [sender.py:send():379] send: stats |
|
2023-11-17 08:41:30,601 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:41:30,601 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:41:30,602 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:41:34,765 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:41:34,766 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:41:34,767 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:41:34,769 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:41:35,702 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:41:35,769 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:41:40,770 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:41:41,147 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:41:41,148 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:41:41,148 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:41:41,149 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:41:41,704 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:41:45,601 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:41:45,601 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:41:46,687 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:41:47,110 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:41:47,111 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:41:47,112 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:41:47,112 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:41:47,707 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:41:52,113 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:41:53,652 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:41:53,653 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:41:53,653 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:41:53,653 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:41:53,709 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:41:57,655 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:41:59,731 DEBUG SenderThread:5676 [sender.py:send():379] send: stats |
|
2023-11-17 08:42:00,602 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:42:00,602 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:42:01,024 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:42:01,025 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:42:01,026 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:42:01,026 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:42:01,712 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:42:03,027 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:42:07,450 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:42:07,451 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:42:07,451 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:42:07,452 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:42:07,715 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:42:08,457 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:42:13,458 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:42:13,974 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:42:13,976 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:42:13,976 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:42:13,976 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:42:14,717 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:42:15,968 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:42:15,968 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:42:19,059 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:42:21,348 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:42:21,349 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:42:21,349 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:42:21,350 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:42:21,720 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:42:24,350 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:42:29,351 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:42:29,732 DEBUG SenderThread:5676 [sender.py:send():379] send: stats |
|
2023-11-17 08:42:29,864 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:42:29,865 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:42:29,866 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:42:29,866 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:42:30,723 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:42:30,968 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:42:30,969 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:42:35,018 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:42:38,032 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:42:38,033 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:42:38,034 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:42:38,034 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:42:38,726 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:42:40,035 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:42:45,036 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:42:45,430 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:42:45,431 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:42:45,431 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:42:45,431 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:42:45,729 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:42:45,969 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:42:45,970 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:42:50,107 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:42:51,807 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:42:51,808 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:42:51,809 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:42:51,809 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:42:52,732 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:42:55,810 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:42:58,318 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:42:58,319 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:42:58,320 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:42:58,320 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:42:58,734 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:42:59,733 DEBUG SenderThread:5676 [sender.py:send():379] send: stats |
|
2023-11-17 08:43:00,968 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:43:00,969 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:43:00,970 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:43:05,708 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:43:05,709 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:43:05,709 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:43:05,710 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:43:05,744 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:43:06,710 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:43:10,829 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:43:10,830 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:43:10,831 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:43:10,831 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:43:11,747 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:43:11,832 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:43:15,969 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:43:15,970 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:43:17,008 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:43:17,158 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:43:17,159 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:43:17,159 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:43:17,160 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:43:17,750 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:43:22,161 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:43:24,520 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:43:24,521 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:43:24,521 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:43:24,522 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:43:24,752 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:43:27,523 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:43:29,733 DEBUG SenderThread:5676 [sender.py:send():379] send: stats |
|
2023-11-17 08:43:30,970 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:43:30,970 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:43:31,554 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:43:31,556 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:43:31,556 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:43:31,556 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:43:31,755 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:43:32,557 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:43:37,558 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:43:37,890 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:43:37,892 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:43:37,893 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:43:37,893 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:43:37,893 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: partial_history |
|
2023-11-17 08:43:37,894 WARNING HandlerThread:5676 [handler.py:handle_request_partial_history():553] Step 1 < 86. Dropping entry: {'loss/epoch': 0.11442129172352251, '_timestamp': 1700210617.890885}. |
|
2023-11-17 08:43:38,758 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:43:39,758 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/output.log |
|
2023-11-17 08:43:41,759 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/output.log |
|
2023-11-17 08:43:42,759 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/output.log |
|
2023-11-17 08:43:42,894 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:43:44,760 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/output.log |
|
2023-11-17 08:43:45,970 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-11-17 08:43:45,970 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: stop_status |
|
2023-11-17 08:43:46,761 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/output.log |
|
2023-11-17 08:43:48,106 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:43:48,762 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/output.log |
|
2023-11-17 08:43:50,762 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/output.log |
|
2023-11-17 08:43:51,526 DEBUG SenderThread:5676 [sender.py:send():379] send: exit |
|
2023-11-17 08:43:51,526 INFO SenderThread:5676 [sender.py:send_exit():584] handling exit code: 1 |
|
2023-11-17 08:43:51,526 INFO SenderThread:5676 [sender.py:send_exit():586] handling runtime: 591 |
|
2023-11-17 08:43:51,526 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:43:51,527 INFO SenderThread:5676 [sender.py:send_exit():592] send defer |
|
2023-11-17 08:43:51,527 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: defer |
|
2023-11-17 08:43:51,527 INFO HandlerThread:5676 [handler.py:handle_request_defer():170] handle defer: 0 |
|
2023-11-17 08:43:51,527 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: defer |
|
2023-11-17 08:43:51,527 INFO SenderThread:5676 [sender.py:send_request_defer():608] handle sender defer: 0 |
|
2023-11-17 08:43:51,527 INFO SenderThread:5676 [sender.py:transition_state():612] send defer: 1 |
|
2023-11-17 08:43:51,527 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: defer |
|
2023-11-17 08:43:51,527 INFO HandlerThread:5676 [handler.py:handle_request_defer():170] handle defer: 1 |
|
2023-11-17 08:43:51,528 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: defer |
|
2023-11-17 08:43:51,528 INFO SenderThread:5676 [sender.py:send_request_defer():608] handle sender defer: 1 |
|
2023-11-17 08:43:51,528 INFO SenderThread:5676 [sender.py:transition_state():612] send defer: 2 |
|
2023-11-17 08:43:51,528 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: defer |
|
2023-11-17 08:43:51,528 INFO HandlerThread:5676 [handler.py:handle_request_defer():170] handle defer: 2 |
|
2023-11-17 08:43:51,528 INFO HandlerThread:5676 [system_monitor.py:finish():190] Stopping system monitor |
|
2023-11-17 08:43:51,528 DEBUG SystemMonitor:5676 [system_monitor.py:_start():166] Finished system metrics aggregation loop |
|
2023-11-17 08:43:51,528 DEBUG SystemMonitor:5676 [system_monitor.py:_start():170] Publishing last batch of metrics |
|
2023-11-17 08:43:51,529 INFO HandlerThread:5676 [interfaces.py:finish():202] Joined cpu monitor |
|
2023-11-17 08:43:51,529 INFO HandlerThread:5676 [interfaces.py:finish():202] Joined disk monitor |
|
2023-11-17 08:43:51,538 INFO HandlerThread:5676 [interfaces.py:finish():202] Joined gpu monitor |
|
2023-11-17 08:43:51,538 INFO HandlerThread:5676 [interfaces.py:finish():202] Joined memory monitor |
|
2023-11-17 08:43:51,538 INFO HandlerThread:5676 [interfaces.py:finish():202] Joined network monitor |
|
2023-11-17 08:43:51,539 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: defer |
|
2023-11-17 08:43:51,539 INFO SenderThread:5676 [sender.py:send_request_defer():608] handle sender defer: 2 |
|
2023-11-17 08:43:51,539 INFO SenderThread:5676 [sender.py:transition_state():612] send defer: 3 |
|
2023-11-17 08:43:51,539 DEBUG SenderThread:5676 [sender.py:send():379] send: stats |
|
2023-11-17 08:43:51,539 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: defer |
|
2023-11-17 08:43:51,540 INFO HandlerThread:5676 [handler.py:handle_request_defer():170] handle defer: 3 |
|
2023-11-17 08:43:51,540 DEBUG SenderThread:5676 [sender.py:send():379] send: history |
|
2023-11-17 08:43:51,541 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: summary_record |
|
2023-11-17 08:43:51,541 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:43:51,541 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: defer |
|
2023-11-17 08:43:51,541 INFO SenderThread:5676 [sender.py:send_request_defer():608] handle sender defer: 3 |
|
2023-11-17 08:43:51,541 INFO SenderThread:5676 [sender.py:transition_state():612] send defer: 4 |
|
2023-11-17 08:43:51,542 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: defer |
|
2023-11-17 08:43:51,542 INFO HandlerThread:5676 [handler.py:handle_request_defer():170] handle defer: 4 |
|
2023-11-17 08:43:51,542 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: defer |
|
2023-11-17 08:43:51,542 INFO SenderThread:5676 [sender.py:send_request_defer():608] handle sender defer: 4 |
|
2023-11-17 08:43:51,542 INFO SenderThread:5676 [sender.py:transition_state():612] send defer: 5 |
|
2023-11-17 08:43:51,542 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: defer |
|
2023-11-17 08:43:51,542 INFO HandlerThread:5676 [handler.py:handle_request_defer():170] handle defer: 5 |
|
2023-11-17 08:43:51,543 DEBUG SenderThread:5676 [sender.py:send():379] send: summary |
|
2023-11-17 08:43:51,543 INFO SenderThread:5676 [sender.py:_save_file():1376] saving file wandb-summary.json with policy end |
|
2023-11-17 08:43:51,543 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: defer |
|
2023-11-17 08:43:51,543 INFO SenderThread:5676 [sender.py:send_request_defer():608] handle sender defer: 5 |
|
2023-11-17 08:43:51,543 INFO SenderThread:5676 [sender.py:transition_state():612] send defer: 6 |
|
2023-11-17 08:43:51,543 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: defer |
|
2023-11-17 08:43:51,543 INFO HandlerThread:5676 [handler.py:handle_request_defer():170] handle defer: 6 |
|
2023-11-17 08:43:51,544 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: defer |
|
2023-11-17 08:43:51,544 INFO SenderThread:5676 [sender.py:send_request_defer():608] handle sender defer: 6 |
|
2023-11-17 08:43:51,544 INFO SenderThread:5676 [sender.py:transition_state():612] send defer: 7 |
|
2023-11-17 08:43:51,544 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: status_report |
|
2023-11-17 08:43:51,544 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: defer |
|
2023-11-17 08:43:51,544 INFO HandlerThread:5676 [handler.py:handle_request_defer():170] handle defer: 7 |
|
2023-11-17 08:43:51,545 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: defer |
|
2023-11-17 08:43:51,545 INFO SenderThread:5676 [sender.py:send_request_defer():608] handle sender defer: 7 |
|
2023-11-17 08:43:51,763 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:43:52,526 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: poll_exit |
|
2023-11-17 08:43:52,764 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/output.log |
|
2023-11-17 08:43:54,657 INFO SenderThread:5676 [sender.py:transition_state():612] send defer: 8 |
|
2023-11-17 08:43:54,657 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: poll_exit |
|
2023-11-17 08:43:54,657 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: defer |
|
2023-11-17 08:43:54,657 INFO HandlerThread:5676 [handler.py:handle_request_defer():170] handle defer: 8 |
|
2023-11-17 08:43:54,658 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: defer |
|
2023-11-17 08:43:54,658 INFO SenderThread:5676 [sender.py:send_request_defer():608] handle sender defer: 8 |
|
2023-11-17 08:43:54,658 INFO SenderThread:5676 [job_builder.py:build():280] Attempting to build job artifact |
|
2023-11-17 08:43:54,658 INFO SenderThread:5676 [job_builder.py:_get_source_type():389] is repo sourced job |
|
2023-11-17 08:43:54,665 INFO SenderThread:5676 [job_builder.py:build():363] adding wandb-job metadata file |
|
2023-11-17 08:43:54,671 INFO SenderThread:5676 [sender.py:transition_state():612] send defer: 9 |
|
2023-11-17 08:43:54,671 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: defer |
|
2023-11-17 08:43:54,671 INFO HandlerThread:5676 [handler.py:handle_request_defer():170] handle defer: 9 |
|
2023-11-17 08:43:54,672 DEBUG SenderThread:5676 [sender.py:send():379] send: artifact |
|
2023-11-17 08:43:54,764 INFO Thread-12 :5676 [dir_watcher.py:_on_file_modified():289] file/dir modified: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/output.log |
|
2023-11-17 08:43:55,177 INFO wandb-upload_0:5676 [upload_job.py:push():86] Skipped uploading /root/.local/share/wandb/artifacts/staging/tmpaxl62r1n |
|
2023-11-17 08:43:55,516 INFO wandb-upload_1:5676 [upload_job.py:push():89] Uploaded file /root/.local/share/wandb/artifacts/staging/tmpbhp9625f |
|
2023-11-17 08:43:56,385 INFO SenderThread:5676 [sender.py:send_artifact():1476] sent artifact job-https___github.com_qaneel_kohya-trainer_sdxl_train_network.py - {'id': 'QXJ0aWZhY3Q6NjQxMzE0Njcx', 'digest': 'd545101fca5d4391fee1f3b40d5981eb', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjExNjM5NTUzOQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NjQwNjQ0OTYz', 'versionIndex': 0}}, 'version': 'latest'} |
|
2023-11-17 08:43:56,385 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: defer |
|
2023-11-17 08:43:56,385 INFO SenderThread:5676 [sender.py:send_request_defer():608] handle sender defer: 9 |
|
2023-11-17 08:43:56,386 INFO SenderThread:5676 [dir_watcher.py:finish():359] shutting down directory watcher |
|
2023-11-17 08:43:56,765 INFO SenderThread:5676 [dir_watcher.py:finish():389] scan: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files |
|
2023-11-17 08:43:56,765 INFO SenderThread:5676 [dir_watcher.py:finish():403] scan save: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/config.yaml config.yaml |
|
2023-11-17 08:43:56,766 INFO SenderThread:5676 [dir_watcher.py:finish():403] scan save: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/output.log output.log |
|
2023-11-17 08:43:56,768 INFO SenderThread:5676 [dir_watcher.py:finish():403] scan save: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json wandb-summary.json |
|
2023-11-17 08:43:56,769 INFO SenderThread:5676 [dir_watcher.py:finish():403] scan save: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-metadata.json wandb-metadata.json |
|
2023-11-17 08:43:56,769 INFO SenderThread:5676 [dir_watcher.py:finish():403] scan save: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/requirements.txt requirements.txt |
|
2023-11-17 08:43:56,774 INFO SenderThread:5676 [sender.py:transition_state():612] send defer: 10 |
|
2023-11-17 08:43:56,774 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: defer |
|
2023-11-17 08:43:56,774 INFO HandlerThread:5676 [handler.py:handle_request_defer():170] handle defer: 10 |
|
2023-11-17 08:43:56,777 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: defer |
|
2023-11-17 08:43:56,778 INFO SenderThread:5676 [sender.py:send_request_defer():608] handle sender defer: 10 |
|
2023-11-17 08:43:56,778 INFO SenderThread:5676 [file_pusher.py:finish():159] shutting down file pusher |
|
2023-11-17 08:43:56,983 INFO wandb-upload_2:5676 [upload_job.py:push():131] Uploaded file /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/output.log |
|
2023-11-17 08:43:57,302 INFO wandb-upload_0:5676 [upload_job.py:push():131] Uploaded file /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/config.yaml |
|
2023-11-17 08:43:57,303 INFO wandb-upload_1:5676 [upload_job.py:push():131] Uploaded file /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/wandb-summary.json |
|
2023-11-17 08:43:57,379 INFO wandb-upload_3:5676 [upload_job.py:push():131] Uploaded file /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/files/requirements.txt |
|
2023-11-17 08:43:57,527 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: keepalive |
|
2023-11-17 08:43:57,579 INFO Thread-11 (_thread_body):5676 [sender.py:transition_state():612] send defer: 11 |
|
2023-11-17 08:43:57,580 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: defer |
|
2023-11-17 08:43:57,580 INFO HandlerThread:5676 [handler.py:handle_request_defer():170] handle defer: 11 |
|
2023-11-17 08:43:57,580 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: defer |
|
2023-11-17 08:43:57,581 INFO SenderThread:5676 [sender.py:send_request_defer():608] handle sender defer: 11 |
|
2023-11-17 08:43:57,581 INFO SenderThread:5676 [file_pusher.py:join():164] waiting for file pusher |
|
2023-11-17 08:43:57,581 INFO SenderThread:5676 [sender.py:transition_state():612] send defer: 12 |
|
2023-11-17 08:43:57,581 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: defer |
|
2023-11-17 08:43:57,581 INFO HandlerThread:5676 [handler.py:handle_request_defer():170] handle defer: 12 |
|
2023-11-17 08:43:57,581 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: defer |
|
2023-11-17 08:43:57,581 INFO SenderThread:5676 [sender.py:send_request_defer():608] handle sender defer: 12 |
|
2023-11-17 08:43:57,810 INFO SenderThread:5676 [sender.py:transition_state():612] send defer: 13 |
|
2023-11-17 08:43:57,811 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: defer |
|
2023-11-17 08:43:57,811 INFO HandlerThread:5676 [handler.py:handle_request_defer():170] handle defer: 13 |
|
2023-11-17 08:43:57,811 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: defer |
|
2023-11-17 08:43:57,811 INFO SenderThread:5676 [sender.py:send_request_defer():608] handle sender defer: 13 |
|
2023-11-17 08:43:57,811 INFO SenderThread:5676 [sender.py:transition_state():612] send defer: 14 |
|
2023-11-17 08:43:57,812 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: defer |
|
2023-11-17 08:43:57,812 INFO HandlerThread:5676 [handler.py:handle_request_defer():170] handle defer: 14 |
|
2023-11-17 08:43:57,812 DEBUG SenderThread:5676 [sender.py:send():379] send: final |
|
2023-11-17 08:43:57,812 DEBUG SenderThread:5676 [sender.py:send():379] send: footer |
|
2023-11-17 08:43:57,813 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: defer |
|
2023-11-17 08:43:57,813 INFO SenderThread:5676 [sender.py:send_request_defer():608] handle sender defer: 14 |
|
2023-11-17 08:43:57,813 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: poll_exit |
|
2023-11-17 08:43:57,813 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: poll_exit |
|
2023-11-17 08:43:57,814 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: server_info |
|
2023-11-17 08:43:57,814 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: server_info |
|
2023-11-17 08:43:57,816 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: get_summary |
|
2023-11-17 08:43:57,817 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: sampled_history |
|
2023-11-17 08:43:57,818 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: job_info |
|
2023-11-17 08:43:57,837 DEBUG SenderThread:5676 [sender.py:send_request():406] send_request: job_info |
|
2023-11-17 08:43:57,838 INFO MainThread:5676 [wandb_run.py:_footer_history_summary_info():3475] rendering history |
|
2023-11-17 08:43:57,838 INFO MainThread:5676 [wandb_run.py:_footer_history_summary_info():3507] rendering summary |
|
2023-11-17 08:43:57,838 INFO MainThread:5676 [wandb_run.py:_footer_sync_info():3434] logging synced files |
|
2023-11-17 08:43:57,839 DEBUG HandlerThread:5676 [handler.py:handle_request():144] handle_request: shutdown |
|
2023-11-17 08:43:57,839 INFO HandlerThread:5676 [handler.py:finish():857] shutting down handler |
|
2023-11-17 08:43:58,818 INFO WriterThread:5676 [datastore.py:close():294] close: /content/LoRA/logs/20231117083158/wandb/run-20231117_083359-f858xji5/run-f858xji5.wandb |
|
2023-11-17 08:43:58,838 INFO SenderThread:5676 [sender.py:finish():1552] shutting down sender |
|
2023-11-17 08:43:58,838 INFO SenderThread:5676 [file_pusher.py:finish():159] shutting down file pusher |
|
2023-11-17 08:43:58,838 INFO SenderThread:5676 [file_pusher.py:join():164] waiting for file pusher |
|
|