ColorfulLlama2-7b-Alpaca-Tune-4epochs
/
wandb
/run-20240218_171505-jggsueon
/logs
/debug-internal.log
| 2024-02-18 17:15:05,334 INFO StreamThr :3454 [internal.py:wandb_internal():86] W&B internal server running at pid: 3454, started at: 2024-02-18 17:15:05.332907 | |
| 2024-02-18 17:15:05,340 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: status | |
| 2024-02-18 17:15:05,343 INFO WriterThread:3454 [datastore.py:open_for_write():87] open: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/run-jggsueon.wandb | |
| 2024-02-18 17:15:05,344 DEBUG SenderThread:3454 [sender.py:send():382] send: header | |
| 2024-02-18 17:15:05,351 DEBUG SenderThread:3454 [sender.py:send():382] send: run | |
| 2024-02-18 17:15:05,353 INFO SenderThread:3454 [sender.py:_maybe_setup_resume():763] checking resume status for None/colorful-llama/jggsueon | |
| 2024-02-18 17:15:05,893 INFO SenderThread:3454 [dir_watcher.py:__init__():211] watching files in: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files | |
| 2024-02-18 17:15:05,894 INFO SenderThread:3454 [sender.py:_start_run_threads():1136] run started: jggsueon with start time 1708276505.332364 | |
| 2024-02-18 17:15:05,905 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: check_version | |
| 2024-02-18 17:15:05,906 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: check_version | |
| 2024-02-18 17:15:06,222 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: run_start | |
| 2024-02-18 17:15:06,246 DEBUG HandlerThread:3454 [system_info.py:__init__():27] System info init | |
| 2024-02-18 17:15:06,246 DEBUG HandlerThread:3454 [system_info.py:__init__():42] System info init done | |
| 2024-02-18 17:15:06,246 INFO HandlerThread:3454 [system_monitor.py:start():194] Starting system monitor | |
| 2024-02-18 17:15:06,247 INFO SystemMonitor:3454 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-02-18 17:15:06,249 INFO HandlerThread:3454 [system_monitor.py:probe():214] Collecting system info | |
| 2024-02-18 17:15:06,250 INFO SystemMonitor:3454 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-02-18 17:15:06,255 INFO SystemMonitor:3454 [interfaces.py:start():190] Started disk monitoring | |
| 2024-02-18 17:15:06,262 INFO SystemMonitor:3454 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-02-18 17:15:06,280 INFO SystemMonitor:3454 [interfaces.py:start():190] Started memory monitoring | |
| 2024-02-18 17:15:06,284 INFO SystemMonitor:3454 [interfaces.py:start():190] Started network monitoring | |
| 2024-02-18 17:15:06,303 DEBUG HandlerThread:3454 [system_info.py:probe():151] Probing system | |
| 2024-02-18 17:15:06,308 DEBUG HandlerThread:3454 [system_info.py:_probe_git():136] Probing git | |
| 2024-02-18 17:15:06,322 DEBUG HandlerThread:3454 [system_info.py:_probe_git():144] Probing git done | |
| 2024-02-18 17:15:06,322 DEBUG HandlerThread:3454 [system_info.py:probe():199] Probing system done | |
| 2024-02-18 17:15:06,323 DEBUG HandlerThread:3454 [system_monitor.py:probe():223] {'os': 'Linux-6.2.0-37-generic-x86_64-with-glibc2.35', 'python': '3.10.12', 'heartbeatAt': '2024-02-18T17:15:06.303360', 'startedAt': '2024-02-18T17:15:05.322666', 'docker': None, 'cuda': None, 'args': ('--config', './basic_config.yaml'), 'state': 'running', 'program': '/home/ubuntu/torchtune-colorful-llama/colorful/./full_finetune.py', 'codePathLocal': 'full_finetune.py', 'codePath': 'colorful/full_finetune.py', 'git': {'remote': 'git@github.com:laurencer/torchtune-colorful-llama.git', 'commit': 'c9b276ef1c5e20d67371637975da5acd64541005'}, 'email': None, 'root': '/home/ubuntu/torchtune-colorful-llama', 'host': '209-20-159-120', 'username': 'ubuntu', 'executable': '/usr/bin/python3', 'cpu_count': 26, 'cpu_count_logical': 26, 'cpu_freq': {'current': 2000.0, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.0, 'min': 0.0, 'max': 0.0}, {'current': 2000.0, 'min': 0.0, 'max': 0.0}, {'current': 2000.0, 'min': 0.0, 'max': 0.0}, {'current': 2000.0, 'min': 0.0, 'max': 0.0}, {'current': 2000.0, 'min': 0.0, 'max': 0.0}, {'current': 2000.0, 'min': 0.0, 'max': 0.0}, {'current': 2000.0, 'min': 0.0, 'max': 0.0}, {'current': 2000.0, 'min': 0.0, 'max': 0.0}, {'current': 2000.0, 'min': 0.0, 'max': 0.0}, {'current': 2000.0, 'min': 0.0, 'max': 0.0}, {'current': 2000.0, 'min': 0.0, 'max': 0.0}, {'current': 2000.0, 'min': 0.0, 'max': 0.0}, {'current': 2000.0, 'min': 0.0, 'max': 0.0}, {'current': 2000.0, 'min': 0.0, 'max': 0.0}, {'current': 2000.0, 'min': 0.0, 'max': 0.0}, {'current': 2000.0, 'min': 0.0, 'max': 0.0}, {'current': 2000.0, 'min': 0.0, 'max': 0.0}, {'current': 2000.0, 'min': 0.0, 'max': 0.0}, {'current': 2000.0, 'min': 0.0, 'max': 0.0}, {'current': 2000.0, 'min': 0.0, 'max': 0.0}, {'current': 2000.0, 'min': 0.0, 'max': 0.0}, {'current': 2000.0, 'min': 0.0, 'max': 0.0}, {'current': 2000.0, 'min': 0.0, 'max': 0.0}, {'current': 2000.0, 'min': 0.0, 'max': 0.0}, {'current': 2000.0, 'min': 0.0, 'max': 0.0}, {'current': 2000.0, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 992.2479553222656, 'used': 57.34247589111328}}, 'gpu': 'NVIDIA H100 PCIe', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA H100 PCIe', 'memory_total': 85520809984}], 'memory': {'total': 196.56491470336914}} | |
| 2024-02-18 17:15:06,323 INFO HandlerThread:3454 [system_monitor.py:probe():224] Finished collecting system info | |
| 2024-02-18 17:15:06,323 INFO HandlerThread:3454 [system_monitor.py:probe():227] Publishing system info | |
| 2024-02-18 17:15:06,324 INFO HandlerThread:3454 [system_monitor.py:probe():229] Finished publishing system info | |
| 2024-02-18 17:15:06,336 DEBUG SenderThread:3454 [sender.py:send():382] send: files | |
| 2024-02-18 17:15:06,336 INFO SenderThread:3454 [sender.py:_save_file():1403] saving file wandb-metadata.json with policy now | |
| 2024-02-18 17:15:06,351 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: python_packages | |
| 2024-02-18 17:15:06,352 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: python_packages | |
| 2024-02-18 17:15:06,356 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-18 17:15:06,356 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-18 17:15:06,358 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-18 17:15:06,897 INFO Thread-12 :3454 [dir_watcher.py:_on_file_created():271] file/dir created: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/requirements.txt | |
| 2024-02-18 17:15:06,898 INFO Thread-12 :3454 [dir_watcher.py:_on_file_created():271] file/dir created: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/wandb-metadata.json | |
| 2024-02-18 17:15:06,902 DEBUG SenderThread:3454 [sender.py:send():382] send: telemetry | |
| 2024-02-18 17:15:07,023 INFO wandb-upload_0:3454 [upload_job.py:push():131] Uploaded file /tmp/tmp3j9a5bmbwandb/zl4crxjh-wandb-metadata.json | |
| 2024-02-18 17:15:07,894 INFO Thread-12 :3454 [dir_watcher.py:_on_file_created():271] file/dir created: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/output.log | |
| 2024-02-18 17:15:09,896 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/output.log | |
| 2024-02-18 17:15:10,907 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-18 17:15:15,908 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-18 17:15:21,353 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-18 17:15:21,354 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-18 17:15:21,358 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-18 17:15:21,455 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-18 17:15:23,902 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/output.log | |
| 2024-02-18 17:15:24,184 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-18 17:15:24,957 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-18 17:15:24,959 DEBUG SenderThread:3454 [sender.py:send():382] send: history | |
| 2024-02-18 17:15:24,960 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-18 17:15:24,960 INFO SenderThread:3454 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-18 17:15:25,904 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/output.log | |
| 2024-02-18 17:15:25,904 INFO Thread-12 :3454 [dir_watcher.py:_on_file_created():271] file/dir created: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/wandb-summary.json | |
| 2024-02-18 17:15:25,927 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-18 17:15:25,929 DEBUG SenderThread:3454 [sender.py:send():382] send: history | |
| 2024-02-18 17:15:25,930 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-18 17:15:25,931 INFO SenderThread:3454 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-18 17:15:26,756 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-18 17:15:26,905 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/wandb-summary.json | |
| 2024-02-18 17:15:27,067 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-18 17:15:27,069 DEBUG SenderThread:3454 [sender.py:send():382] send: history | |
| 2024-02-18 17:15:27,070 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-18 17:15:27,070 INFO SenderThread:3454 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-18 17:15:27,905 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/output.log | |
| 2024-02-18 17:15:27,906 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/wandb-summary.json | |
| 2024-02-18 17:15:27,999 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-18 17:15:28,001 DEBUG SenderThread:3454 [sender.py:send():382] send: history | |
| 2024-02-18 17:15:28,002 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-18 17:15:28,002 INFO SenderThread:3454 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-18 17:15:28,906 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/wandb-summary.json | |
| 2024-02-18 17:15:28,923 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-18 17:15:28,925 DEBUG SenderThread:3454 [sender.py:send():382] send: history | |
| 2024-02-18 17:15:28,926 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-18 17:15:28,926 INFO SenderThread:3454 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-18 17:15:29,796 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-18 17:15:29,798 DEBUG SenderThread:3454 [sender.py:send():382] send: history | |
| 2024-02-18 17:15:29,798 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-18 17:15:29,798 INFO SenderThread:3454 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-18 17:15:29,906 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/output.log | |
| 2024-02-18 17:15:29,907 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/wandb-summary.json | |
| 2024-02-18 17:15:30,913 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-18 17:15:30,915 DEBUG SenderThread:3454 [sender.py:send():382] send: history | |
| 2024-02-18 17:15:30,916 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-18 17:15:30,916 INFO SenderThread:3454 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-18 17:15:31,907 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/output.log | |
| 2024-02-18 17:15:31,908 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/wandb-summary.json | |
| 2024-02-18 17:15:32,076 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-18 17:15:32,078 DEBUG SenderThread:3454 [sender.py:send():382] send: history | |
| 2024-02-18 17:15:32,079 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-18 17:15:32,079 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-18 17:15:32,080 INFO SenderThread:3454 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-18 17:15:32,908 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/wandb-summary.json | |
| 2024-02-18 17:15:33,136 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-18 17:15:33,138 DEBUG SenderThread:3454 [sender.py:send():382] send: history | |
| 2024-02-18 17:15:33,139 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-18 17:15:33,139 INFO SenderThread:3454 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-18 17:15:33,909 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/output.log | |
| 2024-02-18 17:15:33,909 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/wandb-summary.json | |
| 2024-02-18 17:15:34,200 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-18 17:15:34,202 DEBUG SenderThread:3454 [sender.py:send():382] send: history | |
| 2024-02-18 17:15:34,203 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-18 17:15:34,206 INFO SenderThread:3454 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-18 17:15:34,909 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/wandb-summary.json | |
| 2024-02-18 17:15:35,386 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-18 17:15:35,388 DEBUG SenderThread:3454 [sender.py:send():382] send: history | |
| 2024-02-18 17:15:35,388 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-18 17:15:35,389 INFO SenderThread:3454 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-18 17:15:35,910 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/output.log | |
| 2024-02-18 17:15:35,911 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/wandb-summary.json | |
| 2024-02-18 17:15:36,352 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-18 17:15:36,354 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-18 17:15:36,357 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-18 17:15:36,664 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-18 17:15:36,666 DEBUG SenderThread:3454 [sender.py:send():382] send: history | |
| 2024-02-18 17:15:36,666 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-18 17:15:36,669 INFO SenderThread:3454 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-18 17:15:36,911 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/wandb-summary.json | |
| 2024-02-18 17:15:37,293 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-18 17:15:37,632 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-18 17:15:37,633 DEBUG SenderThread:3454 [sender.py:send():382] send: history | |
| 2024-02-18 17:15:37,633 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-18 17:15:37,634 INFO SenderThread:3454 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-18 17:15:37,912 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/output.log | |
| 2024-02-18 17:15:37,912 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/config.yaml | |
| 2024-02-18 17:15:37,912 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/wandb-summary.json | |
| 2024-02-18 17:15:38,431 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-18 17:15:38,433 DEBUG SenderThread:3454 [sender.py:send():382] send: history | |
| 2024-02-18 17:15:38,434 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-18 17:15:38,434 INFO SenderThread:3454 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-18 17:15:38,912 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/wandb-summary.json | |
| 2024-02-18 17:15:39,436 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-18 17:15:39,438 DEBUG SenderThread:3454 [sender.py:send():382] send: history | |
| 2024-02-18 17:15:39,438 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-18 17:15:39,439 INFO SenderThread:3454 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-18 17:15:39,913 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/output.log | |
| 2024-02-18 17:15:39,914 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/wandb-summary.json | |
| 2024-02-18 17:15:40,628 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-18 17:15:40,630 DEBUG SenderThread:3454 [sender.py:send():382] send: history | |
| 2024-02-18 17:15:40,630 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-18 17:15:40,631 INFO SenderThread:3454 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-18 17:15:40,914 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/wandb-summary.json | |
| 2024-02-18 17:15:41,914 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/output.log | |
| 2024-02-18 17:15:41,948 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-18 17:15:41,949 DEBUG SenderThread:3454 [sender.py:send():382] send: history | |
| 2024-02-18 17:15:41,950 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-18 17:15:41,950 INFO SenderThread:3454 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-18 17:15:42,615 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-18 17:15:42,915 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/wandb-summary.json | |
| 2024-02-18 17:15:43,014 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-18 17:15:43,016 DEBUG SenderThread:3454 [sender.py:send():382] send: history | |
| 2024-02-18 17:15:43,016 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-18 17:15:43,017 INFO SenderThread:3454 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-18 17:15:43,915 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/output.log | |
| 2024-02-18 17:15:43,916 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/wandb-summary.json | |
| 2024-02-18 17:15:44,059 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-18 17:15:44,061 DEBUG SenderThread:3454 [sender.py:send():382] send: history | |
| 2024-02-18 17:15:44,061 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-18 17:15:44,062 INFO SenderThread:3454 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-18 17:15:44,916 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/wandb-summary.json | |
| 2024-02-18 17:15:45,269 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-18 17:15:45,271 DEBUG SenderThread:3454 [sender.py:send():382] send: history | |
| 2024-02-18 17:15:45,271 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-18 17:15:45,272 INFO SenderThread:3454 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-18 17:15:45,917 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/output.log | |
| 2024-02-18 17:15:45,917 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/wandb-summary.json | |
| 2024-02-18 17:15:46,317 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-18 17:15:46,320 DEBUG SenderThread:3454 [sender.py:send():382] send: history | |
| 2024-02-18 17:15:46,321 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-18 17:15:46,325 INFO SenderThread:3454 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-18 17:15:46,918 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/wandb-summary.json | |
| 2024-02-18 17:15:47,292 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-18 17:15:47,294 DEBUG SenderThread:3454 [sender.py:send():382] send: history | |
| 2024-02-18 17:15:47,295 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-18 17:15:47,296 INFO SenderThread:3454 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-18 17:15:47,915 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-18 17:15:47,918 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/output.log | |
| 2024-02-18 17:15:47,919 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/wandb-summary.json | |
| 2024-02-18 17:15:48,334 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-18 17:15:48,336 DEBUG SenderThread:3454 [sender.py:send():382] send: history | |
| 2024-02-18 17:15:48,336 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-18 17:15:48,338 INFO SenderThread:3454 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-18 17:15:48,919 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/wandb-summary.json | |
| 2024-02-18 17:15:49,411 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-18 17:15:49,414 DEBUG SenderThread:3454 [sender.py:send():382] send: history | |
| 2024-02-18 17:15:49,415 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-18 17:15:49,417 INFO SenderThread:3454 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-18 17:15:49,920 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/output.log | |
| 2024-02-18 17:15:49,920 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/wandb-summary.json | |
| 2024-02-18 17:15:50,596 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-18 17:15:50,598 DEBUG SenderThread:3454 [sender.py:send():382] send: history | |
| 2024-02-18 17:15:50,599 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-18 17:15:50,601 INFO SenderThread:3454 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-18 17:15:50,920 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/wandb-summary.json | |
| 2024-02-18 17:15:51,353 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-18 17:15:51,355 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-18 17:15:51,360 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-18 17:15:51,720 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-18 17:15:51,722 DEBUG SenderThread:3454 [sender.py:send():382] send: history | |
| 2024-02-18 17:15:51,723 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-18 17:15:51,725 INFO SenderThread:3454 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-18 17:15:51,921 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/output.log | |
| 2024-02-18 17:15:51,922 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/wandb-summary.json | |
| 2024-02-18 17:15:52,775 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-18 17:15:52,778 DEBUG SenderThread:3454 [sender.py:send():382] send: history | |
| 2024-02-18 17:15:52,778 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-18 17:15:52,780 INFO SenderThread:3454 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-18 17:15:52,922 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/wandb-summary.json | |
| 2024-02-18 17:15:53,386 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-18 17:15:53,752 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-18 17:15:53,754 DEBUG SenderThread:3454 [sender.py:send():382] send: history | |
| 2024-02-18 17:15:53,755 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-18 17:15:53,756 INFO SenderThread:3454 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-18 17:15:53,923 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/output.log | |
| 2024-02-18 17:15:53,923 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/wandb-summary.json | |
| 2024-02-18 17:15:54,563 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-18 17:15:54,565 DEBUG SenderThread:3454 [sender.py:send():382] send: history | |
| 2024-02-18 17:15:54,566 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-18 17:15:54,568 INFO SenderThread:3454 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-18 17:15:54,924 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/wandb-summary.json | |
| 2024-02-18 17:15:55,396 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-18 17:15:55,399 DEBUG SenderThread:3454 [sender.py:send():382] send: history | |
| 2024-02-18 17:15:55,399 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-18 17:15:55,401 INFO SenderThread:3454 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-18 17:15:55,924 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/output.log | |
| 2024-02-18 17:15:55,925 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/wandb-summary.json | |
| 2024-02-18 17:15:56,541 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-18 17:15:56,543 DEBUG SenderThread:3454 [sender.py:send():382] send: history | |
| 2024-02-18 17:15:56,544 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-18 17:15:56,546 INFO SenderThread:3454 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-18 17:15:56,925 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/wandb-summary.json | |
| 2024-02-18 17:15:57,698 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-18 17:15:57,701 DEBUG SenderThread:3454 [sender.py:send():382] send: history | |
| 2024-02-18 17:15:57,701 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-18 17:15:57,702 INFO SenderThread:3454 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-18 17:15:57,926 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/output.log | |
| 2024-02-18 17:15:57,926 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/wandb-summary.json | |
| 2024-02-18 17:15:58,628 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-18 17:15:58,629 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-18 17:15:58,631 DEBUG SenderThread:3454 [sender.py:send():382] send: history | |
| 2024-02-18 17:15:58,631 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-18 17:15:58,632 INFO SenderThread:3454 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-18 17:15:58,926 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/wandb-summary.json | |
| 2024-02-18 17:15:59,910 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-18 17:15:59,913 DEBUG SenderThread:3454 [sender.py:send():382] send: history | |
| 2024-02-18 17:15:59,914 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-18 17:15:59,915 INFO SenderThread:3454 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-18 17:15:59,927 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/output.log | |
| 2024-02-18 17:15:59,928 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/wandb-summary.json | |
| 2024-02-18 17:16:01,158 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-18 17:16:01,161 DEBUG SenderThread:3454 [sender.py:send():382] send: history | |
| 2024-02-18 17:16:01,162 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-18 17:16:01,162 INFO SenderThread:3454 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-18 17:16:01,929 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/output.log | |
| 2024-02-18 17:16:01,929 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/wandb-summary.json | |
| 2024-02-18 17:16:02,298 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-18 17:16:02,300 DEBUG SenderThread:3454 [sender.py:send():382] send: history | |
| 2024-02-18 17:16:02,301 DEBUG SenderThread:3454 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-18 17:16:02,302 INFO SenderThread:3454 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-18 17:16:02,929 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/wandb-summary.json | |
| 2024-02-18 17:16:03,930 INFO Thread-12 :3454 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/ubuntu/torchtune-colorful-llama/colorful/wandb/run-20240218_171505-jggsueon/files/output.log | |
| 2024-02-18 17:16:04,244 INFO MainThread:3454 [internal.py:handle_exit():76] Internal process exited | |
| 2024-02-18 17:16:04,304 DEBUG HandlerThread:3454 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-18 17:16:04,313 INFO memory :3454 [interfaces.py:monitor():140] Process proc.memory.rssMB has exited. | |
| 2024-02-18 17:16:04,314 DEBUG SystemMonitor:3454 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-02-18 17:16:04,314 DEBUG SystemMonitor:3454 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-02-18 17:16:04,315 DEBUG SystemMonitor:3454 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-02-18 17:16:04,318 DEBUG SenderThread:3454 [sender.py:send():382] send: stats | |