2024-05-26 14:13:05,028 INFO StreamThr :225 [internal.py:wandb_internal():86] W&B internal server running at pid: 225, started at: 2024-05-26 14:13:05.028130 2024-05-26 14:13:05,030 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status 2024-05-26 14:13:06,044 INFO WriterThread:225 [datastore.py:open_for_write():87] open: /kaggle/working/wandb/run-20240526_141305-wfyhopk0/run-wfyhopk0.wandb 2024-05-26 14:13:06,044 DEBUG SenderThread:225 [sender.py:send():379] send: header 2024-05-26 14:13:06,047 DEBUG SenderThread:225 [sender.py:send():379] send: run 2024-05-26 14:13:06,468 INFO SenderThread:225 [dir_watcher.py:__init__():211] watching files in: /kaggle/working/wandb/run-20240526_141305-wfyhopk0/files 2024-05-26 14:13:06,469 INFO SenderThread:225 [sender.py:_start_run_threads():1124] run started: wfyhopk0 with start time 1716732785.02907 2024-05-26 14:13:06,477 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: check_version 2024-05-26 14:13:06,477 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: check_version 2024-05-26 14:13:06,534 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: run_start 2024-05-26 14:13:06,545 DEBUG HandlerThread:225 [system_info.py:__init__():26] System info init 2024-05-26 14:13:06,545 DEBUG HandlerThread:225 [system_info.py:__init__():41] System info init done 2024-05-26 14:13:06,545 INFO HandlerThread:225 [system_monitor.py:start():194] Starting system monitor 2024-05-26 14:13:06,546 INFO SystemMonitor:225 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-05-26 14:13:06,546 INFO HandlerThread:225 [system_monitor.py:probe():214] Collecting system info 2024-05-26 14:13:06,546 INFO SystemMonitor:225 [interfaces.py:start():190] Started cpu monitoring 2024-05-26 14:13:06,547 INFO SystemMonitor:225 [interfaces.py:start():190] Started disk monitoring 2024-05-26 14:13:06,549 INFO SystemMonitor:225 [interfaces.py:start():190] Started gpu monitoring 2024-05-26 14:13:06,550 INFO SystemMonitor:225 [interfaces.py:start():190] Started memory monitoring 2024-05-26 14:13:06,550 INFO SystemMonitor:225 [interfaces.py:start():190] Started network monitoring 2024-05-26 14:13:06,561 DEBUG HandlerThread:225 [system_info.py:probe():150] Probing system 2024-05-26 14:13:06,563 DEBUG HandlerThread:225 [gitlib.py:_init_repo():56] git repository is invalid 2024-05-26 14:13:06,563 DEBUG HandlerThread:225 [system_info.py:probe():198] Probing system done 2024-05-26 14:13:06,563 DEBUG HandlerThread:225 [system_monitor.py:probe():223] {'os': 'Linux-5.15.133+-x86_64-with-glibc2.31', 'python': '3.10.13', 'heartbeatAt': '2024-05-26T14:13:06.561396', 'startedAt': '2024-05-26T14:13:05.022165', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'kaggle.ipynb', 'codePathLocal': None, 'root': '/kaggle/working', 'host': '142d81d78b72', 'username': 'root', 'executable': '/opt/conda/bin/python3.10', 'cpu_count': 2, 'cpu_count_logical': 4, 'cpu_freq': {'current': 2000.164, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.164, 'min': 0.0, 'max': 0.0}, {'current': 2000.164, 'min': 0.0, 'max': 0.0}, {'current': 2000.164, 'min': 0.0, 'max': 0.0}, {'current': 2000.164, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 8062.387607574463, 'used': 5597.954666137695}}, 'gpu': 'Tesla P100-PCIE-16GB', 'gpu_count': 1, 'gpu_devices': [{'name': 'Tesla P100-PCIE-16GB', 'memory_total': 17179869184}], 'memory': {'total': 31.357559204101562}} 2024-05-26 14:13:06,563 INFO HandlerThread:225 [system_monitor.py:probe():224] Finished collecting system info 2024-05-26 14:13:06,563 INFO HandlerThread:225 [system_monitor.py:probe():227] Publishing system info 2024-05-26 14:13:06,563 DEBUG HandlerThread:225 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment 2024-05-26 14:13:07,470 INFO Thread-12 :225 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240526_141305-wfyhopk0/files/conda-environment.yaml 2024-05-26 14:13:21,580 ERROR HandlerThread:225 [system_info.py:_save_conda():221] Error saving conda packages: Command '['conda', 'env', 'export']' timed out after 15 seconds Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/internal/system/system_info.py", line 214, in _save_conda subprocess.call( File "/opt/conda/lib/python3.10/subprocess.py", line 347, in call return p.wait(timeout=timeout) File "/opt/conda/lib/python3.10/subprocess.py", line 1209, in wait return self._wait(timeout=timeout) File "/opt/conda/lib/python3.10/subprocess.py", line 1951, in _wait raise TimeoutExpired(self.args, timeout) subprocess.TimeoutExpired: Command '['conda', 'env', 'export']' timed out after 15 seconds 2024-05-26 14:13:21,583 DEBUG HandlerThread:225 [system_info.py:_save_conda():222] Saving conda packages done 2024-05-26 14:13:21,583 INFO HandlerThread:225 [system_monitor.py:probe():229] Finished publishing system info 2024-05-26 14:13:21,591 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:13:21,592 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: keepalive 2024-05-26 14:13:21,592 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:13:21,592 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: keepalive 2024-05-26 14:13:21,592 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:13:21,592 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: keepalive 2024-05-26 14:13:21,592 DEBUG SenderThread:225 [sender.py:send():379] send: files 2024-05-26 14:13:21,593 INFO SenderThread:225 [sender.py:_save_file():1390] saving file wandb-metadata.json with policy now 2024-05-26 14:13:22,239 INFO wandb-upload_0:225 [upload_job.py:push():131] Uploaded file /tmp/tmpfxjhfb1zwandb/k20mz4b6-wandb-metadata.json 2024-05-26 14:13:22,474 INFO Thread-12 :225 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240526_141305-wfyhopk0/files/wandb-metadata.json 2024-05-26 14:13:22,557 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: python_packages 2024-05-26 14:13:22,558 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: python_packages 2024-05-26 14:13:22,560 DEBUG SenderThread:225 [sender.py:send():379] send: telemetry 2024-05-26 14:13:22,573 DEBUG SenderThread:225 [sender.py:send():379] send: config 2024-05-26 14:13:22,575 DEBUG SenderThread:225 [sender.py:send():379] send: metric 2024-05-26 14:13:22,575 DEBUG SenderThread:225 [sender.py:send():379] send: telemetry 2024-05-26 14:13:22,575 DEBUG SenderThread:225 [sender.py:send():379] send: metric 2024-05-26 14:13:22,575 WARNING SenderThread:225 [sender.py:send_metric():1341] Seen metric with glob (shouldn't happen) 2024-05-26 14:13:22,575 DEBUG SenderThread:225 [sender.py:send():379] send: telemetry 2024-05-26 14:13:22,576 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:13:22,577 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:13:22,577 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:13:23,474 INFO Thread-12 :225 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240526_141305-wfyhopk0/files/requirements.txt 2024-05-26 14:13:23,475 INFO Thread-12 :225 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240526_141305-wfyhopk0/files/output.log 2024-05-26 14:13:25,475 INFO Thread-12 :225 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240526_141305-wfyhopk0/files/output.log 2024-05-26 14:13:25,533 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:13:30,534 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:13:35,540 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:13:36,479 INFO Thread-12 :225 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240526_141305-wfyhopk0/files/config.yaml 2024-05-26 14:13:37,729 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:13:37,730 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:13:37,732 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:13:40,943 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:13:45,944 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:13:50,945 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:13:52,677 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:13:52,677 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:13:52,717 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:13:56,897 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:14:01,898 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:14:06,550 DEBUG SystemMonitor:225 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-05-26 14:14:06,552 DEBUG SenderThread:225 [sender.py:send():379] send: stats 2024-05-26 14:14:07,553 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:14:07,975 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:14:07,976 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:14:07,977 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:14:13,194 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:14:18,196 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:14:22,965 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:14:22,974 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:14:22,975 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:14:24,158 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:14:29,159 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:14:34,160 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:14:36,553 DEBUG SenderThread:225 [sender.py:send():379] send: stats 2024-05-26 14:14:37,966 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:14:37,975 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:14:37,975 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:14:39,207 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:14:44,208 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:14:49,210 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:14:53,045 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:14:53,066 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:14:53,067 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:14:54,265 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:14:59,266 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:15:04,267 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:15:06,554 DEBUG SenderThread:225 [sender.py:send():379] send: stats 2024-05-26 14:15:08,034 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:15:08,034 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:15:08,035 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:15:10,176 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:15:15,177 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:15:20,179 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:15:23,224 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:15:23,225 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:15:23,228 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:15:25,427 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:15:30,428 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:15:35,428 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:15:36,555 DEBUG SenderThread:225 [sender.py:send():379] send: stats 2024-05-26 14:15:38,194 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:15:38,195 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:15:38,195 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:15:41,356 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:15:46,356 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:15:51,357 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:15:53,194 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:15:53,195 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:15:53,195 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:15:56,402 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:16:01,403 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:16:06,404 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:16:06,555 DEBUG SenderThread:225 [sender.py:send():379] send: stats 2024-05-26 14:16:08,194 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:16:08,195 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:16:08,195 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:16:11,441 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:16:16,442 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:16:21,443 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:16:23,331 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:16:23,332 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:16:23,332 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:16:26,538 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:16:31,539 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:16:36,540 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:16:36,556 DEBUG SenderThread:225 [sender.py:send():379] send: stats 2024-05-26 14:16:38,437 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:16:38,438 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:16:38,439 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:16:41,579 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:16:46,580 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:16:51,581 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:16:53,400 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:16:53,401 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:16:53,401 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:16:56,646 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:17:01,647 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:17:06,557 DEBUG SenderThread:225 [sender.py:send():379] send: stats 2024-05-26 14:17:07,558 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:17:08,401 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:17:08,401 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:17:08,401 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:17:12,573 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:17:17,574 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:17:22,575 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:17:23,457 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:17:23,457 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:17:23,461 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:17:27,603 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:17:32,604 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:17:36,558 DEBUG SenderThread:225 [sender.py:send():379] send: stats 2024-05-26 14:17:38,511 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:17:38,512 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:17:38,512 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:17:38,513 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:17:43,750 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:17:48,751 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:17:53,602 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:17:53,602 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:17:53,643 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:17:53,851 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:17:58,853 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:18:03,854 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:18:06,559 DEBUG SenderThread:225 [sender.py:send():379] send: stats 2024-05-26 14:18:08,729 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:18:08,730 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:18:08,769 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:18:08,886 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:18:13,886 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:18:18,887 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:18:23,707 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:18:23,708 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:18:23,748 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:18:23,935 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:18:28,935 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:18:33,937 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:18:34,444 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: partial_history 2024-05-26 14:18:34,446 DEBUG SenderThread:225 [sender.py:send():379] send: metric 2024-05-26 14:18:34,446 DEBUG SenderThread:225 [sender.py:send():379] send: metric 2024-05-26 14:18:34,447 DEBUG SenderThread:225 [sender.py:send():379] send: metric 2024-05-26 14:18:34,447 DEBUG SenderThread:225 [sender.py:send():379] send: metric 2024-05-26 14:18:34,447 DEBUG SenderThread:225 [sender.py:send():379] send: history 2024-05-26 14:18:34,447 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: summary_record 2024-05-26 14:18:34,447 INFO SenderThread:225 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-26 14:18:34,595 INFO Thread-12 :225 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240526_141305-wfyhopk0/files/wandb-summary.json 2024-05-26 14:18:36,560 DEBUG SenderThread:225 [sender.py:send():379] send: stats 2024-05-26 14:18:38,725 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:18:38,726 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:18:38,728 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:18:38,980 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:18:43,989 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:18:44,541 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: partial_history 2024-05-26 14:18:44,543 DEBUG SenderThread:225 [sender.py:send():379] send: metric 2024-05-26 14:18:44,543 DEBUG SenderThread:225 [sender.py:send():379] send: metric 2024-05-26 14:18:44,544 DEBUG SenderThread:225 [sender.py:send():379] send: metric 2024-05-26 14:18:44,544 DEBUG SenderThread:225 [sender.py:send():379] send: metric 2024-05-26 14:18:44,544 DEBUG SenderThread:225 [sender.py:send():379] send: metric 2024-05-26 14:18:44,544 DEBUG SenderThread:225 [sender.py:send():379] send: history 2024-05-26 14:18:44,544 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: summary_record 2024-05-26 14:18:44,545 INFO SenderThread:225 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-26 14:18:44,599 INFO Thread-12 :225 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240526_141305-wfyhopk0/files/config.yaml 2024-05-26 14:18:44,599 INFO Thread-12 :225 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240526_141305-wfyhopk0/files/wandb-summary.json 2024-05-26 14:18:47,600 INFO Thread-12 :225 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240526_141305-wfyhopk0/files/output.log 2024-05-26 14:18:49,772 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:18:53,717 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:18:53,717 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:18:53,728 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:18:54,981 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:18:59,983 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:19:04,983 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:19:06,561 DEBUG SenderThread:225 [sender.py:send():379] send: stats 2024-05-26 14:19:08,998 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:19:08,999 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:19:08,999 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:19:10,173 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:19:15,182 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:19:15,611 INFO Thread-12 :225 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240526_141305-wfyhopk0/files/config.yaml 2024-05-26 14:19:20,444 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:19:24,110 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:19:24,111 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:19:24,112 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:19:26,358 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:19:31,359 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:19:36,360 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:19:36,561 DEBUG SenderThread:225 [sender.py:send():379] send: stats 2024-05-26 14:19:39,080 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:19:39,094 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:19:39,095 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:19:42,342 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:19:47,343 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:19:52,344 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:19:54,133 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:19:54,134 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:19:54,134 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:19:57,378 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:20:02,379 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:20:06,562 DEBUG SenderThread:225 [sender.py:send():379] send: stats 2024-05-26 14:20:07,563 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:20:09,161 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:20:09,162 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:20:09,202 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:20:13,403 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:20:18,404 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:20:23,405 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:20:24,256 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:20:24,257 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:20:24,258 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:20:28,448 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:20:33,449 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:20:36,563 DEBUG SenderThread:225 [sender.py:send():379] send: stats 2024-05-26 14:20:38,564 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:20:39,217 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:20:39,229 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:20:39,229 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:20:44,376 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:20:49,378 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:20:54,323 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:20:54,324 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:20:54,325 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:20:54,536 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:20:59,537 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:21:04,539 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:21:06,564 DEBUG SenderThread:225 [sender.py:send():379] send: stats 2024-05-26 14:21:09,282 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:21:09,282 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:21:09,283 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:21:10,414 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:21:15,415 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:21:20,416 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:21:24,408 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:21:24,408 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:21:24,448 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:21:25,636 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:21:30,637 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:21:35,638 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:21:36,565 DEBUG SenderThread:225 [sender.py:send():379] send: stats 2024-05-26 14:21:39,424 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:21:39,425 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:21:39,465 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:21:41,638 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:21:46,639 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:21:51,640 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:21:54,587 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:21:54,588 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:21:54,588 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:21:56,779 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:22:01,780 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:22:06,566 DEBUG SenderThread:225 [sender.py:send():379] send: stats 2024-05-26 14:22:07,567 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:22:09,559 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:22:09,560 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:22:09,560 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:22:12,730 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:22:17,730 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:22:22,731 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:22:24,792 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:22:24,794 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:22:24,794 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:22:28,053 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:22:33,054 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:22:36,567 DEBUG SenderThread:225 [sender.py:send():379] send: stats 2024-05-26 14:22:38,568 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:22:39,741 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:22:39,742 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:22:39,782 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:22:43,915 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:22:48,916 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:22:53,917 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:22:54,741 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:22:54,741 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:22:54,782 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:22:59,906 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:23:04,907 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:23:06,568 DEBUG SenderThread:225 [sender.py:send():379] send: stats 2024-05-26 14:23:09,808 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:23:09,809 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:23:09,849 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:23:10,024 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:23:15,025 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:23:20,026 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:23:24,784 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:23:24,785 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:23:24,826 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:23:26,014 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:23:31,015 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:23:36,016 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:23:36,569 DEBUG SenderThread:225 [sender.py:send():379] send: stats 2024-05-26 14:23:39,780 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:23:39,784 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:23:39,784 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:23:41,930 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:23:46,931 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:23:51,933 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:23:55,089 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:23:55,090 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:23:55,091 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:23:55,427 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: partial_history 2024-05-26 14:23:55,428 DEBUG SenderThread:225 [sender.py:send():379] send: history 2024-05-26 14:23:55,428 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: summary_record 2024-05-26 14:23:55,429 INFO SenderThread:225 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-26 14:23:55,720 INFO Thread-12 :225 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240526_141305-wfyhopk0/files/wandb-summary.json 2024-05-26 14:23:57,430 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:24:02,431 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:24:05,670 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: partial_history 2024-05-26 14:24:05,672 DEBUG SenderThread:225 [sender.py:send():379] send: history 2024-05-26 14:24:05,672 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: summary_record 2024-05-26 14:24:05,672 INFO SenderThread:225 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-26 14:24:05,725 INFO Thread-12 :225 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240526_141305-wfyhopk0/files/wandb-summary.json 2024-05-26 14:24:06,569 DEBUG SenderThread:225 [sender.py:send():379] send: stats 2024-05-26 14:24:07,570 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:24:09,929 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-26 14:24:09,930 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: stop_status 2024-05-26 14:24:09,931 DEBUG SenderThread:225 [sender.py:send_request():406] send_request: stop_status 2024-05-26 14:24:10,727 INFO Thread-12 :225 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240526_141305-wfyhopk0/files/output.log 2024-05-26 14:24:13,110 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:24:18,111 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report 2024-05-26 14:24:23,113 DEBUG HandlerThread:225 [handler.py:handle_request():146] handle_request: status_report