2024-05-22 11:34:14,002 INFO StreamThr :290 [internal.py:wandb_internal():86] W&B internal server running at pid: 290, started at: 2024-05-22 11:34:14.001974 2024-05-22 11:34:14,004 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status 2024-05-22 11:34:14,541 INFO WriterThread:290 [datastore.py:open_for_write():87] open: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/run-8mudzhjp.wandb 2024-05-22 11:34:14,542 DEBUG SenderThread:290 [sender.py:send():379] send: header 2024-05-22 11:34:14,545 DEBUG SenderThread:290 [sender.py:send():379] send: run 2024-05-22 11:34:14,770 INFO SenderThread:290 [dir_watcher.py:__init__():211] watching files in: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files 2024-05-22 11:34:14,770 INFO SenderThread:290 [sender.py:_start_run_threads():1124] run started: 8mudzhjp with start time 1716377654.002618 2024-05-22 11:34:14,778 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: check_version 2024-05-22 11:34:14,779 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: check_version 2024-05-22 11:34:14,850 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: run_start 2024-05-22 11:34:14,861 DEBUG HandlerThread:290 [system_info.py:__init__():26] System info init 2024-05-22 11:34:14,861 DEBUG HandlerThread:290 [system_info.py:__init__():41] System info init done 2024-05-22 11:34:14,861 INFO HandlerThread:290 [system_monitor.py:start():194] Starting system monitor 2024-05-22 11:34:14,862 INFO SystemMonitor:290 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-05-22 11:34:14,862 INFO HandlerThread:290 [system_monitor.py:probe():214] Collecting system info 2024-05-22 11:34:14,862 INFO SystemMonitor:290 [interfaces.py:start():190] Started cpu monitoring 2024-05-22 11:34:14,863 INFO SystemMonitor:290 [interfaces.py:start():190] Started disk monitoring 2024-05-22 11:34:14,864 INFO SystemMonitor:290 [interfaces.py:start():190] Started gpu monitoring 2024-05-22 11:34:14,866 INFO SystemMonitor:290 [interfaces.py:start():190] Started memory monitoring 2024-05-22 11:34:14,866 INFO SystemMonitor:290 [interfaces.py:start():190] Started network monitoring 2024-05-22 11:34:14,876 DEBUG HandlerThread:290 [system_info.py:probe():150] Probing system 2024-05-22 11:34:14,878 DEBUG HandlerThread:290 [gitlib.py:_init_repo():56] git repository is invalid 2024-05-22 11:34:14,878 DEBUG HandlerThread:290 [system_info.py:probe():198] Probing system done 2024-05-22 11:34:14,878 DEBUG HandlerThread:290 [system_monitor.py:probe():223] {'os': 'Linux-5.15.133+-x86_64-with-glibc2.31', 'python': '3.10.13', 'heartbeatAt': '2024-05-22T11:34:14.876249', 'startedAt': '2024-05-22T11:34:13.994936', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'kaggle.ipynb', 'codePathLocal': None, 'root': '/kaggle/working', 'host': '2c1b614ec68f', 'username': 'root', 'executable': '/opt/conda/bin/python3.10', 'cpu_count': 2, 'cpu_count_logical': 4, 'cpu_freq': {'current': 2000.144, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.144, 'min': 0.0, 'max': 0.0}, {'current': 2000.144, 'min': 0.0, 'max': 0.0}, {'current': 2000.144, 'min': 0.0, 'max': 0.0}, {'current': 2000.144, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 8062.387607574463, 'used': 5656.421318054199}}, 'gpu': 'Tesla P100-PCIE-16GB', 'gpu_count': 1, 'gpu_devices': [{'name': 'Tesla P100-PCIE-16GB', 'memory_total': 17179869184}], 'memory': {'total': 31.357563018798828}} 2024-05-22 11:34:14,878 INFO HandlerThread:290 [system_monitor.py:probe():224] Finished collecting system info 2024-05-22 11:34:14,878 INFO HandlerThread:290 [system_monitor.py:probe():227] Publishing system info 2024-05-22 11:34:14,878 DEBUG HandlerThread:290 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment 2024-05-22 11:34:15,772 INFO Thread-12 :290 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/conda-environment.yaml 2024-05-22 11:34:29,895 ERROR HandlerThread:290 [system_info.py:_save_conda():221] Error saving conda packages: Command '['conda', 'env', 'export']' timed out after 15 seconds Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/internal/system/system_info.py", line 214, in _save_conda subprocess.call( File "/opt/conda/lib/python3.10/subprocess.py", line 347, in call return p.wait(timeout=timeout) File "/opt/conda/lib/python3.10/subprocess.py", line 1209, in wait return self._wait(timeout=timeout) File "/opt/conda/lib/python3.10/subprocess.py", line 1951, in _wait raise TimeoutExpired(self.args, timeout) subprocess.TimeoutExpired: Command '['conda', 'env', 'export']' timed out after 15 seconds 2024-05-22 11:34:29,898 DEBUG HandlerThread:290 [system_info.py:_save_conda():222] Saving conda packages done 2024-05-22 11:34:29,898 INFO HandlerThread:290 [system_monitor.py:probe():229] Finished publishing system info 2024-05-22 11:34:29,905 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:34:29,905 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: keepalive 2024-05-22 11:34:29,905 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:34:29,905 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: keepalive 2024-05-22 11:34:29,905 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:34:29,905 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: keepalive 2024-05-22 11:34:29,906 DEBUG SenderThread:290 [sender.py:send():379] send: files 2024-05-22 11:34:29,906 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-metadata.json with policy now 2024-05-22 11:34:30,205 INFO wandb-upload_0:290 [upload_job.py:push():131] Uploaded file /tmp/tmpr3076z41wandb/p0q9p97i-wandb-metadata.json 2024-05-22 11:34:30,775 INFO Thread-12 :290 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-metadata.json 2024-05-22 11:34:30,855 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: python_packages 2024-05-22 11:34:30,856 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: python_packages 2024-05-22 11:34:30,859 DEBUG SenderThread:290 [sender.py:send():379] send: telemetry 2024-05-22 11:34:30,864 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:34:30,864 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:34:30,876 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:34:30,931 DEBUG SenderThread:290 [sender.py:send():379] send: config 2024-05-22 11:34:30,933 DEBUG SenderThread:290 [sender.py:send():379] send: metric 2024-05-22 11:34:30,934 DEBUG SenderThread:290 [sender.py:send():379] send: telemetry 2024-05-22 11:34:30,934 DEBUG SenderThread:290 [sender.py:send():379] send: metric 2024-05-22 11:34:30,934 WARNING SenderThread:290 [sender.py:send_metric():1341] Seen metric with glob (shouldn't happen) 2024-05-22 11:34:30,934 DEBUG SenderThread:290 [sender.py:send():379] send: telemetry 2024-05-22 11:34:30,934 DEBUG SenderThread:290 [sender.py:send():379] send: telemetry 2024-05-22 11:34:30,934 DEBUG SenderThread:290 [sender.py:send():379] send: config 2024-05-22 11:34:31,776 INFO Thread-12 :290 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/output.log 2024-05-22 11:34:31,776 INFO Thread-12 :290 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/requirements.txt 2024-05-22 11:34:31,792 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: log_artifact 2024-05-22 11:34:31,793 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: log_artifact 2024-05-22 11:34:32,556 INFO wandb-upload_3:290 [upload_job.py:push():89] Uploaded file /tmp/tmp_p0hvowz/model_architecture.txt 2024-05-22 11:34:32,561 INFO wandb-upload_0:290 [upload_job.py:push():89] Uploaded file /tmp/tmp_p0hvowz/README.md 2024-05-22 11:34:32,569 INFO wandb-upload_1:290 [upload_job.py:push():89] Uploaded file /tmp/tmp_p0hvowz/adapter_config.json 2024-05-22 11:34:32,795 INFO wandb-upload_2:290 [upload_job.py:push():89] Uploaded file /tmp/tmp_p0hvowz/adapter_model.safetensors 2024-05-22 11:34:33,350 INFO SenderThread:290 [sender.py:send_request_log_artifact():1456] logged artifact model-8mudzhjp - {'id': 'QXJ0aWZhY3Q6ODQ2MDQyODAx', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE3Nzk3MzExMA==', 'latestArtifact': None}} 2024-05-22 11:34:33,777 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/output.log 2024-05-22 11:34:35,351 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:34:35,777 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/output.log 2024-05-22 11:34:40,352 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:34:45,362 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:34:45,781 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/config.yaml 2024-05-22 11:34:45,856 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:34:45,857 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:34:45,859 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:34:50,964 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:34:55,965 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:35:00,856 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:35:00,857 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:35:00,897 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:35:01,929 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:35:06,930 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:35:11,931 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:35:14,866 DEBUG SystemMonitor:290 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-05-22 11:35:14,868 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:35:15,856 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:35:15,857 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:35:15,897 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:35:16,958 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:35:21,959 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:35:26,960 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:35:30,857 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:35:30,857 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:35:30,897 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:35:32,942 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:35:37,944 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:35:42,945 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:35:44,869 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:35:45,857 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:35:45,857 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:35:45,897 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:35:47,948 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:35:52,949 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:35:57,950 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:36:00,857 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:36:00,858 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:36:00,898 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:36:02,970 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:36:07,971 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:36:12,972 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:36:14,870 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:36:15,857 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:36:15,858 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:36:15,898 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:36:18,016 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:36:23,017 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:36:28,018 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:36:30,857 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:36:30,858 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:36:30,898 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:36:33,990 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:36:38,991 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:36:43,992 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:36:44,870 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:36:45,857 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:36:45,858 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:36:45,898 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:36:49,920 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:36:54,921 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:36:59,921 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:37:00,857 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:37:00,858 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:37:00,898 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:37:04,951 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:37:09,952 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:37:14,871 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:37:15,858 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:37:15,858 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:37:15,898 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:37:15,945 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:37:20,946 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:37:25,946 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:37:30,858 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:37:30,858 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:37:30,899 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:37:31,017 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:37:36,018 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:37:38,442 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 11:37:38,445 DEBUG SenderThread:290 [sender.py:send():379] send: metric 2024-05-22 11:37:38,445 DEBUG SenderThread:290 [sender.py:send():379] send: metric 2024-05-22 11:37:38,445 DEBUG SenderThread:290 [sender.py:send():379] send: metric 2024-05-22 11:37:38,445 DEBUG SenderThread:290 [sender.py:send():379] send: metric 2024-05-22 11:37:38,445 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 11:37:38,446 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 11:37:38,447 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 11:37:38,844 INFO Thread-12 :290 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 11:37:41,449 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:37:44,872 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:37:45,858 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:37:45,859 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:37:45,861 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:37:46,938 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:37:51,944 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:37:53,869 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/config.yaml 2024-05-22 11:37:57,869 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:38:00,858 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:38:00,859 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:38:00,899 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:38:02,949 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:38:07,950 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:38:12,951 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:38:14,873 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:38:15,858 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:38:15,859 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:38:15,899 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:38:17,960 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:38:22,961 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:38:27,962 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:38:31,097 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:38:31,097 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:38:31,138 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:38:33,173 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:38:38,174 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:38:43,175 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:38:44,874 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:38:46,085 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:38:46,086 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:38:46,126 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:38:49,160 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:38:54,161 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:38:59,162 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:39:01,084 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:39:01,085 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:39:01,086 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:39:04,170 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:39:09,171 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:39:14,172 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:39:14,875 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:39:16,084 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:39:16,085 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:39:16,085 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:39:19,245 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:39:24,247 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:39:29,247 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:39:31,084 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:39:31,086 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:39:31,087 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:39:35,179 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:39:40,180 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:39:44,875 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:39:45,876 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:39:46,084 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:39:46,086 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:39:46,087 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:39:51,240 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:39:56,242 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:40:01,085 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:40:01,086 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:40:01,086 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:40:02,194 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:40:03,044 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 11:40:03,046 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 11:40:03,046 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 11:40:03,047 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 11:40:03,919 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 11:40:08,048 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:40:13,049 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:40:14,877 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:40:16,085 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:40:16,086 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:40:16,086 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:40:18,182 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:40:23,184 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:40:28,184 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:40:31,085 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:40:31,086 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:40:31,086 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:40:33,213 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:40:38,214 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:40:43,215 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:40:44,877 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:40:46,085 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:40:46,086 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:40:46,087 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:40:49,193 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:40:54,194 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:40:59,195 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:41:01,085 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:41:01,087 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:41:01,087 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:41:04,203 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:41:09,204 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:41:14,205 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:41:14,878 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:41:16,085 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:41:16,087 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:41:16,087 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:41:20,203 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:41:25,204 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:41:29,488 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 11:41:29,489 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 11:41:29,490 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 11:41:29,490 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 11:41:29,955 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 11:41:30,491 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:41:31,086 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:41:31,086 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:41:31,087 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:41:36,243 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:41:41,244 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:41:44,879 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:41:46,085 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:41:46,087 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:41:46,088 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:41:47,181 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:41:52,182 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:41:57,183 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:42:01,086 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:42:01,087 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:42:01,087 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:42:02,231 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:42:07,232 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:42:12,233 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:42:14,880 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:42:16,086 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:42:16,087 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:42:16,087 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:42:17,241 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:42:22,242 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:42:27,243 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:42:31,086 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:42:31,087 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:42:31,088 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:42:33,199 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:42:38,200 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:42:43,201 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:42:44,881 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:42:46,086 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:42:46,087 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:42:46,088 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:42:49,184 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:42:54,185 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:42:59,185 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:43:01,086 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:43:01,088 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:43:01,088 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:43:04,229 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:43:09,230 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:43:14,231 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:43:14,882 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:43:16,086 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:43:16,087 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:43:16,088 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:43:16,324 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 11:43:16,325 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 11:43:16,326 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 11:43:16,326 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 11:43:16,994 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 11:43:19,327 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:43:24,328 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:43:29,329 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:43:31,087 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:43:31,088 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:43:31,088 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:43:35,213 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:43:40,214 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:43:44,883 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:43:45,883 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:43:46,087 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:43:46,088 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:43:46,088 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:43:51,240 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:43:56,241 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:44:01,087 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:44:01,088 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:44:01,089 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:44:01,261 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:44:06,262 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:44:11,263 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:44:14,884 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:44:16,087 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:44:16,088 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:44:16,089 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:44:17,242 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:44:22,243 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:44:27,244 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:44:31,087 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:44:31,089 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:44:31,089 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:44:33,241 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:44:38,242 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:44:43,243 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:44:44,884 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:44:46,088 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:44:46,088 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:44:46,089 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:44:48,261 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:44:53,262 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:44:58,262 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:45:01,089 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:45:01,091 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:45:01,130 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:45:01,651 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 11:45:01,652 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 11:45:01,652 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 11:45:01,653 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 11:45:02,033 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 11:45:03,653 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:45:08,655 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:45:13,656 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:45:14,885 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:45:16,088 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:45:16,089 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:45:16,089 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:45:19,179 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:45:24,180 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:45:29,181 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:45:31,088 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:45:31,089 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:45:31,089 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:45:34,214 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:45:39,215 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:45:44,216 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:45:44,886 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:45:46,088 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:45:46,089 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:45:46,089 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:45:50,216 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:45:55,216 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:46:00,217 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:46:01,088 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:46:01,089 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:46:01,090 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:46:06,206 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:46:11,207 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:46:14,887 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:46:16,088 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:46:16,089 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:46:16,090 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:46:16,229 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:46:19,889 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 11:46:19,890 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 11:46:19,891 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 11:46:19,891 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 11:46:20,063 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 11:46:21,892 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:46:26,893 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:46:31,089 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:46:31,090 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:46:31,091 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:46:32,208 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:46:37,209 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:46:42,209 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:46:44,888 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:46:46,089 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:46:46,089 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:46:46,090 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:46:48,179 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:46:53,180 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:46:58,181 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:47:01,089 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:47:01,090 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:47:01,090 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:47:04,016 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:47:09,017 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:47:14,018 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:47:14,889 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:47:16,089 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:47:16,090 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:47:16,090 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:47:19,177 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:47:24,178 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:47:29,180 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:47:31,089 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:47:31,090 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:47:31,090 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:47:35,168 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:47:40,169 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:47:44,890 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:47:45,890 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:47:46,089 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:47:46,090 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:47:46,090 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:47:51,193 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:47:52,653 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 11:47:52,654 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 11:47:52,655 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 11:47:52,655 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 11:47:53,097 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 11:47:56,656 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:48:01,090 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:48:01,090 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:48:01,091 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:48:02,204 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:48:07,205 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:48:12,206 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:48:14,890 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:48:16,090 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:48:16,091 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:48:16,092 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:48:17,271 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:48:22,272 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:48:27,273 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:48:31,090 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:48:31,091 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:48:31,092 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:48:33,218 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:48:38,219 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:48:43,220 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:48:44,891 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:48:46,090 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:48:46,091 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:48:46,092 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:48:48,260 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:48:53,261 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:48:58,262 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:49:01,092 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:49:01,092 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:49:01,093 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:49:04,208 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:49:09,209 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:49:14,210 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:49:14,892 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:49:16,090 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:49:16,092 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:49:16,092 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:49:20,154 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:49:22,737 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 11:49:22,738 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 11:49:22,738 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 11:49:22,739 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 11:49:23,131 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 11:49:25,740 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:49:30,741 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:49:31,090 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:49:31,092 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:49:31,093 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:49:36,204 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:49:41,204 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:49:44,893 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:49:46,090 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:49:46,092 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:49:46,092 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:49:47,178 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:49:52,179 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:49:57,180 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:50:01,091 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:50:01,092 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:50:01,092 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:50:02,231 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:50:07,232 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:50:12,233 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:50:14,894 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:50:16,091 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:50:16,092 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:50:16,092 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:50:17,265 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:50:22,267 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:50:27,268 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:50:31,091 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:50:31,092 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:50:31,093 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:50:33,191 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:50:38,192 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:50:43,193 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:50:44,895 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:50:46,091 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:50:46,093 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:50:46,093 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:50:49,161 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:50:54,162 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:50:59,162 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:51:01,091 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:51:01,093 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:51:01,094 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:51:04,168 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:51:09,169 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:51:14,170 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:51:14,211 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 11:51:14,217 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 11:51:14,218 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 11:51:14,218 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 11:51:14,896 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:51:15,173 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 11:51:16,091 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:51:16,093 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:51:16,094 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:51:19,190 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:51:24,191 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:51:29,192 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:51:31,091 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:51:31,093 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:51:31,094 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:51:35,188 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:51:40,189 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:51:44,897 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:51:45,898 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:51:46,091 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:51:46,093 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:51:46,094 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:51:51,168 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:51:56,168 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:52:01,091 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:52:01,094 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:52:01,094 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:52:01,247 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:52:06,248 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:52:11,249 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:52:14,898 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:52:16,092 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:52:16,094 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:52:16,094 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:52:17,181 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:52:22,182 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:52:27,183 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:52:31,103 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:52:31,104 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:52:31,106 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:52:32,271 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:52:37,272 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:52:42,273 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:52:44,899 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:52:46,100 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:52:46,102 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:52:46,103 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:52:48,232 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:52:53,233 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:52:58,233 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:53:01,100 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:53:01,102 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:53:01,103 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:53:04,187 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:53:09,188 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:53:14,189 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:53:14,899 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:53:16,100 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:53:16,102 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:53:16,103 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:53:19,263 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:53:24,264 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:53:26,227 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 11:53:26,228 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 11:53:26,228 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 11:53:26,229 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 11:53:27,223 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 11:53:30,230 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:53:31,100 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:53:31,103 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:53:31,103 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:53:36,203 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:53:41,204 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:53:44,900 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:53:46,100 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:53:46,103 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:53:46,103 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:53:47,186 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:53:52,187 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:53:57,187 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:54:01,101 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:54:01,103 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:54:01,103 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:54:02,194 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:54:07,195 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:54:08,574 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 11:54:08,575 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 11:54:08,576 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 11:54:08,576 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 11:54:09,239 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 11:54:12,577 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:54:14,901 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:54:16,100 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:54:16,103 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:54:16,103 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:54:18,178 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:54:23,179 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:54:28,180 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:54:31,101 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:54:31,103 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:54:31,104 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:54:33,191 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:54:38,192 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:54:43,193 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:54:44,902 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:54:46,101 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:54:46,103 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:54:46,104 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:54:48,263 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:54:53,264 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:54:58,265 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:55:01,101 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:55:01,103 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:55:01,104 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:55:03,280 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:55:03,909 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 11:55:03,910 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 11:55:03,910 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 11:55:03,911 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 11:55:04,259 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 11:55:08,912 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:55:13,913 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:55:14,903 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:55:16,101 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:55:16,103 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:55:16,104 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:55:19,165 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:55:24,166 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:55:29,167 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:55:31,101 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:55:31,104 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:55:31,104 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:55:34,254 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:55:39,255 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:55:44,256 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:55:44,904 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:55:46,101 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:55:46,104 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:55:46,104 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:55:48,523 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 11:55:48,525 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 11:55:48,525 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 11:55:48,526 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 11:55:49,277 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 11:55:49,526 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:55:54,527 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:55:59,528 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:56:01,101 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:56:01,104 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:56:01,107 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:56:05,275 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:56:10,276 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:56:14,905 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:56:15,906 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:56:16,102 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:56:16,104 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:56:16,104 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:56:21,277 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:56:26,277 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:56:31,102 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:56:31,105 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:56:31,105 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:56:32,239 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:56:37,240 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:56:42,241 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:56:44,906 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:56:46,102 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:56:46,104 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:56:46,105 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:56:48,233 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:56:51,919 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 11:56:51,920 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 11:56:51,920 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 11:56:51,921 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 11:56:52,301 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 11:56:53,922 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:56:58,923 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:57:01,102 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:57:01,105 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:57:01,105 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:57:04,217 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:57:09,218 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:57:14,219 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:57:14,907 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:57:16,102 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:57:16,105 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:57:16,105 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:57:19,244 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:57:24,245 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:57:29,245 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:57:31,102 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:57:31,105 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:57:31,105 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:57:34,203 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 11:57:34,204 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 11:57:34,204 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 11:57:34,205 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 11:57:34,317 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 11:57:35,206 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:57:40,206 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:57:44,908 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:57:45,909 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:57:46,102 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:57:46,105 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:57:46,105 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:57:51,218 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:57:56,219 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:58:01,108 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:58:01,109 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:58:01,111 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:58:02,199 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:58:07,200 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:58:12,201 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:58:14,909 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:58:16,108 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:58:16,109 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:58:16,149 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:58:17,226 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:58:22,227 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:58:27,228 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:58:31,109 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:58:31,109 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:58:31,110 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:58:33,182 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:58:35,612 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 11:58:35,613 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 11:58:35,614 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 11:58:35,614 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 11:58:36,340 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 11:58:38,615 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:58:43,616 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:58:44,909 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:58:46,109 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:58:46,109 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:58:46,110 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:58:49,207 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:58:54,207 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:58:59,208 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:59:01,109 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:59:01,109 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:59:01,149 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:59:04,253 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:59:09,253 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:59:14,255 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:59:14,910 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:59:16,108 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:59:16,109 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:59:16,149 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:59:20,203 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:59:23,240 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 11:59:23,241 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 11:59:23,241 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 11:59:23,242 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 11:59:23,357 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 11:59:25,243 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:59:30,243 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:59:31,108 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:59:31,109 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:59:31,110 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:59:35,278 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:59:40,279 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:59:44,911 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 11:59:45,912 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:59:46,109 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 11:59:46,109 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 11:59:46,110 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 11:59:51,207 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 11:59:56,207 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:00:01,109 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:00:01,109 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:00:01,110 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:00:02,184 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:00:06,061 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:00:06,062 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:00:06,062 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:00:06,063 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:00:06,373 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:00:08,063 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:00:13,064 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:00:14,912 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:00:16,109 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:00:16,109 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:00:16,110 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:00:18,187 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:00:23,188 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:00:28,189 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:00:31,109 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:00:31,110 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:00:31,110 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:00:33,265 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:00:38,266 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:00:43,267 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:00:44,913 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:00:46,110 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:00:46,110 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:00:46,110 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:00:49,252 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:00:54,253 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:00:54,683 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:00:54,684 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:00:54,684 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:00:54,684 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:00:55,391 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:00:59,686 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:01:01,110 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:01:01,110 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:01:01,110 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:01:05,636 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:01:10,637 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:01:14,914 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:01:15,915 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:01:16,110 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:01:16,110 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:01:16,110 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:01:21,248 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:01:26,249 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:01:31,110 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:01:31,110 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:01:31,110 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:01:32,232 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:01:37,233 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:01:42,234 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:01:44,915 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:01:46,110 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:01:46,110 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:01:46,111 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:01:47,278 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:01:47,824 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:01:47,825 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:01:47,826 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:01:47,826 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:01:48,411 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:01:52,827 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:01:57,828 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:02:01,110 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:02:01,110 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:02:01,111 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:02:03,214 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:02:08,215 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:02:13,216 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:02:14,916 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:02:16,111 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:02:16,111 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:02:16,112 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:02:18,257 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:02:23,258 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:02:28,259 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:02:31,111 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:02:31,111 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:02:31,111 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:02:33,263 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:02:35,923 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:02:35,925 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:02:35,925 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:02:35,926 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:02:36,430 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:02:38,927 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:02:43,928 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:02:44,917 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:02:46,110 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:02:46,111 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:02:46,112 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:02:49,197 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:02:54,198 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:02:59,199 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:03:01,111 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:03:01,111 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:03:01,112 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:03:04,243 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:03:09,244 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:03:14,245 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:03:14,918 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:03:16,110 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:03:16,111 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:03:16,151 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:03:19,261 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:03:24,262 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:03:27,064 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:03:27,065 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:03:27,066 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:03:27,066 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:03:27,450 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:03:30,067 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:03:31,111 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:03:31,111 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:03:31,112 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:03:35,205 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:03:40,206 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:03:44,919 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:03:45,920 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:03:46,111 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:03:46,111 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:03:46,152 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:03:51,248 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:03:56,249 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:04:01,111 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:04:01,112 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:04:01,112 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:04:01,266 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:04:06,267 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:04:11,268 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:04:12,325 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:04:12,326 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:04:12,327 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:04:12,327 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:04:12,468 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:04:14,920 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:04:16,111 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:04:16,111 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:04:16,112 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:04:17,185 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:04:22,186 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:04:27,187 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:04:31,112 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:04:31,112 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:04:31,112 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:04:32,214 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:04:37,215 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:04:42,216 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:04:44,920 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:04:46,111 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:04:46,112 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:04:46,152 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:04:47,249 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:04:52,250 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:04:55,580 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:04:55,581 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:04:55,581 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:04:55,582 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:04:56,486 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:04:57,582 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:05:01,111 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:05:01,112 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:05:01,112 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:05:03,224 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:05:08,225 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:05:13,227 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:05:14,921 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:05:16,112 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:05:16,113 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:05:16,113 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:05:19,202 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:05:24,203 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:05:29,204 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:05:31,112 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:05:31,113 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:05:31,113 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:05:35,199 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:05:40,200 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:05:44,922 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:05:45,923 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:05:46,113 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:05:46,113 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:05:46,113 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:05:51,200 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:05:56,201 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:06:01,117 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:06:01,118 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:06:01,119 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:06:01,119 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:06:01,158 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:06:01,158 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:06:01,159 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:06:01,263 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:06:01,512 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:06:06,265 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:06:11,265 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:06:14,923 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:06:16,114 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:06:16,115 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:06:16,115 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:06:17,208 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:06:22,209 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:06:27,210 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:06:31,115 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:06:31,115 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:06:31,115 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:06:33,207 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:06:38,208 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:06:43,209 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:06:44,924 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:06:46,115 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:06:46,115 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:06:46,116 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:06:48,225 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:06:53,226 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:06:54,663 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:06:54,664 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:06:54,664 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:06:54,665 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:06:55,533 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:06:58,666 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:07:01,114 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:07:01,115 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:07:01,118 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:07:04,205 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:07:09,206 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:07:14,207 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:07:14,925 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:07:16,115 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:07:16,116 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:07:16,116 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:07:19,272 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:07:24,273 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:07:29,274 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:07:31,115 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:07:31,115 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:07:31,155 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:07:35,220 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:07:40,221 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:07:44,926 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:07:45,926 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:07:46,115 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:07:46,116 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:07:46,116 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:07:51,270 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:07:56,271 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:08:01,116 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:08:01,116 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:08:01,116 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:08:02,260 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:08:07,261 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:08:12,029 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:08:12,031 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:08:12,031 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:08:12,031 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:08:12,564 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:08:13,032 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:08:14,927 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:08:16,115 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:08:16,116 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:08:16,116 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:08:18,248 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:08:23,249 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:08:28,250 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:08:31,115 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:08:31,116 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:08:31,156 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:08:33,263 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:08:38,264 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:08:43,265 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:08:44,927 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:08:46,116 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:08:46,117 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:08:46,117 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:08:47,353 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:08:47,354 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:08:47,355 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:08:47,355 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:08:47,578 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:08:48,356 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:08:53,357 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:08:58,358 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:09:01,115 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:09:01,116 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:09:01,117 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:09:04,291 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:09:09,292 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:09:14,293 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:09:14,928 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:09:16,116 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:09:16,116 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:09:16,156 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:09:18,353 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:09:18,355 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:09:18,355 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:09:18,356 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:09:18,590 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:09:19,356 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:09:24,357 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:09:29,358 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:09:31,117 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:09:31,117 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:09:31,120 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:09:35,202 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:09:40,203 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:09:44,929 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:09:45,930 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:09:46,116 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:09:46,116 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:09:46,157 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:09:51,252 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:09:52,835 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:09:52,837 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:09:52,837 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:09:52,837 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:09:53,605 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:09:56,838 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:10:01,116 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:10:01,117 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:10:01,117 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:10:02,277 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:10:07,278 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:10:12,279 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:10:14,930 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:10:16,116 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:10:16,117 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:10:16,157 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:10:18,209 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:10:23,211 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:10:28,212 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:10:30,478 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:10:30,479 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:10:30,479 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:10:30,480 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:10:30,622 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:10:31,116 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:10:31,117 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:10:31,118 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:10:34,188 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:10:39,189 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:10:44,190 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:10:44,931 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:10:46,118 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:10:46,118 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:10:46,118 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:10:49,192 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:10:54,193 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:10:59,193 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:11:01,117 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:11:01,117 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:11:01,158 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:11:04,239 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:11:05,479 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:11:05,481 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:11:05,481 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:11:05,481 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:11:05,636 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:11:09,482 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:11:14,483 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:11:14,936 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:11:16,117 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:11:16,118 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:11:16,118 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:11:20,223 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:11:25,224 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:11:30,225 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:11:31,118 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:11:31,118 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:11:31,118 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:11:36,219 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:11:39,007 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:11:39,008 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:11:39,008 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:11:39,009 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:11:39,649 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:11:42,010 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:11:44,933 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:11:46,118 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:11:46,118 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:11:46,118 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:11:47,290 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:11:52,291 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:11:57,292 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:12:01,117 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:12:01,118 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:12:01,158 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:12:03,275 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:12:08,276 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:12:13,254 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:12:13,256 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:12:13,256 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:12:13,258 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:12:13,663 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:12:14,258 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:12:14,934 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:12:16,117 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:12:16,118 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:12:16,120 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:12:20,238 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:12:25,239 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:12:30,240 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:12:31,118 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:12:31,119 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:12:31,119 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:12:35,267 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:12:40,268 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:12:44,936 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:12:45,937 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:12:46,119 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:12:46,120 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:12:46,120 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:12:46,920 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:12:46,921 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:12:46,922 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:12:46,922 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:12:47,676 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:12:51,923 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:12:56,924 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:13:01,118 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:13:01,118 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:13:01,119 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:13:02,249 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:13:07,250 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:13:12,251 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:13:14,935 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:13:16,118 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:13:16,119 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:13:16,159 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:13:17,266 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:13:22,267 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:13:27,268 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:13:31,119 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:13:31,119 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:13:31,120 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:13:31,195 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:13:31,197 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:13:31,197 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:13:31,198 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:13:31,694 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:13:33,199 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:13:38,200 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:13:43,200 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:13:44,936 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:13:46,119 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:13:46,119 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:13:46,120 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:13:48,253 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:13:53,255 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:13:58,256 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:14:01,120 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:14:01,120 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:14:01,120 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:14:03,297 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:14:03,362 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:14:03,363 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:14:03,364 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:14:03,364 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:14:03,707 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:14:08,365 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:14:13,366 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:14:14,937 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:14:16,118 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:14:16,119 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:14:16,120 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:14:19,226 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:14:24,227 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:14:29,228 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:14:31,118 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:14:31,119 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:14:31,159 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:14:34,246 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:14:35,588 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:14:35,589 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:14:35,590 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:14:35,590 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:14:35,719 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:14:39,591 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:14:44,592 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:14:44,938 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:14:46,119 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:14:46,120 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:14:46,120 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:14:50,210 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:14:55,211 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:15:00,212 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:15:01,131 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:15:01,132 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:15:01,172 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:15:05,216 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:15:10,217 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:15:11,041 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:15:11,042 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:15:11,043 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:15:11,044 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:15:11,733 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:15:14,939 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:15:15,940 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:15:16,124 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:15:16,130 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:15:16,130 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:15:21,219 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:15:26,220 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:15:31,125 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:15:31,130 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:15:31,130 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:15:32,199 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:15:37,200 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:15:42,201 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:15:44,940 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:15:46,125 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:15:46,130 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:15:46,131 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:15:47,275 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:15:51,378 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:15:51,379 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:15:51,379 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:15:51,380 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:15:51,748 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:15:52,381 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:15:57,382 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:16:01,126 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:16:01,130 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:16:01,131 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:16:03,233 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:16:08,234 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:16:13,235 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:16:14,940 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:16:16,125 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:16:16,130 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:16:16,131 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:16:18,312 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:16:23,313 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:16:28,314 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:16:31,127 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:16:31,130 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:16:31,131 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:16:31,171 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:16:31,234 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:16:31,234 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:16:31,235 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:16:31,763 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:16:34,236 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:16:39,237 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:16:44,237 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:16:44,941 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:16:46,127 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:16:46,131 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:16:46,131 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:16:49,270 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:16:54,271 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:16:59,272 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:17:01,127 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:17:01,131 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:17:01,131 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:17:02,014 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:17:02,016 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:17:02,016 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:17:02,016 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:17:02,775 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:17:05,017 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:17:10,018 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:17:14,942 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:17:15,943 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:17:16,127 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:17:16,131 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:17:16,131 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:17:21,239 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:17:26,240 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:17:31,127 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:17:31,131 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:17:31,132 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:17:32,221 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:17:33,437 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:17:33,438 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:17:33,438 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:17:33,439 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:17:33,787 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:17:37,440 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:17:42,441 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:17:44,943 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:17:46,127 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:17:46,131 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:17:46,131 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:17:48,196 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:17:53,197 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:17:58,198 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:18:01,127 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:18:01,131 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:18:01,132 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:18:03,235 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:18:07,451 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:18:07,453 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:18:07,453 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:18:07,454 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:18:07,799 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:18:08,454 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:18:13,455 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:18:14,944 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:18:16,127 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:18:16,131 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:18:16,132 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:18:19,238 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:18:24,239 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:18:29,240 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:18:31,128 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:18:31,131 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:18:31,132 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:18:34,256 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:18:39,257 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:18:42,378 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:18:42,379 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:18:42,379 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:18:42,379 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:18:42,812 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:18:44,380 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:18:44,945 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:18:46,127 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:18:46,131 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:18:46,132 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:18:50,242 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:18:55,243 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:19:00,244 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:19:01,128 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:19:01,132 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:19:01,132 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:19:05,277 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:19:10,278 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:19:14,945 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:19:15,946 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:19:16,128 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:19:16,132 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:19:16,132 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:19:16,315 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:19:16,316 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:19:16,316 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:19:16,317 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:19:16,825 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:19:21,318 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:19:26,319 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:19:31,128 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:19:31,132 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:19:31,132 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:19:32,245 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:19:37,247 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:19:42,248 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:19:44,946 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:19:46,128 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:19:46,132 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:19:46,132 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:19:47,249 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:19:52,250 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:19:53,613 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:19:53,614 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:19:53,614 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:19:53,614 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:19:53,839 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:19:57,615 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:20:01,128 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:20:01,132 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:20:01,133 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:20:03,279 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:20:08,281 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:20:13,281 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:20:14,947 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:20:16,128 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:20:16,132 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:20:16,133 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:20:19,243 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:20:24,243 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:20:25,853 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:20:25,854 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:20:25,854 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:20:25,855 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:20:26,852 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:20:29,856 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:20:31,128 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:20:31,132 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:20:31,133 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:20:35,271 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:20:40,273 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:20:44,948 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:20:45,949 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:20:46,129 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:20:46,132 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:20:46,133 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:20:51,228 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:20:56,229 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:21:00,815 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:21:00,816 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:21:00,817 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:21:00,817 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:21:00,864 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:21:01,129 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:21:01,133 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:21:01,133 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:21:01,285 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:21:06,286 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:21:11,288 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:21:14,949 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:21:16,129 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:21:16,133 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:21:16,133 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:21:17,258 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:21:22,259 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:21:27,260 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:21:31,129 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:21:31,133 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:21:31,133 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:21:33,259 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:21:35,849 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:21:35,850 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:21:35,850 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:21:35,851 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:21:35,877 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:21:38,852 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:21:43,853 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:21:44,950 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:21:46,129 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:21:46,133 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:21:46,133 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:21:49,310 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:21:54,311 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:21:59,312 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:22:01,129 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:22:01,133 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:22:01,133 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:22:05,300 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:22:10,300 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:22:12,197 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:22:12,198 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:22:12,198 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:22:12,199 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:22:12,891 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:22:14,951 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:22:15,951 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:22:16,130 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:22:16,133 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:22:16,134 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:22:21,288 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:22:26,289 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:22:31,130 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:22:31,133 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:22:31,134 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:22:32,280 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:22:37,281 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:22:42,282 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:22:44,416 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:22:44,417 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:22:44,418 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:22:44,418 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:22:44,904 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:22:44,951 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:22:46,130 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:22:46,133 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:22:46,134 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:22:48,249 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:22:53,251 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:22:58,251 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:23:01,130 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:23:01,133 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:23:01,134 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:23:04,251 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:23:09,252 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:23:14,253 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:23:14,952 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:23:16,130 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:23:16,133 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:23:16,134 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:23:17,969 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:23:17,970 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:23:17,970 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:23:17,971 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:23:18,917 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:23:19,972 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:23:24,973 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:23:29,974 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:23:31,130 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:23:31,134 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:23:31,134 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:23:35,260 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:23:40,261 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:23:44,953 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:23:45,954 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:23:46,131 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:23:46,134 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:23:46,134 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:23:50,184 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:23:50,186 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:23:50,186 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:23:50,187 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:23:50,929 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:23:51,187 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:23:56,188 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:24:01,131 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:24:01,134 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:24:01,135 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:24:01,287 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:24:06,289 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:24:11,290 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:24:14,954 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:24:16,131 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:24:16,134 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:24:16,135 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:24:17,254 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:24:22,255 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:24:22,884 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:24:22,885 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:24:22,885 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:24:22,886 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:24:22,941 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:24:27,887 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:24:31,131 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:24:31,134 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:24:31,135 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:24:33,268 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:24:38,269 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:24:43,270 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:24:44,955 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:24:46,132 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:24:46,134 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:24:46,135 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:24:48,318 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:24:53,319 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:24:55,600 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:24:55,601 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:24:55,601 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:24:55,601 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:24:55,954 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:24:58,602 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:25:01,131 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:25:01,134 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:25:01,135 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:25:04,280 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:25:09,282 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:25:14,283 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:25:14,956 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:25:16,131 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:25:16,134 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:25:16,135 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:25:19,291 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:25:24,292 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:25:29,293 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:25:31,132 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:25:31,135 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:25:31,135 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:25:32,518 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:25:32,519 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:25:32,520 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:25:32,520 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:25:32,969 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:25:34,521 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:25:39,522 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:25:44,523 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:25:44,956 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:25:46,132 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:25:46,135 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:25:46,135 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:25:50,311 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:25:55,312 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:26:00,313 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:26:01,132 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:26:01,135 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:26:01,135 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:26:06,239 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:26:11,241 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:26:12,581 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:26:12,582 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:26:12,582 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:26:12,583 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:26:12,984 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:26:14,957 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:26:16,133 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:26:16,135 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:26:16,135 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:26:16,271 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:26:21,272 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:26:26,273 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:26:31,132 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:26:31,135 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:26:31,135 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:26:32,249 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:26:37,249 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:26:42,250 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:26:44,958 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:26:46,132 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:26:46,135 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:26:46,135 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:26:47,258 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:26:48,764 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:26:48,765 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:26:48,765 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:26:48,766 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:26:48,998 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:26:52,767 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:26:57,768 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:27:01,133 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:27:01,135 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:27:01,135 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:27:03,237 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:27:08,238 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:27:13,239 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:27:14,959 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:27:16,134 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:27:16,136 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:27:16,137 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:27:18,247 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:27:22,391 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:27:22,392 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:27:22,392 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:27:22,393 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:27:23,012 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:27:23,394 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:27:28,394 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:27:31,133 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:27:31,135 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:27:31,136 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:27:34,281 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:27:39,282 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:27:44,283 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:27:44,960 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:27:46,134 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:27:46,136 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:27:46,136 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:27:50,247 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:27:55,201 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:27:55,202 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:27:55,203 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:27:55,203 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:27:56,025 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:27:56,204 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:28:01,133 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:28:01,136 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:28:01,136 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:28:01,224 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:28:06,225 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:28:11,226 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:28:14,961 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:28:16,134 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:28:16,136 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:28:16,136 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:28:16,299 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:28:21,300 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:28:26,301 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:28:31,134 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:28:31,136 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:28:31,136 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:28:32,207 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:28:37,208 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:28:42,209 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:28:44,962 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:28:46,134 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:28:46,136 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:28:46,136 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:28:47,245 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:28:52,246 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:28:57,247 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:28:58,440 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:28:58,441 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:28:58,441 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:28:58,442 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:28:59,051 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:29:01,134 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:29:01,136 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:29:01,136 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:29:02,250 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:29:07,251 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:29:12,252 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:29:14,963 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:29:16,134 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:29:16,136 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:29:16,137 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:29:17,293 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:29:22,294 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:29:27,295 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:29:30,068 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:29:30,069 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:29:30,070 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:29:30,070 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:29:31,065 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:29:31,134 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:29:31,136 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:29:31,137 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:29:33,273 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:29:38,274 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:29:43,275 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:29:44,964 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:29:46,136 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:29:46,137 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:29:46,137 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:29:49,260 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:29:54,261 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:29:59,262 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:30:01,135 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:30:01,137 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:30:01,137 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:30:02,704 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:30:02,707 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:30:02,707 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:30:02,708 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:30:03,078 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:30:04,709 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:30:09,710 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:30:14,711 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:30:14,970 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:30:16,135 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:30:16,137 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:30:16,137 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:30:20,197 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:30:25,198 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:30:30,199 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:30:31,135 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:30:31,137 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:30:31,137 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:30:35,224 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:30:39,170 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:30:39,171 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:30:39,172 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:30:39,172 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:30:40,092 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:30:41,173 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:30:44,965 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:30:46,135 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:30:46,137 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:30:46,137 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:30:46,306 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:30:51,307 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:30:56,308 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:31:01,135 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:31:01,137 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:31:01,137 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:31:02,234 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:31:07,235 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:31:11,778 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:31:11,779 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:31:11,780 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:31:11,782 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:31:12,104 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:31:12,782 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:31:14,966 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:31:16,136 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:31:16,137 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:31:16,137 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:31:18,258 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:31:23,259 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:31:28,260 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:31:31,136 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:31:31,137 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:31:31,138 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:31:34,247 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:31:39,248 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:31:44,249 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:31:44,969 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:31:46,136 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:31:46,137 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:31:46,138 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:31:47,634 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:31:47,636 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:31:47,636 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:31:47,636 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:31:48,118 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:31:49,637 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:31:54,638 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:31:59,639 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:32:01,136 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:32:01,137 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:32:01,138 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:32:05,254 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:32:10,255 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:32:14,968 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:32:15,968 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:32:16,136 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:32:16,138 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:32:16,138 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:32:21,252 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:32:22,213 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:32:22,214 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:32:22,214 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:32:22,215 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:32:23,132 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:32:27,216 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:32:31,136 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:32:31,138 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:32:31,138 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:32:32,226 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:32:37,227 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:32:42,228 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:32:44,969 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:32:46,137 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:32:46,138 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:32:46,138 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:32:47,256 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:32:52,257 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:32:57,258 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:33:00,883 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:33:00,885 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:33:00,885 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:33:00,886 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:33:01,137 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:33:01,138 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:33:01,139 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:33:01,147 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:33:02,285 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:33:07,286 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:33:12,288 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:33:14,969 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:33:16,137 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:33:16,138 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:33:16,138 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:33:18,263 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:33:23,264 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:33:28,265 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:33:31,137 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:33:31,138 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:33:31,138 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:33:33,316 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:33:33,793 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:33:33,794 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:33:33,794 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:33:33,795 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:33:34,159 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:33:38,796 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:33:43,797 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:33:44,971 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:33:46,138 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:33:46,139 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:33:46,139 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:33:49,238 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:33:54,239 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:33:59,240 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:34:01,137 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:34:01,139 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:34:01,139 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:34:04,265 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:34:09,192 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:34:09,193 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:34:09,193 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:34:09,194 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:34:10,173 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:34:10,194 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:34:14,971 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:34:15,972 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:34:16,137 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:34:16,139 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:34:16,139 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:34:21,263 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:34:26,264 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:34:31,138 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:34:31,139 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:34:31,139 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:34:32,042 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:34:37,043 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:34:42,044 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:34:44,972 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:34:45,324 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:34:45,325 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:34:45,326 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:34:45,326 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:34:46,138 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:34:46,139 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:34:46,140 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:34:46,186 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:34:47,213 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:34:52,214 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:34:57,215 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:35:01,138 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:35:01,139 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:35:01,140 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:35:02,220 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:35:07,221 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:35:12,222 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:35:14,973 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:35:16,138 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:35:16,139 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:35:16,140 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:35:17,257 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:35:19,301 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:35:19,302 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:35:19,303 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:35:19,303 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:35:20,200 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:35:22,304 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:35:27,305 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:35:31,138 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:35:31,140 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:35:31,140 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:35:33,231 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:35:38,232 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:35:43,233 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:35:44,974 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:35:46,138 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:35:46,140 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:35:46,140 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:35:48,291 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:35:53,292 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:35:54,509 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:35:54,510 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:35:54,510 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:35:54,511 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:35:55,214 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:35:58,512 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:36:01,139 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:36:01,140 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:36:01,140 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:36:03,514 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:36:08,515 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:36:13,516 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:36:14,975 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:36:16,139 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:36:16,140 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:36:16,141 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:36:19,226 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:36:24,226 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:36:26,671 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:36:26,672 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:36:26,673 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:36:26,673 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:36:27,227 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:36:29,674 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:36:31,139 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:36:31,140 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:36:31,141 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:36:35,302 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:36:40,303 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:36:44,976 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:36:45,977 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:36:46,139 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:36:46,141 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:36:46,141 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:36:51,218 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:36:56,219 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:37:01,139 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:37:01,141 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:37:01,141 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:37:01,182 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:37:01,292 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:37:01,292 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:37:01,293 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:37:01,293 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:37:02,241 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:37:06,294 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:37:11,295 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:37:14,977 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:37:16,139 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:37:16,141 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:37:16,141 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:37:17,257 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:37:22,258 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:37:27,259 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:37:31,140 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:37:31,141 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:37:31,141 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:37:32,271 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:37:36,524 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:37:36,525 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:37:36,525 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:37:36,526 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:37:37,255 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:37:37,527 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:37:42,528 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:37:44,978 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:37:46,140 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:37:46,141 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:37:46,142 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:37:48,303 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:37:53,303 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:37:58,304 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:38:01,140 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:38:01,141 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:38:01,142 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:38:03,319 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:38:08,320 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:38:13,320 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:38:14,979 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:38:16,045 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:38:16,046 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:38:16,047 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:38:16,047 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:38:16,140 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:38:16,141 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:38:16,142 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:38:16,270 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:38:19,254 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:38:24,255 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:38:29,256 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:38:31,143 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:38:31,143 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:38:31,144 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:38:34,317 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:38:39,318 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:38:44,319 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:38:44,979 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:38:46,141 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:38:46,142 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:38:46,142 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:38:50,223 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:38:54,118 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:38:54,119 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:38:54,120 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:38:54,120 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:38:54,284 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:38:56,121 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:39:01,122 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:39:01,141 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:39:01,142 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:39:01,142 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:39:06,268 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:39:11,269 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:39:14,980 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:39:16,141 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:39:16,142 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:39:16,143 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:39:16,307 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:39:21,308 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:39:26,309 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:39:31,141 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:39:31,142 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:39:31,143 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:39:32,296 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:39:32,579 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:39:32,580 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:39:32,580 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:39:32,581 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:39:33,299 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:39:37,582 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:39:42,583 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:39:44,981 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:39:46,141 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:39:46,142 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:39:46,143 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:39:48,305 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:39:53,306 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:39:58,307 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:40:01,141 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:40:01,143 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:40:01,143 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:40:04,251 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:40:08,033 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:40:08,035 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:40:08,035 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:40:08,035 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:40:08,312 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:40:10,036 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:40:14,982 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:40:15,983 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:40:16,142 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:40:16,143 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:40:16,144 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:40:21,316 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:40:26,317 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:40:31,142 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:40:31,143 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:40:31,143 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:40:32,280 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:40:37,280 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:40:42,281 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:40:44,194 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:40:44,196 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:40:44,196 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:40:44,196 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:40:44,325 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:40:44,983 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:40:46,142 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:40:46,143 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:40:46,144 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:40:47,306 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:40:52,307 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:40:57,308 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:41:01,142 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:41:01,143 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:41:01,144 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:41:03,290 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:41:08,291 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:41:13,292 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:41:14,984 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:41:16,143 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:41:16,151 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:41:16,151 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:41:19,238 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:41:22,339 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:41:22,340 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:41:22,341 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:41:22,341 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:41:23,339 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:41:24,342 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:41:29,342 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:41:31,143 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:41:31,150 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:41:31,150 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:41:35,264 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:41:40,265 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:41:44,984 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:41:45,985 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:41:46,143 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:41:46,150 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:41:46,151 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:41:51,317 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:41:55,020 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:41:55,022 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:41:55,022 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:41:55,022 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:41:55,351 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:41:57,023 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:42:01,143 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:42:01,150 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:42:01,151 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:42:02,233 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:42:07,234 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:42:12,235 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:42:14,985 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:42:16,143 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:42:16,150 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:42:16,151 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:42:17,305 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:42:22,306 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:42:27,307 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:42:30,262 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:42:30,264 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:42:30,264 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:42:30,265 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:42:30,364 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:42:31,143 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:42:31,150 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:42:31,151 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:42:33,271 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:42:38,272 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:42:43,273 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:42:44,986 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:42:46,143 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:42:46,151 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:42:46,151 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:42:49,266 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:42:54,267 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:42:59,268 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:43:01,144 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:43:01,151 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:43:01,151 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:43:03,674 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:43:03,675 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:43:03,675 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:43:03,676 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:43:04,377 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:43:04,676 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:43:09,677 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:43:14,678 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:43:14,987 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:43:16,144 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:43:16,151 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:43:16,151 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:43:20,267 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:43:25,268 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:43:30,269 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:43:31,144 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:43:31,151 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:43:31,151 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:43:35,293 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:43:39,616 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:43:39,618 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:43:39,618 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:43:39,618 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:43:40,392 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:43:40,619 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:43:44,988 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:43:45,989 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:43:46,144 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:43:46,151 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:43:46,152 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:43:51,330 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:43:56,331 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:44:01,144 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:44:01,151 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:44:01,152 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:44:02,220 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:44:07,221 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:44:12,222 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:44:14,989 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:44:16,144 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:44:16,151 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:44:16,152 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:44:17,820 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:44:18,915 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:44:18,916 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:44:18,917 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:44:18,917 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:44:19,407 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:44:22,918 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:44:27,919 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:44:31,144 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:44:31,151 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:44:31,152 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:44:33,319 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:44:38,320 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:44:43,321 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:44:44,989 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:44:46,145 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:44:46,152 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:44:46,152 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:44:49,230 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:44:50,942 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:44:50,943 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:44:50,944 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:44:50,944 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:44:51,419 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:44:54,945 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:44:59,947 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:45:01,144 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:45:01,152 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:45:01,152 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:45:05,761 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:45:10,762 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:45:14,990 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:45:15,991 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:45:16,145 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:45:16,152 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:45:16,152 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:45:21,263 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:45:26,264 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:45:31,145 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:45:31,152 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:45:31,152 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:45:31,303 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:45:34,058 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:45:34,059 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:45:34,059 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:45:34,060 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:45:34,435 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:45:37,061 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:45:42,062 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:45:44,991 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:45:46,145 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:45:46,152 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:45:46,152 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:45:47,248 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:45:52,250 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:45:57,250 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:46:01,145 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:46:01,152 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:46:01,153 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:46:02,322 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:46:06,148 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:46:06,149 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:46:06,150 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:46:06,150 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:46:06,448 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:46:08,151 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:46:13,152 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:46:14,992 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:46:16,146 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:46:16,152 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:46:16,153 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:46:18,277 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:46:23,278 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:46:28,279 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:46:31,146 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:46:31,152 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:46:31,153 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:46:33,310 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:46:38,311 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:46:39,485 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:46:39,487 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:46:39,487 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:46:39,487 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:46:40,460 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:46:43,488 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:46:44,993 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:46:46,146 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:46:46,153 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:46:46,153 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:46:49,236 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:46:54,237 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:46:59,238 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:47:01,145 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:47:01,153 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:47:01,153 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:47:04,281 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:47:09,282 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:47:11,022 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:47:11,023 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:47:11,024 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:47:11,024 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:47:11,472 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:47:14,994 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:47:14,996 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:47:16,146 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:47:16,153 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:47:16,153 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:47:20,260 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:47:25,261 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:47:30,262 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:47:31,146 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:47:31,153 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:47:31,153 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:47:35,287 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:47:40,288 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:47:44,850 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:47:44,851 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:47:44,852 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:47:44,852 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:47:44,995 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:47:45,485 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:47:45,995 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:47:46,146 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:47:46,153 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:47:46,154 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:47:51,299 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:47:56,301 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:48:01,146 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:48:01,153 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:48:01,154 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:48:02,332 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:48:07,334 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:48:12,335 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:48:14,996 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:48:16,146 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:48:16,158 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:48:16,158 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:48:18,179 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:48:18,180 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:48:18,181 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:48:18,181 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:48:18,182 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:48:18,497 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:48:23,183 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:48:28,183 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:48:31,146 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:48:31,158 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:48:31,158 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:48:33,296 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:48:38,297 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:48:43,298 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:48:44,996 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:48:46,147 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:48:46,158 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:48:46,158 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:48:48,315 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:48:51,742 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:48:51,744 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:48:51,744 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:48:51,744 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:48:52,511 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:48:53,745 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:48:58,746 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:49:01,147 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:49:01,158 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:49:01,158 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:49:04,302 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:49:09,303 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:49:14,304 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:49:15,002 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:49:16,147 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:49:16,158 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:49:16,159 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:49:20,244 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:49:24,547 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:49:24,549 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:49:24,549 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:49:24,550 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:49:25,524 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:49:25,550 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:49:30,551 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:49:31,148 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:49:31,158 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:49:31,158 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:49:36,275 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:49:41,277 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:49:44,998 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:49:46,147 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:49:46,158 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:49:46,159 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:49:47,269 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:49:52,269 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:49:56,772 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:49:56,773 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:49:56,774 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:49:56,776 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:49:57,535 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:49:57,777 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:50:01,147 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:50:01,158 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:50:01,159 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:50:03,315 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:50:08,316 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:50:13,317 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:50:14,999 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:50:16,148 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:50:16,158 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:50:16,159 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:50:19,260 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:50:24,261 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:50:29,261 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:50:30,881 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:50:30,882 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:50:30,883 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:50:30,885 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:50:31,148 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:50:31,159 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:50:31,159 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:50:31,547 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:50:34,313 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:50:39,314 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:50:44,315 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:50:45,000 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:50:46,148 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:50:46,159 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:50:46,159 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:50:50,240 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:50:55,241 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:51:00,242 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:51:01,148 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:51:01,159 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:51:01,159 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:51:03,682 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:51:03,683 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:51:03,683 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:51:03,686 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:51:04,560 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:51:05,687 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:51:10,687 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:51:15,001 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:51:16,001 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:51:16,148 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:51:16,159 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:51:16,159 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:51:21,330 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:51:26,331 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:51:31,148 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:51:31,159 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:51:31,160 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:51:32,322 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:51:35,261 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:51:35,262 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:51:35,263 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:51:35,265 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:51:35,571 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:51:38,265 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:51:43,267 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:51:45,002 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:51:46,148 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:51:46,159 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:51:46,160 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:51:49,260 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:51:54,261 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:51:59,262 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:52:01,149 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:52:01,159 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:52:01,160 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:52:04,264 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:52:09,265 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:52:09,628 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:52:09,630 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:52:09,630 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:52:09,632 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:52:10,585 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:52:14,633 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:52:15,002 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:52:16,149 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:52:16,160 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:52:16,160 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:52:20,320 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:52:25,321 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:52:30,322 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:52:31,149 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:52:31,160 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:52:31,160 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:52:36,265 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:52:41,266 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:52:45,004 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:52:46,149 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:52:46,160 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:52:46,160 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:52:47,224 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:52:52,226 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:52:57,098 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:52:57,100 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:52:57,100 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:52:57,100 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:52:57,602 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:52:58,101 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:53:01,149 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:53:01,160 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:53:01,160 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:53:03,340 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:53:08,341 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:53:13,342 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:53:15,004 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:53:16,149 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:53:16,163 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:53:16,163 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:53:19,313 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:53:24,314 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:53:29,315 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:53:31,150 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:53:31,151 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:53:31,152 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:53:31,153 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:53:31,153 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:53:31,163 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:53:31,163 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:53:31,616 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:53:35,300 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:53:40,301 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:53:45,005 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:53:46,006 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:53:46,150 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:53:46,163 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:53:46,163 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:53:51,272 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:53:56,273 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:54:01,150 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:54:01,163 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:54:01,163 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:54:02,245 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:54:03,757 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:54:03,758 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:54:03,758 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:54:03,759 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:54:04,629 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:54:07,760 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:54:12,760 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:54:15,005 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:54:16,150 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:54:16,163 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:54:16,164 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:54:18,241 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:54:23,242 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:54:28,243 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:54:31,150 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:54:31,163 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:54:31,164 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:54:33,307 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:54:37,417 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:54:37,418 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:54:37,418 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:54:37,419 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:54:37,642 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:54:38,419 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:54:43,420 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:54:45,006 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:54:46,151 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:54:46,164 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:54:46,164 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:54:49,328 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:54:54,329 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:54:59,330 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:55:01,151 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:55:01,164 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:55:01,164 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:55:05,273 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:55:10,274 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:55:14,272 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:55:14,273 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:55:14,273 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:55:14,274 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:55:14,656 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:55:15,007 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:55:16,008 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:55:16,151 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:55:16,164 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:55:16,164 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:55:21,316 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:55:26,317 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:55:31,151 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:55:31,164 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:55:31,164 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:55:32,303 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:55:37,304 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:55:42,305 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:55:45,008 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:55:45,831 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:55:45,833 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:55:45,833 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:55:45,833 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:55:46,151 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:55:46,164 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:55:46,165 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:55:46,668 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:55:47,306 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:55:52,307 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:55:57,308 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:56:01,151 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:56:01,164 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:56:01,164 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:56:02,317 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:56:07,317 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:56:12,318 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:56:15,009 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:56:16,152 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:56:16,165 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:56:16,165 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:56:17,321 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:56:20,699 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:56:20,700 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:56:20,701 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:56:20,701 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:56:21,682 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:56:22,702 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:56:27,703 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:56:31,152 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:56:31,164 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:56:31,165 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:56:33,308 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:56:38,309 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:56:43,310 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:56:45,010 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:56:46,152 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:56:46,165 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:56:46,165 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:56:48,323 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:56:53,324 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:56:53,964 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:56:53,965 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:56:53,965 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:56:53,966 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:56:54,694 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:56:58,967 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:57:01,152 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:57:01,165 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:57:01,166 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:57:04,306 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:57:09,307 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:57:14,308 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:57:15,010 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:57:16,152 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:57:16,165 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:57:16,165 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:57:19,310 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:57:24,311 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:57:25,566 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:57:25,567 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:57:25,568 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:57:25,568 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:57:25,706 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:57:29,569 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:57:31,152 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:57:31,165 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:57:31,165 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:57:35,293 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:57:40,294 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:57:45,011 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:57:46,012 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:57:46,152 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:57:46,165 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:57:46,165 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:57:51,306 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:57:56,307 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:58:01,152 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:58:01,166 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:58:01,166 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:58:01,309 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:58:02,783 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:58:02,785 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:58:02,785 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:58:02,785 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:58:03,721 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:58:06,787 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:58:11,788 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:58:15,012 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:58:16,153 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:58:16,165 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:58:16,166 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:58:17,319 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:58:22,320 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:58:27,321 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:58:31,153 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:58:31,165 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:58:31,166 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:58:33,308 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:58:38,309 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:58:39,225 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:58:39,226 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:58:39,227 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:58:39,227 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:58:39,734 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:58:44,228 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:58:45,013 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:58:46,153 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:58:46,166 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:58:46,166 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:58:50,224 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:58:55,225 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:59:00,226 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:59:01,153 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:59:01,166 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:59:01,166 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:59:05,295 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:59:10,296 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:59:11,344 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:59:11,345 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:59:11,346 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:59:11,346 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:59:11,746 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:59:15,014 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:59:16,015 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:59:16,164 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:59:16,166 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:59:16,166 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:59:21,302 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:59:26,303 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:59:31,164 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:59:31,166 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:59:31,166 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:59:31,318 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:59:36,319 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:59:41,320 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:59:45,014 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 12:59:46,163 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 12:59:46,166 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 12:59:46,166 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 12:59:47,316 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:59:49,071 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 12:59:49,072 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 12:59:49,073 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 12:59:49,073 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 12:59:49,761 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 12:59:53,074 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 12:59:58,075 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:00:01,164 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:00:01,166 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:00:01,166 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:00:03,296 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:00:08,296 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:00:13,297 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:00:15,015 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:00:16,164 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:00:16,166 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:00:16,166 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:00:18,305 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:00:22,786 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:00:22,787 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:00:22,788 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:00:22,788 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:00:23,774 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:00:23,789 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:00:28,789 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:00:31,164 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:00:31,166 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:00:31,166 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:00:34,341 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:00:39,342 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:00:44,343 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:00:45,016 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:00:46,164 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:00:46,166 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:00:46,167 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:00:50,266 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:00:55,267 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:00:59,067 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:00:59,068 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:00:59,068 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:00:59,069 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:00:59,788 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:01:01,069 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:01:01,164 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:01:01,166 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:01:01,167 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:01:06,276 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:01:11,277 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:01:15,017 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:01:16,164 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:01:16,167 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:01:16,167 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:01:17,251 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:01:22,252 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:01:27,253 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:01:31,164 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:01:31,167 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:01:31,167 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:01:32,381 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:01:32,382 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:01:32,383 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:01:32,383 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:01:32,383 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:01:32,801 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:01:37,385 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:01:42,385 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:01:45,018 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:01:46,164 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:01:46,167 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:01:46,167 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:01:48,233 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:01:53,234 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:01:58,235 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:02:01,165 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:02:01,167 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:02:01,167 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:02:03,320 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:02:05,225 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:02:05,226 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:02:05,226 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:02:05,227 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:02:05,813 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:02:09,228 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:02:14,229 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:02:15,018 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:02:16,165 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:02:16,167 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:02:16,167 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:02:19,327 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:02:24,328 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:02:29,329 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:02:31,165 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:02:31,167 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:02:31,167 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:02:35,293 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:02:36,082 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:02:36,083 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:02:36,083 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:02:36,084 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:02:36,824 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:02:41,085 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:02:45,019 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:02:46,165 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:02:46,166 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:02:46,167 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:02:46,168 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:02:51,303 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:02:56,304 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:03:01,165 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:03:01,167 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:03:01,168 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:03:02,271 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:03:07,272 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:03:12,272 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:03:14,468 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:03:14,469 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:03:14,469 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:03:14,470 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:03:14,838 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:03:15,020 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:03:16,165 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:03:16,167 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:03:16,168 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:03:18,244 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:03:23,245 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:03:28,246 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:03:31,166 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:03:31,168 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:03:31,168 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:03:33,253 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:03:38,254 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:03:43,255 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:03:45,021 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:03:46,166 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:03:46,168 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:03:46,168 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:03:48,270 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:03:48,606 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:03:48,607 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:03:48,607 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:03:48,608 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:03:48,850 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:03:53,609 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:03:58,610 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:04:01,166 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:04:01,168 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:04:01,168 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:04:04,264 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:04:09,265 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:04:14,266 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:04:15,021 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:04:16,166 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:04:16,168 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:04:16,169 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:04:19,296 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:04:21,282 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:04:21,283 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:04:21,284 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:04:21,284 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:04:21,862 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:04:25,285 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:04:30,286 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:04:31,166 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:04:31,168 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:04:31,169 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:04:36,236 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:04:41,237 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:04:45,022 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:04:46,166 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:04:46,169 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:04:46,169 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:04:46,237 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:04:51,238 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:04:56,239 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:05:01,167 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:05:01,169 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:05:01,169 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:05:01,284 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:05:02,598 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:05:02,599 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:05:02,599 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:05:02,600 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:05:02,878 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:05:06,601 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:05:11,602 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:05:15,023 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:05:16,166 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:05:16,169 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:05:16,169 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:05:17,336 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:05:22,337 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:05:27,338 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:05:31,167 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:05:31,169 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:05:31,169 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:05:32,352 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:05:35,325 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:05:35,326 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:05:35,326 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:05:35,327 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:05:35,890 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:05:38,328 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:05:43,329 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:05:45,024 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:05:46,167 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:05:46,169 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:05:46,169 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:05:48,338 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:05:53,339 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:05:58,340 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:06:01,167 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:06:01,169 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:06:01,169 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:06:04,226 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:06:07,521 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:06:07,523 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:06:07,523 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:06:07,523 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:06:07,903 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:06:08,903 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/output.log 2024-05-22 13:06:10,119 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:06:15,025 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:06:16,025 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:06:16,167 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:06:16,169 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:06:16,169 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:06:21,282 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:06:26,283 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:06:31,167 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:06:31,169 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:06:31,170 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:06:32,264 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:06:37,265 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:06:40,881 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:06:40,882 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:06:40,883 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:06:40,883 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:06:40,915 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:06:42,884 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:06:45,025 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:06:46,167 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:06:46,169 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:06:46,170 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:06:48,245 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:06:53,246 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:06:58,247 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:07:01,169 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:07:01,170 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:07:01,170 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:07:03,269 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:07:08,270 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:07:13,271 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:07:15,026 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:07:16,168 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:07:16,170 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:07:16,170 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:07:17,754 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:07:17,755 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:07:17,756 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:07:17,756 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:07:17,929 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:07:18,757 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:07:23,758 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:07:28,760 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:07:31,168 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:07:31,170 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:07:31,170 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:07:34,307 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:07:39,307 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:07:44,308 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:07:45,031 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:07:46,168 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:07:46,170 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:07:46,170 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:07:50,269 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:07:50,540 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:07:50,542 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:07:50,542 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:07:50,542 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:07:50,942 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:07:55,544 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:08:00,545 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:08:01,169 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:08:01,170 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:08:01,170 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:08:06,296 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:08:11,297 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:08:15,028 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:08:16,169 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:08:16,170 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:08:16,171 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:08:16,301 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:08:21,302 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:08:26,303 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:08:31,169 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:08:31,170 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:08:31,171 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:08:31,306 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:08:34,882 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:08:34,883 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:08:34,884 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:08:34,886 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:08:34,958 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:08:36,887 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:08:41,887 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:08:45,029 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:08:46,169 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:08:46,170 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:08:46,171 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:08:47,257 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:08:52,258 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:08:57,259 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:09:01,169 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:09:01,171 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:09:01,171 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:09:03,246 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:09:07,318 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:09:07,319 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:09:07,320 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:09:07,321 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:09:07,971 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:09:08,322 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:09:13,324 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:09:15,029 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:09:16,170 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:09:16,171 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:09:16,172 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:09:19,315 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:09:24,316 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:09:29,317 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:09:31,169 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:09:31,171 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:09:31,172 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:09:34,339 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:09:39,340 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:09:39,634 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:09:39,636 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:09:39,636 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:09:39,638 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:09:39,982 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:09:44,639 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:09:45,030 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:09:46,170 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:09:46,171 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:09:46,172 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:09:50,257 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:09:55,259 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:10:00,260 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:10:01,170 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:10:01,171 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:10:01,172 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:10:05,261 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:10:10,262 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:10:15,032 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:10:15,042 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:10:15,043 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:10:15,043 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:10:15,043 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:10:15,995 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:10:16,044 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:10:16,170 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:10:16,172 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:10:16,172 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:10:21,300 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:10:26,301 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:10:31,170 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:10:31,172 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:10:31,172 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:10:32,253 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:10:37,254 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:10:42,255 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:10:45,032 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:10:46,170 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:10:46,172 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:10:46,172 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:10:47,309 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:10:52,201 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:10:52,202 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:10:52,203 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:10:52,203 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:10:53,012 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:10:53,204 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:10:58,205 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:11:01,171 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:11:01,172 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:11:01,172 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:11:03,309 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:11:08,310 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:11:13,311 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:11:15,033 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:11:16,171 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:11:16,172 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:11:16,173 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:11:19,272 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:11:24,273 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:11:26,518 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:11:26,519 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:11:26,520 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:11:26,520 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:11:27,025 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:11:29,521 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:11:31,172 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:11:31,173 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:11:31,174 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:11:35,314 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:11:40,315 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:11:45,033 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:11:46,034 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:11:46,171 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:11:46,172 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:11:46,173 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:11:51,273 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:11:56,274 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:11:58,185 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:11:58,187 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:11:58,187 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:11:58,187 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:11:59,035 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:12:01,171 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:12:01,172 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:12:01,173 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:12:01,325 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:12:06,326 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:12:11,327 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:12:15,034 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:12:16,171 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:12:16,173 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:12:16,173 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:12:17,324 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:12:22,325 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:12:27,326 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:12:31,172 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:12:31,173 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:12:31,174 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:12:31,489 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:12:31,490 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:12:31,490 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:12:31,491 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:12:32,046 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:12:32,492 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:12:37,493 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:12:42,494 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:12:45,035 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:12:46,172 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:12:46,173 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:12:46,173 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:12:48,324 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:12:53,325 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:12:58,326 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:13:01,172 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:13:01,173 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:13:01,174 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:13:03,335 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:13:04,929 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:13:04,930 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:13:04,930 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:13:04,930 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:13:05,058 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:13:08,931 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:13:13,932 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:13:15,035 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:13:16,172 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:13:16,173 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:13:16,174 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:13:19,253 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:13:24,254 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:13:29,255 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:13:31,173 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:13:31,174 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:13:31,174 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:13:35,249 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:13:39,122 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:13:39,123 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:13:39,123 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:13:39,124 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:13:40,070 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:13:41,125 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:13:45,036 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:13:46,172 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:13:46,173 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:13:46,174 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:13:46,174 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:13:51,288 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:13:56,289 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:14:01,172 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:14:01,174 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:14:01,174 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:14:02,253 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:14:07,254 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:14:12,255 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:14:13,251 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:14:13,253 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:14:13,253 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:14:13,254 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:14:14,082 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:14:15,037 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:14:16,173 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:14:16,174 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:14:16,174 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:14:18,248 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:14:23,249 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:14:28,250 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:14:31,173 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:14:31,174 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:14:31,174 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:14:33,320 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:14:38,321 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:14:43,322 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:14:45,038 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:14:46,173 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:14:46,174 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:14:46,175 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:14:46,533 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:14:46,534 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:14:46,534 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:14:46,535 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:14:47,093 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:14:48,535 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:14:53,537 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:14:58,537 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:15:01,173 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:15:01,175 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:15:01,175 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:15:04,342 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:15:09,343 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:15:14,344 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:15:15,039 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:15:16,176 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:15:16,176 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:15:16,177 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:15:20,260 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:15:21,653 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:15:21,654 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:15:21,654 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:15:21,654 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:15:22,106 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:15:25,655 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:15:30,656 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:15:31,174 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:15:31,174 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:15:31,175 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:15:36,337 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:15:41,338 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:15:45,040 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:15:46,174 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:15:46,175 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:15:46,176 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:15:47,301 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:15:52,302 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:15:55,861 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:15:55,863 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:15:55,863 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:15:55,863 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:15:56,122 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:15:57,864 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:16:01,174 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:16:01,175 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:16:01,175 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:16:03,253 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:16:08,254 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:16:13,255 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:16:15,040 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:16:16,174 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:16:16,175 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:16:16,175 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:16:18,287 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:16:23,288 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:16:28,289 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:16:31,174 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:16:31,175 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:16:31,176 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:16:32,739 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:16:32,740 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:16:32,741 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:16:32,741 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:16:33,141 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:16:33,742 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:16:38,743 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:16:43,743 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:16:45,041 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:16:46,174 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:16:46,175 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:16:46,175 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:16:49,252 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:16:54,253 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:16:59,254 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:17:01,175 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:17:01,177 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:17:01,177 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:17:04,270 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:17:09,271 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:17:09,402 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:17:09,403 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:17:09,403 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:17:09,404 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:17:10,161 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:17:14,405 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:17:15,042 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:17:16,175 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:17:16,176 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:17:16,176 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:17:20,286 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:17:25,287 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:17:30,288 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:17:31,175 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:17:31,176 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:17:31,176 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:17:36,282 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:17:41,283 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:17:42,699 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:17:42,700 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:17:42,701 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:17:42,701 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:17:43,177 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:17:45,043 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:17:46,175 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:17:46,176 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:17:46,176 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:17:46,353 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:17:51,354 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:17:56,355 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:18:01,175 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:18:01,176 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:18:01,177 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:18:01,892 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:18:06,893 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:18:11,894 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:18:14,779 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:18:14,781 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:18:14,781 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:18:14,781 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:18:15,044 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:18:15,194 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:18:16,175 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:18:16,176 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:18:16,177 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:18:17,296 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:18:22,297 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:18:27,298 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:18:31,176 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:18:31,177 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:18:31,178 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:18:32,347 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:18:37,347 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:18:42,348 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:18:45,045 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:18:46,177 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:18:46,178 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:18:46,218 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:18:48,327 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:18:50,179 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:18:50,180 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:18:50,180 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:18:50,181 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:18:50,207 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:18:54,182 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:18:59,183 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:19:01,176 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:19:01,176 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:19:01,177 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:19:04,253 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:19:09,254 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:19:14,255 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:19:15,046 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:19:16,176 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:19:16,177 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:19:16,177 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:19:19,330 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:19:22,950 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:19:22,951 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:19:22,951 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:19:22,952 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:19:23,225 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:19:24,953 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:19:29,954 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:19:31,176 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:19:31,177 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:19:31,178 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:19:35,263 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:19:40,264 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:19:45,047 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:19:46,047 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:19:46,178 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:19:46,178 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:19:46,179 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:19:51,300 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:19:56,301 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:19:59,183 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:19:59,184 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:19:59,185 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:19:59,185 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:19:59,239 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:20:01,177 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:20:01,177 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:20:01,178 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:20:01,303 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:20:06,304 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:20:11,305 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:20:15,047 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:20:16,177 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:20:16,177 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:20:16,178 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:20:17,295 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:20:22,296 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:20:27,297 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:20:31,177 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:20:31,178 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:20:31,179 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:20:32,312 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:20:37,312 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:20:41,990 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:20:41,992 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:20:41,992 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:20:41,992 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:20:42,254 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:20:42,993 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:20:45,049 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:20:46,177 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:20:46,178 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:20:46,179 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:20:48,348 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:20:53,349 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:20:58,350 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:21:01,177 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:21:01,178 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:21:01,179 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:21:04,340 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:21:09,341 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:21:14,341 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:21:15,049 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:21:16,178 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:21:16,179 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:21:16,179 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:21:20,289 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:21:21,236 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:21:21,237 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:21:21,237 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:21:21,238 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:21:21,278 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:21:26,239 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:21:31,178 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:21:31,178 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:21:31,179 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:21:31,274 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:21:36,275 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:21:41,276 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:21:45,050 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:21:46,178 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:21:46,179 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:21:46,179 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:21:46,337 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:21:51,338 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:21:54,769 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:21:54,770 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:21:54,771 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:21:54,771 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:21:55,298 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:21:56,772 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:22:01,178 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:22:01,179 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:22:01,179 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:22:02,314 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:22:07,315 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:22:12,316 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:22:15,051 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:22:16,178 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:22:16,179 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:22:16,179 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:22:18,261 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:22:23,262 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:22:28,263 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:22:31,179 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:22:31,179 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:22:31,179 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:22:32,268 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:22:32,269 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:22:32,270 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:22:32,270 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:22:32,318 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:22:33,271 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:22:38,272 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:22:43,273 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:22:45,052 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:22:46,178 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:22:46,180 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:22:46,180 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:22:48,330 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:22:53,331 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:22:58,332 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:23:01,179 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:23:01,180 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:23:01,181 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:23:03,194 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:23:03,195 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:23:03,196 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:23:03,196 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:23:03,332 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:23:04,197 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:23:09,198 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:23:14,199 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:23:15,052 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:23:16,179 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:23:16,180 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:23:16,180 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:23:19,280 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:23:24,281 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:23:29,282 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:23:31,180 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:23:31,180 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:23:31,180 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:23:34,351 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:23:38,336 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:23:38,337 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:23:38,337 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:23:38,338 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:23:38,350 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:23:40,338 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:23:45,053 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:23:46,054 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:23:46,180 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:23:46,180 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:23:46,180 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:23:51,327 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:23:56,328 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:24:01,180 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:24:01,180 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:24:01,181 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:24:01,333 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:24:06,334 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:24:11,335 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:24:13,806 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:24:13,807 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:24:13,808 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:24:13,808 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:24:14,362 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:24:15,054 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:24:16,180 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:24:16,180 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:24:16,181 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:24:17,281 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:24:22,282 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:24:27,283 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:24:31,199 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:24:31,200 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:24:31,200 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:24:32,347 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:24:37,348 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:24:42,349 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:24:45,055 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:24:46,198 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:24:46,199 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:24:46,199 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:24:48,346 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:24:50,432 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:24:50,434 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:24:50,434 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:24:50,435 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:24:51,383 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:24:53,435 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:24:58,436 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:25:01,198 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:25:01,199 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:25:01,199 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:25:04,335 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:25:09,337 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:25:14,338 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:25:15,056 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:25:16,199 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:25:16,199 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:25:16,200 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:25:20,326 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:25:25,327 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:25:30,328 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:25:31,199 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:25:31,199 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:25:31,200 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:25:35,336 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:25:40,339 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:25:45,057 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:25:46,057 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:25:46,199 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:25:46,199 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:25:46,200 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:25:51,356 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:25:54,338 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:25:54,339 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:25:54,340 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:25:54,340 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:25:54,407 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:25:57,341 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:26:01,199 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:26:01,200 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:26:01,200 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:26:02,378 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:26:07,379 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:26:12,380 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:26:15,057 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:26:16,200 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:26:16,201 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:26:16,201 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:26:18,338 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:26:23,339 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:26:28,340 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:26:30,419 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:26:30,424 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:26:30,425 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:26:30,425 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:26:31,201 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:26:31,201 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:26:31,242 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:26:31,420 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:26:33,370 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:26:38,372 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:26:43,372 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:26:45,058 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:26:46,200 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:26:46,200 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:26:46,201 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:26:49,286 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:26:54,287 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:26:59,288 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:27:01,201 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:27:01,202 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:27:01,202 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:27:04,340 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:27:05,310 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:27:05,311 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:27:05,311 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:27:05,312 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:27:05,433 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:27:10,313 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:27:15,059 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:27:16,060 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:27:16,202 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:27:16,203 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:27:16,243 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:27:21,375 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:27:26,377 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:27:31,200 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:27:31,201 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:27:31,201 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:27:32,348 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:27:37,349 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:27:41,389 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:27:41,390 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:27:41,391 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:27:41,391 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:27:41,446 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:27:42,391 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:27:45,060 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:27:46,200 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:27:46,201 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:27:46,201 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:27:48,312 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:27:53,313 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:27:58,314 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:28:01,201 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:28:01,201 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:28:01,201 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:28:03,365 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:28:08,366 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:28:13,368 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:28:14,296 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:28:14,297 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:28:14,297 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:28:14,298 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:28:14,460 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:28:15,061 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:28:16,200 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:28:16,201 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:28:16,201 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:28:18,370 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:28:23,371 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:28:28,372 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:28:31,201 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:28:31,201 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:28:31,202 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:28:34,331 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:28:39,332 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:28:44,333 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:28:45,062 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:28:46,200 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:28:46,201 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:28:46,202 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:28:48,148 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:28:48,150 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:28:48,150 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:28:48,150 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:28:48,472 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:28:50,151 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:28:55,152 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:29:00,153 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:29:01,201 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:29:01,201 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:29:01,202 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:29:05,351 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:29:10,352 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:29:15,063 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:29:16,063 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:29:16,201 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:29:16,202 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:29:16,202 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:29:21,350 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:29:21,485 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:29:21,487 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:29:21,487 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:29:21,487 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:29:22,485 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:29:26,488 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:29:31,201 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:29:31,202 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:29:31,202 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:29:32,295 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:29:37,296 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:29:42,297 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:29:45,064 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:29:46,201 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:29:46,202 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:29:46,202 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:29:48,289 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:29:53,291 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:29:55,715 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:29:55,717 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:29:55,717 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:29:55,717 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:29:56,497 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:29:58,718 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:30:01,202 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:30:01,202 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:30:01,202 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:30:04,335 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:30:09,336 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:30:14,337 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:30:15,064 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:30:16,202 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:30:16,202 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:30:16,202 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:30:20,301 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:30:25,302 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:30:27,262 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:30:27,263 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:30:27,263 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:30:27,264 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:30:27,508 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:30:31,202 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:30:31,202 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:30:31,202 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:30:31,203 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:30:36,321 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:30:41,322 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:30:45,066 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:30:46,202 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:30:46,203 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:30:46,203 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:30:46,369 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:30:51,370 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:30:56,371 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:31:01,202 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:31:01,203 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:31:01,203 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:31:02,339 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:31:06,284 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:31:06,285 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:31:06,286 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:31:06,286 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:31:06,521 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:31:08,287 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:31:13,288 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:31:15,067 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:31:16,202 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:31:16,203 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:31:16,203 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:31:18,317 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:31:23,318 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:31:28,319 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:31:31,203 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:31:31,203 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:31:31,203 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:31:33,324 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:31:38,325 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:31:41,191 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:31:41,192 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:31:41,192 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:31:41,192 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:31:41,533 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:31:44,193 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:31:45,068 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:31:46,202 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:31:46,203 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:31:46,203 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:31:49,311 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:31:54,312 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:31:59,313 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:32:01,203 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:32:01,203 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:32:01,203 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:32:04,371 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:32:09,373 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:32:13,421 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:32:13,422 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:32:13,423 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:32:13,423 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:32:13,548 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:32:14,424 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:32:15,068 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:32:16,202 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:32:16,203 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:32:16,204 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:32:20,324 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:32:25,325 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:32:30,326 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:32:31,203 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:32:31,203 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:32:31,204 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:32:35,378 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:32:40,379 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:32:45,069 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:32:46,070 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:32:46,203 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:32:46,203 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:32:46,204 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:32:47,777 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:32:47,778 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:32:47,778 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:32:47,779 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:32:48,566 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:32:51,780 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:32:56,781 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:33:01,203 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:33:01,204 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:33:01,204 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:33:02,362 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:33:07,363 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:33:12,364 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:33:15,070 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:33:16,203 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:33:16,204 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:33:16,204 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:33:18,341 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:33:22,978 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:33:22,979 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:33:22,979 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:33:22,979 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:33:23,585 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:33:23,980 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:33:28,981 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:33:31,203 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:33:31,204 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:33:31,204 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:33:34,311 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:33:39,312 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:33:44,313 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:33:45,071 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:33:46,205 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:33:46,205 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:33:46,206 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:33:49,373 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:33:54,374 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:33:58,189 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:33:58,190 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:33:58,191 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:33:58,191 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:33:58,605 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:34:00,192 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:34:01,204 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:34:01,204 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:34:01,205 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:34:05,352 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:34:10,353 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:34:15,072 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:34:16,073 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:34:16,204 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:34:16,204 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:34:16,205 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:34:21,314 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:34:26,315 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:34:31,205 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:34:31,206 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:34:31,206 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:34:31,321 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:34:33,566 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:34:33,567 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:34:33,568 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:34:33,568 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:34:33,622 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:34:36,569 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:34:41,570 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:34:45,073 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:34:46,204 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:34:46,205 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:34:46,205 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:34:47,278 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:34:52,280 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:34:57,281 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:35:01,205 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:35:01,205 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:35:01,206 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:35:02,317 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:35:07,318 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:35:07,573 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:35:07,574 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:35:07,574 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:35:07,575 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:35:07,641 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:35:12,576 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:35:15,073 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:35:16,204 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:35:16,205 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:35:16,205 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:35:18,287 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:35:23,288 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:35:28,289 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:35:31,205 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:35:31,205 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:35:31,206 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:35:33,333 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:35:38,335 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:35:40,915 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:35:40,917 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:35:40,917 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:35:40,917 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:35:41,657 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:35:43,918 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:35:45,074 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:35:46,205 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:35:46,205 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:35:46,206 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:35:49,336 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:35:54,337 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:35:59,338 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:36:01,205 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:36:01,205 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:36:01,206 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:36:04,366 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:36:09,367 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:36:14,107 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:36:14,108 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:36:14,109 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:36:14,109 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:36:14,674 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:36:15,075 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:36:15,076 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:36:16,205 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:36:16,206 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:36:16,206 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:36:20,346 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:36:25,348 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:36:30,349 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:36:31,205 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:36:31,206 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:36:31,206 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:36:36,344 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:36:41,345 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:36:45,076 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:36:46,206 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:36:46,206 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:36:46,206 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:36:47,016 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:36:47,018 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:36:47,018 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:36:47,018 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:36:47,019 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:36:47,690 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:36:52,020 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:36:57,021 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:37:01,205 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:37:01,206 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:37:01,206 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:37:02,377 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:37:07,378 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:37:12,379 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:37:15,077 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:37:16,206 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:37:16,206 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:37:16,206 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:37:18,310 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:37:21,508 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:37:21,509 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:37:21,509 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:37:21,510 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:37:21,702 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:37:23,510 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:37:28,512 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:37:31,206 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:37:31,207 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:37:31,207 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:37:34,366 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:37:39,368 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:37:44,369 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:37:45,078 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:37:46,206 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:37:46,206 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:37:46,207 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:37:50,309 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:37:55,310 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:37:55,593 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:37:55,595 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:37:55,595 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:37:55,596 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:37:55,713 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:38:00,597 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:38:01,206 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:38:01,206 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:38:01,207 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:38:06,357 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:38:11,359 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:38:15,079 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:38:16,206 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:38:16,206 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:38:16,207 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:38:17,272 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:38:22,273 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:38:27,274 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:38:28,840 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:38:28,841 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:38:28,841 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:38:28,842 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:38:29,730 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:38:31,206 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:38:31,206 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:38:31,207 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:38:32,348 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:38:37,349 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:38:42,350 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:38:45,080 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:38:46,207 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:38:46,207 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:38:46,207 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:38:48,313 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:38:53,314 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:38:58,315 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:39:01,206 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:39:01,207 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:39:01,207 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:39:02,952 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:39:02,953 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:39:02,953 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:39:02,954 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:39:03,748 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:39:03,954 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:39:08,956 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:39:13,957 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:39:15,081 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:39:16,207 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:39:16,207 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:39:16,208 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:39:19,385 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:39:24,386 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:39:29,387 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:39:31,209 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:39:31,209 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:39:31,209 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:39:35,279 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:39:37,063 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:39:37,064 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:39:37,065 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:39:37,065 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:39:37,763 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:39:41,066 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:39:45,082 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:39:46,082 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:39:46,207 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:39:46,207 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:39:46,208 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:39:51,316 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:39:56,317 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:40:01,208 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:40:01,208 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:40:01,209 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:40:01,343 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:40:06,345 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:40:09,269 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:40:09,270 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:40:09,271 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:40:09,271 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:40:09,779 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:40:12,272 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:40:15,083 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:40:16,207 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:40:16,207 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:40:16,208 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:40:17,317 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:40:22,318 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:40:27,319 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:40:31,208 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:40:31,208 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:40:31,208 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:40:32,370 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:40:37,371 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:40:42,372 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:40:45,083 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:40:46,208 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:40:46,208 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:40:46,208 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:40:46,705 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:40:46,706 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:40:46,707 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:40:46,707 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:40:46,793 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:40:47,708 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:40:52,709 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:40:57,710 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:41:01,207 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:41:01,208 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:41:01,209 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:41:03,289 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:41:08,290 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:41:13,291 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:41:15,084 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:41:16,208 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:41:16,208 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:41:16,209 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:41:18,369 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:41:21,509 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:41:21,510 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:41:21,510 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:41:21,510 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:41:21,806 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:41:23,511 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:41:28,512 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:41:31,208 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:41:31,209 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:41:31,209 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:41:34,365 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:41:39,366 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:41:44,367 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:41:45,085 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:41:46,209 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:41:46,209 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:41:46,209 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:41:50,312 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:41:54,335 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:41:54,336 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:41:54,337 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:41:54,337 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:41:54,820 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:41:55,338 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:42:00,339 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:42:01,208 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:42:01,209 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:42:01,209 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:42:06,335 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:42:11,336 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:42:15,086 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:42:16,209 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:42:16,209 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:42:16,209 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:42:16,373 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:42:21,374 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:42:26,375 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:42:31,209 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:42:31,209 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:42:31,210 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:42:32,292 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:42:35,197 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:42:35,198 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:42:35,198 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:42:35,199 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:42:35,837 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:42:38,200 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:42:43,201 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:42:45,087 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:42:46,210 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:42:46,211 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:42:46,211 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:42:48,311 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:42:53,312 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:42:58,313 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:43:01,210 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:43:01,211 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:43:01,211 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:43:04,283 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:43:09,284 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:43:09,999 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:43:10,000 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:43:10,001 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:43:10,001 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:43:10,854 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:43:15,002 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:43:15,087 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:43:16,209 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:43:16,210 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:43:16,211 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:43:20,310 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:43:25,311 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:43:30,312 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:43:31,209 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:43:31,210 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:43:31,210 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:43:35,364 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:43:40,365 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:43:43,878 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:43:43,879 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:43:43,879 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:43:43,880 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:43:44,868 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:43:45,088 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:43:46,089 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:43:46,209 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:43:46,210 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:43:46,210 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:43:51,374 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:43:56,375 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:44:01,210 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:44:01,210 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:44:01,211 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:44:01,381 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:44:06,382 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:44:11,383 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:44:15,089 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:44:15,384 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:44:15,385 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:44:15,386 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:44:15,386 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:44:15,882 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:44:16,209 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:44:16,210 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:44:16,211 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:44:17,354 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:44:22,355 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:44:27,356 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:44:31,210 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:44:31,211 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:44:31,211 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:44:32,369 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:44:37,370 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:44:42,371 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:44:45,090 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:44:46,210 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:44:46,211 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:44:46,211 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:44:48,233 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:44:48,234 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:44:48,234 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:44:48,235 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:44:48,235 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:44:48,894 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:44:53,236 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:44:58,237 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:45:01,210 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:45:01,211 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:45:01,211 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:45:03,334 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:45:08,334 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:45:13,335 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:45:15,096 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:45:16,211 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:45:16,211 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:45:16,211 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:45:19,321 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:45:24,322 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:45:28,796 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:45:28,797 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:45:28,798 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:45:28,798 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:45:28,912 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:45:29,799 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:45:31,210 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:45:31,211 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:45:31,211 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:45:35,343 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:45:40,344 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:45:45,092 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:45:46,093 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:45:46,211 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:45:46,211 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:45:46,212 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:45:51,365 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:45:56,366 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:46:01,211 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:46:01,211 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:46:01,212 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:46:02,253 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:46:04,907 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:46:04,908 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:46:04,908 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:46:04,911 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:46:04,929 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:46:07,912 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:46:12,913 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:46:15,093 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:46:16,211 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:46:16,211 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:46:16,212 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:46:18,342 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:46:23,343 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:46:28,344 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:46:31,211 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:46:31,211 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:46:31,212 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:46:33,354 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:46:36,481 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:46:36,482 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:46:36,483 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:46:36,485 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:46:36,942 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:46:38,486 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:46:43,486 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:46:45,094 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:46:46,211 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:46:46,212 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:46:46,212 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:46:49,350 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:46:54,351 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:46:59,352 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:47:01,212 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:47:01,213 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:47:01,214 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:47:05,282 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:47:07,304 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:47:07,306 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:47:07,306 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:47:07,308 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:47:07,954 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:47:10,309 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:47:15,095 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:47:16,096 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:47:16,211 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:47:16,212 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:47:16,212 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:47:21,296 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:47:26,297 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:47:31,212 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:47:31,212 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:47:31,213 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:47:31,376 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:47:36,377 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:47:41,377 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:47:45,097 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:47:46,212 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:47:46,212 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:47:46,213 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:47:47,335 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:47:52,336 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:47:52,735 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:47:52,737 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:47:52,737 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:47:52,737 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:47:52,977 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:47:57,738 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:48:01,213 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:48:01,213 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:48:01,214 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:48:03,313 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:48:08,314 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:48:13,315 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:48:15,097 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:48:16,212 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:48:16,212 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:48:16,213 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:48:18,356 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:48:23,357 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:48:28,359 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:48:31,212 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:48:31,213 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:48:31,213 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:48:32,005 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:48:32,006 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:48:32,007 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:48:32,007 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:48:32,997 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:48:34,008 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:48:39,009 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:48:44,010 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:48:45,097 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:48:46,213 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:48:46,213 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:48:46,213 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:48:49,358 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:48:54,359 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:48:59,360 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:49:01,212 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:49:01,213 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:49:01,213 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:49:05,354 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:49:06,119 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:49:06,121 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:49:06,121 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:49:06,122 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:49:07,010 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:49:11,122 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:49:15,098 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:49:16,212 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:49:16,213 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:49:16,213 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:49:16,214 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:49:21,312 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:49:26,313 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:49:31,213 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:49:31,213 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:49:31,214 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:49:31,315 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:49:36,316 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:49:39,280 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:49:39,281 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:49:39,281 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:49:39,282 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:49:40,022 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:49:42,283 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:49:45,100 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:49:46,213 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:49:46,213 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:49:46,214 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:49:47,374 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:49:52,375 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:49:57,376 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:50:01,213 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:50:01,214 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:50:01,214 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:50:02,904 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:50:07,905 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:50:12,906 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:50:15,100 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:50:16,213 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:50:16,214 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:50:16,214 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:50:16,721 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:50:16,722 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:50:16,723 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:50:16,723 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:50:17,040 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:50:18,724 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:50:23,725 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:50:28,726 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:50:31,213 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:50:31,214 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:50:31,214 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:50:34,334 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:50:39,335 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:50:44,336 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:50:45,101 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:50:46,214 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:50:46,214 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:50:46,215 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:50:49,919 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:50:49,920 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:50:49,921 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:50:49,921 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:50:49,922 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:50:50,055 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:50:54,923 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:50:59,923 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:51:01,214 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:51:01,214 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:51:01,215 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:51:05,368 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:51:10,369 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:51:15,102 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:51:16,103 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:51:16,214 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:51:16,214 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:51:16,215 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:51:21,343 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:51:23,273 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:51:23,275 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:51:23,275 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:51:23,275 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:51:24,072 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:51:27,276 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:51:31,214 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:51:31,215 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:51:31,216 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:51:32,328 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:51:37,329 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:51:42,330 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:51:45,103 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:51:46,215 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:51:46,215 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:51:46,215 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:51:48,296 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:51:53,297 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:51:58,298 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:52:00,282 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:52:00,283 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:52:00,284 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:52:00,284 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:52:01,089 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:52:01,215 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:52:01,215 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:52:01,216 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:52:03,342 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:52:08,343 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:52:13,343 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:52:15,104 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:52:16,215 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:52:16,215 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:52:16,216 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:52:19,301 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:52:24,302 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:52:29,302 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:52:31,215 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:52:31,216 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:52:31,216 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:52:32,419 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:52:32,420 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:52:32,420 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:52:32,421 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:52:33,101 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:52:34,422 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:52:39,423 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:52:44,424 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:52:45,105 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:52:46,215 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:52:46,215 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:52:46,216 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:52:50,354 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:52:55,355 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:53:00,356 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:53:01,216 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:53:01,216 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:53:01,217 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:53:04,539 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:53:04,540 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:53:04,540 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:53:04,541 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:53:05,118 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:53:05,541 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:53:10,542 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:53:15,106 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:53:16,106 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:53:16,215 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:53:16,216 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:53:16,216 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:53:21,315 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:53:26,316 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:53:31,216 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:53:31,216 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:53:31,217 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:53:31,337 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:53:36,338 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:53:39,061 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:53:39,062 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:53:39,062 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:53:39,063 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:53:39,131 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:53:42,064 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:53:45,106 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:53:46,215 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:53:46,216 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:53:46,217 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:53:47,364 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:53:52,366 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:53:57,367 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:54:01,218 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:54:01,219 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:54:01,259 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:54:03,282 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:54:08,283 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:54:13,284 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:54:15,107 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:54:16,216 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:54:16,217 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:54:16,257 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:54:18,350 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:54:18,861 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:54:18,863 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:54:18,863 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:54:18,863 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:54:19,153 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:54:23,864 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:54:28,865 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:54:31,216 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:54:31,217 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:54:31,217 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:54:34,346 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:54:39,347 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:54:44,348 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:54:45,108 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:54:46,217 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:54:46,217 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:54:46,218 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:54:49,376 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:54:54,377 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:54:55,040 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:54:55,041 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:54:55,041 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:54:55,042 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:54:55,169 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:55:00,043 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:55:01,217 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:55:01,217 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:55:01,218 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:55:05,299 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:55:10,301 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:55:15,109 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:55:16,110 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:55:16,217 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:55:16,217 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:55:16,218 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:55:21,309 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:55:26,310 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:55:29,153 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:55:29,154 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:55:29,155 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:55:29,155 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:55:29,182 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:55:31,217 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:55:31,218 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:55:31,218 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:55:31,361 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:55:36,362 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:55:41,363 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:55:45,110 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:55:46,217 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:55:46,218 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:55:46,218 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:55:47,342 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:55:52,343 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:55:57,344 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:56:01,217 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:56:01,218 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:56:01,218 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:56:02,393 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:56:05,777 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:56:05,778 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:56:05,778 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:56:05,779 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:56:06,198 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:56:07,780 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:56:12,780 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:56:15,111 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:56:16,217 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:56:16,218 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:56:16,218 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:56:18,421 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:56:23,421 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:56:28,422 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:56:31,219 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:56:31,220 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:56:31,220 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:56:34,349 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:56:39,350 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:56:40,761 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:56:40,762 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:56:40,762 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:56:40,763 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:56:41,219 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:56:44,764 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:56:45,112 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:56:46,218 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:56:46,218 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:56:46,219 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:56:50,316 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:56:55,317 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:57:00,318 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:57:01,226 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:57:01,226 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:57:01,266 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:57:05,393 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:57:10,394 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:57:14,822 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:57:14,823 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:57:14,823 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:57:14,824 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:57:15,113 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:57:15,237 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:57:16,114 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:57:16,223 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:57:16,224 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:57:16,227 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:57:21,371 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:57:26,372 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:57:31,224 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:57:31,224 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:57:31,265 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:57:32,311 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:57:37,312 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:57:42,313 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:57:45,114 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:57:46,224 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:57:46,224 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:57:46,265 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:57:47,058 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:57:47,059 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:57:47,059 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:57:47,060 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:57:47,253 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:57:48,060 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:57:53,061 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:57:58,062 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:58:01,224 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:58:01,224 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:58:01,226 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:58:03,363 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:58:08,364 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:58:13,365 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:58:15,115 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:58:16,224 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:58:16,224 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:58:16,265 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:58:18,387 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:58:19,260 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:58:19,262 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:58:19,262 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:58:19,262 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:58:19,264 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:58:24,263 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:58:29,264 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:58:31,224 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:58:31,225 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:58:31,227 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:58:34,379 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:58:39,380 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:58:44,381 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:58:45,115 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:58:46,224 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:58:46,225 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:58:46,265 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:58:50,374 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:58:51,345 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:58:51,347 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:58:51,347 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:58:51,347 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:58:52,277 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:58:56,348 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:59:01,224 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:59:01,225 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:59:01,227 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:59:01,385 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:59:06,386 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:59:11,387 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:59:15,116 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:59:16,225 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:59:16,225 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:59:16,265 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:59:17,308 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:59:22,309 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:59:27,310 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:59:27,749 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 13:59:27,750 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 13:59:27,751 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 13:59:27,751 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 13:59:28,291 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 13:59:31,225 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:59:31,225 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:59:31,229 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:59:32,384 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:59:37,385 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:59:42,386 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:59:45,117 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 13:59:46,225 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 13:59:46,225 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 13:59:46,266 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 13:59:48,323 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:59:53,324 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 13:59:58,325 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:00:01,225 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:00:01,225 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:00:01,266 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:00:04,297 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:00:05,389 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 14:00:05,390 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 14:00:05,391 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 14:00:05,391 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 14:00:06,310 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 14:00:09,392 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:00:14,393 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:00:15,118 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 14:00:16,225 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:00:16,225 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:00:16,228 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:00:20,383 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:00:25,384 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:00:30,385 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:00:31,225 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:00:31,226 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:00:31,266 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:00:36,370 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:00:37,580 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 14:00:37,581 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 14:00:37,581 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 14:00:37,582 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 14:00:38,323 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 14:00:41,583 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:00:45,119 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 14:00:46,225 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:00:46,226 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:00:46,229 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:00:47,321 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:00:52,322 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:00:57,323 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:01:01,226 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:01:01,226 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:01:01,266 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:01:03,319 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:01:08,321 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:01:11,114 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 14:01:11,115 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 14:01:11,116 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 14:01:11,116 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 14:01:11,341 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 14:01:14,117 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:01:15,119 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 14:01:16,227 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:01:16,227 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:01:16,228 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:01:19,354 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:01:24,355 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:01:29,355 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:01:31,226 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:01:31,227 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:01:31,267 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:01:34,401 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:01:39,402 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:01:44,403 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:01:45,120 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 14:01:46,226 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:01:46,227 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:01:46,267 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:01:50,343 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:01:50,656 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 14:01:50,657 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 14:01:50,657 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 14:01:50,658 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 14:01:51,363 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 14:01:55,659 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:02:00,660 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:02:01,228 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:02:01,228 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:02:01,231 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:02:06,305 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:02:11,306 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:02:15,121 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 14:02:16,226 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:02:16,227 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:02:16,267 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:02:16,379 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:02:21,380 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:02:26,381 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:02:27,170 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 14:02:27,171 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 14:02:27,171 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 14:02:27,172 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 14:02:27,382 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 14:02:31,227 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:02:31,227 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:02:31,230 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:02:32,366 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:02:37,367 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:02:42,368 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:02:45,122 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 14:02:46,227 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:02:46,227 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:02:46,268 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:02:48,299 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:02:53,300 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:02:58,301 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:02:59,306 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 14:02:59,307 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 14:02:59,307 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 14:02:59,308 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 14:02:59,403 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 14:03:01,227 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:03:01,227 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:03:01,230 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:03:03,317 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:03:08,318 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:03:13,319 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:03:15,122 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 14:03:16,229 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:03:16,230 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:03:16,270 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:03:18,371 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:03:23,372 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:03:28,373 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:03:31,227 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:03:31,228 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:03:31,268 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:03:33,400 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:03:34,753 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 14:03:34,754 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 14:03:34,754 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 14:03:34,755 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 14:03:35,417 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 14:03:38,756 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:03:43,757 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:03:45,128 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 14:03:46,227 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:03:46,228 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:03:46,230 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:03:49,304 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:03:54,305 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:03:59,306 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:04:01,228 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:04:01,228 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:04:01,269 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:04:05,300 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:04:08,042 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 14:04:08,044 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 14:04:08,044 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 14:04:08,044 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 14:04:08,434 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 14:04:11,045 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:04:15,124 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 14:04:16,125 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:04:16,228 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:04:16,228 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:04:16,231 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:04:21,348 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:04:26,348 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:04:31,228 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:04:31,228 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:04:31,269 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:04:32,340 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:04:37,341 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:04:41,290 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 14:04:41,292 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 14:04:41,292 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 14:04:41,294 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 14:04:41,447 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 14:04:41,868 DEBUG SenderThread:290 [sender.py:send():379] send: telemetry 2024-05-22 14:04:41,869 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 14:04:41,871 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: partial_history 2024-05-22 14:04:41,872 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 14:04:41,872 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 14:04:41,873 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 14:04:41,873 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 14:04:41,874 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 14:04:41,874 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 14:04:41,874 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 14:04:41,874 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 14:04:41,875 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: pause 2024-05-22 14:04:41,875 INFO HandlerThread:290 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-05-22 14:04:41,875 INFO HandlerThread:290 [system_monitor.py:finish():203] Stopping system monitor 2024-05-22 14:04:41,876 DEBUG SystemMonitor:290 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-05-22 14:04:41,876 DEBUG SystemMonitor:290 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-05-22 14:04:41,877 INFO HandlerThread:290 [interfaces.py:finish():202] Joined cpu monitor 2024-05-22 14:04:41,877 INFO HandlerThread:290 [interfaces.py:finish():202] Joined disk monitor 2024-05-22 14:04:41,878 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 14:04:41,878 DEBUG SenderThread:290 [sender.py:send():379] send: history 2024-05-22 14:04:41,878 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: summary_record 2024-05-22 14:04:41,879 INFO SenderThread:290 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-05-22 14:04:41,884 INFO HandlerThread:290 [interfaces.py:finish():202] Joined gpu monitor 2024-05-22 14:04:41,884 INFO HandlerThread:290 [interfaces.py:finish():202] Joined memory monitor 2024-05-22 14:04:41,884 INFO HandlerThread:290 [interfaces.py:finish():202] Joined network monitor 2024-05-22 14:04:41,885 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 14:04:42,448 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/wandb-summary.json 2024-05-22 14:04:42,886 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:04:43,448 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/output.log 2024-05-22 14:04:46,228 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:04:46,229 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:04:46,229 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:04:48,307 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:04:53,316 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:04:54,496 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/config.yaml 2024-05-22 14:04:58,497 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:05:01,228 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:05:01,229 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:05:01,269 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:05:04,326 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:05:09,328 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:05:14,329 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:05:16,229 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:05:16,229 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:05:16,269 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:05:19,385 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:05:24,386 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:05:29,387 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:05:31,229 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:05:31,229 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:05:31,270 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:05:35,309 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:05:40,310 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:05:45,311 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:05:46,229 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:05:46,229 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:05:46,270 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:05:50,403 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:05:55,404 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:06:00,405 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:06:01,229 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:06:01,229 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:06:01,270 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:06:06,317 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:06:11,318 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:06:16,229 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:06:16,230 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:06:16,270 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:06:17,307 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:06:22,308 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:06:27,309 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:06:31,229 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:06:31,230 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:06:31,270 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:06:32,397 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:06:37,398 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:06:42,399 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:06:46,230 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:06:46,230 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:06:46,270 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:06:48,362 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:06:53,363 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:06:58,364 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:07:01,230 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:07:01,230 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:07:01,271 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:07:03,940 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:07:08,942 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:07:13,943 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:07:16,230 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:07:16,230 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:07:16,271 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:07:19,344 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:07:24,345 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:07:29,346 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:07:31,230 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:07:31,231 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:07:31,271 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:07:34,384 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:07:39,385 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:07:44,386 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:07:46,230 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:07:46,230 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:07:46,271 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:07:49,387 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:07:54,389 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:07:59,390 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:08:01,230 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:08:01,231 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:08:01,271 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:08:05,346 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:08:10,347 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:08:15,349 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:08:16,230 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:08:16,231 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:08:16,272 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:08:21,325 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:08:26,325 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:08:31,231 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:08:31,231 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:08:31,272 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:08:31,367 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:08:36,369 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:08:41,370 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:08:46,231 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:08:46,231 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:08:46,272 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:08:47,348 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:08:52,349 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:08:57,350 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:09:01,231 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:09:01,232 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:09:01,272 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:09:02,391 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:09:07,392 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:09:12,393 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:09:16,231 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:09:16,232 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:09:16,272 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:09:18,350 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:09:23,351 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:09:28,353 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:09:31,231 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:09:31,232 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:09:31,272 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:09:34,310 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:09:39,311 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:09:44,312 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:09:46,231 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:09:46,232 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:09:46,272 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:09:49,417 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:09:54,418 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:09:59,419 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:10:01,232 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:10:01,232 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:10:01,273 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:10:05,351 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:10:10,352 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:10:15,354 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:10:16,232 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:10:16,232 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:10:16,233 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:10:21,307 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:10:26,308 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:10:31,232 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:10:31,232 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:10:31,233 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:10:31,383 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:10:36,384 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:10:41,385 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:10:46,232 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:10:46,233 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:10:46,233 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:10:47,340 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:10:52,342 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:10:57,343 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:11:01,233 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:11:01,233 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:11:01,233 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:11:03,309 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:11:08,311 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:11:13,312 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:11:16,233 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:11:16,233 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:11:16,234 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:11:18,402 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:11:23,403 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:11:28,404 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:11:31,233 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:11:31,233 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:11:31,234 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:11:34,341 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:11:39,343 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:11:44,344 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:11:46,233 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:11:46,233 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:11:46,234 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:11:49,379 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:11:54,380 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:11:59,381 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:12:01,233 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:12:01,234 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:12:01,234 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:12:05,341 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:12:10,342 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:12:15,343 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:12:16,234 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:12:16,234 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:12:16,234 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:12:20,383 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:12:25,384 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:12:30,385 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:12:31,234 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:12:31,234 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:12:31,235 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:12:36,368 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:12:41,369 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:12:46,234 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:12:46,234 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:12:46,235 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:12:46,370 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:12:51,372 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:12:56,372 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:13:01,234 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:13:01,234 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:13:01,235 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:13:02,326 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:13:07,328 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:13:12,329 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:13:16,234 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:13:16,234 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:13:16,235 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:13:17,346 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:13:22,347 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:13:27,348 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:13:31,235 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:13:31,235 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:13:31,235 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:13:33,329 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:13:38,330 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:13:43,331 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:13:46,235 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:13:46,235 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:13:46,235 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:13:48,360 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:13:53,361 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:13:58,362 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:14:01,235 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:14:01,235 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:14:01,236 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:14:03,387 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:14:08,388 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:14:13,390 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:14:16,235 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:14:16,235 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:14:16,276 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:14:18,761 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:14:23,763 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:14:28,763 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:14:31,236 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:14:31,236 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:14:31,236 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:14:34,388 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:14:39,389 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:14:44,391 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:14:46,236 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:14:46,236 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:14:46,236 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:14:49,422 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:14:52,958 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: resume 2024-05-22 14:14:52,958 INFO HandlerThread:290 [handler.py:handle_request_resume():699] starting system metrics thread 2024-05-22 14:14:52,958 INFO HandlerThread:290 [system_monitor.py:start():194] Starting system monitor 2024-05-22 14:14:52,959 INFO SystemMonitor:290 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-05-22 14:14:52,959 INFO SystemMonitor:290 [interfaces.py:start():190] Started cpu monitoring 2024-05-22 14:14:52,960 INFO SystemMonitor:290 [interfaces.py:start():190] Started disk monitoring 2024-05-22 14:14:52,960 INFO SystemMonitor:290 [interfaces.py:start():190] Started gpu monitoring 2024-05-22 14:14:52,961 INFO SystemMonitor:290 [interfaces.py:start():190] Started memory monitoring 2024-05-22 14:14:52,962 INFO SystemMonitor:290 [interfaces.py:start():190] Started network monitoring 2024-05-22 14:14:53,744 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/output.log 2024-05-22 14:14:54,438 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: pause 2024-05-22 14:14:54,438 INFO HandlerThread:290 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-05-22 14:14:54,438 INFO HandlerThread:290 [system_monitor.py:finish():203] Stopping system monitor 2024-05-22 14:14:54,439 DEBUG SystemMonitor:290 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-05-22 14:14:54,439 DEBUG SystemMonitor:290 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-05-22 14:14:54,439 DEBUG SystemMonitor:290 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-05-22 14:14:54,440 INFO HandlerThread:290 [interfaces.py:finish():202] Joined cpu monitor 2024-05-22 14:14:54,440 INFO HandlerThread:290 [interfaces.py:finish():202] Joined disk monitor 2024-05-22 14:14:54,446 INFO HandlerThread:290 [interfaces.py:finish():202] Joined gpu monitor 2024-05-22 14:14:54,446 INFO HandlerThread:290 [interfaces.py:finish():202] Joined memory monitor 2024-05-22 14:14:54,447 INFO HandlerThread:290 [interfaces.py:finish():202] Joined network monitor 2024-05-22 14:14:54,447 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 14:14:54,447 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:14:56,745 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/output.log 2024-05-22 14:14:59,449 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:15:01,236 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:15:01,236 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:15:01,237 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:15:05,347 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:15:10,348 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:15:15,350 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:15:16,236 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:15:16,236 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:15:16,236 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:15:20,350 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:15:25,351 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:15:26,187 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: resume 2024-05-22 14:15:26,187 INFO HandlerThread:290 [handler.py:handle_request_resume():699] starting system metrics thread 2024-05-22 14:15:26,187 INFO HandlerThread:290 [system_monitor.py:start():194] Starting system monitor 2024-05-22 14:15:26,187 INFO SystemMonitor:290 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-05-22 14:15:26,188 INFO SystemMonitor:290 [interfaces.py:start():190] Started cpu monitoring 2024-05-22 14:15:26,189 INFO SystemMonitor:290 [interfaces.py:start():190] Started disk monitoring 2024-05-22 14:15:26,190 INFO SystemMonitor:290 [interfaces.py:start():190] Started gpu monitoring 2024-05-22 14:15:26,191 INFO SystemMonitor:290 [interfaces.py:start():190] Started memory monitoring 2024-05-22 14:15:26,192 INFO SystemMonitor:290 [interfaces.py:start():190] Started network monitoring 2024-05-22 14:15:27,757 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/output.log 2024-05-22 14:15:30,375 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:15:30,758 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/output.log 2024-05-22 14:15:31,236 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:15:31,237 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:15:31,237 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:15:35,381 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:15:40,417 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:15:45,447 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:15:46,236 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:15:46,237 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:15:46,237 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:15:50,491 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:15:55,523 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:16:00,569 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:16:01,237 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:16:01,237 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:16:01,239 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:16:05,598 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:16:10,637 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:16:15,638 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:16:16,237 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:16:16,237 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:16:16,238 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:16:20,673 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:16:25,349 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: pause 2024-05-22 14:16:25,349 INFO HandlerThread:290 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-05-22 14:16:25,349 INFO HandlerThread:290 [system_monitor.py:finish():203] Stopping system monitor 2024-05-22 14:16:25,349 DEBUG SystemMonitor:290 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-05-22 14:16:25,349 DEBUG SystemMonitor:290 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-05-22 14:16:25,349 DEBUG SystemMonitor:290 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-05-22 14:16:25,350 INFO HandlerThread:290 [interfaces.py:finish():202] Joined cpu monitor 2024-05-22 14:16:25,351 INFO HandlerThread:290 [interfaces.py:finish():202] Joined disk monitor 2024-05-22 14:16:25,357 INFO HandlerThread:290 [interfaces.py:finish():202] Joined gpu monitor 2024-05-22 14:16:25,357 INFO HandlerThread:290 [interfaces.py:finish():202] Joined memory monitor 2024-05-22 14:16:25,358 INFO HandlerThread:290 [interfaces.py:finish():202] Joined network monitor 2024-05-22 14:16:25,358 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 14:16:26,359 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:16:26,781 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/output.log 2024-05-22 14:16:29,691 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: resume 2024-05-22 14:16:29,692 INFO HandlerThread:290 [handler.py:handle_request_resume():699] starting system metrics thread 2024-05-22 14:16:29,692 INFO HandlerThread:290 [system_monitor.py:start():194] Starting system monitor 2024-05-22 14:16:29,692 INFO SystemMonitor:290 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-05-22 14:16:29,693 INFO SystemMonitor:290 [interfaces.py:start():190] Started cpu monitoring 2024-05-22 14:16:29,693 INFO SystemMonitor:290 [interfaces.py:start():190] Started disk monitoring 2024-05-22 14:16:29,694 INFO SystemMonitor:290 [interfaces.py:start():190] Started gpu monitoring 2024-05-22 14:16:29,695 INFO SystemMonitor:290 [interfaces.py:start():190] Started memory monitoring 2024-05-22 14:16:29,695 INFO SystemMonitor:290 [interfaces.py:start():190] Started network monitoring 2024-05-22 14:16:31,237 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:16:31,238 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:16:31,238 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:16:31,370 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:16:31,783 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/output.log 2024-05-22 14:16:32,784 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/output.log 2024-05-22 14:16:36,400 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:16:41,446 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:16:44,750 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: pause 2024-05-22 14:16:44,750 INFO HandlerThread:290 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-05-22 14:16:44,750 INFO HandlerThread:290 [system_monitor.py:finish():203] Stopping system monitor 2024-05-22 14:16:44,751 DEBUG SystemMonitor:290 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-05-22 14:16:44,751 DEBUG SystemMonitor:290 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-05-22 14:16:44,751 DEBUG SystemMonitor:290 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-05-22 14:16:44,751 INFO HandlerThread:290 [interfaces.py:finish():202] Joined cpu monitor 2024-05-22 14:16:44,752 INFO HandlerThread:290 [interfaces.py:finish():202] Joined disk monitor 2024-05-22 14:16:44,758 INFO HandlerThread:290 [interfaces.py:finish():202] Joined gpu monitor 2024-05-22 14:16:44,759 INFO HandlerThread:290 [interfaces.py:finish():202] Joined memory monitor 2024-05-22 14:16:44,759 INFO HandlerThread:290 [interfaces.py:finish():202] Joined network monitor 2024-05-22 14:16:44,759 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 14:16:46,237 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:16:46,238 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:16:46,238 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:16:46,790 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/output.log 2024-05-22 14:16:47,404 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:16:52,405 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:16:57,406 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:17:01,238 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:17:01,238 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:17:01,238 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:17:03,348 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:17:08,349 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:17:13,350 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:17:16,238 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:17:16,238 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:17:16,239 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:17:18,408 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:17:23,409 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:17:28,410 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:17:31,239 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:17:31,239 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:17:31,239 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:17:34,381 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:17:39,382 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:17:44,383 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:17:46,238 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:17:46,238 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:17:46,239 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:17:50,358 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:17:55,358 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:18:00,360 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:18:01,239 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:18:01,239 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:18:01,239 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:18:06,338 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:18:11,340 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:18:16,239 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:18:16,239 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:18:16,279 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:18:16,375 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:18:21,377 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:18:26,378 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:18:31,239 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:18:31,239 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:18:31,240 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:18:31,400 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:18:36,400 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:18:41,402 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:18:46,239 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:18:46,239 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:18:46,240 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:18:47,351 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:18:52,352 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:18:57,353 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:19:01,239 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:19:01,239 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:19:01,280 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:19:02,417 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:19:07,417 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:19:12,418 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:19:16,239 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:19:16,240 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:19:16,240 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:19:18,347 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:19:23,348 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:19:28,349 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:19:31,239 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:19:31,240 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:19:31,240 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:19:33,364 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:19:38,365 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:19:43,366 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:19:46,239 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:19:46,240 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:19:46,240 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:19:49,356 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:19:54,357 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:19:59,357 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:20:01,240 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:20:01,240 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:20:01,241 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:20:05,312 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:20:10,313 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:20:15,314 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:20:16,240 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:20:16,240 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:20:16,241 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:20:20,367 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:20:25,368 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:20:30,369 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:20:31,240 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:20:31,240 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:20:31,241 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:20:35,419 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:20:40,420 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:20:45,421 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:20:46,240 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:20:46,240 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:20:46,241 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:20:51,315 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:20:56,316 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:21:01,241 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:21:01,241 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:21:01,241 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:21:02,305 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:21:07,306 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:21:12,307 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:21:16,240 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:21:16,241 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:21:16,242 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:21:17,352 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:21:22,354 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:21:27,355 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:21:31,241 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:21:31,241 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:21:31,242 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:21:32,911 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:21:37,912 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:21:42,913 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:21:46,241 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:21:46,241 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:21:46,242 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:21:48,342 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:21:53,344 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:21:58,345 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:22:01,241 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:22:01,241 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:22:01,242 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:22:04,322 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:22:09,323 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:22:14,324 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:22:16,241 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:22:16,242 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:22:16,242 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:22:19,354 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:22:24,356 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:22:29,357 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:22:31,242 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:22:31,242 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:22:31,243 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:22:35,339 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:22:40,340 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:22:45,341 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:22:46,242 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:22:46,242 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:22:46,243 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:22:50,407 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:22:55,408 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:23:00,409 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:23:01,242 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:23:01,242 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:23:01,243 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:23:06,165 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:23:11,167 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:23:14,137 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: resume 2024-05-22 14:23:14,137 INFO HandlerThread:290 [handler.py:handle_request_resume():699] starting system metrics thread 2024-05-22 14:23:14,137 INFO HandlerThread:290 [system_monitor.py:start():194] Starting system monitor 2024-05-22 14:23:14,137 INFO SystemMonitor:290 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-05-22 14:23:14,138 INFO SystemMonitor:290 [interfaces.py:start():190] Started cpu monitoring 2024-05-22 14:23:14,139 INFO SystemMonitor:290 [interfaces.py:start():190] Started disk monitoring 2024-05-22 14:23:14,141 INFO SystemMonitor:290 [interfaces.py:start():190] Started gpu monitoring 2024-05-22 14:23:14,142 INFO SystemMonitor:290 [interfaces.py:start():190] Started memory monitoring 2024-05-22 14:23:14,142 INFO SystemMonitor:290 [interfaces.py:start():190] Started network monitoring 2024-05-22 14:23:16,168 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:23:16,248 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:23:16,249 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:23:16,249 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:23:16,353 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: pause 2024-05-22 14:23:16,354 INFO HandlerThread:290 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-05-22 14:23:16,354 INFO HandlerThread:290 [system_monitor.py:finish():203] Stopping system monitor 2024-05-22 14:23:16,354 DEBUG SystemMonitor:290 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-05-22 14:23:16,354 DEBUG SystemMonitor:290 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-05-22 14:23:16,354 DEBUG SystemMonitor:290 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-05-22 14:23:16,355 INFO HandlerThread:290 [interfaces.py:finish():202] Joined cpu monitor 2024-05-22 14:23:16,355 INFO HandlerThread:290 [interfaces.py:finish():202] Joined disk monitor 2024-05-22 14:23:16,361 INFO HandlerThread:290 [interfaces.py:finish():202] Joined gpu monitor 2024-05-22 14:23:16,362 INFO HandlerThread:290 [interfaces.py:finish():202] Joined memory monitor 2024-05-22 14:23:16,362 INFO HandlerThread:290 [interfaces.py:finish():202] Joined network monitor 2024-05-22 14:23:16,377 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 14:23:21,379 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:23:26,380 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:23:31,248 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:23:31,248 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:23:31,248 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:23:31,389 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:23:36,391 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:23:41,393 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:23:46,248 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:23:46,248 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:23:46,248 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:23:47,375 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:23:52,376 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:23:57,377 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:24:01,248 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:24:01,248 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:24:01,249 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:24:02,399 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:24:07,400 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:24:12,401 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:24:16,247 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:24:16,248 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:24:16,249 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:24:18,322 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:24:23,323 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:24:28,324 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:24:31,249 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:24:31,249 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:24:31,249 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:24:33,380 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:24:38,381 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:24:43,383 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:24:46,248 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:24:46,248 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:24:46,249 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:24:48,387 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:24:53,388 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:24:58,390 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:25:01,249 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:25:01,249 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:25:01,250 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:25:04,347 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:25:09,348 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:25:14,349 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:25:16,248 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:25:16,249 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:25:16,250 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:25:19,387 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:25:24,388 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:25:29,389 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:25:31,249 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:25:31,249 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:25:31,250 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:25:35,319 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:25:40,320 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:25:45,321 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:25:46,249 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:25:46,249 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:25:46,250 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:25:50,417 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:25:55,419 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:26:00,420 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:26:01,249 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:26:01,250 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:26:01,250 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:26:06,331 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:26:11,333 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:26:16,249 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:26:16,249 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:26:16,250 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:26:16,349 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:26:18,733 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: resume 2024-05-22 14:26:18,733 INFO HandlerThread:290 [handler.py:handle_request_resume():699] starting system metrics thread 2024-05-22 14:26:18,733 INFO HandlerThread:290 [system_monitor.py:start():194] Starting system monitor 2024-05-22 14:26:18,733 INFO SystemMonitor:290 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-05-22 14:26:18,734 INFO SystemMonitor:290 [interfaces.py:start():190] Started cpu monitoring 2024-05-22 14:26:18,734 INFO SystemMonitor:290 [interfaces.py:start():190] Started disk monitoring 2024-05-22 14:26:18,735 INFO SystemMonitor:290 [interfaces.py:start():190] Started gpu monitoring 2024-05-22 14:26:18,736 INFO SystemMonitor:290 [interfaces.py:start():190] Started memory monitoring 2024-05-22 14:26:18,738 INFO SystemMonitor:290 [interfaces.py:start():190] Started network monitoring 2024-05-22 14:26:19,624 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: pause 2024-05-22 14:26:19,624 INFO HandlerThread:290 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-05-22 14:26:19,624 INFO HandlerThread:290 [system_monitor.py:finish():203] Stopping system monitor 2024-05-22 14:26:19,625 INFO HandlerThread:290 [interfaces.py:finish():202] Joined cpu monitor 2024-05-22 14:26:19,625 DEBUG SystemMonitor:290 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-05-22 14:26:19,625 DEBUG SystemMonitor:290 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-05-22 14:26:19,625 DEBUG SystemMonitor:290 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-05-22 14:26:19,627 INFO HandlerThread:290 [interfaces.py:finish():202] Joined disk monitor 2024-05-22 14:26:19,632 INFO HandlerThread:290 [interfaces.py:finish():202] Joined gpu monitor 2024-05-22 14:26:19,633 INFO HandlerThread:290 [interfaces.py:finish():202] Joined memory monitor 2024-05-22 14:26:19,633 INFO HandlerThread:290 [interfaces.py:finish():202] Joined network monitor 2024-05-22 14:26:19,633 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 14:26:20,037 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/output.log 2024-05-22 14:26:21,634 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:26:26,636 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:26:31,249 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:26:31,250 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:26:31,250 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:26:32,333 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:26:37,334 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:26:42,336 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:26:46,250 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:26:46,250 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:26:46,250 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:26:47,339 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:26:52,340 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:26:57,341 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:27:01,250 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:27:01,250 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:27:01,251 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:27:02,431 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:27:07,432 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:27:12,433 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:27:16,250 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:27:16,250 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:27:16,251 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:27:18,327 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:27:23,328 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:27:28,329 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:27:31,250 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:27:31,251 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:27:31,251 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:27:33,406 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:27:38,407 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:27:43,408 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:27:46,250 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:27:46,251 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:27:46,251 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:27:49,353 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:27:54,355 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:27:59,356 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:28:01,250 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:28:01,251 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:28:01,251 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:28:04,407 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:28:09,408 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:28:14,409 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:28:16,251 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:28:16,251 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:28:16,251 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:28:20,361 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:28:25,362 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:28:30,363 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:28:31,251 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:28:31,251 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:28:31,252 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:28:35,406 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:28:40,407 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:28:45,408 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:28:46,251 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:28:46,251 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:28:46,252 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:28:51,325 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:28:56,327 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:29:01,251 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:29:01,251 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:29:01,252 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:29:01,433 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:29:06,434 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:29:11,436 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:29:16,251 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:29:16,252 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:29:16,252 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:29:17,404 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:29:22,406 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:29:27,407 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:29:31,251 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:29:31,252 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:29:31,252 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:29:33,395 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:29:38,396 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:29:43,397 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:29:46,252 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:29:46,252 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:29:46,252 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:29:49,324 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:29:54,325 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:29:59,327 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:30:01,252 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:30:01,252 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:30:01,253 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:30:04,420 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:30:09,422 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:30:14,423 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:30:16,252 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:30:16,253 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:30:16,253 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:30:20,348 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:30:25,349 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:30:30,350 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:30:31,252 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:30:31,253 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:30:31,253 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:30:35,358 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:30:40,360 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:30:45,361 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:30:46,253 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:30:46,253 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:30:46,253 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:30:50,375 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:30:55,377 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:31:00,378 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:31:01,253 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:31:01,253 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:31:01,254 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:31:05,419 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:31:10,420 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:31:15,421 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:31:16,253 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:31:16,254 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:31:16,254 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:31:21,323 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:31:26,324 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:31:31,253 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:31:31,254 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:31:31,254 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:31:31,383 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:31:36,385 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:31:41,386 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:31:46,254 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:31:46,254 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:31:46,294 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:31:47,361 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:31:52,362 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:31:57,363 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:32:01,254 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:32:01,254 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:32:01,254 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:32:02,414 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:32:07,415 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:32:12,416 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:32:16,254 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:32:16,254 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:32:16,255 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:32:18,329 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:32:23,330 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:32:28,331 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:32:31,254 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:32:31,255 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:32:31,255 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:32:33,416 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:32:38,417 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:32:43,418 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:32:46,254 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:32:46,255 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:32:46,255 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:32:48,429 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:32:53,431 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:32:58,432 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:33:01,255 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:33:01,255 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:33:01,255 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:33:04,379 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:33:09,380 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:33:14,381 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:33:16,254 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:33:16,255 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:33:16,256 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:33:19,416 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:33:24,417 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:33:29,418 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:33:31,255 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:33:31,255 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:33:31,256 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:33:34,425 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:33:39,426 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:33:44,427 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:33:46,255 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:33:46,255 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:33:46,256 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:33:50,338 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:33:55,339 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:34:00,340 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:34:00,493 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: resume 2024-05-22 14:34:00,494 INFO HandlerThread:290 [handler.py:handle_request_resume():699] starting system metrics thread 2024-05-22 14:34:00,494 INFO HandlerThread:290 [system_monitor.py:start():194] Starting system monitor 2024-05-22 14:34:00,494 INFO SystemMonitor:290 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-05-22 14:34:00,495 INFO SystemMonitor:290 [interfaces.py:start():190] Started cpu monitoring 2024-05-22 14:34:00,496 INFO SystemMonitor:290 [interfaces.py:start():190] Started disk monitoring 2024-05-22 14:34:00,497 INFO SystemMonitor:290 [interfaces.py:start():190] Started gpu monitoring 2024-05-22 14:34:00,498 INFO SystemMonitor:290 [interfaces.py:start():190] Started memory monitoring 2024-05-22 14:34:00,498 INFO SystemMonitor:290 [interfaces.py:start():190] Started network monitoring 2024-05-22 14:34:00,985 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: pause 2024-05-22 14:34:00,985 INFO HandlerThread:290 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-05-22 14:34:00,985 INFO HandlerThread:290 [system_monitor.py:finish():203] Stopping system monitor 2024-05-22 14:34:00,985 DEBUG SystemMonitor:290 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-05-22 14:34:00,986 INFO HandlerThread:290 [interfaces.py:finish():202] Joined cpu monitor 2024-05-22 14:34:00,986 DEBUG SystemMonitor:290 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-05-22 14:34:00,986 INFO HandlerThread:290 [interfaces.py:finish():202] Joined disk monitor 2024-05-22 14:34:00,986 DEBUG SystemMonitor:290 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-05-22 14:34:00,995 INFO HandlerThread:290 [interfaces.py:finish():202] Joined gpu monitor 2024-05-22 14:34:00,995 INFO HandlerThread:290 [interfaces.py:finish():202] Joined memory monitor 2024-05-22 14:34:00,995 INFO HandlerThread:290 [interfaces.py:finish():202] Joined network monitor 2024-05-22 14:34:00,996 DEBUG SenderThread:290 [sender.py:send():379] send: stats 2024-05-22 14:34:01,255 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:34:01,256 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:34:01,256 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:34:03,236 INFO Thread-12 :290 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240522_113413-8mudzhjp/files/output.log 2024-05-22 14:34:06,337 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:34:11,338 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:34:16,255 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: internal_messages 2024-05-22 14:34:16,256 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: stop_status 2024-05-22 14:34:16,256 DEBUG SenderThread:290 [sender.py:send_request():406] send_request: stop_status 2024-05-22 14:34:16,354 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: status_report 2024-05-22 14:34:16,411 DEBUG HandlerThread:290 [handler.py:handle_request():146] handle_request: resume 2024-05-22 14:34:16,411 INFO HandlerThread:290 [handler.py:handle_request_resume():699] starting system metrics thread 2024-05-22 14:34:16,411 INFO HandlerThread:290 [system_monitor.py:start():194] Starting system monitor 2024-05-22 14:34:16,411 INFO SystemMonitor:290 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-05-22 14:34:16,412 INFO SystemMonitor:290 [interfaces.py:start():190] Started cpu monitoring 2024-05-22 14:34:16,412 INFO SystemMonitor:290 [interfaces.py:start():190] Started disk monitoring 2024-05-22 14:34:16,413 INFO SystemMonitor:290 [interfaces.py:start():190] Started gpu monitoring 2024-05-22 14:34:16,414 INFO SystemMonitor:290 [interfaces.py:start():190] Started memory monitoring 2024-05-22 14:34:16,415 INFO SystemMonitor:290 [interfaces.py:start():190] Started network monitoring