Spaces:
Runtime error
Runtime error
| 2024-02-10 21:13:29,170 INFO StreamThr :3896 [internal.py:wandb_internal():86] W&B internal server running at pid: 3896, started at: 2024-02-10 21:13:29.169015 | |
| 2024-02-10 21:13:29,173 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status | |
| 2024-02-10 21:13:29,174 INFO WriterThread:3896 [datastore.py:open_for_write():87] open: c:\Users\hiroga\Documents\GitHub\til\computer-science\machine-learning\_src\pokemon-palworld\notebooks\wandb\run-20240210_211321-s7dtxxtb\run-s7dtxxtb.wandb | |
| 2024-02-10 21:13:29,175 DEBUG SenderThread:3896 [sender.py:send():382] send: header | |
| 2024-02-10 21:13:29,183 DEBUG SenderThread:3896 [sender.py:send():382] send: run | |
| 2024-02-10 21:13:29,743 INFO SenderThread:3896 [dir_watcher.py:__init__():211] watching files in: c:\Users\hiroga\Documents\GitHub\til\computer-science\machine-learning\_src\pokemon-palworld\notebooks\wandb\run-20240210_211321-s7dtxxtb\files | |
| 2024-02-10 21:13:29,744 INFO SenderThread:3896 [sender.py:_start_run_threads():1136] run started: s7dtxxtb with start time 1707567208.843146 | |
| 2024-02-10 21:13:29,772 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: check_version | |
| 2024-02-10 21:13:29,772 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: check_version | |
| 2024-02-10 21:13:30,022 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: run_start | |
| 2024-02-10 21:13:30,029 DEBUG HandlerThread:3896 [system_info.py:__init__():27] System info init | |
| 2024-02-10 21:13:30,029 DEBUG HandlerThread:3896 [system_info.py:__init__():42] System info init done | |
| 2024-02-10 21:13:30,029 INFO HandlerThread:3896 [system_monitor.py:start():194] Starting system monitor | |
| 2024-02-10 21:13:30,030 INFO SystemMonitor:3896 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-02-10 21:13:30,030 INFO HandlerThread:3896 [system_monitor.py:probe():214] Collecting system info | |
| 2024-02-10 21:13:30,033 INFO SystemMonitor:3896 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-02-10 21:13:30,034 INFO SystemMonitor:3896 [interfaces.py:start():190] Started disk monitoring | |
| 2024-02-10 21:13:30,036 DEBUG HandlerThread:3896 [system_info.py:probe():151] Probing system | |
| 2024-02-10 21:13:30,037 INFO SystemMonitor:3896 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-02-10 21:13:30,045 INFO SystemMonitor:3896 [interfaces.py:start():190] Started memory monitoring | |
| 2024-02-10 21:13:30,051 INFO SystemMonitor:3896 [interfaces.py:start():190] Started network monitoring | |
| 2024-02-10 21:13:30,052 DEBUG HandlerThread:3896 [system_info.py:_probe_git():136] Probing git | |
| 2024-02-10 21:13:30,102 DEBUG HandlerThread:3896 [system_info.py:_probe_git():144] Probing git done | |
| 2024-02-10 21:13:30,102 DEBUG HandlerThread:3896 [system_info.py:probe():199] Probing system done | |
| 2024-02-10 21:13:30,102 DEBUG HandlerThread:3896 [system_monitor.py:probe():223] {'os': 'Windows-11-10.0.22621-SP0', 'python': '3.12.1', 'heartbeatAt': '2024-02-10T12:13:30.037960', 'startedAt': '2024-02-10T12:13:21.136668', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': '<python with no main file>', 'codePathLocal': None, 'git': {'remote': 'https://github.com/xhiroga/til.git', 'commit': '17367d9d9913edc9a0a685e866a07120b938e31e'}, 'email': '13391129+xhiroga@users.noreply.github.com', 'root': 'C:/Users/hiroga/Documents/GitHub/til', 'host': 'HIROGA-RTX4090', 'username': 'hiroga', 'executable': 'c:\\Users\\hiroga\\miniconda3\\envs\\pokemon-palworld-v2\\python.exe', 'cpu_count': 14, 'cpu_count_logical': 20, 'cpu_freq': {'current': 2500.0, 'min': 0.0, 'max': 2500.0}, 'cpu_freq_per_core': [{'current': 2500.0, 'min': 0.0, 'max': 2500.0}], 'disk': {'/': {'total': 1862.2490196228027, 'used': 953.5622978210449}}, 'gpu': 'NVIDIA GeForce RTX 4090', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 4090', 'memory_total': 25757220864}], 'memory': {'total': 31.770225524902344}} | |
| 2024-02-10 21:13:30,102 INFO HandlerThread:3896 [system_monitor.py:probe():224] Finished collecting system info | |
| 2024-02-10 21:13:30,102 INFO HandlerThread:3896 [system_monitor.py:probe():227] Publishing system info | |
| 2024-02-10 21:13:30,103 DEBUG HandlerThread:3896 [system_info.py:_save_conda():208] Saving list of conda packages installed into the current environment | |
| 2024-02-10 21:13:30,104 ERROR HandlerThread:3896 [system_info.py:_save_conda():219] Error saving conda packages: [WinError 2] The system cannot find the file specified | |
| Traceback (most recent call last): | |
| File "c:\Users\hiroga\miniconda3\envs\pokemon-palworld-v2\Lib\site-packages\wandb\sdk\internal\system\system_info.py", line 215, in _save_conda | |
| subprocess.call( | |
| File "c:\Users\hiroga\miniconda3\envs\pokemon-palworld-v2\Lib\subprocess.py", line 389, in call | |
| with Popen(*popenargs, **kwargs) as p: | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "c:\Users\hiroga\miniconda3\envs\pokemon-palworld-v2\Lib\subprocess.py", line 1026, in __init__ | |
| self._execute_child(args, executable, preexec_fn, close_fds, | |
| File "c:\Users\hiroga\miniconda3\envs\pokemon-palworld-v2\Lib\subprocess.py", line 1538, in _execute_child | |
| hp, ht, pid, tid = _winapi.CreateProcess(executable, args, | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| FileNotFoundError: [WinError 2] The system cannot find the file specified | |
| 2024-02-10 21:13:30,105 DEBUG HandlerThread:3896 [system_info.py:_save_conda():220] Saving conda packages done | |
| 2024-02-10 21:13:30,115 INFO HandlerThread:3896 [system_monitor.py:probe():229] Finished publishing system info | |
| 2024-02-10 21:13:30,115 DEBUG SenderThread:3896 [sender.py:send():382] send: files | |
| 2024-02-10 21:13:30,116 INFO SenderThread:3896 [sender.py:_save_file():1403] saving file wandb-metadata.json with policy now | |
| 2024-02-10 21:13:30,136 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: python_packages | |
| 2024-02-10 21:13:30,136 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: python_packages | |
| 2024-02-10 21:13:30,139 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-10 21:13:30,140 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-10 21:13:30,143 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-10 21:13:30,529 DEBUG SenderThread:3896 [sender.py:send():382] send: telemetry | |
| 2024-02-10 21:13:30,745 INFO Thread-32 :3896 [dir_watcher.py:_on_file_created():271] file/dir created: c:\Users\hiroga\Documents\GitHub\til\computer-science\machine-learning\_src\pokemon-palworld\notebooks\wandb\run-20240210_211321-s7dtxxtb\files\output.log | |
| 2024-02-10 21:13:30,745 INFO Thread-32 :3896 [dir_watcher.py:_on_file_created():271] file/dir created: c:\Users\hiroga\Documents\GitHub\til\computer-science\machine-learning\_src\pokemon-palworld\notebooks\wandb\run-20240210_211321-s7dtxxtb\files\requirements.txt | |
| 2024-02-10 21:13:30,745 INFO Thread-32 :3896 [dir_watcher.py:_on_file_created():271] file/dir created: c:\Users\hiroga\Documents\GitHub\til\computer-science\machine-learning\_src\pokemon-palworld\notebooks\wandb\run-20240210_211321-s7dtxxtb\files\conda-environment.yaml | |
| 2024-02-10 21:13:30,745 INFO Thread-32 :3896 [dir_watcher.py:_on_file_created():271] file/dir created: c:\Users\hiroga\Documents\GitHub\til\computer-science\machine-learning\_src\pokemon-palworld\notebooks\wandb\run-20240210_211321-s7dtxxtb\files\wandb-metadata.json | |
| 2024-02-10 21:13:30,969 INFO wandb-upload_0:3896 [upload_job.py:push():131] Uploaded file C:\Users\hiroga\AppData\Local\Temp\tmph4b94e87wandb\6gfnu8ay-wandb-metadata.json | |
| 2024-02-10 21:13:32,762 INFO Thread-32 :3896 [dir_watcher.py:_on_file_modified():288] file/dir modified: c:\Users\hiroga\Documents\GitHub\til\computer-science\machine-learning\_src\pokemon-palworld\notebooks\wandb\run-20240210_211321-s7dtxxtb\files\output.log | |
| 2024-02-10 21:13:34,832 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:13:39,895 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:13:44,963 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:13:45,149 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-10 21:13:45,150 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-10 21:13:45,150 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-10 21:13:50,444 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:13:55,480 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:14:00,163 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-10 21:14:00,164 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-10 21:14:00,165 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-10 21:14:01,372 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:14:06,127 INFO Thread-32 :3896 [dir_watcher.py:_on_file_modified():288] file/dir modified: c:\Users\hiroga\Documents\GitHub\til\computer-science\machine-learning\_src\pokemon-palworld\notebooks\wandb\run-20240210_211321-s7dtxxtb\files\config.yaml | |
| 2024-02-10 21:14:10,455 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:14:15,165 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-10 21:14:15,167 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-10 21:14:15,182 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-10 21:14:16,440 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:14:21,468 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:14:26,526 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:14:30,054 DEBUG SystemMonitor:3896 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-02-10 21:14:30,059 DEBUG SenderThread:3896 [sender.py:send():382] send: stats | |
| 2024-02-10 21:14:30,180 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-10 21:14:30,180 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-10 21:14:30,197 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-10 21:14:32,379 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:14:37,440 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:14:42,471 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:14:45,177 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-10 21:14:45,178 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-10 21:14:45,209 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-10 21:14:48,473 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:14:53,537 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:14:58,570 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:15:00,066 DEBUG SenderThread:3896 [sender.py:send():382] send: stats | |
| 2024-02-10 21:15:00,192 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-10 21:15:00,194 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-10 21:15:00,225 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-10 21:15:04,427 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:15:09,488 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:15:14,548 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:15:15,202 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-10 21:15:15,205 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-10 21:15:15,233 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-10 21:15:20,440 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:15:25,490 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:15:30,081 DEBUG SenderThread:3896 [sender.py:send():382] send: stats | |
| 2024-02-10 21:15:30,208 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-10 21:15:30,209 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-10 21:15:30,223 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-10 21:15:31,440 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:15:36,496 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:15:41,535 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:15:45,217 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-10 21:15:45,218 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-10 21:15:45,231 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-10 21:15:47,487 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:15:52,542 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:15:57,582 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:16:00,093 DEBUG SenderThread:3896 [sender.py:send():382] send: stats | |
| 2024-02-10 21:16:00,215 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-10 21:16:00,215 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-10 21:16:00,232 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-10 21:16:03,472 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:16:08,530 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:16:13,591 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:16:15,225 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-10 21:16:15,229 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-10 21:16:15,230 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-10 21:16:19,524 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:16:24,584 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:16:29,636 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:16:30,104 DEBUG SenderThread:3896 [sender.py:send():382] send: stats | |
| 2024-02-10 21:16:30,228 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-10 21:16:30,229 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-10 21:16:30,243 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-10 21:16:35,541 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:16:40,583 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:16:45,226 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-10 21:16:45,227 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-10 21:16:45,255 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-10 21:16:46,471 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:16:51,520 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:16:56,584 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:17:00,105 DEBUG SenderThread:3896 [sender.py:send():382] send: stats | |
| 2024-02-10 21:17:00,226 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-10 21:17:00,226 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-10 21:17:00,257 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-10 21:17:02,474 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:17:07,502 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:17:12,567 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:17:15,230 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-10 21:17:15,231 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-10 21:17:15,261 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-10 21:17:18,458 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:17:22,966 DEBUG SenderThread:3896 [sender.py:send():382] send: telemetry | |
| 2024-02-10 21:17:22,969 DEBUG SenderThread:3896 [sender.py:send():382] send: exit | |
| 2024-02-10 21:17:22,969 INFO SenderThread:3896 [sender.py:send_exit():589] handling exit code: 0 | |
| 2024-02-10 21:17:22,969 INFO SenderThread:3896 [sender.py:send_exit():591] handling runtime: 232 | |
| 2024-02-10 21:17:22,970 INFO SenderThread:3896 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-10 21:17:22,970 INFO SenderThread:3896 [sender.py:send_exit():597] send defer | |
| 2024-02-10 21:17:22,971 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: defer | |
| 2024-02-10 21:17:22,971 INFO HandlerThread:3896 [handler.py:handle_request_defer():172] handle defer: 0 | |
| 2024-02-10 21:17:22,972 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: defer | |
| 2024-02-10 21:17:22,972 INFO SenderThread:3896 [sender.py:send_request_defer():613] handle sender defer: 0 | |
| 2024-02-10 21:17:22,972 INFO SenderThread:3896 [sender.py:transition_state():617] send defer: 1 | |
| 2024-02-10 21:17:22,972 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: defer | |
| 2024-02-10 21:17:22,972 INFO HandlerThread:3896 [handler.py:handle_request_defer():172] handle defer: 1 | |
| 2024-02-10 21:17:22,972 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: defer | |
| 2024-02-10 21:17:22,972 INFO SenderThread:3896 [sender.py:send_request_defer():613] handle sender defer: 1 | |
| 2024-02-10 21:17:22,972 INFO SenderThread:3896 [sender.py:transition_state():617] send defer: 2 | |
| 2024-02-10 21:17:22,973 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: defer | |
| 2024-02-10 21:17:22,973 INFO HandlerThread:3896 [handler.py:handle_request_defer():172] handle defer: 2 | |
| 2024-02-10 21:17:22,973 INFO HandlerThread:3896 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-02-10 21:17:22,973 DEBUG SystemMonitor:3896 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-02-10 21:17:22,980 DEBUG SystemMonitor:3896 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-02-10 21:17:22,980 INFO HandlerThread:3896 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-02-10 21:17:22,986 INFO HandlerThread:3896 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-02-10 21:17:23,020 INFO HandlerThread:3896 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-02-10 21:17:23,020 INFO HandlerThread:3896 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-02-10 21:17:23,020 INFO HandlerThread:3896 [interfaces.py:finish():202] Joined network monitor | |
| 2024-02-10 21:17:23,020 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: defer | |
| 2024-02-10 21:17:23,020 INFO SenderThread:3896 [sender.py:send_request_defer():613] handle sender defer: 2 | |
| 2024-02-10 21:17:23,020 INFO SenderThread:3896 [sender.py:transition_state():617] send defer: 3 | |
| 2024-02-10 21:17:23,021 DEBUG SenderThread:3896 [sender.py:send():382] send: stats | |
| 2024-02-10 21:17:23,021 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: defer | |
| 2024-02-10 21:17:23,021 INFO HandlerThread:3896 [handler.py:handle_request_defer():172] handle defer: 3 | |
| 2024-02-10 21:17:23,021 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: defer | |
| 2024-02-10 21:17:23,021 INFO SenderThread:3896 [sender.py:send_request_defer():613] handle sender defer: 3 | |
| 2024-02-10 21:17:23,021 INFO SenderThread:3896 [sender.py:transition_state():617] send defer: 4 | |
| 2024-02-10 21:17:23,021 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: defer | |
| 2024-02-10 21:17:23,021 INFO HandlerThread:3896 [handler.py:handle_request_defer():172] handle defer: 4 | |
| 2024-02-10 21:17:23,022 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: defer | |
| 2024-02-10 21:17:23,022 INFO SenderThread:3896 [sender.py:send_request_defer():613] handle sender defer: 4 | |
| 2024-02-10 21:17:23,022 INFO SenderThread:3896 [sender.py:transition_state():617] send defer: 5 | |
| 2024-02-10 21:17:23,022 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: defer | |
| 2024-02-10 21:17:23,022 INFO HandlerThread:3896 [handler.py:handle_request_defer():172] handle defer: 5 | |
| 2024-02-10 21:17:23,022 DEBUG SenderThread:3896 [sender.py:send():382] send: summary | |
| 2024-02-10 21:17:23,022 INFO SenderThread:3896 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-10 21:17:23,023 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: defer | |
| 2024-02-10 21:17:23,023 INFO SenderThread:3896 [sender.py:send_request_defer():613] handle sender defer: 5 | |
| 2024-02-10 21:17:23,023 INFO SenderThread:3896 [sender.py:transition_state():617] send defer: 6 | |
| 2024-02-10 21:17:23,023 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: defer | |
| 2024-02-10 21:17:23,023 INFO HandlerThread:3896 [handler.py:handle_request_defer():172] handle defer: 6 | |
| 2024-02-10 21:17:23,023 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: defer | |
| 2024-02-10 21:17:23,023 INFO SenderThread:3896 [sender.py:send_request_defer():613] handle sender defer: 6 | |
| 2024-02-10 21:17:23,025 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-10 21:17:23,209 INFO SenderThread:3896 [sender.py:transition_state():617] send defer: 7 | |
| 2024-02-10 21:17:23,209 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: defer | |
| 2024-02-10 21:17:23,209 INFO HandlerThread:3896 [handler.py:handle_request_defer():172] handle defer: 7 | |
| 2024-02-10 21:17:23,210 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: defer | |
| 2024-02-10 21:17:23,210 INFO SenderThread:3896 [sender.py:send_request_defer():613] handle sender defer: 7 | |
| 2024-02-10 21:17:23,282 INFO Thread-32 :3896 [dir_watcher.py:_on_file_modified():288] file/dir modified: c:\Users\hiroga\Documents\GitHub\til\computer-science\machine-learning\_src\pokemon-palworld\notebooks\wandb\run-20240210_211321-s7dtxxtb\files\config.yaml | |
| 2024-02-10 21:17:23,283 INFO Thread-32 :3896 [dir_watcher.py:_on_file_created():271] file/dir created: c:\Users\hiroga\Documents\GitHub\til\computer-science\machine-learning\_src\pokemon-palworld\notebooks\wandb\run-20240210_211321-s7dtxxtb\files\wandb-summary.json | |
| 2024-02-10 21:17:23,984 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: poll_exit | |
| 2024-02-10 21:17:24,613 INFO SenderThread:3896 [sender.py:transition_state():617] send defer: 8 | |
| 2024-02-10 21:17:24,613 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: poll_exit | |
| 2024-02-10 21:17:24,613 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: defer | |
| 2024-02-10 21:17:24,614 INFO HandlerThread:3896 [handler.py:handle_request_defer():172] handle defer: 8 | |
| 2024-02-10 21:17:24,614 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: defer | |
| 2024-02-10 21:17:24,614 INFO SenderThread:3896 [sender.py:send_request_defer():613] handle sender defer: 8 | |
| 2024-02-10 21:17:24,614 INFO SenderThread:3896 [job_builder.py:build():296] Attempting to build job artifact | |
| 2024-02-10 21:17:24,615 INFO SenderThread:3896 [job_builder.py:_get_source_type():426] is repo sourced job | |
| 2024-02-10 21:17:24,616 INFO SenderThread:3896 [job_builder.py:_get_program_relpath():444] run is notebook based run | |
| 2024-02-10 21:17:24,616 INFO SenderThread:3896 [sender.py:transition_state():617] send defer: 9 | |
| 2024-02-10 21:17:24,616 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: defer | |
| 2024-02-10 21:17:24,616 INFO HandlerThread:3896 [handler.py:handle_request_defer():172] handle defer: 9 | |
| 2024-02-10 21:17:24,616 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: defer | |
| 2024-02-10 21:17:24,616 INFO SenderThread:3896 [sender.py:send_request_defer():613] handle sender defer: 9 | |
| 2024-02-10 21:17:24,616 INFO SenderThread:3896 [dir_watcher.py:finish():358] shutting down directory watcher | |
| 2024-02-10 21:17:24,987 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: poll_exit | |
| 2024-02-10 21:17:25,304 INFO SenderThread:3896 [dir_watcher.py:_on_file_modified():288] file/dir modified: c:\Users\hiroga\Documents\GitHub\til\computer-science\machine-learning\_src\pokemon-palworld\notebooks\wandb\run-20240210_211321-s7dtxxtb\files\output.log | |
| 2024-02-10 21:17:25,305 INFO SenderThread:3896 [dir_watcher.py:finish():388] scan: c:\Users\hiroga\Documents\GitHub\til\computer-science\machine-learning\_src\pokemon-palworld\notebooks\wandb\run-20240210_211321-s7dtxxtb\files | |
| 2024-02-10 21:17:25,306 INFO SenderThread:3896 [dir_watcher.py:finish():402] scan save: c:\Users\hiroga\Documents\GitHub\til\computer-science\machine-learning\_src\pokemon-palworld\notebooks\wandb\run-20240210_211321-s7dtxxtb\files\conda-environment.yaml conda-environment.yaml | |
| 2024-02-10 21:17:25,307 INFO SenderThread:3896 [dir_watcher.py:finish():402] scan save: c:\Users\hiroga\Documents\GitHub\til\computer-science\machine-learning\_src\pokemon-palworld\notebooks\wandb\run-20240210_211321-s7dtxxtb\files\config.yaml config.yaml | |
| 2024-02-10 21:17:25,307 INFO SenderThread:3896 [dir_watcher.py:finish():402] scan save: c:\Users\hiroga\Documents\GitHub\til\computer-science\machine-learning\_src\pokemon-palworld\notebooks\wandb\run-20240210_211321-s7dtxxtb\files\output.log output.log | |
| 2024-02-10 21:17:25,311 INFO SenderThread:3896 [dir_watcher.py:finish():402] scan save: c:\Users\hiroga\Documents\GitHub\til\computer-science\machine-learning\_src\pokemon-palworld\notebooks\wandb\run-20240210_211321-s7dtxxtb\files\requirements.txt requirements.txt | |
| 2024-02-10 21:17:25,313 INFO SenderThread:3896 [dir_watcher.py:finish():402] scan save: c:\Users\hiroga\Documents\GitHub\til\computer-science\machine-learning\_src\pokemon-palworld\notebooks\wandb\run-20240210_211321-s7dtxxtb\files\wandb-metadata.json wandb-metadata.json | |
| 2024-02-10 21:17:25,313 INFO SenderThread:3896 [dir_watcher.py:finish():402] scan save: c:\Users\hiroga\Documents\GitHub\til\computer-science\machine-learning\_src\pokemon-palworld\notebooks\wandb\run-20240210_211321-s7dtxxtb\files\wandb-summary.json wandb-summary.json | |
| 2024-02-10 21:17:25,319 INFO SenderThread:3896 [sender.py:transition_state():617] send defer: 10 | |
| 2024-02-10 21:17:25,319 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: poll_exit | |
| 2024-02-10 21:17:25,319 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: defer | |
| 2024-02-10 21:17:25,320 INFO HandlerThread:3896 [handler.py:handle_request_defer():172] handle defer: 10 | |
| 2024-02-10 21:17:25,321 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: defer | |
| 2024-02-10 21:17:25,323 INFO SenderThread:3896 [sender.py:send_request_defer():613] handle sender defer: 10 | |
| 2024-02-10 21:17:25,324 INFO SenderThread:3896 [file_pusher.py:finish():172] shutting down file pusher | |
| 2024-02-10 21:17:25,721 INFO wandb-upload_0:3896 [upload_job.py:push():131] Uploaded file c:\Users\hiroga\Documents\GitHub\til\computer-science\machine-learning\_src\pokemon-palworld\notebooks\wandb\run-20240210_211321-s7dtxxtb\files\config.yaml | |
| 2024-02-10 21:17:26,001 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: poll_exit | |
| 2024-02-10 21:17:26,002 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: poll_exit | |
| 2024-02-10 21:17:26,098 INFO wandb-upload_2:3896 [upload_job.py:push():131] Uploaded file c:\Users\hiroga\Documents\GitHub\til\computer-science\machine-learning\_src\pokemon-palworld\notebooks\wandb\run-20240210_211321-s7dtxxtb\files\requirements.txt | |
| 2024-02-10 21:17:26,362 INFO wandb-upload_1:3896 [upload_job.py:push():131] Uploaded file c:\Users\hiroga\Documents\GitHub\til\computer-science\machine-learning\_src\pokemon-palworld\notebooks\wandb\run-20240210_211321-s7dtxxtb\files\output.log | |
| 2024-02-10 21:17:26,489 INFO wandb-upload_3:3896 [upload_job.py:push():131] Uploaded file c:\Users\hiroga\Documents\GitHub\til\computer-science\machine-learning\_src\pokemon-palworld\notebooks\wandb\run-20240210_211321-s7dtxxtb\files\wandb-summary.json | |
| 2024-02-10 21:17:26,705 INFO Thread-31 (_thread_body):3896 [sender.py:transition_state():617] send defer: 11 | |
| 2024-02-10 21:17:26,705 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: defer | |
| 2024-02-10 21:17:26,705 INFO HandlerThread:3896 [handler.py:handle_request_defer():172] handle defer: 11 | |
| 2024-02-10 21:17:26,705 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: defer | |
| 2024-02-10 21:17:26,705 INFO SenderThread:3896 [sender.py:send_request_defer():613] handle sender defer: 11 | |
| 2024-02-10 21:17:26,706 INFO SenderThread:3896 [file_pusher.py:join():178] waiting for file pusher | |
| 2024-02-10 21:17:26,706 INFO SenderThread:3896 [sender.py:transition_state():617] send defer: 12 | |
| 2024-02-10 21:17:26,706 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: defer | |
| 2024-02-10 21:17:26,706 INFO HandlerThread:3896 [handler.py:handle_request_defer():172] handle defer: 12 | |
| 2024-02-10 21:17:26,706 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: defer | |
| 2024-02-10 21:17:26,706 INFO SenderThread:3896 [sender.py:send_request_defer():613] handle sender defer: 12 | |
| 2024-02-10 21:17:26,706 INFO SenderThread:3896 [file_stream.py:finish():595] file stream finish called | |
| 2024-02-10 21:17:27,015 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: poll_exit | |
| 2024-02-10 21:17:27,243 INFO SenderThread:3896 [file_stream.py:finish():599] file stream finish is done | |
| 2024-02-10 21:17:27,243 INFO SenderThread:3896 [sender.py:transition_state():617] send defer: 13 | |
| 2024-02-10 21:17:27,244 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: poll_exit | |
| 2024-02-10 21:17:27,244 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: defer | |
| 2024-02-10 21:17:27,245 INFO HandlerThread:3896 [handler.py:handle_request_defer():172] handle defer: 13 | |
| 2024-02-10 21:17:27,246 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: defer | |
| 2024-02-10 21:17:27,246 INFO SenderThread:3896 [sender.py:send_request_defer():613] handle sender defer: 13 | |
| 2024-02-10 21:17:27,246 INFO SenderThread:3896 [sender.py:transition_state():617] send defer: 14 | |
| 2024-02-10 21:17:27,246 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: defer | |
| 2024-02-10 21:17:27,247 INFO HandlerThread:3896 [handler.py:handle_request_defer():172] handle defer: 14 | |
| 2024-02-10 21:17:27,247 DEBUG SenderThread:3896 [sender.py:send():382] send: final | |
| 2024-02-10 21:17:27,247 DEBUG SenderThread:3896 [sender.py:send():382] send: footer | |
| 2024-02-10 21:17:27,248 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: defer | |
| 2024-02-10 21:17:27,248 INFO SenderThread:3896 [sender.py:send_request_defer():613] handle sender defer: 14 | |
| 2024-02-10 21:17:27,250 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: poll_exit | |
| 2024-02-10 21:17:27,250 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: poll_exit | |
| 2024-02-10 21:17:27,252 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-10 21:17:27,253 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: server_info | |
| 2024-02-10 21:17:27,253 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: server_info | |
| 2024-02-10 21:17:27,253 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: get_summary | |
| 2024-02-10 21:17:27,255 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: sampled_history | |
| 2024-02-10 21:17:27,255 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: job_info | |
| 2024-02-10 21:17:27,425 DEBUG SenderThread:3896 [sender.py:send_request():409] send_request: job_info | |
| 2024-02-10 21:17:27,427 DEBUG HandlerThread:3896 [handler.py:handle_request():146] handle_request: shutdown | |
| 2024-02-10 21:17:27,427 INFO HandlerThread:3896 [handler.py:finish():869] shutting down handler | |
| 2024-02-10 21:17:28,266 INFO WriterThread:3896 [datastore.py:close():296] close: c:\Users\hiroga\Documents\GitHub\til\computer-science\machine-learning\_src\pokemon-palworld\notebooks\wandb\run-20240210_211321-s7dtxxtb\run-s7dtxxtb.wandb | |
| 2024-02-10 21:17:28,437 INFO SenderThread:3896 [sender.py:finish():1572] shutting down sender | |
| 2024-02-10 21:17:28,437 INFO SenderThread:3896 [file_pusher.py:finish():172] shutting down file pusher | |
| 2024-02-10 21:17:28,438 INFO SenderThread:3896 [file_pusher.py:join():178] waiting for file pusher | |