| 2024-02-14 12:54:29,380 INFO StreamThr :35088 [internal.py:wandb_internal():86] W&B internal server running at pid: 35088, started at: 2024-02-14 12:54:29.379470 | |
| 2024-02-14 12:54:29,381 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status | |
| 2024-02-14 12:54:30,447 INFO WriterThread:35088 [datastore.py:open_for_write():87] open: /content/wandb/run-20240214_125429-fe3s7gan/run-fe3s7gan.wandb | |
| 2024-02-14 12:54:30,447 DEBUG SenderThread:35088 [sender.py:send():382] send: header | |
| 2024-02-14 12:54:30,450 DEBUG SenderThread:35088 [sender.py:send():382] send: run | |
| 2024-02-14 12:54:30,674 INFO SenderThread:35088 [dir_watcher.py:__init__():211] watching files in: /content/wandb/run-20240214_125429-fe3s7gan/files | |
| 2024-02-14 12:54:30,674 INFO SenderThread:35088 [sender.py:_start_run_threads():1136] run started: fe3s7gan with start time 1707915269.380721 | |
| 2024-02-14 12:54:30,681 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: check_version | |
| 2024-02-14 12:54:30,682 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: check_version | |
| 2024-02-14 12:54:30,815 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: run_start | |
| 2024-02-14 12:54:30,824 DEBUG HandlerThread:35088 [system_info.py:__init__():27] System info init | |
| 2024-02-14 12:54:30,824 DEBUG HandlerThread:35088 [system_info.py:__init__():42] System info init done | |
| 2024-02-14 12:54:30,824 INFO HandlerThread:35088 [system_monitor.py:start():194] Starting system monitor | |
| 2024-02-14 12:54:30,824 INFO SystemMonitor:35088 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-02-14 12:54:30,825 INFO HandlerThread:35088 [system_monitor.py:probe():214] Collecting system info | |
| 2024-02-14 12:54:30,825 INFO SystemMonitor:35088 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-02-14 12:54:30,826 INFO SystemMonitor:35088 [interfaces.py:start():190] Started disk monitoring | |
| 2024-02-14 12:54:30,826 INFO SystemMonitor:35088 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-02-14 12:54:30,828 INFO SystemMonitor:35088 [interfaces.py:start():190] Started memory monitoring | |
| 2024-02-14 12:54:30,828 INFO SystemMonitor:35088 [interfaces.py:start():190] Started network monitoring | |
| 2024-02-14 12:54:30,832 DEBUG HandlerThread:35088 [system_info.py:probe():151] Probing system | |
| 2024-02-14 12:54:30,833 DEBUG HandlerThread:35088 [gitlib.py:_init_repo():56] git repository is invalid | |
| 2024-02-14 12:54:30,833 DEBUG HandlerThread:35088 [system_info.py:probe():199] Probing system done | |
| 2024-02-14 12:54:30,833 DEBUG HandlerThread:35088 [system_monitor.py:probe():223] {'os': 'Linux-6.1.58+-x86_64-with-glibc2.35', 'python': '3.10.12', 'heartbeatAt': '2024-02-14T12:54:30.832517', 'startedAt': '2024-02-14T12:54:29.373808', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'Smart_Agriculture_Falcon.ipynb', 'codePathLocal': None, 'colab': 'https://colab.research.google.com/notebook#fileId=1kv2VO7q0c8xX_DQt5NERLeJJZtHW6zHO', 'host': 'f4b889916ade', 'username': 'root', 'executable': '/usr/bin/python3', 'cpu_count': 4, 'cpu_count_logical': 8, 'cpu_freq': {'current': 2299.998, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2299.998, 'min': 0.0, 'max': 0.0}, {'current': 2299.998, 'min': 0.0, 'max': 0.0}, {'current': 2299.998, 'min': 0.0, 'max': 0.0}, {'current': 2299.998, 'min': 0.0, 'max': 0.0}, {'current': 2299.998, 'min': 0.0, 'max': 0.0}, {'current': 2299.998, 'min': 0.0, 'max': 0.0}, {'current': 2299.998, 'min': 0.0, 'max': 0.0}, {'current': 2299.998, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 166.77415084838867, 'used': 66.21718978881836}}, 'gpu': 'Tesla T4', 'gpu_count': 1, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 50.993690490722656}} | |
| 2024-02-14 12:54:30,834 INFO HandlerThread:35088 [system_monitor.py:probe():224] Finished collecting system info | |
| 2024-02-14 12:54:30,834 INFO HandlerThread:35088 [system_monitor.py:probe():227] Publishing system info | |
| 2024-02-14 12:54:30,834 INFO HandlerThread:35088 [system_monitor.py:probe():229] Finished publishing system info | |
| 2024-02-14 12:54:30,840 DEBUG SenderThread:35088 [sender.py:send():382] send: files | |
| 2024-02-14 12:54:30,840 INFO SenderThread:35088 [sender.py:_save_file():1403] saving file wandb-metadata.json with policy now | |
| 2024-02-14 12:54:30,856 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: python_packages | |
| 2024-02-14 12:54:30,856 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: python_packages | |
| 2024-02-14 12:54:30,896 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 12:54:30,896 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 12:54:30,897 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 12:54:31,103 DEBUG SenderThread:35088 [sender.py:send():382] send: telemetry | |
| 2024-02-14 12:54:31,103 DEBUG SenderThread:35088 [sender.py:send():382] send: config | |
| 2024-02-14 12:54:31,104 DEBUG SenderThread:35088 [sender.py:send():382] send: metric | |
| 2024-02-14 12:54:31,104 DEBUG SenderThread:35088 [sender.py:send():382] send: telemetry | |
| 2024-02-14 12:54:31,104 DEBUG SenderThread:35088 [sender.py:send():382] send: metric | |
| 2024-02-14 12:54:31,104 WARNING SenderThread:35088 [sender.py:send_metric():1354] Seen metric with glob (shouldn't happen) | |
| 2024-02-14 12:54:31,104 DEBUG SenderThread:35088 [sender.py:send():382] send: telemetry | |
| 2024-02-14 12:54:31,178 INFO wandb-upload_0:35088 [upload_job.py:push():131] Uploaded file /tmp/tmp668dgil2wandb/pe1jrqzh-wandb-metadata.json | |
| 2024-02-14 12:54:31,675 INFO Thread-12 :35088 [dir_watcher.py:_on_file_created():271] file/dir created: /content/wandb/run-20240214_125429-fe3s7gan/files/requirements.txt | |
| 2024-02-14 12:54:31,675 INFO Thread-12 :35088 [dir_watcher.py:_on_file_created():271] file/dir created: /content/wandb/run-20240214_125429-fe3s7gan/files/wandb-metadata.json | |
| 2024-02-14 12:54:35,105 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:54:40,106 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:54:45,107 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:54:45,857 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 12:54:45,857 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 12:54:45,897 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 12:54:50,987 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:54:55,988 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:55:00,857 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 12:55:00,857 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 12:55:00,897 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 12:55:01,012 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:55:01,684 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/config.yaml | |
| 2024-02-14 12:55:06,125 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:55:11,126 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:55:15,857 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 12:55:15,857 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 12:55:15,897 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 12:55:16,924 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:55:21,925 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:55:26,926 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:55:30,828 DEBUG SystemMonitor:35088 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-02-14 12:55:30,829 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 12:55:30,857 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 12:55:30,857 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 12:55:30,897 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 12:55:31,981 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:55:36,982 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:55:41,983 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:55:45,857 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 12:55:45,858 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 12:55:45,898 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 12:55:47,973 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:55:51,040 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-14 12:55:51,042 DEBUG SenderThread:35088 [sender.py:send():382] send: metric | |
| 2024-02-14 12:55:51,042 DEBUG SenderThread:35088 [sender.py:send():382] send: metric | |
| 2024-02-14 12:55:51,042 DEBUG SenderThread:35088 [sender.py:send():382] send: metric | |
| 2024-02-14 12:55:51,042 DEBUG SenderThread:35088 [sender.py:send():382] send: history | |
| 2024-02-14 12:55:51,043 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-14 12:55:51,045 INFO SenderThread:35088 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-14 12:55:51,700 INFO Thread-12 :35088 [dir_watcher.py:_on_file_created():271] file/dir created: /content/wandb/run-20240214_125429-fe3s7gan/files/wandb-summary.json | |
| 2024-02-14 12:55:51,701 INFO Thread-12 :35088 [dir_watcher.py:_on_file_created():271] file/dir created: /content/wandb/run-20240214_125429-fe3s7gan/files/output.log | |
| 2024-02-14 12:55:53,082 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:55:53,701 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/output.log | |
| 2024-02-14 12:55:58,083 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:56:00,830 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 12:56:00,857 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 12:56:00,858 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 12:56:00,858 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |