Hugging Face's logo

salahyahya
/
text_correct

Text2Text Generation
Transformers
Safetensors
t5
Generated from Trainer
text-generation-inference
Inference Endpoints
Model card Files Community
salahyahya commited on
Commit
d705c3c
•
1 Parent(s): 307f1e7

Upload folder using huggingface_hub

Browse files
Files changed (2) hide show
  1. debug-internal.log +505 -0
  2. debug.log +24 -0
debug-internal.log CHANGED
@@ -11706,3 +11706,508 @@
11706
  2024-06-12 06:41:26,524 INFO SystemMonitor:3156 [interfaces.py:start():188] Started gpu monitoring
11707
  2024-06-12 06:41:26,527 INFO SystemMonitor:3156 [interfaces.py:start():188] Started memory monitoring
11708
  2024-06-12 06:41:26,532 INFO SystemMonitor:3156 [interfaces.py:start():188] Started network monitoring
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
11706
  2024-06-12 06:41:26,524 INFO SystemMonitor:3156 [interfaces.py:start():188] Started gpu monitoring
11707
  2024-06-12 06:41:26,527 INFO SystemMonitor:3156 [interfaces.py:start():188] Started memory monitoring
11708
  2024-06-12 06:41:26,532 INFO SystemMonitor:3156 [interfaces.py:start():188] Started network monitoring
11709
+ 2024-06-12 06:41:27,471 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11710
+ 2024-06-12 06:41:28,471 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11711
+ 2024-06-12 06:41:29,472 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11712
+ 2024-06-12 06:41:29,491 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
11713
+ 2024-06-12 06:41:30,472 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11714
+ 2024-06-12 06:41:31,472 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11715
+ 2024-06-12 06:41:32,472 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11716
+ 2024-06-12 06:41:33,309 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: pause
11717
+ 2024-06-12 06:41:33,309 INFO HandlerThread:3156 [handler.py:handle_request_pause():724] stopping system metrics thread
11718
+ 2024-06-12 06:41:33,309 INFO HandlerThread:3156 [system_monitor.py:finish():203] Stopping system monitor
11719
+ 2024-06-12 06:41:33,309 DEBUG SystemMonitor:3156 [system_monitor.py:_start():172] Starting system metrics aggregation loop
11720
+ 2024-06-12 06:41:33,310 DEBUG SystemMonitor:3156 [system_monitor.py:_start():179] Finished system metrics aggregation loop
11721
+ 2024-06-12 06:41:33,310 DEBUG SystemMonitor:3156 [system_monitor.py:_start():183] Publishing last batch of metrics
11722
+ 2024-06-12 06:41:33,310 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined cpu monitor
11723
+ 2024-06-12 06:41:33,311 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined disk monitor
11724
+ 2024-06-12 06:41:33,320 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined gpu monitor
11725
+ 2024-06-12 06:41:33,321 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined memory monitor
11726
+ 2024-06-12 06:41:33,321 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined network monitor
11727
+ 2024-06-12 06:41:33,322 DEBUG SenderThread:3156 [sender.py:send():379] send: stats
11728
+ 2024-06-12 06:41:33,472 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11729
+ 2024-06-12 06:41:34,252 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: stop_status
11730
+ 2024-06-12 06:41:34,253 DEBUG SenderThread:3156 [sender.py:send_request():406] send_request: stop_status
11731
+ 2024-06-12 06:41:34,472 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11732
+ 2024-06-12 06:41:34,506 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
11733
+ 2024-06-12 06:41:35,472 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11734
+ 2024-06-12 06:41:36,472 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11735
+ 2024-06-12 06:41:37,473 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11736
+ 2024-06-12 06:41:38,473 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11737
+ 2024-06-12 06:41:39,473 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11738
+ 2024-06-12 06:41:40,472 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
11739
+ 2024-06-12 06:41:40,473 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11740
+ 2024-06-12 06:41:41,473 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11741
+ 2024-06-12 06:41:42,473 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11742
+ 2024-06-12 06:41:43,473 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11743
+ 2024-06-12 06:41:44,473 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11744
+ 2024-06-12 06:41:45,473 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11745
+ 2024-06-12 06:41:46,472 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
11746
+ 2024-06-12 06:41:46,473 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11747
+ 2024-06-12 06:41:47,474 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11748
+ 2024-06-12 06:41:48,474 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11749
+ 2024-06-12 06:41:49,252 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: stop_status
11750
+ 2024-06-12 06:41:49,253 DEBUG SenderThread:3156 [sender.py:send_request():406] send_request: stop_status
11751
+ 2024-06-12 06:41:49,474 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11752
+ 2024-06-12 06:41:50,474 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11753
+ 2024-06-12 06:41:51,472 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
11754
+ 2024-06-12 06:41:51,474 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11755
+ 2024-06-12 06:41:52,474 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11756
+ 2024-06-12 06:41:53,474 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11757
+ 2024-06-12 06:41:54,474 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11758
+ 2024-06-12 06:41:55,474 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11759
+ 2024-06-12 06:41:56,474 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11760
+ 2024-06-12 06:41:57,472 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
11761
+ 2024-06-12 06:41:57,475 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11762
+ 2024-06-12 06:41:58,475 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11763
+ 2024-06-12 06:41:59,475 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11764
+ 2024-06-12 06:42:00,475 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11765
+ 2024-06-12 06:42:01,475 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11766
+ 2024-06-12 06:42:02,472 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
11767
+ 2024-06-12 06:42:02,475 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11768
+ 2024-06-12 06:42:03,476 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11769
+ 2024-06-12 06:42:04,252 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: stop_status
11770
+ 2024-06-12 06:42:04,253 DEBUG SenderThread:3156 [sender.py:send_request():406] send_request: stop_status
11771
+ 2024-06-12 06:42:04,476 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11772
+ 2024-06-12 06:42:05,476 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11773
+ 2024-06-12 06:42:06,476 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11774
+ 2024-06-12 06:42:07,472 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
11775
+ 2024-06-12 06:42:07,476 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11776
+ 2024-06-12 06:42:08,476 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11777
+ 2024-06-12 06:42:09,476 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11778
+ 2024-06-12 06:42:10,476 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11779
+ 2024-06-12 06:42:11,476 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11780
+ 2024-06-12 06:42:12,476 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11781
+ 2024-06-12 06:42:13,472 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
11782
+ 2024-06-12 06:42:13,477 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11783
+ 2024-06-12 06:42:14,477 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11784
+ 2024-06-12 06:42:15,477 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11785
+ 2024-06-12 06:42:16,477 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11786
+ 2024-06-12 06:42:17,477 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11787
+ 2024-06-12 06:42:18,473 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
11788
+ 2024-06-12 06:42:18,477 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11789
+ 2024-06-12 06:42:19,252 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: stop_status
11790
+ 2024-06-12 06:42:19,253 DEBUG SenderThread:3156 [sender.py:send_request():406] send_request: stop_status
11791
+ 2024-06-12 06:42:19,478 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11792
+ 2024-06-12 06:42:20,478 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11793
+ 2024-06-12 06:42:21,478 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11794
+ 2024-06-12 06:42:22,478 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11795
+ 2024-06-12 06:42:23,478 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11796
+ 2024-06-12 06:42:24,473 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
11797
+ 2024-06-12 06:42:24,478 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11798
+ 2024-06-12 06:42:25,478 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11799
+ 2024-06-12 06:42:26,478 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11800
+ 2024-06-12 06:42:27,478 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11801
+ 2024-06-12 06:42:28,479 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11802
+ 2024-06-12 06:42:29,474 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
11803
+ 2024-06-12 06:42:29,479 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11804
+ 2024-06-12 06:42:30,479 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11805
+ 2024-06-12 06:42:31,479 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11806
+ 2024-06-12 06:42:32,479 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11807
+ 2024-06-12 06:42:33,479 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11808
+ 2024-06-12 06:42:34,253 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: stop_status
11809
+ 2024-06-12 06:42:34,253 DEBUG SenderThread:3156 [sender.py:send_request():406] send_request: stop_status
11810
+ 2024-06-12 06:42:34,479 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11811
+ 2024-06-12 06:42:34,573 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
11812
+ 2024-06-12 06:42:35,479 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11813
+ 2024-06-12 06:42:36,479 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11814
+ 2024-06-12 06:42:37,480 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11815
+ 2024-06-12 06:42:38,480 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11816
+ 2024-06-12 06:42:39,480 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11817
+ 2024-06-12 06:42:40,473 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
11818
+ 2024-06-12 06:42:40,480 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11819
+ 2024-06-12 06:42:41,480 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11820
+ 2024-06-12 06:42:42,480 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11821
+ 2024-06-12 06:42:43,480 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11822
+ 2024-06-12 06:42:44,481 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11823
+ 2024-06-12 06:42:45,474 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
11824
+ 2024-06-12 06:42:45,481 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11825
+ 2024-06-12 06:42:46,481 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11826
+ 2024-06-12 06:42:47,481 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11827
+ 2024-06-12 06:42:48,481 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11828
+ 2024-06-12 06:42:49,253 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: stop_status
11829
+ 2024-06-12 06:42:49,253 DEBUG SenderThread:3156 [sender.py:send_request():406] send_request: stop_status
11830
+ 2024-06-12 06:42:49,481 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11831
+ 2024-06-12 06:42:50,482 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11832
+ 2024-06-12 06:42:50,511 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
11833
+ 2024-06-12 06:42:51,481 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11834
+ 2024-06-12 06:42:52,482 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11835
+ 2024-06-12 06:42:53,482 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11836
+ 2024-06-12 06:42:54,482 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11837
+ 2024-06-12 06:42:55,482 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11838
+ 2024-06-12 06:42:56,474 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
11839
+ 2024-06-12 06:42:56,482 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11840
+ 2024-06-12 06:42:57,482 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11841
+ 2024-06-12 06:42:58,483 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11842
+ 2024-06-12 06:42:59,483 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11843
+ 2024-06-12 06:43:00,483 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11844
+ 2024-06-12 06:43:00,718 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: resume
11845
+ 2024-06-12 06:43:00,718 INFO HandlerThread:3156 [handler.py:handle_request_resume():715] starting system metrics thread
11846
+ 2024-06-12 06:43:00,718 INFO HandlerThread:3156 [system_monitor.py:start():194] Starting system monitor
11847
+ 2024-06-12 06:43:00,718 INFO SystemMonitor:3156 [system_monitor.py:_start():158] Starting system asset monitoring threads
11848
+ 2024-06-12 06:43:00,727 INFO SystemMonitor:3156 [interfaces.py:start():188] Started cpu monitoring
11849
+ 2024-06-12 06:43:00,729 INFO SystemMonitor:3156 [interfaces.py:start():188] Started disk monitoring
11850
+ 2024-06-12 06:43:00,734 INFO SystemMonitor:3156 [interfaces.py:start():188] Started gpu monitoring
11851
+ 2024-06-12 06:43:00,735 INFO SystemMonitor:3156 [interfaces.py:start():188] Started memory monitoring
11852
+ 2024-06-12 06:43:00,736 INFO SystemMonitor:3156 [interfaces.py:start():188] Started network monitoring
11853
+ 2024-06-12 06:43:01,474 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
11854
+ 2024-06-12 06:43:01,483 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11855
+ 2024-06-12 06:43:02,331 INFO Thread-12 :3156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240612_042407-8t92icsm/files/output.log
11856
+ 2024-06-12 06:43:02,483 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11857
+ 2024-06-12 06:43:02,662 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: pause
11858
+ 2024-06-12 06:43:02,662 INFO HandlerThread:3156 [handler.py:handle_request_pause():724] stopping system metrics thread
11859
+ 2024-06-12 06:43:02,663 INFO HandlerThread:3156 [system_monitor.py:finish():203] Stopping system monitor
11860
+ 2024-06-12 06:43:02,666 DEBUG SystemMonitor:3156 [system_monitor.py:_start():172] Starting system metrics aggregation loop
11861
+ 2024-06-12 06:43:02,666 DEBUG SystemMonitor:3156 [system_monitor.py:_start():179] Finished system metrics aggregation loop
11862
+ 2024-06-12 06:43:02,666 DEBUG SystemMonitor:3156 [system_monitor.py:_start():183] Publishing last batch of metrics
11863
+ 2024-06-12 06:43:02,667 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined cpu monitor
11864
+ 2024-06-12 06:43:02,667 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined disk monitor
11865
+ 2024-06-12 06:43:02,682 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined gpu monitor
11866
+ 2024-06-12 06:43:02,682 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined memory monitor
11867
+ 2024-06-12 06:43:02,682 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined network monitor
11868
+ 2024-06-12 06:43:02,683 DEBUG SenderThread:3156 [sender.py:send():379] send: stats
11869
+ 2024-06-12 06:43:03,483 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11870
+ 2024-06-12 06:43:04,253 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: stop_status
11871
+ 2024-06-12 06:43:04,253 DEBUG SenderThread:3156 [sender.py:send_request():406] send_request: stop_status
11872
+ 2024-06-12 06:43:04,339 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: resume
11873
+ 2024-06-12 06:43:04,339 INFO HandlerThread:3156 [handler.py:handle_request_resume():715] starting system metrics thread
11874
+ 2024-06-12 06:43:04,339 INFO HandlerThread:3156 [system_monitor.py:start():194] Starting system monitor
11875
+ 2024-06-12 06:43:04,343 INFO SystemMonitor:3156 [system_monitor.py:_start():158] Starting system asset monitoring threads
11876
+ 2024-06-12 06:43:04,343 INFO SystemMonitor:3156 [interfaces.py:start():188] Started cpu monitoring
11877
+ 2024-06-12 06:43:04,350 INFO SystemMonitor:3156 [interfaces.py:start():188] Started disk monitoring
11878
+ 2024-06-12 06:43:04,350 INFO SystemMonitor:3156 [interfaces.py:start():188] Started gpu monitoring
11879
+ 2024-06-12 06:43:04,350 INFO SystemMonitor:3156 [interfaces.py:start():188] Started memory monitoring
11880
+ 2024-06-12 06:43:04,356 INFO SystemMonitor:3156 [interfaces.py:start():188] Started network monitoring
11881
+ 2024-06-12 06:43:04,483 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11882
+ 2024-06-12 06:43:04,557 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: pause
11883
+ 2024-06-12 06:43:04,557 INFO HandlerThread:3156 [handler.py:handle_request_pause():724] stopping system metrics thread
11884
+ 2024-06-12 06:43:04,557 INFO HandlerThread:3156 [system_monitor.py:finish():203] Stopping system monitor
11885
+ 2024-06-12 06:43:04,557 DEBUG SystemMonitor:3156 [system_monitor.py:_start():172] Starting system metrics aggregation loop
11886
+ 2024-06-12 06:43:04,557 DEBUG SystemMonitor:3156 [system_monitor.py:_start():179] Finished system metrics aggregation loop
11887
+ 2024-06-12 06:43:04,557 DEBUG SystemMonitor:3156 [system_monitor.py:_start():183] Publishing last batch of metrics
11888
+ 2024-06-12 06:43:04,559 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined cpu monitor
11889
+ 2024-06-12 06:43:04,559 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined disk monitor
11890
+ 2024-06-12 06:43:04,569 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined gpu monitor
11891
+ 2024-06-12 06:43:04,569 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined memory monitor
11892
+ 2024-06-12 06:43:04,570 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined network monitor
11893
+ 2024-06-12 06:43:04,570 DEBUG SenderThread:3156 [sender.py:send():379] send: stats
11894
+ 2024-06-12 06:43:05,484 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11895
+ 2024-06-12 06:43:06,332 INFO Thread-12 :3156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240612_042407-8t92icsm/files/output.log
11896
+ 2024-06-12 06:43:06,483 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11897
+ 2024-06-12 06:43:07,474 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
11898
+ 2024-06-12 06:43:07,484 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11899
+ 2024-06-12 06:43:08,484 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11900
+ 2024-06-12 06:43:09,323 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: resume
11901
+ 2024-06-12 06:43:09,323 INFO HandlerThread:3156 [handler.py:handle_request_resume():715] starting system metrics thread
11902
+ 2024-06-12 06:43:09,323 INFO HandlerThread:3156 [system_monitor.py:start():194] Starting system monitor
11903
+ 2024-06-12 06:43:09,331 INFO SystemMonitor:3156 [system_monitor.py:_start():158] Starting system asset monitoring threads
11904
+ 2024-06-12 06:43:09,331 INFO SystemMonitor:3156 [interfaces.py:start():188] Started cpu monitoring
11905
+ 2024-06-12 06:43:09,332 INFO SystemMonitor:3156 [interfaces.py:start():188] Started disk monitoring
11906
+ 2024-06-12 06:43:09,337 INFO SystemMonitor:3156 [interfaces.py:start():188] Started gpu monitoring
11907
+ 2024-06-12 06:43:09,339 INFO SystemMonitor:3156 [interfaces.py:start():188] Started memory monitoring
11908
+ 2024-06-12 06:43:09,351 INFO SystemMonitor:3156 [interfaces.py:start():188] Started network monitoring
11909
+ 2024-06-12 06:43:09,484 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11910
+ 2024-06-12 06:43:10,484 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11911
+ 2024-06-12 06:43:10,635 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: pause
11912
+ 2024-06-12 06:43:10,636 INFO HandlerThread:3156 [handler.py:handle_request_pause():724] stopping system metrics thread
11913
+ 2024-06-12 06:43:10,636 INFO HandlerThread:3156 [system_monitor.py:finish():203] Stopping system monitor
11914
+ 2024-06-12 06:43:10,637 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined cpu monitor
11915
+ 2024-06-12 06:43:10,637 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined disk monitor
11916
+ 2024-06-12 06:43:10,644 DEBUG SystemMonitor:3156 [system_monitor.py:_start():172] Starting system metrics aggregation loop
11917
+ 2024-06-12 06:43:10,644 DEBUG SystemMonitor:3156 [system_monitor.py:_start():179] Finished system metrics aggregation loop
11918
+ 2024-06-12 06:43:10,644 DEBUG SystemMonitor:3156 [system_monitor.py:_start():183] Publishing last batch of metrics
11919
+ 2024-06-12 06:43:10,651 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined gpu monitor
11920
+ 2024-06-12 06:43:10,651 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined memory monitor
11921
+ 2024-06-12 06:43:10,651 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined network monitor
11922
+ 2024-06-12 06:43:10,652 DEBUG SenderThread:3156 [sender.py:send():379] send: stats
11923
+ 2024-06-12 06:43:11,484 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11924
+ 2024-06-12 06:43:12,484 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11925
+ 2024-06-12 06:43:13,474 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
11926
+ 2024-06-12 06:43:13,484 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11927
+ 2024-06-12 06:43:14,485 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11928
+ 2024-06-12 06:43:15,485 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11929
+ 2024-06-12 06:43:16,485 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11930
+ 2024-06-12 06:43:17,485 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11931
+ 2024-06-12 06:43:18,485 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11932
+ 2024-06-12 06:43:19,253 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: stop_status
11933
+ 2024-06-12 06:43:19,253 DEBUG SenderThread:3156 [sender.py:send_request():406] send_request: stop_status
11934
+ 2024-06-12 06:43:19,485 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11935
+ 2024-06-12 06:43:19,530 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
11936
+ 2024-06-12 06:43:20,485 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11937
+ 2024-06-12 06:43:21,485 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11938
+ 2024-06-12 06:43:22,485 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11939
+ 2024-06-12 06:43:23,486 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11940
+ 2024-06-12 06:43:24,486 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11941
+ 2024-06-12 06:43:25,475 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
11942
+ 2024-06-12 06:43:25,486 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11943
+ 2024-06-12 06:43:26,486 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11944
+ 2024-06-12 06:43:27,486 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11945
+ 2024-06-12 06:43:28,486 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11946
+ 2024-06-12 06:43:29,487 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11947
+ 2024-06-12 06:43:30,475 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
11948
+ 2024-06-12 06:43:30,487 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11949
+ 2024-06-12 06:43:31,487 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11950
+ 2024-06-12 06:43:32,487 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11951
+ 2024-06-12 06:43:33,487 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11952
+ 2024-06-12 06:43:34,253 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: stop_status
11953
+ 2024-06-12 06:43:34,254 DEBUG SenderThread:3156 [sender.py:send_request():406] send_request: stop_status
11954
+ 2024-06-12 06:43:34,487 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11955
+ 2024-06-12 06:43:35,487 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11956
+ 2024-06-12 06:43:35,577 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
11957
+ 2024-06-12 06:43:36,487 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11958
+ 2024-06-12 06:43:37,488 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11959
+ 2024-06-12 06:43:38,488 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11960
+ 2024-06-12 06:43:39,488 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11961
+ 2024-06-12 06:43:40,488 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11962
+ 2024-06-12 06:43:41,475 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
11963
+ 2024-06-12 06:43:41,488 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11964
+ 2024-06-12 06:43:42,488 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11965
+ 2024-06-12 06:43:43,489 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11966
+ 2024-06-12 06:43:44,491 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11967
+ 2024-06-12 06:43:45,489 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11968
+ 2024-06-12 06:43:45,821 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: resume
11969
+ 2024-06-12 06:43:45,822 INFO HandlerThread:3156 [handler.py:handle_request_resume():715] starting system metrics thread
11970
+ 2024-06-12 06:43:45,822 INFO HandlerThread:3156 [system_monitor.py:start():194] Starting system monitor
11971
+ 2024-06-12 06:43:45,827 INFO SystemMonitor:3156 [system_monitor.py:_start():158] Starting system asset monitoring threads
11972
+ 2024-06-12 06:43:45,827 INFO SystemMonitor:3156 [interfaces.py:start():188] Started cpu monitoring
11973
+ 2024-06-12 06:43:45,832 INFO SystemMonitor:3156 [interfaces.py:start():188] Started disk monitoring
11974
+ 2024-06-12 06:43:45,838 INFO SystemMonitor:3156 [interfaces.py:start():188] Started gpu monitoring
11975
+ 2024-06-12 06:43:45,841 INFO SystemMonitor:3156 [interfaces.py:start():188] Started memory monitoring
11976
+ 2024-06-12 06:43:45,849 INFO SystemMonitor:3156 [interfaces.py:start():188] Started network monitoring
11977
+ 2024-06-12 06:43:46,338 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: pause
11978
+ 2024-06-12 06:43:46,338 INFO HandlerThread:3156 [handler.py:handle_request_pause():724] stopping system metrics thread
11979
+ 2024-06-12 06:43:46,338 INFO HandlerThread:3156 [system_monitor.py:finish():203] Stopping system monitor
11980
+ 2024-06-12 06:43:46,341 DEBUG SystemMonitor:3156 [system_monitor.py:_start():172] Starting system metrics aggregation loop
11981
+ 2024-06-12 06:43:46,341 DEBUG SystemMonitor:3156 [system_monitor.py:_start():179] Finished system metrics aggregation loop
11982
+ 2024-06-12 06:43:46,341 DEBUG SystemMonitor:3156 [system_monitor.py:_start():183] Publishing last batch of metrics
11983
+ 2024-06-12 06:43:46,344 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined cpu monitor
11984
+ 2024-06-12 06:43:46,344 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined disk monitor
11985
+ 2024-06-12 06:43:46,351 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined gpu monitor
11986
+ 2024-06-12 06:43:46,351 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined memory monitor
11987
+ 2024-06-12 06:43:46,351 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined network monitor
11988
+ 2024-06-12 06:43:46,351 DEBUG SenderThread:3156 [sender.py:send():379] send: stats
11989
+ 2024-06-12 06:43:46,475 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
11990
+ 2024-06-12 06:43:46,489 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11991
+ 2024-06-12 06:43:47,489 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11992
+ 2024-06-12 06:43:48,489 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11993
+ 2024-06-12 06:43:49,253 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: stop_status
11994
+ 2024-06-12 06:43:49,254 DEBUG SenderThread:3156 [sender.py:send_request():406] send_request: stop_status
11995
+ 2024-06-12 06:43:49,490 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11996
+ 2024-06-12 06:43:50,490 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11997
+ 2024-06-12 06:43:51,490 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
11998
+ 2024-06-12 06:43:52,475 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
11999
+ 2024-06-12 06:43:52,490 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12000
+ 2024-06-12 06:43:53,490 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12001
+ 2024-06-12 06:43:54,490 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12002
+ 2024-06-12 06:43:55,490 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12003
+ 2024-06-12 06:43:56,490 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12004
+ 2024-06-12 06:43:57,475 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
12005
+ 2024-06-12 06:43:57,490 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12006
+ 2024-06-12 06:43:58,491 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12007
+ 2024-06-12 06:43:59,491 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12008
+ 2024-06-12 06:44:00,491 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12009
+ 2024-06-12 06:44:01,491 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12010
+ 2024-06-12 06:44:02,491 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12011
+ 2024-06-12 06:44:03,475 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
12012
+ 2024-06-12 06:44:03,491 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12013
+ 2024-06-12 06:44:04,253 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: stop_status
12014
+ 2024-06-12 06:44:04,254 DEBUG SenderThread:3156 [sender.py:send_request():406] send_request: stop_status
12015
+ 2024-06-12 06:44:04,491 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12016
+ 2024-06-12 06:44:05,492 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12017
+ 2024-06-12 06:44:06,492 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12018
+ 2024-06-12 06:44:07,492 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12019
+ 2024-06-12 06:44:08,475 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
12020
+ 2024-06-12 06:44:08,492 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12021
+ 2024-06-12 06:44:09,492 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12022
+ 2024-06-12 06:44:10,492 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12023
+ 2024-06-12 06:44:11,492 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12024
+ 2024-06-12 06:44:12,492 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12025
+ 2024-06-12 06:44:13,476 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
12026
+ 2024-06-12 06:44:13,493 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12027
+ 2024-06-12 06:44:14,493 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12028
+ 2024-06-12 06:44:15,493 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12029
+ 2024-06-12 06:44:16,493 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12030
+ 2024-06-12 06:44:17,494 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12031
+ 2024-06-12 06:44:18,481 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
12032
+ 2024-06-12 06:44:18,494 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12033
+ 2024-06-12 06:44:19,254 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: stop_status
12034
+ 2024-06-12 06:44:19,254 DEBUG SenderThread:3156 [sender.py:send_request():406] send_request: stop_status
12035
+ 2024-06-12 06:44:19,494 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12036
+ 2024-06-12 06:44:20,494 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12037
+ 2024-06-12 06:44:21,494 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12038
+ 2024-06-12 06:44:22,494 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12039
+ 2024-06-12 06:44:23,495 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12040
+ 2024-06-12 06:44:24,476 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
12041
+ 2024-06-12 06:44:24,495 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12042
+ 2024-06-12 06:44:25,495 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12043
+ 2024-06-12 06:44:26,495 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12044
+ 2024-06-12 06:44:27,495 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12045
+ 2024-06-12 06:44:28,495 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12046
+ 2024-06-12 06:44:29,495 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12047
+ 2024-06-12 06:44:30,476 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
12048
+ 2024-06-12 06:44:30,495 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12049
+ 2024-06-12 06:44:31,496 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12050
+ 2024-06-12 06:44:32,496 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12051
+ 2024-06-12 06:44:33,496 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12052
+ 2024-06-12 06:44:34,254 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: stop_status
12053
+ 2024-06-12 06:44:34,254 DEBUG SenderThread:3156 [sender.py:send_request():406] send_request: stop_status
12054
+ 2024-06-12 06:44:34,496 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12055
+ 2024-06-12 06:44:35,496 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12056
+ 2024-06-12 06:44:35,553 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
12057
+ 2024-06-12 06:44:36,496 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12058
+ 2024-06-12 06:44:37,496 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12059
+ 2024-06-12 06:44:38,496 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12060
+ 2024-06-12 06:44:39,496 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12061
+ 2024-06-12 06:44:40,497 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12062
+ 2024-06-12 06:44:41,476 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
12063
+ 2024-06-12 06:44:41,497 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12064
+ 2024-06-12 06:44:42,497 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12065
+ 2024-06-12 06:44:43,497 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12066
+ 2024-06-12 06:44:44,497 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12067
+ 2024-06-12 06:44:45,497 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12068
+ 2024-06-12 06:44:46,476 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
12069
+ 2024-06-12 06:44:46,497 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12070
+ 2024-06-12 06:44:47,498 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12071
+ 2024-06-12 06:44:48,498 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12072
+ 2024-06-12 06:44:49,254 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: stop_status
12073
+ 2024-06-12 06:44:49,254 DEBUG SenderThread:3156 [sender.py:send_request():406] send_request: stop_status
12074
+ 2024-06-12 06:44:49,498 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12075
+ 2024-06-12 06:44:50,498 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12076
+ 2024-06-12 06:44:50,949 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: resume
12077
+ 2024-06-12 06:44:50,950 INFO HandlerThread:3156 [handler.py:handle_request_resume():715] starting system metrics thread
12078
+ 2024-06-12 06:44:50,950 INFO HandlerThread:3156 [system_monitor.py:start():194] Starting system monitor
12079
+ 2024-06-12 06:44:50,950 INFO SystemMonitor:3156 [system_monitor.py:_start():158] Starting system asset monitoring threads
12080
+ 2024-06-12 06:44:50,950 INFO SystemMonitor:3156 [interfaces.py:start():188] Started cpu monitoring
12081
+ 2024-06-12 06:44:50,951 INFO SystemMonitor:3156 [interfaces.py:start():188] Started disk monitoring
12082
+ 2024-06-12 06:44:50,957 INFO SystemMonitor:3156 [interfaces.py:start():188] Started gpu monitoring
12083
+ 2024-06-12 06:44:50,963 INFO SystemMonitor:3156 [interfaces.py:start():188] Started memory monitoring
12084
+ 2024-06-12 06:44:50,964 INFO SystemMonitor:3156 [interfaces.py:start():188] Started network monitoring
12085
+ 2024-06-12 06:44:51,477 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
12086
+ 2024-06-12 06:44:51,506 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12087
+ 2024-06-12 06:44:56,478 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
12088
+ 2024-06-12 06:45:01,479 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
12089
+ 2024-06-12 06:45:06,479 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
12090
+ 2024-06-12 06:45:08,497 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12091
+ 2024-06-12 06:45:08,498 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: stop_status
12092
+ 2024-06-12 06:45:08,498 DEBUG SenderThread:3156 [sender.py:send_request():406] send_request: stop_status
12093
+ 2024-06-12 06:45:09,496 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12094
+ 2024-06-12 06:45:10,496 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12095
+ 2024-06-12 06:45:11,497 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12096
+ 2024-06-12 06:45:11,744 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
12097
+ 2024-06-12 06:45:12,483 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: pause
12098
+ 2024-06-12 06:45:12,483 INFO HandlerThread:3156 [handler.py:handle_request_pause():724] stopping system metrics thread
12099
+ 2024-06-12 06:45:12,483 INFO HandlerThread:3156 [system_monitor.py:finish():203] Stopping system monitor
12100
+ 2024-06-12 06:45:12,483 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined cpu monitor
12101
+ 2024-06-12 06:45:12,484 DEBUG SystemMonitor:3156 [system_monitor.py:_start():172] Starting system metrics aggregation loop
12102
+ 2024-06-12 06:45:12,484 DEBUG SystemMonitor:3156 [system_monitor.py:_start():179] Finished system metrics aggregation loop
12103
+ 2024-06-12 06:45:12,484 DEBUG SystemMonitor:3156 [system_monitor.py:_start():183] Publishing last batch of metrics
12104
+ 2024-06-12 06:45:12,484 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined disk monitor
12105
+ 2024-06-12 06:45:12,498 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined gpu monitor
12106
+ 2024-06-12 06:45:12,498 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined memory monitor
12107
+ 2024-06-12 06:45:12,499 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined network monitor
12108
+ 2024-06-12 06:45:12,499 DEBUG SenderThread:3156 [sender.py:send():379] send: stats
12109
+ 2024-06-12 06:45:12,524 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: resume
12110
+ 2024-06-12 06:45:12,524 INFO HandlerThread:3156 [handler.py:handle_request_resume():715] starting system metrics thread
12111
+ 2024-06-12 06:45:12,524 INFO HandlerThread:3156 [system_monitor.py:start():194] Starting system monitor
12112
+ 2024-06-12 06:45:12,531 INFO SystemMonitor:3156 [system_monitor.py:_start():158] Starting system asset monitoring threads
12113
+ 2024-06-12 06:45:12,531 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12114
+ 2024-06-12 06:45:12,532 INFO SystemMonitor:3156 [interfaces.py:start():188] Started cpu monitoring
12115
+ 2024-06-12 06:45:12,533 INFO SystemMonitor:3156 [interfaces.py:start():188] Started disk monitoring
12116
+ 2024-06-12 06:45:12,534 INFO SystemMonitor:3156 [interfaces.py:start():188] Started gpu monitoring
12117
+ 2024-06-12 06:45:12,540 INFO SystemMonitor:3156 [interfaces.py:start():188] Started memory monitoring
12118
+ 2024-06-12 06:45:12,540 INFO SystemMonitor:3156 [interfaces.py:start():188] Started network monitoring
12119
+ 2024-06-12 06:45:17,502 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
12120
+ 2024-06-12 06:45:18,221 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12121
+ 2024-06-12 06:45:18,227 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: pause
12122
+ 2024-06-12 06:45:18,228 INFO HandlerThread:3156 [handler.py:handle_request_pause():724] stopping system metrics thread
12123
+ 2024-06-12 06:45:18,228 INFO HandlerThread:3156 [system_monitor.py:finish():203] Stopping system monitor
12124
+ 2024-06-12 06:45:18,228 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined cpu monitor
12125
+ 2024-06-12 06:45:18,228 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined disk monitor
12126
+ 2024-06-12 06:45:18,228 DEBUG SystemMonitor:3156 [system_monitor.py:_start():172] Starting system metrics aggregation loop
12127
+ 2024-06-12 06:45:18,229 DEBUG SystemMonitor:3156 [system_monitor.py:_start():179] Finished system metrics aggregation loop
12128
+ 2024-06-12 06:45:18,229 DEBUG SystemMonitor:3156 [system_monitor.py:_start():183] Publishing last batch of metrics
12129
+ 2024-06-12 06:45:18,243 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined gpu monitor
12130
+ 2024-06-12 06:45:18,244 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined memory monitor
12131
+ 2024-06-12 06:45:18,244 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined network monitor
12132
+ 2024-06-12 06:45:18,245 DEBUG SenderThread:3156 [sender.py:send():379] send: stats
12133
+ 2024-06-12 06:45:19,220 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12134
+ 2024-06-12 06:45:20,220 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12135
+ 2024-06-12 06:45:21,220 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12136
+ 2024-06-12 06:45:22,221 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12137
+ 2024-06-12 06:45:23,221 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12138
+ 2024-06-12 06:45:23,222 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
12139
+ 2024-06-12 06:45:23,496 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: stop_status
12140
+ 2024-06-12 06:45:23,496 DEBUG SenderThread:3156 [sender.py:send_request():406] send_request: stop_status
12141
+ 2024-06-12 06:45:24,221 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12142
+ 2024-06-12 06:45:25,221 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12143
+ 2024-06-12 06:45:26,221 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12144
+ 2024-06-12 06:45:27,221 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12145
+ 2024-06-12 06:45:28,221 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12146
+ 2024-06-12 06:45:28,223 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
12147
+ 2024-06-12 06:45:29,222 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12148
+ 2024-06-12 06:45:30,222 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12149
+ 2024-06-12 06:45:31,222 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12150
+ 2024-06-12 06:45:32,222 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12151
+ 2024-06-12 06:45:33,222 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12152
+ 2024-06-12 06:45:34,222 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
12153
+ 2024-06-12 06:45:34,222 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12154
+ 2024-06-12 06:45:35,222 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12155
+ 2024-06-12 06:45:36,222 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12156
+ 2024-06-12 06:45:37,223 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12157
+ 2024-06-12 06:45:37,989 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: resume
12158
+ 2024-06-12 06:45:37,990 INFO HandlerThread:3156 [handler.py:handle_request_resume():715] starting system metrics thread
12159
+ 2024-06-12 06:45:37,990 INFO HandlerThread:3156 [system_monitor.py:start():194] Starting system monitor
12160
+ 2024-06-12 06:45:38,000 INFO SystemMonitor:3156 [system_monitor.py:_start():158] Starting system asset monitoring threads
12161
+ 2024-06-12 06:45:38,006 INFO SystemMonitor:3156 [interfaces.py:start():188] Started cpu monitoring
12162
+ 2024-06-12 06:45:38,007 INFO SystemMonitor:3156 [interfaces.py:start():188] Started disk monitoring
12163
+ 2024-06-12 06:45:38,008 INFO SystemMonitor:3156 [interfaces.py:start():188] Started gpu monitoring
12164
+ 2024-06-12 06:45:38,010 INFO SystemMonitor:3156 [interfaces.py:start():188] Started memory monitoring
12165
+ 2024-06-12 06:45:38,018 INFO SystemMonitor:3156 [interfaces.py:start():188] Started network monitoring
12166
+ 2024-06-12 06:45:38,223 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12167
+ 2024-06-12 06:45:38,496 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: stop_status
12168
+ 2024-06-12 06:45:38,497 DEBUG SenderThread:3156 [sender.py:send_request():406] send_request: stop_status
12169
+ 2024-06-12 06:45:39,806 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
12170
+ 2024-06-12 06:45:42,209 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12171
+ 2024-06-12 06:45:42,213 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: pause
12172
+ 2024-06-12 06:45:42,213 INFO HandlerThread:3156 [handler.py:handle_request_pause():724] stopping system metrics thread
12173
+ 2024-06-12 06:45:42,213 INFO HandlerThread:3156 [system_monitor.py:finish():203] Stopping system monitor
12174
+ 2024-06-12 06:45:42,213 DEBUG SystemMonitor:3156 [system_monitor.py:_start():172] Starting system metrics aggregation loop
12175
+ 2024-06-12 06:45:42,213 DEBUG SystemMonitor:3156 [system_monitor.py:_start():179] Finished system metrics aggregation loop
12176
+ 2024-06-12 06:45:42,213 DEBUG SystemMonitor:3156 [system_monitor.py:_start():183] Publishing last batch of metrics
12177
+ 2024-06-12 06:45:42,214 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined cpu monitor
12178
+ 2024-06-12 06:45:42,214 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined disk monitor
12179
+ 2024-06-12 06:45:42,224 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined gpu monitor
12180
+ 2024-06-12 06:45:42,224 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined memory monitor
12181
+ 2024-06-12 06:45:42,224 INFO HandlerThread:3156 [interfaces.py:finish():200] Joined network monitor
12182
+ 2024-06-12 06:45:42,225 DEBUG SenderThread:3156 [sender.py:send():379] send: stats
12183
+ 2024-06-12 06:45:43,204 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12184
+ 2024-06-12 06:45:44,205 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12185
+ 2024-06-12 06:45:45,205 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12186
+ 2024-06-12 06:45:45,222 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
12187
+ 2024-06-12 06:45:46,205 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12188
+ 2024-06-12 06:45:47,205 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12189
+ 2024-06-12 06:45:48,205 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12190
+ 2024-06-12 06:45:49,205 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12191
+ 2024-06-12 06:45:50,205 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12192
+ 2024-06-12 06:45:50,222 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
12193
+ 2024-06-12 06:45:51,206 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12194
+ 2024-06-12 06:45:52,206 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12195
+ 2024-06-12 06:45:53,206 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12196
+ 2024-06-12 06:45:53,496 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: stop_status
12197
+ 2024-06-12 06:45:53,497 DEBUG SenderThread:3156 [sender.py:send_request():406] send_request: stop_status
12198
+ 2024-06-12 06:45:54,206 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12199
+ 2024-06-12 06:45:55,206 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12200
+ 2024-06-12 06:45:55,774 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: status_report
12201
+ 2024-06-12 06:45:56,206 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12202
+ 2024-06-12 06:45:57,206 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12203
+ 2024-06-12 06:45:58,206 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12204
+ 2024-06-12 06:45:59,206 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: internal_messages
12205
+ 2024-06-12 06:46:00,099 DEBUG HandlerThread:3156 [handler.py:handle_request():158] handle_request: resume
12206
+ 2024-06-12 06:46:00,100 INFO HandlerThread:3156 [handler.py:handle_request_resume():715] starting system metrics thread
12207
+ 2024-06-12 06:46:00,100 INFO HandlerThread:3156 [system_monitor.py:start():194] Starting system monitor
12208
+ 2024-06-12 06:46:00,101 INFO SystemMonitor:3156 [system_monitor.py:_start():158] Starting system asset monitoring threads
12209
+ 2024-06-12 06:46:00,104 INFO SystemMonitor:3156 [interfaces.py:start():188] Started cpu monitoring
12210
+ 2024-06-12 06:46:00,105 INFO SystemMonitor:3156 [interfaces.py:start():188] Started disk monitoring
12211
+ 2024-06-12 06:46:00,106 INFO SystemMonitor:3156 [interfaces.py:start():188] Started gpu monitoring
12212
+ 2024-06-12 06:46:00,110 INFO SystemMonitor:3156 [interfaces.py:start():188] Started memory monitoring
12213
+ 2024-06-12 06:46:00,111 INFO SystemMonitor:3156 [interfaces.py:start():188] Started network monitoring
debug.log CHANGED
@@ -102,3 +102,27 @@ config: {}
102
  2024-06-12 06:41:26,479 INFO MainThread:1920 [jupyter.py:save_ipynb():372] not saving jupyter notebook
103
  2024-06-12 06:41:26,479 INFO MainThread:1920 [wandb_init.py:_pause_backend():431] pausing backend
104
  2024-06-12 06:41:26,513 INFO MainThread:1920 [wandb_init.py:_resume_backend():436] resuming backend
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
102
  2024-06-12 06:41:26,479 INFO MainThread:1920 [jupyter.py:save_ipynb():372] not saving jupyter notebook
103
  2024-06-12 06:41:26,479 INFO MainThread:1920 [wandb_init.py:_pause_backend():431] pausing backend
104
  2024-06-12 06:41:26,513 INFO MainThread:1920 [wandb_init.py:_resume_backend():436] resuming backend
105
+ 2024-06-12 06:41:33,308 INFO MainThread:1920 [jupyter.py:save_ipynb():372] not saving jupyter notebook
106
+ 2024-06-12 06:41:33,309 INFO MainThread:1920 [wandb_init.py:_pause_backend():431] pausing backend
107
+ 2024-06-12 06:43:00,716 INFO MainThread:1920 [wandb_init.py:_resume_backend():436] resuming backend
108
+ 2024-06-12 06:43:02,662 INFO MainThread:1920 [jupyter.py:save_ipynb():372] not saving jupyter notebook
109
+ 2024-06-12 06:43:02,662 INFO MainThread:1920 [wandb_init.py:_pause_backend():431] pausing backend
110
+ 2024-06-12 06:43:04,339 INFO MainThread:1920 [wandb_init.py:_resume_backend():436] resuming backend
111
+ 2024-06-12 06:43:04,556 INFO MainThread:1920 [jupyter.py:save_ipynb():372] not saving jupyter notebook
112
+ 2024-06-12 06:43:04,556 INFO MainThread:1920 [wandb_init.py:_pause_backend():431] pausing backend
113
+ 2024-06-12 06:43:09,322 INFO MainThread:1920 [wandb_init.py:_resume_backend():436] resuming backend
114
+ 2024-06-12 06:43:10,633 INFO MainThread:1920 [jupyter.py:save_ipynb():372] not saving jupyter notebook
115
+ 2024-06-12 06:43:10,634 INFO MainThread:1920 [wandb_init.py:_pause_backend():431] pausing backend
116
+ 2024-06-12 06:43:45,820 INFO MainThread:1920 [wandb_init.py:_resume_backend():436] resuming backend
117
+ 2024-06-12 06:43:46,338 INFO MainThread:1920 [jupyter.py:save_ipynb():372] not saving jupyter notebook
118
+ 2024-06-12 06:43:46,338 INFO MainThread:1920 [wandb_init.py:_pause_backend():431] pausing backend
119
+ 2024-06-12 06:44:50,948 INFO MainThread:1920 [wandb_init.py:_resume_backend():436] resuming backend
120
+ 2024-06-12 06:45:12,482 INFO MainThread:1920 [jupyter.py:save_ipynb():372] not saving jupyter notebook
121
+ 2024-06-12 06:45:12,482 INFO MainThread:1920 [wandb_init.py:_pause_backend():431] pausing backend
122
+ 2024-06-12 06:45:12,495 INFO MainThread:1920 [wandb_init.py:_resume_backend():436] resuming backend
123
+ 2024-06-12 06:45:18,227 INFO MainThread:1920 [jupyter.py:save_ipynb():372] not saving jupyter notebook
124
+ 2024-06-12 06:45:18,227 INFO MainThread:1920 [wandb_init.py:_pause_backend():431] pausing backend
125
+ 2024-06-12 06:45:37,988 INFO MainThread:1920 [wandb_init.py:_resume_backend():436] resuming backend
126
+ 2024-06-12 06:45:42,212 INFO MainThread:1920 [jupyter.py:save_ipynb():372] not saving jupyter notebook
127
+ 2024-06-12 06:45:42,212 INFO MainThread:1920 [wandb_init.py:_pause_backend():431] pausing backend
128
+ 2024-06-12 06:46:00,099 INFO MainThread:1920 [wandb_init.py:_resume_backend():436] resuming backend