2024-07-11 22:26:52 | INFO | model_worker | args: Namespace(host='0.0.0.0', port=40007, worker_address='http://10.140.66.196:40007', controller_address='http://10.140.60.209:10075', model_path='share_internvl/InternVL2-78B/', model_name=None, device='auto', limit_model_concurrency=5, stream_interval=1, load_8bit=False)
2024-07-11 22:26:52 | INFO | model_worker | Loading the model InternVL2-78B on worker 2370fb ...
2024-07-11 22:26:53 | WARNING | transformers.tokenization_utils_base | Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
2024-07-11 22:26:53 | WARNING | transformers.tokenization_utils_base | Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
2024-07-11 22:26:59 | ERROR | stderr | 
Loading checkpoint shards:   0%|          | 0/33 [00:00<?, ?it/s]
2024-07-11 22:27:01 | ERROR | stderr | 
Loading checkpoint shards:   3%|▎         | 1/33 [00:01<01:01,  1.93s/it]
2024-07-11 22:27:03 | ERROR | stderr | 
Loading checkpoint shards:   6%|▌         | 2/33 [00:03<00:58,  1.89s/it]
2024-07-11 22:27:05 | ERROR | stderr | 
Loading checkpoint shards:   9%|▉         | 3/33 [00:05<00:56,  1.87s/it]
2024-07-11 22:27:06 | ERROR | stderr | 
Loading checkpoint shards:  12%|█▏        | 4/33 [00:07<00:55,  1.90s/it]
2024-07-11 22:27:08 | ERROR | stderr | 
Loading checkpoint shards:  15%|█▌        | 5/33 [00:09<00:53,  1.90s/it]
2024-07-11 22:27:10 | ERROR | stderr | 
Loading checkpoint shards:  18%|█▊        | 6/33 [00:11<00:52,  1.93s/it]
2024-07-11 22:27:12 | ERROR | stderr | 
Loading checkpoint shards:  21%|██        | 7/33 [00:13<00:50,  1.93s/it]
2024-07-11 22:27:14 | ERROR | stderr | 
Loading checkpoint shards:  24%|██▍       | 8/33 [00:15<00:48,  1.92s/it]
2024-07-11 22:27:16 | ERROR | stderr | 
Loading checkpoint shards:  27%|██▋       | 9/33 [00:17<00:45,  1.91s/it]
2024-07-11 22:27:18 | ERROR | stderr | 
Loading checkpoint shards:  30%|███       | 10/33 [00:19<00:44,  1.92s/it]
2024-07-11 22:27:20 | ERROR | stderr | 
Loading checkpoint shards:  33%|███▎      | 11/33 [00:21<00:42,  1.92s/it]
2024-07-11 22:27:22 | ERROR | stderr | 
Loading checkpoint shards:  36%|███▋      | 12/33 [00:22<00:39,  1.90s/it]
2024-07-11 22:27:24 | ERROR | stderr | 
Loading checkpoint shards:  39%|███▉      | 13/33 [00:24<00:37,  1.89s/it]
2024-07-11 22:27:26 | ERROR | stderr | 
Loading checkpoint shards:  42%|████▏     | 14/33 [00:26<00:36,  1.90s/it]
2024-07-11 22:27:27 | ERROR | stderr | 
Loading checkpoint shards:  45%|████▌     | 15/33 [00:28<00:33,  1.88s/it]
2024-07-11 22:27:29 | ERROR | stderr | 
Loading checkpoint shards:  48%|████▊     | 16/33 [00:30<00:31,  1.88s/it]
2024-07-11 22:27:31 | ERROR | stderr | 
Loading checkpoint shards:  52%|█████▏    | 17/33 [00:32<00:30,  1.88s/it]
2024-07-11 22:27:33 | ERROR | stderr | 
Loading checkpoint shards:  55%|█████▍    | 18/33 [00:34<00:28,  1.91s/it]
2024-07-11 22:27:35 | ERROR | stderr | 
Loading checkpoint shards:  58%|█████▊    | 19/33 [00:36<00:26,  1.90s/it]
2024-07-11 22:27:38 | ERROR | stderr | 
Loading checkpoint shards:  61%|██████    | 20/33 [00:38<00:27,  2.10s/it]
2024-07-11 22:27:40 | ERROR | stderr | 
Loading checkpoint shards:  64%|██████▎   | 21/33 [00:40<00:24,  2.05s/it]
2024-07-11 22:27:42 | ERROR | stderr | 
Loading checkpoint shards:  67%|██████▋   | 22/33 [00:42<00:22,  2.04s/it]
2024-07-11 22:27:44 | ERROR | stderr | 
Loading checkpoint shards:  70%|██████▉   | 23/33 [00:44<00:20,  2.03s/it]
2024-07-11 22:27:46 | ERROR | stderr | 
Loading checkpoint shards:  73%|███████▎  | 24/33 [00:46<00:18,  2.07s/it]
2024-07-11 22:27:49 | ERROR | stderr | 
Loading checkpoint shards:  76%|███████▌  | 25/33 [00:49<00:18,  2.32s/it]
2024-07-11 22:27:51 | ERROR | stderr | 
Loading checkpoint shards:  79%|███████▉  | 26/33 [00:51<00:15,  2.21s/it]
2024-07-11 22:27:53 | ERROR | stderr | 
Loading checkpoint shards:  82%|████████▏ | 27/33 [00:54<00:13,  2.29s/it]
2024-07-11 22:27:55 | ERROR | stderr | 
Loading checkpoint shards:  85%|████████▍ | 28/33 [00:56<00:10,  2.18s/it]
2024-07-11 22:27:57 | ERROR | stderr | 
Loading checkpoint shards:  88%|████████▊ | 29/33 [00:57<00:08,  2.10s/it]
2024-07-11 22:27:59 | ERROR | stderr | 
Loading checkpoint shards:  91%|█████████ | 30/33 [01:00<00:06,  2.09s/it]
2024-07-11 22:28:01 | ERROR | stderr | 
Loading checkpoint shards:  94%|█████████▍| 31/33 [01:02<00:04,  2.07s/it]
2024-07-11 22:28:03 | ERROR | stderr | 
Loading checkpoint shards:  97%|█████████▋| 32/33 [01:03<00:01,  1.96s/it]
2024-07-11 22:28:04 | ERROR | stderr | 
Loading checkpoint shards: 100%|██████████| 33/33 [01:04<00:00,  1.73s/it]
2024-07-11 22:28:04 | ERROR | stderr | 
Loading checkpoint shards: 100%|██████████| 33/33 [01:04<00:00,  1.97s/it]
2024-07-11 22:28:04 | ERROR | stderr | 
2024-07-11 22:28:05 | INFO | model_worker | Register to controller
2024-07-11 22:28:05 | ERROR | stderr | INFO:     Started server process [28208]
2024-07-11 22:28:05 | ERROR | stderr | INFO:     Waiting for application startup.
2024-07-11 22:28:05 | ERROR | stderr | INFO:     Application startup complete.
2024-07-11 22:28:05 | ERROR | stderr | INFO:     Uvicorn running on http://0.0.0.0:40007 (Press CTRL+C to quit)
2024-07-11 22:28:07 | INFO | stdout | INFO:     10.140.60.209:42234 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:28:10 | INFO | stdout | INFO:     10.140.60.209:42262 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:28:12 | INFO | stdout | INFO:     10.140.60.209:42306 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:28:14 | INFO | stdout | INFO:     10.140.60.209:42350 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:28:14 | INFO | stdout | INFO:     10.140.60.209:42375 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:28:20 | INFO | model_worker | Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
2024-07-11 22:28:22 | INFO | stdout | INFO:     10.140.60.209:42566 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:28:25 | INFO | stdout | INFO:     10.140.60.209:42586 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:28:26 | INFO | stdout | INFO:     10.140.60.209:42610 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:28:26 | INFO | stdout | INFO:     10.140.60.209:42630 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:28:35 | INFO | model_worker | Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
2024-07-11 22:28:36 | INFO | stdout | INFO:     10.140.60.209:42690 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:28:37 | INFO | stdout | INFO:     10.140.60.209:42714 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:28:38 | INFO | stdout | INFO:     10.140.60.209:42738 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:28:50 | INFO | model_worker | Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
2024-07-11 22:29:05 | INFO | model_worker | Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
2024-07-11 22:29:20 | INFO | model_worker | Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
2024-07-11 22:29:35 | INFO | model_worker | Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
2024-07-11 22:29:50 | INFO | model_worker | Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
2024-07-11 22:30:05 | INFO | model_worker | Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
2024-07-11 22:30:20 | INFO | model_worker | Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
2024-07-11 22:30:35 | INFO | model_worker | Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
2024-07-11 22:30:50 | INFO | model_worker | Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
2024-07-11 22:31:05 | INFO | model_worker | Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
2024-07-11 22:31:20 | INFO | model_worker | Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
2024-07-11 22:31:35 | INFO | model_worker | Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
2024-07-11 22:31:50 | INFO | model_worker | Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0
2024-07-11 22:32:05 | INFO | model_worker | Send heart beat. Models: ['InternVL2-78B']. Semaphore: None. global_counter: 0