2024-07-11 22:32:13 | INFO | model_worker | args: Namespace(host='0.0.0.0', port=40006, worker_address='http://10.140.60.25:40006', controller_address='http://10.140.60.209:10075', model_path='share_internvl/InternVL2-40B/', model_name=None, device='auto', limit_model_concurrency=5, stream_interval=1, load_8bit=False)
2024-07-11 22:32:13 | INFO | model_worker | Loading the model InternVL2-40B on worker 762b3d ...
2024-07-11 22:32:13 | WARNING | transformers.tokenization_utils_base | Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
2024-07-11 22:32:13 | WARNING | transformers.tokenization_utils_base | Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
2024-07-11 22:32:15 | ERROR | stderr | /mnt/petrelfs/wangweiyun/miniconda3/envs/internvl-apex/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:397: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `None` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`. This was detected when initializing the generation config instance, which means the corresponding file may hold incorrect parameterization and should be fixed.
2024-07-11 22:32:15 | ERROR | stderr |   warnings.warn(
2024-07-11 22:32:16 | ERROR | stderr | 
Loading checkpoint shards:   0%|          | 0/17 [00:00<?, ?it/s]
2024-07-11 22:32:18 | ERROR | stderr | 
Loading checkpoint shards:   6%|▌         | 1/17 [00:01<00:31,  1.98s/it]
2024-07-11 22:32:20 | ERROR | stderr | 
Loading checkpoint shards:  12%|█▏        | 2/17 [00:03<00:29,  1.95s/it]
2024-07-11 22:32:22 | ERROR | stderr | 
Loading checkpoint shards:  18%|█▊        | 3/17 [00:05<00:27,  1.94s/it]
2024-07-11 22:32:24 | ERROR | stderr | 
Loading checkpoint shards:  24%|██▎       | 4/17 [00:07<00:26,  2.01s/it]
2024-07-11 22:32:26 | ERROR | stderr | 
Loading checkpoint shards:  29%|██▉       | 5/17 [00:10<00:25,  2.09s/it]
2024-07-11 22:32:28 | ERROR | stderr | 
Loading checkpoint shards:  35%|███▌      | 6/17 [00:12<00:22,  2.07s/it]
2024-07-11 22:32:30 | ERROR | stderr | 
Loading checkpoint shards:  41%|████      | 7/17 [00:14<00:20,  2.07s/it]
2024-07-11 22:32:32 | ERROR | stderr | 
Loading checkpoint shards:  47%|████▋     | 8/17 [00:16<00:18,  2.09s/it]
2024-07-11 22:32:34 | ERROR | stderr | 
Loading checkpoint shards:  53%|█████▎    | 9/17 [00:18<00:16,  2.06s/it]
2024-07-11 22:32:36 | ERROR | stderr | 
Loading checkpoint shards:  59%|█████▉    | 10/17 [00:20<00:14,  2.04s/it]
2024-07-11 22:32:38 | ERROR | stderr | 
Loading checkpoint shards:  65%|██████▍   | 11/17 [00:22<00:12,  2.05s/it]
2024-07-11 22:32:40 | ERROR | stderr | 
Loading checkpoint shards:  71%|███████   | 12/17 [00:24<00:10,  2.02s/it]
2024-07-11 22:32:43 | ERROR | stderr | 
Loading checkpoint shards:  76%|███████▋  | 13/17 [00:27<00:08,  2.22s/it]
2024-07-11 22:32:45 | ERROR | stderr | 
Loading checkpoint shards:  82%|████████▏ | 14/17 [00:29<00:06,  2.20s/it]
2024-07-11 22:32:48 | ERROR | stderr | 
Loading checkpoint shards:  88%|████████▊ | 15/17 [00:32<00:04,  2.39s/it]
2024-07-11 22:32:50 | ERROR | stderr | 
Loading checkpoint shards:  94%|█████████▍| 16/17 [00:34<00:02,  2.26s/it]
2024-07-11 22:32:51 | ERROR | stderr | 
Loading checkpoint shards: 100%|██████████| 17/17 [00:35<00:00,  1.93s/it]
2024-07-11 22:32:51 | ERROR | stderr | 
Loading checkpoint shards: 100%|██████████| 17/17 [00:35<00:00,  2.07s/it]
2024-07-11 22:32:51 | ERROR | stderr | 
2024-07-11 22:32:52 | INFO | model_worker | Register to controller
2024-07-11 22:32:52 | ERROR | stderr | INFO:     Started server process [86159]
2024-07-11 22:32:52 | ERROR | stderr | INFO:     Waiting for application startup.
2024-07-11 22:32:52 | ERROR | stderr | INFO:     Application startup complete.
2024-07-11 22:32:52 | ERROR | stderr | INFO:     Uvicorn running on http://0.0.0.0:40006 (Press CTRL+C to quit)
2024-07-11 22:33:07 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-11 22:33:11 | INFO | stdout | INFO:     10.140.60.209:48660 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:33:13 | INFO | stdout | INFO:     10.140.60.209:48714 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:33:13 | INFO | stdout | INFO:     10.140.60.209:48736 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:33:14 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=4, locked=False). global_counter: 1
2024-07-11 22:33:14 | INFO | stdout | INFO:     10.140.60.209:48756 - "POST /worker_generate_stream HTTP/1.1" 200 OK
2024-07-11 22:33:14 | INFO | model_worker | max_input_tile_list: [12]
2024-07-11 22:33:14 | INFO | model_worker | Split images to torch.Size([13, 3, 448, 448])
2024-07-11 22:33:14 | INFO | model_worker | []
2024-07-11 22:33:14 | INFO | model_worker | Generation config: {'num_beams': 1, 'max_new_tokens': 2048, 'do_sample': True, 'temperature': 0.8, 'repetition_penalty': 1.1, 'max_length': 8192, 'top_p': 0.7, 'streamer': <transformers.generation.streamers.TextIteratorStreamer object at 0x7f3dd8422530>}
2024-07-11 22:33:15 | ERROR | stderr | /mnt/petrelfs/wangweiyun/miniconda3/envs/internvl-apex/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:397: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `None` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`. This was detected when initializing the generation config instance, which means the corresponding file may hold incorrect parameterization and should be fixed.
2024-07-11 22:33:15 | ERROR | stderr |   warnings.warn(
2024-07-11 22:33:15 | WARNING | transformers.generation.utils | Both `max_new_tokens` (=2048) and `max_length`(=8192) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
2024-07-11 22:33:18 | ERROR | stderr | Exception in thread Thread-3 (chat):
2024-07-11 22:33:18 | ERROR | stderr | Traceback (most recent call last):
2024-07-11 22:33:18 | ERROR | stderr |   File "/mnt/petrelfs/wangweiyun/miniconda3/envs/internvl-apex/lib/python3.10/threading.py", line 1009, in _bootstrap_inner
2024-07-11 22:33:18 | ERROR | stderr |     self.run()
2024-07-11 22:33:18 | ERROR | stderr |   File "/mnt/petrelfs/wangweiyun/miniconda3/envs/internvl-apex/lib/python3.10/threading.py", line 946, in run
2024-07-11 22:33:18 | ERROR | stderr |     self._target(*self._args, **self._kwargs)
2024-07-11 22:33:18 | ERROR | stderr |   File "/mnt/petrelfs/wangweiyun/.cache/huggingface/modules/transformers_modules/InternVL2-40B/modeling_internvl_chat.py", line 280, in chat
2024-07-11 22:33:18 | ERROR | stderr |     generation_output = self.generate(
2024-07-11 22:33:18 | ERROR | stderr |   File "/mnt/petrelfs/wangweiyun/miniconda3/envs/internvl-apex/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2024-07-11 22:33:18 | ERROR | stderr |     return func(*args, **kwargs)
2024-07-11 22:33:18 | ERROR | stderr |   File "/mnt/petrelfs/wangweiyun/.cache/huggingface/modules/transformers_modules/InternVL2-40B/modeling_internvl_chat.py", line 330, in generate
2024-07-11 22:33:18 | ERROR | stderr |     outputs = self.language_model.generate(
2024-07-11 22:33:18 | ERROR | stderr |   File "/mnt/petrelfs/wangweiyun/miniconda3/envs/internvl-apex/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2024-07-11 22:33:18 | ERROR | stderr |     return func(*args, **kwargs)
2024-07-11 22:33:18 | ERROR | stderr |   File "/mnt/petrelfs/wangweiyun/miniconda3/envs/internvl-apex/lib/python3.10/site-packages/transformers/generation/utils.py", line 1525, in generate
2024-07-11 22:33:18 | ERROR | stderr |     return self.sample(
2024-07-11 22:33:18 | ERROR | stderr |   File "/mnt/petrelfs/wangweiyun/miniconda3/envs/internvl-apex/lib/python3.10/site-packages/transformers/generation/utils.py", line 2641, in sample
2024-07-11 22:33:18 | ERROR | stderr |     next_token_scores = logits_processor(input_ids, next_token_logits)
2024-07-11 22:33:18 | ERROR | stderr |   File "/mnt/petrelfs/wangweiyun/miniconda3/envs/internvl-apex/lib/python3.10/site-packages/transformers/generation/logits_process.py", line 97, in __call__
2024-07-11 22:33:18 | ERROR | stderr |     scores = processor(input_ids, scores)
2024-07-11 22:33:18 | ERROR | stderr |   File "/mnt/petrelfs/wangweiyun/miniconda3/envs/internvl-apex/lib/python3.10/site-packages/transformers/generation/logits_process.py", line 333, in __call__
2024-07-11 22:33:18 | ERROR | stderr |     score = torch.gather(scores, 1, input_ids)
2024-07-11 22:33:18 | ERROR | stderr | RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:2 and cuda:0! (when checking argument for argument index in method wrapper_CUDA_gather)
2024-07-11 22:33:22 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=4, locked=False). global_counter: 1
2024-07-11 22:33:24 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:33:37 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:33:52 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:34:02 | INFO | stdout | INFO:     10.140.60.209:49124 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:34:04 | INFO | stdout | INFO:     10.140.60.209:49144 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:34:05 | INFO | stdout | INFO:     10.140.60.209:49165 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:34:07 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:34:22 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:34:37 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:34:52 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:35:07 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:35:22 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:35:37 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:35:52 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:36:07 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:36:22 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:36:38 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:36:43 | INFO | stdout | INFO:     10.140.60.209:50210 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:36:49 | INFO | stdout | INFO:     10.140.60.209:50228 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:36:53 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:36:57 | INFO | stdout | INFO:     10.140.60.209:50330 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:37:00 | INFO | stdout | INFO:     10.140.60.209:50354 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:37:01 | INFO | stdout | INFO:     10.140.60.209:50374 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:37:08 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:37:23 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:37:38 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:37:53 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:38:08 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:38:23 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:38:38 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:38:53 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:39:08 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:39:23 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:39:38 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:39:53 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:40:08 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:40:23 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:40:38 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:40:53 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:40:59 | INFO | stdout | INFO:     10.140.60.209:52238 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:41:01 | INFO | stdout | INFO:     10.140.60.209:52258 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:41:01 | INFO | stdout | INFO:     10.140.60.209:52278 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:41:08 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:41:23 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:41:38 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:41:53 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:42:08 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:42:23 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:42:38 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:42:53 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:43:07 | INFO | stdout | INFO:     10.140.60.209:52936 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:43:08 | INFO | stdout | INFO:     10.140.60.209:52956 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:43:08 | INFO | stdout | INFO:     10.140.60.209:52976 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-11 22:43:08 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:43:23 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:43:38 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1
2024-07-11 22:43:39 | ERROR | stderr | INFO:     Shutting down