logs/model_worker_35f772.log

2024-07-10 17:34:48 | INFO | model_worker | args: Namespace(host='0.0.0.0', port=40003, worker_address='http://10.140.60.25:40003', controller_address='http://10.140.60.209:10075', model_path='share_internvl/InternVL2-4B/', model_name=None, device='cuda', limit_model_concurrency=5, stream_interval=1, load_8bit=False)
2024-07-10 17:34:48 | INFO | model_worker | Loading the model InternVL2-4B on worker 35f772 ...
2024-07-10 17:34:48 | WARNING | transformers.tokenization_utils_base | Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
2024-07-10 17:34:48 | WARNING | transformers.tokenization_utils_base | Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
2024-07-10 17:34:50 | ERROR | stderr | /mnt/petrelfs/wangweiyun/miniconda3/envs/internvl-apex/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:397: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `None` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`. This was detected when initializing the generation config instance, which means the corresponding file may hold incorrect parameterization and should be fixed.
2024-07-10 17:34:50 | ERROR | stderr |   warnings.warn(
2024-07-10 17:34:50 | ERROR | stderr | 
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
2024-07-10 17:35:04 | ERROR | stderr | 
Loading checkpoint shards:  50%|█████     | 1/2 [00:14<00:14, 14.03s/it]
2024-07-10 17:35:12 | ERROR | stderr | 
Loading checkpoint shards: 100%|██████████| 2/2 [00:21<00:00, 10.38s/it]
2024-07-10 17:35:12 | ERROR | stderr | 
Loading checkpoint shards: 100%|██████████| 2/2 [00:21<00:00, 10.93s/it]
2024-07-10 17:35:12 | ERROR | stderr | 
2024-07-10 17:35:16 | INFO | model_worker | Register to controller
2024-07-10 17:35:16 | ERROR | stderr | INFO:     Started server process [4316]
2024-07-10 17:35:16 | ERROR | stderr | INFO:     Waiting for application startup.
2024-07-10 17:35:16 | ERROR | stderr | INFO:     Application startup complete.
2024-07-10 17:35:16 | ERROR | stderr | INFO:     Uvicorn running on http://0.0.0.0:40003 (Press CTRL+C to quit)
2024-07-10 17:35:19 | INFO | stdout | INFO:     10.140.60.209:38568 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 17:35:31 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:35:46 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:36:01 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:36:16 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:36:31 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:36:46 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:37:01 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:37:16 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:37:31 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:37:46 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:38:01 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:38:15 | INFO | stdout | INFO:     10.140.60.209:39160 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 17:38:16 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:38:17 | INFO | stdout | INFO:     10.140.60.209:39230 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 17:38:26 | INFO | stdout | INFO:     10.140.60.209:39274 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 17:38:29 | INFO | stdout | INFO:     10.140.60.209:39290 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 17:38:30 | INFO | stdout | INFO:     10.140.60.209:39304 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 17:38:31 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:38:46 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:38:50 | INFO | stdout | INFO:     10.140.60.209:39328 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 17:39:01 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:39:16 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:39:31 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:39:42 | INFO | stdout | INFO:     10.140.60.209:39532 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 17:39:46 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:40:01 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:40:16 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:40:31 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:40:46 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:41:01 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:41:16 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:41:31 | INFO | stdout | INFO:     10.140.60.209:39946 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 17:41:31 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:41:46 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:42:01 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:42:16 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:42:31 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:42:46 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:43:01 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:43:16 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:43:31 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:43:46 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:44:01 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:44:16 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:44:31 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:44:46 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:45:01 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:45:16 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:45:31 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:45:46 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:46:01 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:46:16 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:46:32 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:46:47 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:47:02 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:47:17 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:47:32 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:47:47 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:48:02 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:48:17 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:48:32 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:48:47 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:49:02 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:49:17 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:49:32 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:49:47 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:50:02 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:50:17 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:50:32 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:50:47 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:51:02 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:51:17 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:51:32 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:51:47 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:52:02 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:52:17 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:52:32 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:52:47 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:53:02 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:53:17 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:53:32 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:53:47 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:54:02 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:54:17 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:54:32 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:54:47 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:55:02 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:55:17 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:55:32 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:55:47 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:56:02 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:56:17 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:56:32 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:56:33 | INFO | stdout | INFO:     10.140.60.209:43642 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 17:56:36 | INFO | stdout | INFO:     10.140.60.209:43665 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 17:56:37 | INFO | stdout | INFO:     10.140.60.209:43682 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 17:56:47 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:57:02 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:57:17 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:57:32 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:57:48 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:58:03 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:58:18 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:58:33 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:58:48 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:59:03 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:59:18 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:59:33 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 17:59:48 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 18:00:03 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 18:00:15 | INFO | stdout | INFO:     10.140.60.209:44476 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 18:00:18 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 18:00:18 | INFO | stdout | INFO:     10.140.60.209:44556 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 18:00:19 | INFO | stdout | INFO:     10.140.60.209:44574 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 18:00:33 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 18:00:48 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 18:01:03 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 18:01:18 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 18:01:30 | INFO | stdout | INFO:     10.140.60.209:44792 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 18:01:33 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 18:01:33 | INFO | stdout | INFO:     10.140.60.209:44806 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 18:01:36 | INFO | stdout | INFO:     10.140.60.209:44822 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 18:01:38 | INFO | stdout | INFO:     10.140.60.209:44839 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 18:01:48 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 18:02:03 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 18:02:18 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 18:02:33 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 18:02:48 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 18:03:03 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 18:03:18 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 18:03:33 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 18:03:48 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 18:04:03 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 18:04:18 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 18:04:33 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 18:04:48 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 18:05:03 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0
2024-07-10 18:05:18 | INFO | model_worker | Send heart beat. Models: ['InternVL2-4B']. Semaphore: None. global_counter: 0