Repository: open-compass/opencompass
Issue: [Bug] 无法多卡并行评测数据
Root cause: Algorithmic Error

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

# 命令
CUDA_VISIBLE_DEVICES=6,7 opencompass --models vllm_qwen2_5_0_5b_instruct --datasets triviaqa_gen -a vllm --max-num-worker 2

# 执行上述模型文件，应该是运行下面路径的这个配置
./opencompass-main/opencompass/configs/models/qwen2_5/vllm_qwen2_5_0_5b_instruct.py
![image](https://github.com/user-attachments/assets/0ca310fa-8c14-45d1-9231-9051379a07b8)

# GPU只占用了一张卡
![image](https://github.com/user-attachments/assets/d251b6e9-77f5-44a0-9884-35a245c5dab9)


### 重现问题 - 代码/配置示例

见上述

### 重现问题 - 命令或脚本

见上述

### 重现问题 - 错误信息

见上述

### 其他信息

_No response_

# 如果用下面命令进行多卡数据并行，**单模型**，多数据，是可以得到预期
CUDA_VISIBLE_DEVICES=4,5,6,7 python3 run.py --models vllm_qwen2_5_0_5b_instruct --datasets  triviaqa_gen bbh_gen  --max-num-worker 4
# 但是用下面命令进行多卡数据并行，**多模型**，多数据，不能得到预期
CUDA_VISIBLE_DEVICES=4,5,6,7 python3 run.py --models vllm_qwen2_5_0_5b_instruct vllm_qwen2_5_3b_instruct --datasets  triviaqa_gen bbh_gen  --max-num-worker 4
通过查看日志：
1. 首先会进行vllm_qwen2_5_0_5b_instruct模型的推理，使用的是4卡进行数据并行推理，这部分正常，
2. 然后会执行vllm_qwen2_5_3b_instruct模型的推理，希望也是使用4卡进行数据并行推理，可实际上只使用了2卡，
下面是部分截图
一开始显示占用的是5，6显卡，后面又显示占用的是4，7显卡
![image](https://github.com/user-attachments/assets/0c7182b2-3224-497f-8d16-c195d0c6004e)
![image](https://github.com/user-attachments/assets/19d67a69-9498-4f7d-a339-505aaf4913c3)
下面是predictions文件夹显示的信息，显示占用的是5，6显卡的时候，predictions文件夹内会接着生成*_1，*_3.json的文件，
当上面图片显示占用的是4.7显卡的时候，predictions文件夹内会接着生成*_0，*_2.json的文件
![image](https://github.com/user-attachments/assets/f6f44f5d-ba0a-46f9-9d80-338e07561074)
![image](https://github.com/user-attachments/assets/6d76c1ec-f0d4-43ff-9030-8c0c5372c1ff)

# 疑问
为何脚本执行vllm_qwen2_5_3b_instruct 推理的时候显卡只用了2张，预期是希望4张卡进行数据并行




> # 如果用下面命令进行多卡数据并行，**单模型**，多数据，是可以得到预期
> CUDA_VISIBLE_DEVICES=4,5,6,7 python3 run.py --models vllm_qwen2_5_0_5b_instruct --datasets triviaqa_gen bbh_gen --max-num-worker 4
> 
> # 但是用下面命令进行多卡数据并行，**多模型**，多数据，不能得到预期
> CUDA_VISIBLE_DEVICES=4,5,6,7 python3 run.py --models vllm_qwen2_5_0_5b_instruct vllm_qwen2_5_3b_instruct --datasets triviaqa_gen bbh_gen --max-num-worker 4 通过查看日志：
> 
> 1. 首先会进行vllm_qwen2_5_0_5b_instruct模型的推理，使用的是4卡进行数据并行推理，这部分正常，
> 2. 然后会执行vllm_qwen2_5_3b_instruct模型的推理，希望也是使用4卡进行数据并行推理，可实际上只使用了2卡，
>    下面是部分截图
>    一开始显示占用的是5，6显卡，后面又显示占用的是4，7显卡
>    ![image](https://private-user-images.githubusercontent.com/11584869/394704885-0c7182b2-3224-497f-8d16-c195d0c6004e.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzUxNzY0OTEsIm5iZiI6MTczNTE3NjE5MSwicGF0aCI6Ii8xMTU4NDg2OS8zOTQ3MDQ4ODUtMGM3MTgyYjItMzIyNC00OTdmLThkMTYtYzE5NWQwYzYwMDRlLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDEyMjYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQxMjI2VDAxMjMxMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTU3NDE3NDhjMDI4ODg5ZDAxYzViMzAxZWJhN2E4ZTI4ZGVjYjAwN2ZlOTA1MzgwOTI3N2YxOTE1YmZlNjA0YTkmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.UbiEJ4JDWO2WAqJ77WIkUSRVkc-CFT_FC9w6p5IrJIc)
>    ![image](https://private-user-images.githubusercontent.com/11584869/394704804-19d67a69-9498-4f7d-a339-505aaf4913c3.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzUxNzY0OTEsIm5iZiI6MTczNTE3NjE5MSwicGF0aCI6Ii8xMTU4NDg2OS8zOTQ3MDQ4MDQtMTlkNjdhNjktOTQ5OC00ZjdkLWEzMzktNTA1YWFmNDkxM2MzLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDEyMjYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQxMjI2VDAxMjMxMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWYxMDhjYzViOGE0YmE1MGRiZjQ5ZDQ0MGQwZWNhOGE3YmQ0NmI3ZTlmMmY4ODBhNmI5MzJhYzY5ODVmODhlYjUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.uUD5gI6fYe5iF91fjOJmYpzAphawEiuUAFcx7pCCa6Q)
>    下面是predictions文件夹显示的信息，显示占用的是5，6显卡的时候，predictions文件夹内会接着生成*_1，__3.json的文件，
>    当上面图片显示占用的是4.7显卡的时候，predictions文件夹内会接着生成__0，*_2.json的文件
>    ![image](https://private-user-images.githubusercontent.com/11584869/394704949-f6f44f5d-ba0a-46f9-9d80-338e07561074.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzUxNzY0OTEsIm5iZiI6MTczNTE3NjE5MSwicGF0aCI6Ii8xMTU4NDg2OS8zOTQ3MDQ5NDktZjZmNDRmNWQtYmEwYS00NmY5LTlkODAtMzM4ZTA3NTYxMDc0LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDEyMjYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQxMjI2VDAxMjMxMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTYwY2RmYWFlN2NjYWY5MDdhOGMzZDcyZTJkNWI3MzYzNzZkMjNhNWJhNDY1ZDliZGRlODliYmRmMjY1N2ZlMmQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.Ms-_95wGb1YFXsqQ9oFHD_dWYwpMH2_XTyhrceDoZRY)
>    ![image](https://private-user-images.githubusercontent.com/11584869/394705724-6d76c1ec-f0d4-43ff-9030-8c0c5372c1ff.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzUxNzY0OTEsIm5iZiI6MTczNTE3NjE5MSwicGF0aCI6Ii8xMTU4NDg2OS8zOTQ3MDU3MjQtNmQ3NmMxZWMtZjBkNC00M2ZmLTkwMzAtOGMwYzUzNzJjMWZmLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDEyMjYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQxMjI2VDAxMjMxMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPThmMWQyMWI3ZTU3MmExMzVkZDUxYTIzYmQ1ZWMzOGQwZjAxNGM3MmE2NzgyZjk0ZDRkYjkyZjJkYTRkY2U2YjcmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.Bm3Mge-qj_GhvMuItbUQ9G7DtADhZQtylhlv4N1qt0s)
> 
> # 疑问
> 为何脚本执行vllm_qwen2_5_3b_instruct 推理的时候显卡只用了2张，预期是希望4张卡进行数据并行

你多模型也能跑起来吗？我的多模型一直报错，每次运行到第二个模型的时候就会卡住，最后包连接错误的问题。
![image](https://github.com/user-attachments/assets/e0b77131-976e-4c44-9b04-e2d576f4b643
[2024-12-26 01:14:33,214] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting build dataloader
[2024-12-26 01:14:33,215] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
Processed prompts: 100%|███████████████████████████████████████████████| 4/4 [00:05<00:00,  1.47s/it, est. speed input: 40.86 toks/s, output: 470.37 toks/s]
Processed prompts: 100%|███████████████████████████████████████████████| 4/4 [00:03<00:00,  1.22it/s, est. speed input: 58.39 toks/s, output: 552.45 toks/s]
Processed prompts: 100%|███████████████████████████████████████████████| 4/4 [00:05<00:00,  1.47s/it, est. speed input: 56.83 toks/s, output: 449.85 toks/s]
Processed prompts: 100%|███████████████████████████████████████████████| 4/4 [00:05<00:00,  1.48s/it, est. speed input: 64.75 toks/s, output: 536.68 toks/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:20<00:00,  5.24s/it]
12/26 01:14:54 - OpenCompass - INFO - Task [qwen-7b-sft-vllm/demo_gsm8k_0,qwen-7b-sft-vllm/demo_math_0]
INFO 12-26 01:14:54 config.py:478] This model supports multiple tasks: {'score', 'generate', 'embed', 'reward', 'classify'}. Defaulting to 'generate'.
INFO 12-26 01:14:54 config.py:1216] Defaulting to use mp for distributed inference
INFO 12-26 01:14:54 llm_engine.py:249] Initializing an LLM engine (v0.6.5) with config: model='/data3/train_result/llamafactory/qwen2.5-7b-instruct/full/sft/checkpoint-5500/', speculative_config=None, tokenizer='/data3/train_result/llamafactory/qwen2.5-7b-instruct/full/sft/checkpoint-5500/', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=2048, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=/data3/train_result/llamafactory/qwen2.5-7b-instruct/full/sft/checkpoint-5500/, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=True, mm_cache_preprocessor=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"candidate_compile_sizes":[],"compile_sizes":[],"capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=False,
/home/qiushiquan/opencompass/opencompass/__init__.py:19: UserWarning: Starting from v0.4.0, all AMOTIC configuration files currently located in `./configs/datasets`, `./configs/models`, and `./configs/summarizers` will be migrated to the `opencompass/configs/` package. Please update your configuration file paths accordingly.
  _warn_about_config_migration()
(VllmWorkerProcess pid=1531428) INFO 12-26 01:15:00 selector.py:120] Using Flash Attention backend.
(VllmWorkerProcess pid=1531428) INFO 12-26 01:15:00 multiproc_worker_utils.py:222] Worker ready; awaiting tasks
[E1226 01:24:20.242697548 socket.cpp:1011] [c10d] The client socket has timed out after 600000ms while trying to connect to (127.0.0.1, 37915).
[W1226 01:24:20.243528552 TCPStore.cpp:358] [c10d] TCP client failed to connect/validate to host 127.0.0.1:37915 - retrying (try=0, timeout=600000ms, delay=38719ms): The client socket has timed out after 600000ms while trying to connect to (127.0.0.1, 37915).
Exception raised from throwTimeoutError at ../torch/csrc/distributed/c10d/socket.cpp:1013 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fd14b36c446 in /data0/miniconda3/envs/opencompass/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x15e04c6 (0x7fd13654d4c6 in /data0/miniconda3/envs/opencompass/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #2: <unknown function> + 0x6029d95 (0x7fd13af96d95 in /data0/miniconda3/envs/opencompass/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #3: <unknown function> + 0x6029f36 (0x7fd13af96f36 in /data0/miniconda3/envs/opencompass/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #4: <unknown function> + 0x602a3a4 (0x7fd13af973a4 in /data0/miniconda3/envs/opencompass/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #5: <unknown function> + 0x5fe8016 (0x7fd13af55016 in /data0/miniconda3/envs/opencompass/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #6: c10d::TCPStore::TCPStore(std::string, c10d::TCPStoreOptions const&) + 0x20c (0x7fd13af57f7c in /data0/miniconda3/envs/opencompass/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #7: <unknown function> + 0xd9acdd (0x7fd14a93acdd in /data0/miniconda3/envs/opencompass/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #8: <unknown function> + 0x4cb474 (0x7fd14a06b474 in /data0/miniconda3/envs/opencompass/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #9: <unknown function> + 0x172df4 (0x56411107cdf4 in /data0/miniconda3/envs/opencompass/bin/python)
frame #10: _PyObject_MakeTpCall + 0x1f8 (0x564111043db8 in /data0/miniconda3/envs/opencompass/bin/python)
frame #11: <unknown function> + 0xeb5a7 (0x564110ff55a7 in /data0/miniconda3/envs/opencompass/bin/python)
frame #12: _PyObject_Call + 0x295 (0x56411104a495 in /data0/miniconda3/envs/opencompass/bin/python)
frame #13: <unknown function> + 0xb87fc (0x564110fc27fc in /data0/miniconda3/envs/opencompass/bin/python)
frame #14: <unknown function> + 0x153a21 (0x56411105da21 in /data0/miniconda3/envs/opencompass/bin/python)
frame #15: <unknown function> + 0x4c9ccb (0x7fd14a069ccb in /data0/miniconda3/envs/opencompass/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #16: _PyObject_MakeTpCall + 0x1f8 (0x564111043db8 in /data0/miniconda3/envs/opencompass/bin/python)
frame #17: _PyEval_EvalFrameDefault + 0x15b1 (0x5641110e3fb1 in /data0/miniconda3/envs/opencompass/bin/python)
frame #18: <unknown function> + 0x1871eb (0x5641110911eb in /data0/miniconda3/envs/opencompass/bin/python)
frame #19: <unknown function> + 0x105472 (0x56411100f472 in /data0/miniconda3/envs/opencompass/bin/python)
frame #20: <unknown function> + 0x18bb4c (0x564111095b4c in /data0/miniconda3/envs/opencompass/bin/python)


> > # 如果用下面命令进行多卡数据并行，**单模型**，多数据，是可以得到预期
> > CUDA_VISIBLE_DEVICES=4,5,6,7 python3 run.py --models vllm_qwen2_5_0_5b_instruct --datasets triviaqa_gen bbh_gen --max-num-worker 4
> > # 但是用下面命令进行多卡数据并行，**多模型**，多数据，不能得到预期
> > CUDA_VISIBLE_DEVICES=4,5,6,7 python3 run.py --models vllm_qwen2_5_0_5b_instruct vllm_qwen2_5_3b_instruct --datasets triviaqa_gen bbh_gen --max-num-worker 4 通过查看日志：
> > 
> > 1. 首先会进行vllm_qwen2_5_0_5b_instruct模型的推理，使用的是4卡进行数据并行推理，这部分正常，
> > 2. 然后会执行vllm_qwen2_5_3b_instruct模型的推理，希望也是使用4卡进行数据并行推理，可实际上只使用了2卡，
> >    下面是部分截图
> >    一开始显示占用的是5，6显卡，后面又显示占用的是4，7显卡
> >    ![image](https://private-user-images.githubusercontent.com/11584869/394704885-0c7182b2-3224-497f-8d16-c195d0c6004e.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzUxNzY0OTEsIm5iZiI6MTczNTE3NjE5MSwicGF0aCI6Ii8xMTU4NDg2OS8zOTQ3MDQ4ODUtMGM3MTgyYjItMzIyNC00OTdmLThkMTYtYzE5NWQwYzYwMDRlLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDEyMjYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQxMjI2VDAxMjMxMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTU3NDE3NDhjMDI4ODg5ZDAxYzViMzAxZWJhN2E4ZTI4ZGVjYjAwN2ZlOTA1MzgwOTI3N2YxOTE1YmZlNjA0YTkmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.UbiEJ4JDWO2WAqJ77WIkUSRVkc-CFT_FC9w6p5IrJIc)
> >    ![image](https://private-user-images.githubusercontent.com/11584869/394704804-19d67a69-9498-4f7d-a339-505aaf4913c3.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzUxNzY0OTEsIm5iZiI6MTczNTE3NjE5MSwicGF0aCI6Ii8xMTU4NDg2OS8zOTQ3MDQ4MDQtMTlkNjdhNjktOTQ5OC00ZjdkLWEzMzktNTA1YWFmNDkxM2MzLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDEyMjYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQxMjI2VDAxMjMxMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWYxMDhjYzViOGE0YmE1MGRiZjQ5ZDQ0MGQwZWNhOGE3YmQ0NmI3ZTlmMmY4ODBhNmI5MzJhYzY5ODVmODhlYjUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.uUD5gI6fYe5iF91fjOJmYpzAphawEiuUAFcx7pCCa6Q)
> >    下面是predictions文件夹显示的信息，显示占用的是5，6显卡的时候，predictions文件夹内会接着生成*_1，__3.json的文件，
> >    当上面图片显示占用的是4.7显卡的时候，predictions文件夹内会接着生成__0，*_2.json的文件
> >    ![image](https://private-user-images.githubusercontent.com/11584869/394704949-f6f44f5d-ba0a-46f9-9d80-338e07561074.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzUxNzY0OTEsIm5iZiI6MTczNTE3NjE5MSwicGF0aCI6Ii8xMTU4NDg2OS8zOTQ3MDQ5NDktZjZmNDRmNWQtYmEwYS00NmY5LTlkODAtMzM4ZTA3NTYxMDc0LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDEyMjYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQxMjI2VDAxMjMxMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTYwY2RmYWFlN2NjYWY5MDdhOGMzZDcyZTJkNWI3MzYzNzZkMjNhNWJhNDY1ZDliZGRlODliYmRmMjY1N2ZlMmQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.Ms-_95wGb1YFXsqQ9oFHD_dWYwpMH2_XTyhrceDoZRY)
> >    ![image](https://private-user-images.githubusercontent.com/11584869/394705724-6d76c1ec-f0d4-43ff-9030-8c0c5372c1ff.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzUxNzY0OTEsIm5iZiI6MTczNTE3NjE5MSwicGF0aCI6Ii8xMTU4NDg2OS8zOTQ3MDU3MjQtNmQ3NmMxZWMtZjBkNC00M2ZmLTkwMzAtOGMwYzUzNzJjMWZmLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDEyMjYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQxMjI2VDAxMjMxMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPThmMWQyMWI3ZTU3MmExMzVkZDUxYTIzYmQ1ZWMzOGQwZjAxNGM3MmE2NzgyZjk0ZDRkYjkyZjJkYTRkY2U2YjcmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.Bm3Mge-qj_GhvMuItbUQ9G7DtADhZQtylhlv4N1qt0s)
> > 
> > # 疑问
> > 为何脚本执行vllm_qwen2_5_3b_instruct 推理的时候显卡只用了2张，预期是希望4张卡进行数据并行
> 
> 你多模型也能跑起来吗？我的多模型一直报错，每次运行到第二个模型的时候就会卡住，最后包连接错误的问题。 ![image](https://github.com/user-attachments/assets/e0b77131-976e-4c44-9b04-e2d576f4b643 [2024-12-26 01:14:33,214] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting build dataloader [2024-12-26 01:14:33,215] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process... Processed prompts: 100%|███████████████████████████████████████████████| 4/4 [00:05<00:00, 1.47s/it, est. speed input: 40.86 toks/s, output: 470.37 toks/s] Processed prompts: 100%|███████████████████████████████████████████████| 4/4 [00:03<00:00, 1.22it/s, est. speed input: 58.39 toks/s, output: 552.45 toks/s] Processed prompts: 100%|███████████████████████████████████████████████| 4/4 [00:05<00:00, 1.47s/it, est. speed input: 56.83 toks/s, output: 449.85 toks/s] Processed prompts: 100%|███████████████████████████████████████████████| 4/4 [00:05<00:00, 1.48s/it, est. speed input: 64.75 toks/s, output: 536.68 toks/s] 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:20<00:00, 5.24s/it] 12/26 01:14:54 - OpenCompass - INFO - Task [qwen-7b-sft-vllm/demo_gsm8k_0,qwen-7b-sft-vllm/demo_math_0] INFO 12-26 01:14:54 config.py:478] This model supports multiple tasks: {'score', 'generate', 'embed', 'reward', 'classify'}. Defaulting to 'generate'. INFO 12-26 01:14:54 config.py:1216] Defaulting to use mp for distributed inference INFO 12-26 01:14:54 llm_engine.py:249] Initializing an LLM engine (v0.6.5) with config: model='/data3/train_result/llamafactory/qwen2.5-7b-instruct/full/sft/checkpoint-5500/', speculative_config=None, tokenizer='/data3/train_result/llamafactory/qwen2.5-7b-instruct/full/sft/checkpoint-5500/', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=2048, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=/data3/train_result/llamafactory/qwen2.5-7b-instruct/full/sft/checkpoint-5500/, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=True, mm_cache_preprocessor=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"candidate_compile_sizes":[],"compile_sizes":[],"capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=False, /home/qiushiquan/opencompass/opencompass/**init**.py:19: UserWarning: Starting from v0.4.0, all AMOTIC configuration files currently located in `./configs/datasets`, `./configs/models`, and `./configs/summarizers` will be migrated to the `opencompass/configs/` package. Please update your configuration file paths accordingly. _warn_about_config_migration() (VllmWorkerProcess pid=1531428) INFO 12-26 01:15:00 selector.py:120] Using Flash Attention backend. (VllmWorkerProcess pid=1531428) INFO 12-26 01:15:00 multiproc_worker_utils.py:222] Worker ready; awaiting tasks [E1226 01:24:20.242697548 socket.cpp:1011] [c10d] The client socket has timed out after 600000ms while trying to connect to (127.0.0.1, 37915). [W1226 01:24:20.243528552 TCPStore.cpp:358] [c10d] TCP client failed to connect/validate to host 127.0.0.1:37915 - retrying (try=0, timeout=600000ms, delay=38719ms): The client socket has timed out after 600000ms while trying to connect to (127.0.0.1, 37915). Exception raised from throwTimeoutError at ../torch/csrc/distributed/c10d/socket.cpp:1013 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fd14b36c446 in /data0/miniconda3/envs/opencompass/lib/python3.10/site-packages/torch/lib/libc10.so) frame #1: + 0x15e04c6 (0x7fd13654d4c6 in /data0/miniconda3/envs/opencompass/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x6029d95 (0x7fd13af96d95 in /data0/miniconda3/envs/opencompass/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x6029f36 (0x7fd13af96f36 in /data0/miniconda3/envs/opencompass/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) frame #4: + 0x602a3a4 (0x7fd13af973a4 in /data0/miniconda3/envs/opencompass/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) frame #5: + 0x5fe8016 (0x7fd13af55016 in /data0/miniconda3/envs/opencompass/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) frame #6: c10d::TCPStore::TCPStore(std::string, c10d::TCPStoreOptions const&) + 0x20c (0x7fd13af57f7c in /data0/miniconda3/envs/opencompass/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) frame #7: + 0xd9acdd (0x7fd14a93acdd in /data0/miniconda3/envs/opencompass/lib/python3.10/site-packages/torch/lib/libtorch_python.so) frame #8: + 0x4cb474 (0x7fd14a06b474 in /data0/miniconda3/envs/opencompass/lib/python3.10/site-packages/torch/lib/libtorch_python.so) frame #9: + 0x172df4 (0x56411107cdf4 in /data0/miniconda3/envs/opencompass/bin/python) frame #10: _PyObject_MakeTpCall + 0x1f8 (0x564111043db8 in /data0/miniconda3/envs/opencompass/bin/python) frame #11: + 0xeb5a7 (0x564110ff55a7 in /data0/miniconda3/envs/opencompass/bin/python) frame #12: _PyObject_Call + 0x295 (0x56411104a495 in /data0/miniconda3/envs/opencompass/bin/python) frame #13: + 0xb87fc (0x564110fc27fc in /data0/miniconda3/envs/opencompass/bin/python) frame #14: + 0x153a21 (0x56411105da21 in /data0/miniconda3/envs/opencompass/bin/python) frame #15: + 0x4c9ccb (0x7fd14a069ccb in /data0/miniconda3/envs/opencompass/lib/python3.10/site-packages/torch/lib/libtorch_python.so) frame #16: _PyObject_MakeTpCall + 0x1f8 (0x564111043db8 in /data0/miniconda3/envs/opencompass/bin/python) frame #17: _PyEval_EvalFrameDefault + 0x15b1 (0x5641110e3fb1 in /data0/miniconda3/envs/opencompass/bin/python) frame #18: + 0x1871eb (0x5641110911eb in /data0/miniconda3/envs/opencompass/bin/python) frame #19: + 0x105472 (0x56411100f472 in /data0/miniconda3/envs/opencompass/bin/python) frame #20: + 0x18bb4c (0x564111095b4c in /data0/miniconda3/envs/opencompass/bin/python)

感觉你这个是网络超时之类的问题，用的torch版本是哪个，或者重新搭建一下环境

你好，请问这个问题解决了吗, 我不论是单模型还是多模型都无法运行在多卡上
命令为python run.py configs/eval_OC15_llama3.1_qwen2_custom_gen.py --max-num-workers 8 --hf-num-gpus 8。
开启tp的时候虽然日志写了ON GPU0, ... 7，但是我查看资源发现还是只有一张卡在实际跑呢.
另外--max-num-workers 8开启dp试了一下发现也是类似的问题, 它确实是切分评测集了, 但是好像只在单卡上运行1/8的评测集，然后串行执行的接下来的第二个1/8，那这样也起不到加速的效果啊





> 你好，请问这个问题解决了吗, 我不论是单模型还是多模型都无法运行在多卡上 命令为python run.py configs/eval_OC15_llama3.1_qwen2_custom_gen.py --max-num-workers 8 --hf-num-gpus 8。 开启tp的时候虽然日志写了ON GPU0, ... 7，但是我查看资源发现还是只有一张卡在实际跑呢. 另外--max-num-workers 8开启dp试了一下发现也是类似的问题, 它确实是切分评测集了, 但是好像只在单卡上运行1/8的评测集，然后串行执行的接下来的第二个1/8，那这样也起不到加速的效果啊

贴一下这个文件内容，configs/eval_OC15_llama3.1_qwen2_custom_gen.py -


> > 你好，请问这个问题解决了吗, 我不论是单模型还是多模型都无法运行在多卡上 命令为python run.py configs/eval_OC15_llama3.1_qwen2_custom_gen.py --max-num-workers 8 --hf-num-gpus 8。 开启tp的时候虽然日志写了ON GPU0, ... 7，但是我查看资源发现还是只有一张卡在实际跑呢. 另外--max-num-workers 8开启dp试了一下发现也是类似的问题, 它确实是切分评测集了, 但是好像只在单卡上运行1/8的评测集，然后串行执行的接下来的第二个1/8，那这样也起不到加速的效果啊
> 
> 贴一下这个文件内容，configs/eval_OC15_llama3.1_qwen2_custom_gen.py -

from mmengine.config import read_base

with read_base():
    from opencompass.configs.datasets.mmlu.mmlu_gen_4d595a import mmlu_datasets
    from opencompass.configs.models.hf_llama.hf_llama3_1_8b_instruct import models as hf_llama3_1_8b_instruct_model
    from opencompass.configs.summarizers.chat_OC15 import summarizer
work_dir = "outputs/Llama3_1-7B-Instruct"
datasets = sum((v for k, v in locals().items() if k.endswith('_datasets')), [])
models = sum([v for k, v in locals().items() if k.endswith('_model')], [])
hf_llama3_1_8b_instruct.py的配置就是默认的配置(我把num_gpus=1改成8也没用)

> > > 你好，请问这个问题解决了吗, 我不论是单模型还是多模型都无法运行在多卡上 命令为python run.py configs/eval_OC15_llama3.1_qwen2_custom_gen.py --max-num-workers 8 --hf-num-gpus 8。 开启tp的时候虽然日志写了ON GPU0, ... 7，但是我查看资源发现还是只有一张卡在实际跑呢. 另外--max-num-workers 8开启dp试了一下发现也是类似的问题, 它确实是切分评测集了, 但是好像只在单卡上运行1/8的评测集，然后串行执行的接下来的第二个1/8，那这样也起不到加速的效果啊
> > 
> > 
> > 贴一下这个文件内容，configs/eval_OC15_llama3.1_qwen2_custom_gen.py -
> 
> from mmengine.config import read_base
> 
> with read_base(): from opencompass.configs.datasets.mmlu.mmlu_gen_4d595a import mmlu_datasets from opencompass.configs.models.hf_llama.hf_llama3_1_8b_instruct import models as hf_llama3_1_8b_instruct_model from opencompass.configs.summarizers.chat_OC15 import summarizer work_dir = "outputs/Llama3_1-7B-Instruct" datasets = sum((v for k, v in locals().items() if k.endswith('_datasets')), []) models = sum([v for k, v in locals().items() if k.endswith('_model')], []) hf_llama3_1_8b_instruct.py的配置就是默认的配置(我把num_gpus=1改成8也没用)

Any updates on how you actually solve this issue. Many thanks.

> > > > 你好，请问这个问题解决了吗, 我不论是单模型还是多模型都无法运行在多卡上 命令为python run.py configs/eval_OC15_llama3.1_qwen2_custom_gen.py --max-num-workers 8 --hf-num-gpus 8。 开启tp的时候虽然日志写了ON GPU0, ... 7，但是我查看资源发现还是只有一张卡在实际跑呢. 另外--max-num-workers 8开启dp试了一下发现也是类似的问题, 它确实是切分评测集了, 但是好像只在单卡上运行1/8的评测集，然后串行执行的接下来的第二个1/8，那这样也起不到加速的效果啊
> > > 
> > > 
> > > 贴一下这个文件内容，configs/eval_OC15_llama3.1_qwen2_custom_gen.py -
> > 
> > 
> > from mmengine.config import read_base
> > with read_base(): from opencompass.configs.datasets.mmlu.mmlu_gen_4d595a import mmlu_datasets from opencompass.configs.models.hf_llama.hf_llama3_1_8b_instruct import models as hf_llama3_1_8b_instruct_model from opencompass.configs.summarizers.chat_OC15 import summarizer work_dir = "outputs/Llama3_1-7B-Instruct" datasets = sum((v for k, v in locals().items() if k.endswith('_datasets')), []) models = sum([v for k, v in locals().items() if k.endswith('_model')], []) hf_llama3_1_8b_instruct.py的配置就是默认的配置(我把num_gpus=1改成8也没用)
> 
> Any updates on how you actually solve this issue. Many thanks.

Hi, I found that config/**configs.py didn't work on DP and TP, I ended up specifying the model and dataset directly like this: python run.py --models your_model.py --datasets your_datasets.py --max-num-workers 8 for 8 DP, datasets can be customised for integration.