Repository: open-compass/opencompass
Issue: 不知道怎么在单机多卡上推理
Root cause: Documentation Deficiency

### Describe the feature

目前能跑通单机单卡。但是在单机多卡，以及多机多卡我不知道怎么设置。

我期望 
    我的模型比较小，我一台机器8张卡，每个卡均加载一份模型，然后8个卡同时推理，快很多。

但是目前我不知道怎么设置。

看了之前的issue，  run_cfg=dict(num_gpus=1, num_procs=1), 这俩是位大模型加载而服务的

我不太明白max-partition-size 的运作机制，是多开几个并发，那每个并发都会加载一个模型么？并发和GPU使用的数量有啥关系。

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

不需要考虑，它会自动检测GPU数目，然后开启多个进程。

> run_cfg=dict(num_gpus=1, num_procs=1)

这里的配置更多是跑起来一个模型需要的卡数

python run.py configs/eval_demo.py -w outputs/demo   

我直接这么执行，我发现他们都跑一个卡里去执行了，导致显存溢出
eval_demo 如下。

from opencompass.models import HuggingFaceCausalLM
from mmengine.config import read_base

with read_base():
    from .datasets.winograd.winograd_ppl import winograd_datasets
    from .datasets.siqa.siqa_gen import siqa_datasets

    from .datasets.ceval.ceval_gen import ceval_datasets

datasets = [*ceval_datasets]

_meta_template = dict(
    round=[
        dict(role="HUMAN", begin='\n
user\n', end='
'),
        dict(role="BOT", begin="\n
assistant\n", end='
', generate=True),
    ],
)

models = [
    dict(
        type=HuggingFaceCausalLM,
        abbr='qwen-7b-chat-hf',
        path="/mnt/workspace/llm/model/manual/qwen/Qwen-7B-Chat",
        tokenizer_path='/mnt/workspace/llm/model/manual/qwen/Qwen-7B-Chat',
        tokenizer_kwargs=dict(padding_side='left',
                              truncation_side='left',
                              trust_remote_code=True,
                              use_fast=False,
                              pad_token="
",
                              ),
        max_out_len=512,
        max_seq_len=2048,
        batch_size=8,
        meta_template=_meta_template,
        model_kwargs=dict(device_map='auto', trust_remote_code=True),
        run_cfg=dict(num_gpus=1, num_procs=1),
    )
]




launch OpenICLInfer[qwen-7b-chat-hf/ceval-college_economics] on GPU 0                                                                                                                                                                                                                 
launch OpenICLInfer[qwen-7b-chat-hf/ceval-accountant,qwen-7b-chat-hf/ceval-tax_accountant] on GPU 0                                                                                                                                                                                   
launch OpenICLInfer[qwen-7b-chat-hf/ceval-physician,qwen-7b-chat-hf/ceval-civil_servant] on GPU 0                                                                                                                                                                                     
launch OpenICLInfer[qwen-7b-chat-hf/ceval-urban_and_rural_planner,qwen-7b-chat-hf/ceval-teacher_qualification] on GPU 0                                                                                                                                                               
launch OpenICLInfer[qwen-7b-chat-hf/ceval-college_programming,qwen-7b-chat-hf/ceval-electrical_engineer] on GPU 0                                                                                                                                                                     
launch OpenICLInfer[qwen-7b-chat-hf/ceval-business_administration,qwen-7b-chat-hf/ceval-art_studies,qwen-7b-chat-hf/ceval-fire_engineer] on GPU 0                                                                                                                                     
launch OpenICLInfer[qwen-7b-chat-hf/ceval-environmental_impact_assessment_engineer,qwen-7b-chat-hf/ceval-education_science,qwen-7b-chat-hf/ceval-professional_tour_guide] on GPU 0                                                                                                    
launch OpenICLInfer[qwen-7b-chat-hf/ceval-college_chemistry,qwen-7b-chat-hf/ceval-metrology_engineer,qwen-7b-chat-hf/ceval-mao_zedong_thought,qwen-7b-chat-hf/ceval-law] on GPU 0                                                                                                     
launch OpenICLInfer[qwen-7b-chat-hf/ceval-veterinary_medicine,qwen-7b-chat-hf/ceval-modern_chinese_history,qwen-7b-chat-hf/ceval-chinese_language_and_literature,qwen-7b-chat-hf/ceval-legal_professional] on GPU 0                                                                   
launch OpenICLInfer[qwen-7b-chat-hf/ceval-logic,qwen-7b-chat-hf/ceval-middle_school_history,qwen-7b-chat-hf/ceval-plant_protection,qwen-7b-chat-hf/ceval-clinical_medicine] on GPU 0                                                                                                  
launch OpenICLInfer[qwen-7b-chat-hf/ceval-computer_architecture,qwen-7b-chat-hf/ceval-middle_school_biology,qwen-7b-chat-hf/ceval-middle_school_politics,qwen-7b-chat-hf/ceval-middle_school_chemistry] on GPU 0                                                                      
launch OpenICLInfer[qwen-7b-chat-hf/ceval-high_school_history,qwen-7b-chat-hf/ceval-computer_network,qwen-7b-chat-hf/ceval-operating_system,qwen-7b-chat-hf/ceval-college_physics,qwen-7b-chat-hf/ceval-advanced_mathematics] on GPU 0                                                
launch OpenICLInfer[qwen-7b-chat-hf/ceval-high_school_physics,qwen-7b-chat-hf/ceval-high_school_chemistry,qwen-7b-chat-hf/ceval-high_school_biology,qwen-7b-chat-hf/ceval-middle_school_mathematics,qwen-7b-chat-hf/ceval-middle_school_physics] on GPU 0                             
launch OpenICLInfer[qwen-7b-chat-hf/ceval-marxism,qwen-7b-chat-hf/ceval-high_school_politics,qwen-7b-chat-hf/ceval-high_school_geography,qwen-7b-chat-hf/ceval-ideological_and_moral_cultivation,qwen-7b-chat-hf/ceval-high_school_chinese] on GPU 0                                  
launch OpenICLInfer[qwen-7b-chat-hf/ceval-sports_science,qwen-7b-chat-hf/ceval-basic_medicine,qwen-7b-chat-hf/ceval-probability_and_statistics,qwen-7b-chat-hf/ceval-high_school_mathematics,qwen-7b-chat-hf/ceval-discrete_mathematics] on GPU 0                                     
launch OpenICLInfer[qwen-7b-chat-hf/ceval-middle_school_geography] on GPU 0    

我们会测试一下QWen 的运行， 可能你需要debug 这段代码查看具体问题，https://github.com/InternLM/opencompass/blob/90c07a3dfd99f14bfbc5b43f59b96ce48fc4d0ec/opencompass/runners/local.py#L83

好像，我换成你们官方的默认那俩模型，也是自动加载到一个显卡里面去了 。并没识别多个卡，然后均分之类。

(opencompass1) /mnt/workspace/project/alsc_opencompass> python run.py configs/eval_demo.py -w outputs/demo 
/mnt/workspace/opt/anaconda3/envs/opencompass1/lib/python3.8/site-packages/mmengine/utils/manager.py:113: UserWarning: <class 'mmengine.logging.logger.MMLogger'> instance named of OpenCompass has been created, the method `get_instance` should not accept any other arguments
  warnings.warn(
08/17 19:52:21 - OpenCompass - INFO - Partitioned into 42 tasks.
launch OpenICLInfer[opt350m/siqa_0] on GPU 0                                                                                                                                                                                              
launch OpenICLInfer[opt350m/siqa_1] on GPU 0                                                                                                                                                                                              
launch OpenICLInfer[opt350m/siqa_2] on GPU 0                                                                                                                                                                                              
launch OpenICLInfer[opt350m/siqa_3] on GPU 0                                                                                                                                                                                              
launch OpenICLInfer[opt350m/siqa_4] on GPU 0                                                                                                                                                                                              
launch OpenICLInfer[opt350m/siqa_5] on GPU 0                                                                                                                                                                                              
launch OpenICLInfer[opt350m/siqa_6] on GPU 0                                                                                                                                                                                              
launch OpenICLInfer[opt350m/siqa_7] on GPU 0                                                                                                                                                                                              
launch OpenICLInfer[opt350m/siqa_8] on GPU 0                                                                                                                                                                                              
launch OpenICLInfer[opt350m/siqa_9] on GPU 0                                                                                                                                                                                              
launch OpenICLInfer[opt350m/siqa_10] on GPU 0                                                                                                                                                                                             
launch OpenICLInfer[opt350m/siqa_11] on GPU 0                                                                                                                                                                                             
launch OpenICLInfer[opt350m/siqa_12] on GPU 0                                                                                                                                                                                             
launch OpenICLInfer[opt350m/siqa_13] on GPU 0                                                                                                                                                                                             
launch OpenICLInfer[opt350m/siqa_14] on GPU 0                                                                                                                                                                                             
launch OpenICLInfer[opt350m/siqa_15] on GPU 0                                                                                                                                                                                             
launch OpenICLInfer[opt350m/siqa_16] on GPU 0                                                                                                                                                                                             
launch OpenICLInfer[opt350m/siqa_17] on GPU 0                                                                                                                                                                                             
launch OpenICLInfer[opt350m/siqa_18] on GPU 0                                                                                                                                                                                             
launch OpenICLInfer[opt350m/siqa_19] on GPU 0                                                                                                                                                                                             
launch OpenICLInfer[opt350m/winograd] on GPU 0                                                                                                                                                                                            
launch OpenICLInfer[opt125m/siqa_0] on GPU 0                                                                                                                                                                                              
launch OpenICLInfer[opt125m/siqa_1] on GPU 0                                                                                                                                                                                              
launch OpenICLInfer[opt125m/siqa_2] on GPU 0                                                                                                                                                                                              
launch OpenICLInfer[opt125m/siqa_3] on GPU 0                                                                                                                                                                                              
launch OpenICLInfer[opt125m/siqa_4] on GPU 0                                                                                                                                                                                              
launch OpenICLInfer[opt125m/siqa_5] on GPU 0                                                                                                                                                                                              
launch OpenICLInfer[opt125m/siqa_6] on GPU 0                                                                                                                                                                                              
launch OpenICLInfer[opt125m/siqa_7] on GPU 0                                                                                                                                                                                              
launch OpenICLInfer[opt125m/siqa_8] on GPU 0                                                                                                                                                                                              
launch OpenICLInfer[opt125m/siqa_9] on GPU 0                                                                                                                                                                                              
launch OpenICLInfer[opt125m/siqa_10] on GPU 0         





我期望是这样的
launch OpenICLInfer[opt350m/siqa_0] on GPU 0
launch OpenICLInfer[opt350m/siqa_1] on GPU 1
launch OpenICLInfer[opt350m/siqa_2] on GPU 2
launch OpenICLInfer[opt350m/siqa_3] on GPU 3

我知道咋设置了
python run.py configs/eval_demo.py -w outputs/demo --max-num-workers 4 --max-workers-per-gpu 1
我没看见还有个参数。如下
launch OpenICLInfer[qwen-7b-chat-hf/ceval-college_economics] on GPU 0                                                                                                                                                                                                                 
launch OpenICLInfer[qwen-7b-chat-hf/ceval-accountant,qwen-7b-chat-hf/ceval-tax_accountant] on GPU 1                                                                                                                                                                                   
launch OpenICLInfer[qwen-7b-chat-hf/ceval-physician,qwen-7b-chat-hf/ceval-civil_servant] on GPU 2                                                                                                                                                                                     
launch OpenICLInfer[qwen-7b-chat-hf/ceval-urban_and_rural_planner,qwen-7b-chat-hf/ceval-teacher_qualification] on GPU 3    


这里我给一些建议

我发现模型不停加载和卸载，可以做成消费者生产者模式，几个GPU加载几个模型后就等待推理作业即可，在出1进程分发作业，1个做监控进程，这种效率应该是最大的。




更新到最新版本后设置 -max-num-workers 4 --max-workers-per-gpu 1 不管用了 @wellcasa @Ezra-Yu 评测 ceval 都默认加载到一张卡里了，这里现在是怎么配置，可以使用多张 GPU 跑不同的任务？

```
python run.py configs/eval_llama2_7b_ceval.py -w outputs/eval_llama2_7b_ceval --max-num-workers 21 --max-workers-per-gpu 1 --max-partition-size 1000
```
加了 `max-partition-size`参数，ceval 任务可以拆分成 6 个任务在 6 张卡上