Repository: open-compass/opencompass
Issue: [Bug] 无法使用多卡评测
Root cause: Architectural Constraint

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

环境是昇腾卡NPU
{'CUDA available': False,
 'GCC': 'gcc (GCC) 7.3.0',
 'MMEngine': '0.9.1',
 'OpenCV': '4.8.0',
 'PyTorch': '2.1.0',
 'PyTorch compiling details': 'PyTorch built with:\n'
                              '  - GCC 10.2\n'
                              '  - C++ Version: 201703\n'
                              '  - Intel(R) MKL-DNN v3.1.1 (Git Hash '
                              '64f6bcbcbab628e96f33a62c3e975f8535a7bde4)\n'
                              '  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
                              '  - LAPACK is enabled (usually provided by '
                              'MKL)\n'
                              '  - NNPACK is enabled\n'
                              '  - CPU capability usage: NO AVX\n'
                              '  - Build settings: BLAS_INFO=open, '
                              'BUILD_TYPE=Release, '
                              'CXX_COMPILER=/opt/rh/devtoolset-10/root/usr/bin/c++, '
                              'CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 '
                              '-fabi-version=11 -fvisibility-inlines-hidden '
                              '-DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO '
                              '-DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER '
                              '-DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK '
                              '-DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE '
                              '-O2 -fPIC -Wall -Wextra -Werror=return-type '
                              '-Werror=non-virtual-dtor -Werror=bool-operation '
                              '-Wnarrowing -Wno-missing-field-initializers '
                              '-Wno-type-limits -Wno-array-bounds '
                              '-Wno-unknown-pragmas -Wno-unused-parameter '
                              '-Wno-unused-function -Wno-unused-result '
                              '-Wno-strict-overflow -Wno-strict-aliasing '
                              '-Wno-stringop-overflow -Wno-psabi '
                              '-Wno-error=pedantic -Wno-error=old-style-cast '
                              '-Wno-invalid-partial-specialization '
                              '-Wno-unused-private-field '
                              '-Wno-aligned-allocation-unavailable '
                              '-Wno-missing-braces -fdiagnostics-color=always '
                              '-faligned-new -Wno-unused-but-set-variable '
                              '-Wno-maybe-uninitialized -fno-math-errno '
                              '-fno-trapping-math -Werror=format '
                              '-Werror=cast-function-type '
                              '-Wno-stringop-overflow, LAPACK_INFO=open, '
                              'TORCH_DISABLE_GPU_ASSERTS=ON, '
                              'TORCH_VERSION=2.1.0, USE_CUDA=OFF, '
                              'USE_CUDNN=OFF, USE_EIGEN_FOR_BLAS=ON, '
                              'USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, '
                              'USE_GLOG=OFF, USE_MKL=OFF, USE_MKLDNN=ON, '
                              'USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, '
                              'USE_OPENMP=ON, USE_ROCM=OFF, \n',
 'Python': '3.9.10 | packaged by conda-forge | (main, Feb  1 2022, 21:53:27) '
           '[GCC 9.4.0]',
 'TorchVision': '0.16.0',
 'lmdeploy': "not installed:No module named 'lmdeploy'",
 'numpy_random_seed': 2147483648,
 'opencompass': '0.3.9+',
 'sys.platform': 'linux',
 'transformers': '4.43.2'}

### Reproduces the problem - code/configuration sample

from mmengine.config import read_base
with read_base():

    from opencompass.configs.datasets.mmlu.mmlu_gen_4d595a import mmlu_datasets
    from opencompass.configs.models.hf_llama.hf_llama3_1_8b_instruct import models as hf_llama3_1_8b_instruct_model
    from opencompass.configs.models.qwen2_5.hf_qwen2_5_7b_instruct import models as hf_qwen2_5_7b_instruct_model

work_dir = "outputs/Llama3_1_and_Qwen2_5-7B-Instruct"

datasets = sum((v for k, v in locals().items() if k.endswith('_datasets')), [])
models = sum([v for k, v in locals().items() if k.endswith('_model')], [])

### Reproduces the problem - command or script

python run.py configs/eval_OC15_llama3.1_qwen2_custom_gen.py --max-num-workers 8 --num-gpus 8

### Reproduces the problem - error message

![image](https://github.com/user-attachments/assets/a029d16e-9bb6-4576-8e04-50bb6cd691a0)


### Other information

虽然知道7B大小的模型只用单卡即可，但为什么指定分布在8卡上运行实际上只有单卡在运行呢？另外DP指定为8貌似也没用

开启tp的时候虽然日志写了ON GPU0, ... 7，但是我查看资源发现还是只有一张卡在实际跑呢.
另外--max-num-workers 8开启dp试了一下发现也是类似的问题, 它确实是切分评测集了, 但是好像只在单卡上运行1/8的评测集，然后串行执行的接下来的第二个1/8，那这样也起不到加速的效果啊

```
infer = dict(
    partitioner=dict(type=NumWorkerPartitioner, num_worker=8),
    runner=dict(type=LocalRunner, task=dict(type=OpenICLInferTask)),
)
```
在配置文件中可以直接配置分片逻辑，在外部直接执行`python run.py configs/eval_OC15_llama3.1_qwen2_custom_gen.py`

> ```
> infer = dict(
>     partitioner=dict(type=NumWorkerPartitioner, num_worker=8),
>     runner=dict(type=LocalRunner, task=dict(type=OpenICLInferTask)),
> )
> ```
> 
> 在配置文件中可以直接配置分片逻辑，在外部直接执行`python run.py configs/eval_OC15_llama3.1_qwen2_custom_gen.py`

请问这是DP开启8吧，那么如果我的模型上到70B或33B的情况下如何开启TP呢，因为我在python run.py 添加--max-num-workers 8 --hf-num-gpus 8好像都没用呀

> ```
> infer = dict(
>     partitioner=dict(type=NumWorkerPartitioner, num_worker=8),
>     runner=dict(type=LocalRunner, task=dict(type=OpenICLInferTask)),
> )
> ```
> 
> 在配置文件中可以直接配置分片逻辑，在外部直接执行`python run.py configs/eval_OC15_llama3.1_qwen2_custom_gen.py`

我在配置文件中加入了
from opencompass.partitioners.num_worker import NumWorkerPartitioner
from opencompass.runners.local import LocalRunner
from opencompass.tasks.openicl_infer import OpenICLInferTask
infer = dict(
partitioner=dict(type=NumWorkerPartitioner, num_worker=8),
runner=dict(type=LocalRunner, task=dict(type=OpenICLInferTask)),
)
然后报错
Traceback (most recent call last):
File "/opencompass-main/run.py", line 4, in
main()
File "/opencompass-main/opencompass/cli/main.py", line 231, in main
cfg = get_config_from_arg(args)
File "/opencompass-main/opencompass/utils/run.py", line 97, in get_config_from_arg
config = Config.fromfile(args.config, format_python_code=False)
File "/anaconda3/envs/opencompass/lib/python3.10/site-packages/mmengine/config/config.py", line 494, in fromfile
raise e
File "/anaconda3/envs/opencompass/lib/python3.10/site-packages/mmengine/config/config.py", line 492, in fromfile
cfg_dict, imported_names = Config._parse_lazy_import(filename)
File "/anaconda3/envs/opencompass/lib/python3.10/site-packages/mmengine/config/config.py", line 1081, in _parse_lazy_import
_base_cfg_dict, _base_imported_names = Config._parse_lazy_import( # noqa: E501
File "/anaconda3/envs/opencompass/lib/python3.10/site-packages/mmengine/config/config.py", line 1109, in _parse_lazy_import
exec(
File "/opencompass-main/opencompass/partitioners/num_worker.py", line 16, in
@PARTITIONERS.register_module()
File "/anaconda3/envs/opencompass/lib/python3.10/site-packages/mmengine/config/lazy.py", line 205, in call
raise RuntimeError()
RuntimeError

> > ```
> > infer = dict(
> >     partitioner=dict(type=NumWorkerPartitioner, num_worker=8),
> >     runner=dict(type=LocalRunner, task=dict(type=OpenICLInferTask)),
> > )
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > 在配置文件中可以直接配置分片逻辑，在外部直接执行`python run.py configs/eval_OC15_llama3.1_qwen2_custom_gen.py`
> 
> 我在配置文件中加入了 from opencompass.partitioners.num_worker import NumWorkerPartitioner from opencompass.runners.local import LocalRunner from opencompass.tasks.openicl_infer import OpenICLInferTask infer = dict( partitioner=dict(type=NumWorkerPartitioner, num_worker=8), runner=dict(type=LocalRunner, task=dict(type=OpenICLInferTask)), ) 然后报错 Traceback (most recent call last): File "/opencompass-main/run.py", line 4, in main() File "/opencompass-main/opencompass/cli/main.py", line 231, in main cfg = get_config_from_arg(args) File "/opencompass-main/opencompass/utils/run.py", line 97, in get_config_from_arg config = Config.fromfile(args.config, format_python_code=False) File "/anaconda3/envs/opencompass/lib/python3.10/site-packages/mmengine/config/config.py", line 494, in fromfile raise e File "/anaconda3/envs/opencompass/lib/python3.10/site-packages/mmengine/config/config.py", line 492, in fromfile cfg_dict, imported_names = Config._parse_lazy_import(filename) File "/anaconda3/envs/opencompass/lib/python3.10/site-packages/mmengine/config/config.py", line 1081, in _parse_lazy_import _base_cfg_dict, _base_imported_names = Config._parse_lazy_import( # noqa: E501 File "/anaconda3/envs/opencompass/lib/python3.10/site-packages/mmengine/config/config.py", line 1109, in _parse_lazy_import exec( File "/opencompass-main/opencompass/partitioners/num_worker.py", line 16, in @PARTITIONERS.register_module() File "/anaconda3/envs/opencompass/lib/python3.10/site-packages/mmengine/config/lazy.py", line 205, in call raise RuntimeError() RuntimeError

TP在 model 的 config 中设置,可参考`https://github.com/open-compass/opencompass/blob/main/opencompass/configs/models/qwen2_5/lmdeploy_qwen2_5_72b_instruct.py`

> > > ```
> > > infer = dict(
> > >     partitioner=dict(type=NumWorkerPartitioner, num_worker=8),
> > >     runner=dict(type=LocalRunner, task=dict(type=OpenICLInferTask)),
> > > )
> > > ```
> > > 
> > > 
> > >     
> > >       
> > >     
> > > 
> > >       
> > >     
> > > 
> > >     
> > >   
> > > 在配置文件中可以直接配置分片逻辑，在外部直接执行`python run.py configs/eval_OC15_llama3.1_qwen2_custom_gen.py`
> > 
> > 
> > 我在配置文件中加入了 from opencompass.partitioners.num_worker import NumWorkerPartitioner from opencompass.runners.local import LocalRunner from opencompass.tasks.openicl_infer import OpenICLInferTask infer = dict( partitioner=dict(type=NumWorkerPartitioner, num_worker=8), runner=dict(type=LocalRunner, task=dict(type=OpenICLInferTask)), ) 然后报错 Traceback (most recent call last): File "/opencompass-main/run.py", line 4, in main() File "/opencompass-main/opencompass/cli/main.py", line 231, in main cfg = get_config_from_arg(args) File "/opencompass-main/opencompass/utils/run.py", line 97, in get_config_from_arg config = Config.fromfile(args.config, format_python_code=False) File "/anaconda3/envs/opencompass/lib/python3.10/site-packages/mmengine/config/config.py", line 494, in fromfile raise e File "/anaconda3/envs/opencompass/lib/python3.10/site-packages/mmengine/config/config.py", line 492, in fromfile cfg_dict, imported_names = Config._parse_lazy_import(filename) File "/anaconda3/envs/opencompass/lib/python3.10/site-packages/mmengine/config/config.py", line 1081, in _parse_lazy_import _base_cfg_dict, _base_imported_names = Config._parse_lazy_import( # noqa: E501 File "/anaconda3/envs/opencompass/lib/python3.10/site-packages/mmengine/config/config.py", line 1109, in _parse_lazy_import exec( File "/opencompass-main/opencompass/partitioners/num_worker.py", line 16, in @PARTITIONERS.register_module() File "/anaconda3/envs/opencompass/lib/python3.10/site-packages/mmengine/config/lazy.py", line 205, in call raise RuntimeError() RuntimeError
> 
> TP在 model 的 config 中设置,可参考`https://github.com/open-compass/opencompass/blob/main/opencompass/configs/models/qwen2_5/lmdeploy_qwen2_5_72b_instruct.py`

您是指run_cfg=dict(num_gpus)更改吗，我之前这里改动也是没用的，另外您上一条回复的NumWorkerPartitioner我会报错呀

> > > ```
> > > infer = dict(
> > >     partitioner=dict(type=NumWorkerPartitioner, num_worker=8),
> > >     runner=dict(type=LocalRunner, task=dict(type=OpenICLInferTask)),
> > > )
> > > ```
> > > 
> > > 
> > >     
> > >       
> > >     
> > > 
> > >       
> > >     
> > > 
> > >     
> > >   
> > > 在配置文件中可以直接配置分片逻辑，在外部直接执行`python run.py configs/eval_OC15_llama3.1_qwen2_custom_gen.py`
> > 
> > 
> > 我在配置文件中加入了 from opencompass.partitioners.num_worker import NumWorkerPartitioner from opencompass.runners.local import LocalRunner from opencompass.tasks.openicl_infer import OpenICLInferTask infer = dict( partitioner=dict(type=NumWorkerPartitioner, num_worker=8), runner=dict(type=LocalRunner, task=dict(type=OpenICLInferTask)), ) 然后报错 Traceback (most recent call last): File "/opencompass-main/run.py", line 4, in main() File "/opencompass-main/opencompass/cli/main.py", line 231, in main cfg = get_config_from_arg(args) File "/opencompass-main/opencompass/utils/run.py", line 97, in get_config_from_arg config = Config.fromfile(args.config, format_python_code=False) File "/anaconda3/envs/opencompass/lib/python3.10/site-packages/mmengine/config/config.py", line 494, in fromfile raise e File "/anaconda3/envs/opencompass/lib/python3.10/site-packages/mmengine/config/config.py", line 492, in fromfile cfg_dict, imported_names = Config._parse_lazy_import(filename) File "/anaconda3/envs/opencompass/lib/python3.10/site-packages/mmengine/config/config.py", line 1081, in _parse_lazy_import _base_cfg_dict, _base_imported_names = Config._parse_lazy_import( # noqa: E501 File "/anaconda3/envs/opencompass/lib/python3.10/site-packages/mmengine/config/config.py", line 1109, in _parse_lazy_import exec( File "/opencompass-main/opencompass/partitioners/num_worker.py", line 16, in @PARTITIONERS.register_module() File "/anaconda3/envs/opencompass/lib/python3.10/site-packages/mmengine/config/lazy.py", line 205, in call raise RuntimeError() RuntimeError
> 
> TP在 model 的 config 中设置,可参考`https://github.com/open-compass/opencompass/blob/main/opencompass/configs/models/qwen2_5/lmdeploy_qwen2_5_72b_instruct.py`

您好我按照参考修改了模型的config文件

![Image](https://github.com/user-attachments/assets/ee7ef9f0-b40f-4527-8ed0-1f3b9d3f8c7d)
运行时观察npu-smi info还是显示只跑在一张卡上

![Image](https://github.com/user-attachments/assets/1a264f39-b92b-400f-a04e-288ac6ca8fa7)

请问这个问题还有人在跟进吗？

一样的问题，仅设置num_gpus=2没法用两卡。相反，如果始终开启--debug模式，倒是会自动平均到两张卡上

> 请问这个问题还有人在跟进吗？

> 一样的问题，仅设置num_gpus=2没法用两卡。相反，如果始终开启--debug模式，倒是会自动平均到两张卡上

hello, 我在昇腾卡解决了这个, 你如果是用vllm跑的话就可以多卡运行。如果是hf模式，记得去models/下的两个hf model.py，里面有一句if is_npu_available(), 它会指定device='npu', 就是这个导致的设置多卡运行不起来。解决方法是注释掉这句, 在最上from torch_npu.contrib import transfer_to_npu



> > 请问这个问题还有人在跟进吗？
> 
> > 一样的问题，仅设置num_gpus=2没法用两卡。相反，如果始终开启--debug模式，倒是会自动平均到两张卡上
> 
> hello, 我在昇腾卡解决了这个, 你如果是用vllm跑的话就可以多卡运行。如果是hf模式，记得去models/下的两个hf model.py，里面有一句if is_npu_available(), 它会指定device='npu', 就是这个导致的设置多卡运行不起来。解决方法是注释掉这句, 在最上from torch_npu.contrib import transfer_to_npu

感谢，我也有在用npu，但还没在npu上试过，目前是在2卡3090上（无vllm）跑评测（mbpp这种单卡会oom），然后设置num_gpus=2不起作用，还是会在第一张卡上加载直到OOM. 但是如果我打开debug模式，似乎就可以2卡并行。

BTW，可否加个联系方式，NPU上我也有一些问题和经验，希望能一起交流。我的email是guochen_yan@outlook.com

> > 请问这个问题还有人在跟进吗？
> 
> > 一样的问题，仅设置num_gpus=2没法用两卡。相反，如果始终开启--debug模式，倒是会自动平均到两张卡上
> 
> hello, 我在昇腾卡解决了这个, 你如果是用vllm跑的话就可以多卡运行。如果是hf模式，记得去models/下的两个hf model.py，里面有一句if is_npu_available(), 它会指定device='npu', 就是这个导致的设置多卡运行不起来。解决方法是注释掉这句, 在最上from torch_npu.contrib import transfer_to_npu

感谢您的分享，但是在我的环境中似乎没有产生效果 (910B2*8 opencompass 0.4.1), 希望官方可以跟进一下问题

> > > 请问这个问题还有人在跟进吗？
> > 
> > 
> > > 一样的问题，仅设置num_gpus=2没法用两卡。相反，如果始终开启--debug模式，倒是会自动平均到两张卡上
> > 
> > 
> > hello, 我在昇腾卡解决了这个, 你如果是用vllm跑的话就可以多卡运行。如果是hf模式，记得去models/下的两个hf model.py，里面有一句if is_npu_available(), 它会指定device='npu', 就是这个导致的设置多卡运行不起来。解决方法是注释掉这句, 在最上from torch_npu.contrib import transfer_to_npu
> 
> 感谢您的分享，但是在我的环境中似乎没有产生效果 (910B2*8 opencompass 0.4.1), 希望官方可以跟进一下问题

你具体的config是怎么样的，在config中将run_cfg下的num_gpus改成并行数量(注意git下来的config在opencompass-main/下有一个，在opencompass-main/opencompass/下有一个，一般是后者起作用)，另外注意使用python run.py --models ... --datasets ...运行程序. DP或TP不能添加--debug

> > > > 请问这个问题还有人在跟进吗？
> > > 
> > > 
> > > > 一样的问题，仅设置num_gpus=2没法用两卡。相反，如果始终开启--debug模式，倒是会自动平均到两张卡上
> > > 
> > > 
> > > hello, 我在昇腾卡解决了这个, 你如果是用vllm跑的话就可以多卡运行。如果是hf模式，记得去models/下的两个hf model.py，里面有一句if is_npu_available(), 它会指定device='npu', 就是这个导致的设置多卡运行不起来。解决方法是注释掉这句, 在最上from torch_npu.contrib import transfer_to_npu
> > 
> > 
> > 感谢您的分享，但是在我的环境中似乎没有产生效果 (910B2*8 opencompass 0.4.1), 希望官方可以跟进一下问题
> 
> 你具体的config是怎么样的，在config中将run_cfg下的num_gpus改成并行数量(注意git下来的config在opencompass-main/下有一个，在opencompass-main/opencompass/下有一个，一般是后者起作用)，另外注意使用python run.py --models ... --datasets ...运行程序. DP或TP不能添加--debug

感谢您的回复，opencompass升级至0.4.1以上版本外层的config就清空了，会更干净一些。以下是config，平时是使用--hf_path 运行的模型，如果使用config会报torch版本不足的错误
config:
from opencompass.models import HuggingFacewithChatTemplate
from opencompass.utils.text_postprocessors import extract_non_reasoning_content

models = [
    dict(
        type=HuggingFacewithChatTemplate,
        abbr='deepseek-r1-distill-qwen-14b-hf',
        # path='deepseek-ai/DeepSeek-R1-Distill-Qwen-14B',
        path="/output/models/deepseek_r1_distill_qwen_14b_medical_train_raw",
        max_out_len=16384,
        batch_size=16,
        run_cfg=dict(num_gpus=4),
        pred_postprocessor=dict(type=extract_non_reasoning_content)
    )
]
报错：
/work/.env/opencompass041/lib/python3.10/site-packages/torch_npu/contrib/transfer_to_npu.py:260: RuntimeWarning: torch.jit.script and torch.jit.script_method will be disabled by transfer_to_npu, which currently does not support them, if you need to enable them, please do not use transfer_to_npu.
  warnings.warn(msg, RuntimeWarning)
Traceback (most recent call last):
  File "/work/share/luojun/docker/opencompass-main-0.4.1/opencompass/tasks/openicl_infer.py", line 166, in <module>
    inferencer.run()
  File "/work/share/luojun/docker/opencompass-main-0.4.1/opencompass/tasks/openicl_infer.py", line 74, in run
    self.model = build_model_from_cfg(model_cfg)
  File "/work/share/luojun/docker/opencompass-main-0.4.1/opencompass/utils/build.py", line 24, in build_model_from_cfg
    return MODELS.build(model_cfg)
  File "/work/.env/opencompass041/lib/python3.10/site-packages/mmengine/registry/registry.py", line 570, in build
    return self.build_func(cfg, *args, **kwargs, registry=self)
  File "/work/.env/opencompass041/lib/python3.10/site-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg
    obj = obj_cls(**args)  # type: ignore
  File "/work/share/luojun/docker/opencompass-main-0.4.1/opencompass/models/huggingface_above_v4_33.py", line 179, in __init__
    self._load_model(path=path, kwargs=model_kwargs, peft_path=peft_path, peft_kwargs=peft_kwargs)
  File "/work/share/luojun/docker/opencompass-main-0.4.1/opencompass/models/huggingface_above_v4_33.py", line 234, in _load_model
    self.model = AutoModelForCausalLM.from_pretrained(path, **model_kwargs)
  File "/work/.env/opencompass041/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 571, in from_pretrained
    return model_class.from_pretrained(
  File "/work/.env/opencompass041/lib/python3.10/site-packages/transformers/modeling_utils.py", line 279, in _wrapper
    return func(*args, **kwargs)
  File "/work/.env/opencompass041/lib/python3.10/site-packages/transformers/modeling_utils.py", line 4016, in from_pretrained
    raise EnvironmentError("tensor parallel is only supported for `torch>=2.5`.")


> > > > > 请问这个问题还有人在跟进吗？
> > > > 
> > > > 
> > > > > 一样的问题，仅设置num_gpus=2没法用两卡。相反，如果始终开启--debug模式，倒是会自动平均到两张卡上
> > > > 
> > > > 
> > > > hello, 我在昇腾卡解决了这个, 你如果是用vllm跑的话就可以多卡运行。如果是hf模式，记得去models/下的两个hf model.py，里面有一句if is_npu_available(), 它会指定device='npu', 就是这个导致的设置多卡运行不起来。解决方法是注释掉这句, 在最上from torch_npu.contrib import transfer_to_npu
> > > 
> > > 
> > > 感谢您的分享，但是在我的环境中似乎没有产生效果 (910B2*8 opencompass 0.4.1), 希望官方可以跟进一下问题
> > 
> > 
> > 你具体的config是怎么样的，在config中将run_cfg下的num_gpus改成并行数量(注意git下来的config在opencompass-main/下有一个，在opencompass-main/opencompass/下有一个，一般是后者起作用)，另外注意使用python run.py --models ... --datasets ...运行程序. DP或TP不能添加--debug
> 
> 感谢您的回复，opencompass升级至0.4.1以上版本外层的config就清空了，会更干净一些。以下是config，平时是使用--hf_path 运行的模型，如果使用config会报torch版本不足的错误 config: from opencompass.models import HuggingFacewithChatTemplate from opencompass.utils.text_postprocessors import extract_non_reasoning_content
> 
> models = [ dict( type=HuggingFacewithChatTemplate, abbr='deepseek-r1-distill-qwen-14b-hf', # path='deepseek-ai/DeepSeek-R1-Distill-Qwen-14B', path="/output/models/deepseek_r1_distill_qwen_14b_medical_train_raw", max_out_len=16384, batch_size=16, run_cfg=dict(num_gpus=4), pred_postprocessor=dict(type=extract_non_reasoning_content) ) ] 报错： /work/.env/opencompass041/lib/python3.10/site-packages/torch_npu/contrib/transfer_to_npu.py:260: RuntimeWarning: torch.jit.script and torch.jit.script_method will be disabled by transfer_to_npu, which currently does not support them, if you need to enable them, please do not use transfer_to_npu. warnings.warn(msg, RuntimeWarning) Traceback (most recent call last): File "/work/share/luojun/docker/opencompass-main-0.4.1/opencompass/tasks/openicl_infer.py", line 166, in inferencer.run() File "/work/share/luojun/docker/opencompass-main-0.4.1/opencompass/tasks/openicl_infer.py", line 74, in run self.model = build_model_from_cfg(model_cfg) File "/work/share/luojun/docker/opencompass-main-0.4.1/opencompass/utils/build.py", line 24, in build_model_from_cfg return MODELS.build(model_cfg) File "/work/.env/opencompass041/lib/python3.10/site-packages/mmengine/registry/registry.py", line 570, in build return self.build_func(cfg, *args, **kwargs, registry=self) File "/work/.env/opencompass041/lib/python3.10/site-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg obj = obj_cls(**args) # type: ignore File "/work/share/luojun/docker/opencompass-main-0.4.1/opencompass/models/huggingface_above_v4_33.py", line 179, in **init** self._load_model(path=path, kwargs=model_kwargs, peft_path=peft_path, peft_kwargs=peft_kwargs) File "/work/share/luojun/docker/opencompass-main-0.4.1/opencompass/models/huggingface_above_v4_33.py", line 234, in _load_model self.model = AutoModelForCausalLM.from_pretrained(path, **model_kwargs) File "/work/.env/opencompass041/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 571, in from_pretrained return model_class.from_pretrained( File "/work/.env/opencompass041/lib/python3.10/site-packages/transformers/modeling_utils.py", line 279, in _wrapper return func(*args, **kwargs) File "/work/.env/opencompass041/lib/python3.10/site-packages/transformers/modeling_utils.py", line 4016, in from_pretrained raise EnvironmentError("tensor parallel is only supported for `torch>=2.5`.")

我印象中使用config集成model和dataset那种方式运行好像有问题，一直都是用--models --dataset方式启动的。看这个报错去安装torch2.5就好了, torch_npu目前比较好的版本是torch_npu2.5.1-250320包，可以去搜一下vllm_ascend项目去下载

This issue is quite common and serious. We hope the authorities will debug it as soon as possible.

> > 请问这个问题还有人在跟进吗？
> 
> > 一样的问题，仅设置num_gpus=2没法用两卡。相反，如果始终开启--debug模式，倒是会自动平均到两张卡上
> 
> hello, 我在昇腾卡解决了这个, 你如果是用vllm跑的话就可以多卡运行。如果是hf模式，记得去models/下的两个hf model.py，里面有一句if is_npu_available(), 它会指定device='npu', 就是这个导致的设置多卡运行不起来。解决方法是注释掉这句, 在最上from torch_npu.contrib import transfer_to_npu

感谢你的分享，我最近也遇到了该问题，能否和你进一步交流学习一下呢。我的email是1006910551@qq.com