Repository: open-compass/opencompass
Issue: [Bug] Please help.Unable to evaluate Deepseek-R1-Qwen-32B 4-bit quantized version
Root cause: External Dependency Breakage

### Prerequisite

- [x] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [x] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I have modified the code (config is not considered code), or I'm working on my own tasks/models/datasets.

### Environment

```
{'CUDA available': True,
 'CUDA_HOME': '/usr/local/cuda',
 'GCC': 'gcc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0',
 'GPU 0': 'NVIDIA vGPU-32GB',
 'MMEngine': '0.10.7',
 'MUSA available': False,
 'NVCC': 'Cuda compilation tools, release 11.8, V11.8.89',
 'OpenCV': '4.11.0',
 'PyTorch': '2.6.0+cu124',
 'PyTorch compiling details': 'PyTorch built with:\n'
                              '  - GCC 9.3\n'
                              '  - C++ Version: 201703\n'
                              '  - Intel(R) oneAPI Math Kernel Library Version '
                              '2024.2-Product Build 20240605 for Intel(R) 64 '
                              'architecture applications\n'
                              '  - Intel(R) MKL-DNN v3.5.3 (Git Hash '
                              '66f0cb9eb66affd2da3bf5f8d897376f04aae6af)\n'
                              '  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
                              '  - LAPACK is enabled (usually provided by '
                              'MKL)\n'
                              '  - NNPACK is enabled\n'
                              '  - CPU capability usage: AVX512\n'
                              '  - CUDA Runtime 12.4\n'
                              '  - NVCC architecture flags: '
                              '-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90\n'
                              '  - CuDNN 90.1\n'
                              '  - Magma 2.6.1\n'
                              '  - Build settings: BLAS_INFO=mkl, '
                              'BUILD_TYPE=Release, '
                              'COMMIT_SHA=2236df1770800ffea5697b11b0bb0d910b2e59e1, '
                              'CUDA_VERSION=12.4, CUDNN_VERSION=9.1.0, '
                              'CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, '
                              'CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 '
                              '-fabi-version=11 -fvisibility-inlines-hidden '
                              '-DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO '
                              '-DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON '
                              '-DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK '
                              '-DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE '
                              '-O2 -fPIC -Wall -Wextra -Werror=return-type '
                              '-Werror=non-virtual-dtor -Werror=bool-operation '
                              '-Wnarrowing -Wno-missing-field-initializers '
                              '-Wno-type-limits -Wno-array-bounds '
                              '-Wno-unknown-pragmas -Wno-unused-parameter '
                              '-Wno-strict-overflow -Wno-strict-aliasing '
                              '-Wno-stringop-overflow -Wsuggest-override '
                              '-Wno-psabi -Wno-error=old-style-cast '
                              '-Wno-missing-braces -fdiagnostics-color=always '
                              '-faligned-new -Wno-unused-but-set-variable '
                              '-Wno-maybe-uninitialized -fno-math-errno '
                              '-fno-trapping-math -Werror=format '
                              '-Wno-stringop-overflow, LAPACK_INFO=mkl, '
                              'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, '
                              'TORCH_VERSION=2.6.0, USE_CUDA=ON, USE_CUDNN=ON, '
                              'USE_CUSPARSELT=1, USE_EXCEPTION_PTR=1, '
                              'USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, '
                              'USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, '
                              'USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, '
                              'USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, \n',
 'Python': '3.10.16 (main, Dec 11 2024, 16:24:50) [GCC 11.2.0]',
 'lmdeploy': '0.2.2',
 'numpy_random_seed': 2147483648,
 'opencompass': '0.4.1+fff2d51',
 'sys.platform': 'linux',
 'transformers': '4.49.0'}
```

### Reproduces the problem - code/configuration sample

```
# Support AIME-2024 with Repeat8
# Support MATH-500
# Support OlympiadBench
# Support OmniMath
# Support LiveMathBench-202412-Hard

import os.path as osp
from itertools import product
from opencompass.models import OpenAISDK
from mmengine.config import read_base
from opencompass.utils.text_postprocessors import extract_non_reasoning_content
from opencompass.partitioners import NaivePartitioner, NumWorkerPartitioner
from opencompass.tasks import OpenICLInferTask, OpenICLEvalTask
from opencompass.runners import LocalRunner
from opencompass.models import (
    TurboMindModelwithChatTemplate,
)

#######################################################################
#                          PART 1  Datasets List                      #
#######################################################################
with read_base():
    # You can comment out the datasets you don't want to evaluate

    # Datasets
    # from opencompass.configs.datasets.math.math_prm800k_500_llmverify_gen_6ff468 import math_datasets # 1 Run
    from opencompass.configs.datasets.aime2024.aime2024_llmverify_repeat8_gen_e8fcee import aime2024_datasets # 8 Run
    # from opencompass.configs.datasets.OlympiadBench.OlympiadBench_0shot_llmverify_gen_be8b13 import olympiadbench_datasets
    # from opencompass.configs.datasets.omni_math.omni_math_llmverify_gen_ccf9c0 import omnimath_datasets # 1 Run
    # from opencompass.configs.datasets.livemathbench.livemathbench_hard_custom_llmverify_gen_85d0ef import livemathbench_datasets


    # Summarizer
    from opencompass.configs.summarizers.groups.OlympiadBench import OlympiadBenchMath_summary_groups

datasets = sum(
    (v for k, v in locals().items() if k.endswith('_datasets')),
    [],
)

# Set LLM Verifier used for each dataset

verifier_cfg = dict(
        abbr='qwen2-5-32B-Instruct',
        type=OpenAISDK,
        path='Qwen/Qwen2.5-32B-Instruct', # You need to set your own judge model path
        key='xxxxxxxxxxx', # You need to set your own API key
        openai_api_base=[
            'https://api.siliconflow.cn/v1', # You need to set your own API base
        ],
        meta_template=dict(
            round=[
                dict(role='HUMAN', api_role='HUMAN'),
                dict(role='BOT', api_role='BOT', generate=True),
            ], 
        ),
        query_per_second=16,
        batch_size=1024,
        temperature=0.001,
        tokenizer_path='gpt-4o-2024-05-13',
        verbose=True,
        max_out_len=16384,
        # max_seq_len=32768,
        max_seq_len=49152,
)

for item in datasets:
    # item['infer_cfg']['inferencer']['max_out_len'] = 32768 # You can unset this line if you want to avoid length cutoff
    if 'judge_cfg' in item['eval_cfg']['evaluator']:
        item['eval_cfg']['evaluator']['judge_cfg'] = verifier_cfg


#######################################################################
#                          PART 2  Model List                         #
#######################################################################

models = sum([v for k, v in locals().items() if k.endswith('_model')], [])

models += [
    # You can comment out the models you don't want to evaluate
    # All models use sampling mode
    # dict(
    #     type=TurboMindModelwithChatTemplate,
    #     abbr='deepseek-r1-distill-qwen-7b-turbomind',
    #     path='deepseek-ai/DeepSeek-R1-Distill-Qwen-7B',
    #     engine_config=dict(session_len=32768, max_batch_size=128, tp=1),
    #     gen_config=dict(
    #                     do_sample=True,
    #                     temperature=0.6,
    #                     top_p=0.95,
    #                     max_new_tokens=32768),
    #     max_seq_len=32768,
    #     max_out_len=32768,
    #     batch_size=64,
    #     run_cfg=dict(num_gpus=1),
    #     pred_postprocessor=dict(type=extract_non_reasoning_content)
    # ),
    # dict(
    #     type=TurboMindModelwithChatTemplate,
    #     abbr='deepseek-r1-distill-qwen-14b-turbomind',
    #     path='deepseek-ai/DeepSeek-R1-Distill-Qwen-14B',
    #     engine_config=dict(session_len=32768, max_batch_size=128, tp=2),
    #     gen_config=dict(
    #                     do_sample=True,
    #                     temperature=0.6,
    #                     top_p=0.95,
    #                     max_new_tokens=32768),
    #     max_seq_len=32768,
    #     max_out_len=32768,
    #     batch_size=128,
    #     run_cfg=dict(num_gpus=2),
    #     pred_postprocessor=dict(type=extract_non_reasoning_content)
    # ),
    # dict(
    #     type=TurboMindModelwithChatTemplate,
    #     abbr='deepseek-r1-distill-qwen-32b-turbomind',
    #     path='deepseek-ai/DeepSeek-R1-Distill-Qwen-32B',
    #     engine_config=dict(session_len=32768, max_batch_size=128, tp=4),
    #     gen_config=dict(
    #                     do_sample=True,
    #                     temperature=0.6,
    #                     top_p=0.95,
    #                     max_new_tokens=16384),
    #     max_seq_len=32768,
    #     max_out_len=16384,
    #     batch_size=128,
    #     run_cfg=dict(num_gpus=4),
    #     pred_postprocessor=dict(type=extract_non_reasoning_content)
    # ),
    dict(
        type=TurboMindModelwithChatTemplate,
        abbr='deepseek-r1-distill-qwen-32b-int4-turbomind',
        path='/root/autodl-tmp/DeepSeek-R1-Distill-Qwen-32B-gptq-4bit',
        engine_config=dict(session_len=32768, max_batch_size=128, tp=1),
        gen_config=dict(
                        do_sample=True,
                        temperature=0.6,
                        top_p=0.95,
                        max_new_tokens=32768),
        max_seq_len=32768,
        max_out_len=32768,
        batch_size=64,
        run_cfg=dict(num_gpus=1),
        pred_postprocessor=dict(type=extract_non_reasoning_content)
    ),
]

#######################################################################
#                          PART 3  Inference/Evaluation               #
#######################################################################

# Inference configuration
infer = dict(
    partitioner=dict(
        type=NumWorkerPartitioner,
        num_worker=1
        # Similar with data-parallelism, how many workers for evaluation,
        # each worker will evaluate a part of the dataset. Total GPUs = num_worker * num_gpus_per_worker
        # For example, If you have 8 GPUs, for 7B model using 1 GPU for one instance, you can set num_worker=8
        # to max-utilize the GPUs.
        # If you have 8 GPUs, for 14B model using 2 GPUs for one instance, you can set num_worker=4
    ),
    runner=dict(
        type=LocalRunner,
        task=dict(type=OpenICLInferTask)
    ),
)

# Evaluation configuration
eval = dict(
    partitioner=dict(
        type=NaivePartitioner, n=8
    ),
    runner=dict(
        type=LocalRunner,
        task=dict(
            type=OpenICLEvalTask)
    ),
)


#######################################################################
#                          PART 4  Summarizer                         #
#######################################################################


summary_groups = sum(
    [v for k, v in locals().items() if k.endswith('_summary_groups')], []
)

summary_groups.extend([
    {
        'name': 'AIME2024-Aveage8',
        'subsets':[[f'aime2024-run{idx}', 'accuracy'] for idx in range(8)]
    },
    {
        'name': 'LiveMathBench-v202412-Hard-Aveage8',
        'subsets':[[
            f'livemathbench_hard_custom_{split}_run{run_idx}', 'accuracy'] 
                for split, run_idx in product(['hard_cn', 'hard_en'], range(8))
        ]
    }
])

# Summarizer
summarizer = dict(
    dataset_abbrs=[
        'MATH',
        # ['LiveMathBench-k1-n1', 'pass@1'],
        # ['LiveMathBench-v202412-greedy', 'G-Pass@1_0.0'],
        # ['aime2024', 'accuracy'],
        ['math_prm800k_500-llmjudge', 'accuracy'],
        ['AIME2024-Aveage8', 'naive_average'],
        ['LiveMathBench-v202412-Hard-Aveage8', 'naive_average'],
        ['OlympiadBenchMath', 'accuracy'],
        ['OmniMath', 'accuracy'],
    ],
    summary_groups=summary_groups,
)


#######################################################################
#                          PART 5  Utils                              #
#######################################################################

work_dir = 'outputs/deepseek_r1_reasoning'



```

### Reproduces the problem - command or script

```


python run.py /root/opencompass/examples/eval_deepseek_r1.py --debug --dump-eval-details

opencompass /root/autodl-tmp/opencompass/examples/eval_deepseek_r1.py --debug --dump-eval-details
```

### Reproduces the problem - error message

```
nohup: ignoring input
03/05 22:56:58 - OpenCompass - INFO - Current exp folder: outputs/deepseek_r1_reasoning/20250305_225658
03/05 22:56:58 - OpenCompass - WARNING - SlurmRunner is not used, so the partition argument is ignored.
03/05 22:56:58 - OpenCompass - INFO - Partitioned into 1 tasks.
03/05 22:57:01 - OpenCompass - INFO - Task [deepseek-r1-distill-qwen-32b-int4-turbomind/aime2024-run0,deepseek-r1-distill-qwen-32b-int4-turbomind/aime2024-run1,deepseek-r1-distill-qwen-32b-int4-turbomind/aime2024-run2,deepseek-r1-distill-qwen-32b-int4-turbomind/aime2024-run3,deepseek-r1-distill-qwen-32b-int4-turbomind/aime2024-run4,deepseek-r1-distill-qwen-32b-int4-turbomind/aime2024-run5,deepseek-r1-distill-qwen-32b-int4-turbomind/aime2024-run6,deepseek-r1-distill-qwen-32b-int4-turbomind/aime2024-run7]
model_source: hf_model
Traceback (most recent call last):
  File "/root/autodl-tmp/.conda/envs/opencompass/bin/opencompass", line 33, in <module>
    sys.exit(load_entry_point('opencompass', 'console_scripts', 'opencompass')())
  File "/root/opencompass/opencompass/cli/main.py", line 306, in main
    runner(tasks)
  File "/root/opencompass/opencompass/runners/base.py", line 38, in __call__
    status = self.launch(tasks)
  File "/root/opencompass/opencompass/runners/local.py", line 128, in launch
    task.run(cur_model=getattr(self, 'cur_model',
  File "/root/opencompass/opencompass/tasks/openicl_infer.py", line 73, in run
    self.model = build_model_from_cfg(model_cfg)
  File "/root/opencompass/opencompass/utils/build.py", line 24, in build_model_from_cfg
    return MODELS.build(model_cfg)
  File "/root/autodl-tmp/.conda/envs/opencompass/lib/python3.10/site-packages/mmengine/registry/registry.py", line 570, in build
    return self.build_func(cfg, *args, **kwargs, registry=self)
  File "/root/autodl-tmp/.conda/envs/opencompass/lib/python3.10/site-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg
    obj = obj_cls(**args)  # type: ignore
  File "/root/opencompass/opencompass/models/turbomind_with_tf_above_v4_33.py", line 57, in __init__
    self.pipe = self._build_pipe(path, backend, _engine_config)
  File "/root/opencompass/opencompass/models/turbomind_with_tf_above_v4_33.py", line 203, in _build_pipe
    return pipeline(model_path, backend_config=backend_config, log_level='WARNING')
  File "/root/autodl-tmp/.conda/envs/opencompass/lib/python3.10/site-packages/lmdeploy/api.py", line 61, in pipeline
    return AsyncEngine(model_path,
  File "/root/autodl-tmp/.conda/envs/opencompass/lib/python3.10/site-packages/lmdeploy/serve/async_engine.py", line 67, in __init__
    self._build_turbomind(model_path=model_path,
  File "/root/autodl-tmp/.conda/envs/opencompass/lib/python3.10/site-packages/lmdeploy/serve/async_engine.py", line 108, in _build_turbomind
    self.engine = tm.TurboMind.from_pretrained(
  File "/root/autodl-tmp/.conda/envs/opencompass/lib/python3.10/site-packages/lmdeploy/turbomind/turbomind.py", line 438, in from_pretrained
    return cls(model_path=local_path,
  File "/root/autodl-tmp/.conda/envs/opencompass/lib/python3.10/site-packages/lmdeploy/turbomind/turbomind.py", line 196, in __init__
    self.model_comm = self._from_hf(model_source=model_source,
  File "/root/autodl-tmp/.conda/envs/opencompass/lib/python3.10/site-packages/lmdeploy/turbomind/turbomind.py", line 288, in _from_hf
    model_arch = get_model_from_config(model_path)
  File "/root/autodl-tmp/.conda/envs/opencompass/lib/python3.10/site-packages/lmdeploy/turbomind/utils.py", line 148, in get_model_from_config
    return ARCH_MAP[arch]
KeyError: 'Qwen2ForCausalLM'

```

### Other information

the model address: https://huggingface.co/empirischtech/DeepSeek-R1-Distill-Qwen-32B-gptq-4bit

It looks to me that your lmdeploy does not support your model, try to upgrade your lmdeploy version.

> It looks to me that your lmdeploy does not support your model, try to upgrade your lmdeploy version.

Got it, I tried to run this model directly with lmdeploy, but it went wrong. Then I switched to an AWQ model, but how do I specify the quantization type in the script?

> > It looks to me that your lmdeploy does not support your model, try to upgrade your lmdeploy version.
> 
> Got it, I tried to run this model directly with lmdeploy, but it went wrong. Then I switched to an AWQ model, but how do I specify the quantization type in the script?

this command is worked 
`lmdeploy serve api_server /root/autodl-tmp/deepseek-r1-distill-qwen-32b-awq  --server-port 6006 --model-format awq`

but this command is cant worked
`nohup python run.py /root/opencompass/examples/eval_deepseek_r1.py --debug --dump-eval-details -a lmdeploy --model-format awq > eval_deepseek_r1.log 2>&1 &`

how can i work by opencompass ?

using vllm ,set environment variable
export MKL_SERVICE_FORCE_INTEL=1

> using vllm ,set environment variable export MKL_SERVICE_FORCE_INTEL=1

please where to set this specific variable? My point is that vllm originally support awq/gptq model, is there any way solve by using vllm?