Repository: open-compass/opencompass
Issue: [Bug] 多卡主观评测
Root cause: Documentation Deficiency

### Prerequisite

- [x] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [x] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

```
{'CUDA available': True,
 'GCC': 'gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0',
 'MMEngine': '0.10.6',
 'MUSA available': False,
 'OpenCV': '4.11.0',
 'PyTorch': '2.5.1',
 'PyTorch compiling details': 'PyTorch built with:\n'
                              '  - GCC 10.2\n'
                              '  - C++ Version: 201703\n'
                              '  - Intel(R) MKL-DNN v3.5.3 (Git Hash '
                              '66f0cb9eb66affd2da3bf5f8d897376f04aae6af)\n'
                              '  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
                              '  - LAPACK is enabled (usually provided by '
                              'MKL)\n'
                              '  - NNPACK is enabled\n'
                              '  - CPU capability usage: NO AVX\n'
                              '  - Build settings: BLAS_INFO=open, '
                              'BUILD_TYPE=Release, '
                              'CXX_COMPILER=/opt/rh/devtoolset-10/root/usr/bin/c++, '
                              'CXX_FLAGS=-ffunction-sections -fdata-sections '
                              '-D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 '
                              '-fvisibility-inlines-hidden -DUSE_PTHREADPOOL '
                              '-DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI '
                              '-DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON '
                              '-DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK '
                              '-DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC '
                              '-Wall -Wextra -Werror=return-type '
                              '-Werror=non-virtual-dtor -Werror=bool-operation '
                              '-Wnarrowing -Wno-missing-field-initializers '
                              '-Wno-type-limits -Wno-array-bounds '
                              '-Wno-unknown-pragmas -Wno-unused-parameter '
                              '-Wno-strict-overflow -Wno-strict-aliasing '
                              '-Wno-stringop-overflow -Wsuggest-override '
                              '-Wno-psabi -Wno-error=old-style-cast '
                              '-Wno-missing-braces -fdiagnostics-color=always '
                              '-faligned-new -Wno-unused-but-set-variable '
                              '-Wno-maybe-uninitialized -fno-math-errno '
                              '-fno-trapping-math -Werror=format '
                              '-Wno-stringop-overflow, LAPACK_INFO=open, '
                              'TORCH_VERSION=2.5.1, USE_CUDA=OFF, '
                              'USE_CUDNN=OFF, USE_CUSPARSELT=OFF, '
                              'USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, '
                              'USE_GLOG=OFF, USE_GLOO=ON, USE_MKLDNN=ON, '
                              'USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, '
                              'USE_OPENMP=ON, USE_ROCM=OFF, '
                              'USE_ROCM_KERNEL_ASSERT=OFF, \n',
 'Python': '3.10.2 (main, Jan  8 2025, 06:13:36) [GCC 9.4.0]',
 'TorchVision': '0.20.1',
 'lmdeploy': "not installed:No module named 'lmdeploy'",
 'numpy_random_seed': 2147483648,
 'opencompass': '0.3.9+',
 'sys.platform': 'linux',
 'transformers': '4.51.3'}
```

### Reproduces the problem - code/configuration sample

```
from mmengine.config import read_base

with read_base():
    from opencompass.configs.datasets.subjective.alignbench.alignbench_judgeby_critiquellm_new import alignbench_datasets
from opencompass.models import VLLMwithChatTemplate, OpenAISDK
from opencompass.partitioners import NaivePartitioner
from opencompass.partitioners.sub_naive import SubjectiveNaivePartitioner
from opencompass.runners import LocalRunner
from opencompass.tasks import OpenICLInferTask
from opencompass.tasks.subjective_eval import SubjectiveEvalTask
from opencompass.summarizers import SubjectiveSummarizer


datasets = sum(
    (v for k, v in locals().items() if k.endswith('_datasets')),
    [],
)

api_meta_template = dict(
    round=[
        dict(role='HUMAN', api_role='HUMAN'),
        dict(role='BOT', api_role='BOT', generate=True),
    ],
    reserved_roles=[dict(role='SYSTEM', api_role='SYSTEM')],
)

# -------------Inference Stage ----------------------------------------
# For subjective evaluation, we often set do sample for models
models = [
    dict(
        type=VLLMwithChatTemplate,
        abbr='qwen2.5-7b-instruct-vllm',
        path='/Qwen-Series/Qwen2.5-7B-Instruct',
        model_kwargs=dict(tensor_parallel_size=1),
        max_out_len=1024,
        batch_size=8,
        generation_kwargs=dict(do_sample=True, top_p=0.8, top_k=20, temperature=0.7),
        run_cfg=dict(num_gpus=1),
    )
]


infer = dict(
    partitioner=dict(type=NaivePartitioner, num_worker=8),
    runner=dict(type=LocalRunner, task=dict(type=OpenICLInferTask)),
)
# -------------Evalation Stage ----------------------------------------

## ------------- JudgeLLM Configuration
judge_models = [dict(
    abbr='vllm-qwen2-5-72b-instruct',
    type=OpenAISDK,
    path='Qwen2_5_72B',
    key='EMPTY',
    tokenizer_path='/Qwen-Series/Qwen2.5-72B-Instruct',
    openai_api_base=['http://0.0.0.0:8080/v1'],
    meta_template=api_meta_template,
    query_per_second=50,
    batch_size=16,
    temperature=0,
    verbose=False,
    max_out_len=16384,
    max_seq_len=32768,
)]

## ------------- Evaluation Configuration
eval = dict(
    partitioner=dict(type=SubjectiveNaivePartitioner, models=models, judge_models=judge_models,),
    runner=dict(type=LocalRunner, max_num_workers=16, task=dict(type=SubjectiveEvalTask)),
)

summarizer = dict(type=SubjectiveSummarizer, function='subjective')
work_dir = 'outputs/Qwen2.5-7B-Instruct/subjective/alignbench'

```

### Reproduces the problem - command or script

```
python run.py configs/eval_subjective.py -r
```

### Reproduces the problem - error message

```
TypeError: NaivePartitioner.__init__() got an unexpected keyword argument 'num_worker'
```
我按照eval_subjective.py自己改了一下参数配置，机器是8卡，想启动类似--max-num-workers 8 的功能于是我在infer中加入partitioner=dict(type=NaivePartitioner, num_worker=8),但这貌似是错误的。
请问我该如何修改我的config使得它支持先使用8卡进行推理，然后复用推理结果起一个API服务使用72B模型进行主观评测呢？

### Other information

_No response_

目前我使用API模型在8卡机器进行评测的方法类似如下
先进行推理
python run.py --models vllm_qwen2_5_7b_instruct --datasets ... --max-num-workers 8
推理结束后卡资源空出起一个8卡72B的API，然后
python run.py --models vllm_qwen2_5_7b_instruct --datasets ... --max-num-workers 8 --reuse
datasets中填入了
```
datasets = sum(
    (v for k, v in locals().items() if k.endswith('_datasets')),
    [],
)

api_meta_template = dict(
    round=[
        dict(role='HUMAN', api_role='HUMAN'),
        dict(role='BOT', api_role='BOT', generate=True),
    ],
    reserved_roles=[dict(role='SYSTEM', api_role='SYSTEM')],
)


verifier_cfg = dict(
    abbr='vllm-qwen2-5-72b-instruct',
    type=OpenAISDK,
    path='Qwen2_5_72B',
    key='EMPTY',
    tokenizer_path='/Qwen-Series/Qwen2.5-72B-Instruct',
    openai_api_base=['http://0.0.0.0:8080/v1'],
    meta_template=api_meta_template,
    query_per_second=50,
    batch_size=16,
    temperature=0,
    verbose=False,
    max_out_len=16384,
    max_seq_len=32768
)

for item in datasets:
    if 'judge_cfg' in item['eval_cfg']['evaluator']:
        item['eval_cfg']['evaluator']['judge_cfg'] = verifier_cfg
```


目前已经解决了数据分片的问题，在infer中改为partitioner=dict(type=NumWorkerPartitioner, num_worker=8, num_split=None),
但请问这样改以后eval字段即
eval = dict(
    partitioner=dict(type=SubjectiveNaivePartitioner, models=models, judge_models=judge_models,),
    runner=dict(type=LocalRunner, max_num_workers=16, task=dict(type=SubjectiveEvalTask)),
)
需要改动吗，上述eval配置在推理结束后能-r复用分片的推理结果进行主观评测吗

> 目前已经解决了数据分片的问题，在infer中改为partitioner=dict(type=NumWorkerPartitioner, num_worker=8, num_split=None), 但请问这样改以后eval字段即 eval = dict( partitioner=dict(type=SubjectiveNaivePartitioner, models=models, judge_models=judge_models,), runner=dict(type=LocalRunner, max_num_workers=16, task=dict(type=SubjectiveEvalTask)), ) 需要改动吗，上述eval配置在推理结束后能-r复用分片的推理结果进行主观评测吗

目前情况是评测也评测完了，但是summary处报错
```
Traceback (most recent call last):
  File "/opencompass-main/run.py", line 6, in <module>
    main()
  File "/opencompass-main/opencompass/cli/main.py", line 377, in main
    summarizer_cfg = dict(type=dataset[0]['summarizer']['type'], config=temp_cfg)
  File "/usr/local/python3.10.2/lib/python3.10/site-packages/mmengine/config/config.py", line 138, in __getitem__
    return self.build_lazy(super().__getitem__(key))
  File "/usr/local/python3.10.2/lib/python3.10/site-packages/mmengine/config/config.py", line 105, in __missing__
    raise KeyError(name)
KeyError: 'summarizer'
```

Please help check this issue @bittersweet1999 

因为你用的是```from opencompass.configs.datasets.subjective.alignbench.alignbench_judgeby_critiquellm_new import alignbench_datasets```,所以```from opencompass.summarizers import SubjectiveSummarizer```改成```from opencompass.summarizers import DefaultSubjectiveSummarizer```就可以了

> 因为你用的是`from opencompass.configs.datasets.subjective.alignbench.alignbench_judgeby_critiquellm_new import alignbench_datasets`,所以`from opencompass.summarizers import SubjectiveSummarizer`改成`from opencompass.summarizers import DefaultSubjectiveSummarizer`就可以了
感谢，另外想请教一下alignbench_judgeby_critiquellm_new 和alignbench_judgeby_critiquellm我发现new少了summarizer多了一个postprocess，请问两者有什么区别呢

没有本质区别，为了兼容新旧配置设置的