Repository: open-compass/opencompass
Issue: [Bug] How generic datasets can be evaluated using llm_judge?
Root cause: Documentation Deficiency

### Prerequisite

- [x] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [x] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

```
{'CUDA available': True,
 'GCC': 'gcc (GCC) 7.3.0',
 'MMEngine': '0.10.6',
 'MUSA available': False,
 'OpenCV': '4.11.0',
 'PyTorch': '2.5.1',
 'PyTorch compiling details': 'PyTorch built with:\n'
                              '  - GCC 10.2\n'
                              '  - C++ Version: 201703\n'
                              '  - Intel(R) MKL-DNN v3.5.3 (Git Hash '
                              '66f0cb9eb66affd2da3bf5f8d897376f04aae6af)\n'
                              '  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
                              '  - LAPACK is enabled (usually provided by '
                              'MKL)\n'
                              '  - NNPACK is enabled\n'
                              '  - CPU capability usage: NO AVX\n'
                              '  - Build settings: BLAS_INFO=open, '
                              'BUILD_TYPE=Release, '
                              'CXX_COMPILER=/opt/rh/devtoolset-10/root/usr/bin/c++, '
                              'CXX_FLAGS=-ffunction-sections -fdata-sections '
                              '-D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 '
                              '-fvisibility-inlines-hidden -DUSE_PTHREADPOOL '
                              '-DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI '
                              '-DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON '
                              '-DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK '
                              '-DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC '
                              '-Wall -Wextra -Werror=return-type '
                              '-Werror=non-virtual-dtor -Werror=bool-operation '
                              '-Wnarrowing -Wno-missing-field-initializers '
                              '-Wno-type-limits -Wno-array-bounds '
                              '-Wno-unknown-pragmas -Wno-unused-parameter '
                              '-Wno-strict-overflow -Wno-strict-aliasing '
                              '-Wno-stringop-overflow -Wsuggest-override '
                              '-Wno-psabi -Wno-error=old-style-cast '
                              '-Wno-missing-braces -fdiagnostics-color=always '
                              '-faligned-new -Wno-unused-but-set-variable '
                              '-Wno-maybe-uninitialized -fno-math-errno '
                              '-fno-trapping-math -Werror=format '
                              '-Wno-stringop-overflow, LAPACK_INFO=open, '
                              'TORCH_VERSION=2.5.1, USE_CUDA=OFF, '
                              'USE_CUDNN=OFF, USE_CUSPARSELT=OFF, '
                              'USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, '
                              'USE_GLOG=OFF, USE_GLOO=ON, USE_MKLDNN=ON, '
                              'USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, '
                              'USE_OPENMP=ON, USE_ROCM=OFF, '
                              'USE_ROCM_KERNEL_ASSERT=OFF, \n',
 'Python': '3.10.16 (main, Dec 11 2024, 16:18:56) [GCC 11.2.0]',
 'TorchVision': '0.20.1',
 'lmdeploy': "not installed:No module named 'lmdeploy'",
 'numpy_random_seed': 2147483648,
 'opencompass': '0.3.9+',
 'sys.platform': 'linux',
 'transformers': '4.48.2'}
```

### Reproduces the problem - code/configuration sample

I would like to use llm judge to eval general evaluation set(like mmlu lawbench, etc), which I modelled on eval_llm_judge.py as follows:
```
from mmengine.config import read_base
from opencompass.models.openai_api import OpenAISDK

with read_base():
    from opencompass.configs.models.qwen2_5.vllm_qwen2_5_7b_instruct import (
        models as vllm_qwen2_5_7b_instruct,
    )
    from opencompass.configs.models.qwen2_5.vllm_qwen2_5_14b_instruct import (
        models as vllm_qwen2_5_14b_instruct,
    )

from opencompass.openicl.icl_prompt_template import PromptTemplate
from opencompass.openicl.icl_retriever import ZeroRetriever
from opencompass.openicl.icl_inferencer import GenInferencer
from opencompass.evaluator import GenericLLMEvaluator
from opencompass.datasets import generic_llmjudge_postprocess
from opencompass.datasets import LawBenchDataset

names = [
    ['1-2', 'knowledge_question_answering'],
    ['3-6', 'case_analysis'],
]

# Template for the LLM judge
JUDGE_TEMPLATE = """
请评估以下回答是否正确地回答了问题。
问题：{question}
参考答案：{answer}
模型回答：{prediction}

模型回答是否正确？如果正确，请回答"A"；如果不正确，请回答"B"。
""".strip()

lawbench_datasets = []

for index, name in names:
    lawbench_reader_cfg = dict(
        input_columns=['instruction', 'question'],
        output_column='answer'
    )
    lawbench_infer_cfg = dict(
        prompt_template=dict(
            type=PromptTemplate,
            template=dict(
                round=[
                    dict(role='HUMAN', prompt='{instruction}\n{question}'),
                ]
            ),
        ),
        retriever=dict(type=ZeroRetriever),
        inferencer=dict(type=GenInferencer, max_out_len=1024)
    )
    
    lawbench_eval_cfg = dict(
        evaluator=dict(
            type=GenericLLMEvaluator,
            prompt_template=dict(
                type=PromptTemplate,
                template=dict(
                    begin=[
                        dict(
                            role='SYSTEM',
                            fallback_role='HUMAN',
                            prompt='你是一个负责评估模型输出正确性和质量的助手。',
                        )
                    ],
                    round=[
                        dict(role='HUMAN', prompt=JUDGE_TEMPLATE),
                    ],
                ),
            ),
        dataset_cfg=dict(
            abbr='lawbench-' + index + '-' + name + '-0-shot',
            type=LawBenchDataset,
            path='opencompass-main/data/lawbench/zero_shot_gen',
            index=index,
            reader_cfg=lawbench_reader_cfg,
        ),
        judge_cfg=vllm_qwen2_5_14b_instruct[0],
        dict_postprocessor=dict(type=generic_llmjudge_postprocess),
        ),
    )
    
    lawbench_datasets.append(
        dict(
            abbr='lawbench-' + index + '-' + name + '-0-shot',
            type=LawBenchDataset,
            path='opencompass-main/data/lawbench/zero_shot_gen',
            index=index,
            reader_cfg=lawbench_reader_cfg,
            infer_cfg=lawbench_infer_cfg,
            eval_cfg=lawbench_eval_cfg
        )
    )

    
models = vllm_qwen2_5_7b_instruct

work_dir = './outputs/llm_judge_eval'
```

### Reproduces the problem - command or script

pass

### Reproduces the problem - error message

I think there are still cases where my code is partially inadequate, for example, how do I assign which gpus the judge model is loaded on? How do I assign which gpus the eval model loads on? How do I turn on DP using a parameter like python run.py --max-num-workers?
I've been using python run.py to start opencompass, which is the way I'm most familiar with, is there any way to use llm judge for py files after the --datasets parameter? If so, how do I define dataset.py?

### Other information

_No response_

Your config is fine, the judge model and the inference model are using your local GPUs in the same manner. If you wish to differentiate between the judge model and the inference model, I suggest deploying a separate judge model service for invocation. Then use it in the following manner:：
```
from opencompass.models import OpenAISDK

judge_cfg = dict(
        abbr='qwen2-5-14b-instruct',
        type=OpenAISDK,
        path="Qwen/Qwen2.5-14B-Instruct",
        key='xxx',
        openai_api_base=[
            'http://xxx.xxx.xxx.xxx:xxx/v1'
        ],
        meta_template=api_meta_template,
        query_per_second=16,
        batch_size=16,
        temperature=0.001,
        verbose=True,
        max_out_len=16384,
        max_seq_len=32768
)
```


> Your config is fine, the judge model and the inference model are using your local GPUs in the same manner. If you wish to differentiate between the judge model and the inference model, I suggest deploying a separate judge model service for invocation. Then use it in the following manner:：
> 
> ```
> from opencompass.models import OpenAISDK
> 
> judge_cfg = dict(
>         abbr='qwen2-5-14b-instruct',
>         type=OpenAISDK,
>         path="Qwen/Qwen2.5-14B-Instruct",
>         key='xxx',
>         openai_api_base=[
>             'http://xxx.xxx.xxx.xxx:xxx/v1'
>         ],
>         meta_template=api_meta_template,
>         query_per_second=16,
>         batch_size=16,
>         temperature=0.001,
>         verbose=True,
>         max_out_len=16384,
>         max_seq_len=32768
> )
> ```

thanks a lot, I refer to aime2024_llmverify.py to know how to modify datasets.py

> Your config is fine, the judge model and the inference model are using your local GPUs in the same manner. If you wish to differentiate between the judge model and the inference model, I suggest deploying a separate judge model service for invocation. Then use it in the following manner:：
> 
> ```
> from opencompass.models import OpenAISDK
> 
> judge_cfg = dict(
>         abbr='qwen2-5-14b-instruct',
>         type=OpenAISDK,
>         path="Qwen/Qwen2.5-14B-Instruct",
>         key='xxx',
>         openai_api_base=[
>             'http://xxx.xxx.xxx.xxx:xxx/v1'
>         ],
>         meta_template=api_meta_template,
>         query_per_second=16,
>         batch_size=16,
>         temperature=0.001,
>         verbose=True,
>         max_out_len=16384,
>         max_seq_len=32768
> )
> ```
btw, what exactly should I put in path="Qwen/Qwen2.5-14B-Instruct"? i mean, is the path here for loading the tokenizer? because if you start the service as openai, wouldn't it be enough to add the address to openai_base_api?


After you deploy the Qwen/Qwen2.5-14B-Instruct as a service using vLLM or LMDeploy, the `path` above-mentioned is the `model` in the `client.chat.completions.create` of the OpenAI SDK. I suggest that you explore the relevant documentation.

> After you deploy the Qwen/Qwen2.5-14B-Instruct as a service using vLLM or LMDeploy, the `path` above-mentioned is the `model` in the `client.chat.completions.create` of the OpenAI SDK. I suggest that you explore the relevant documentation.

thanks a lot, I've successfully used llm judge for evaluation. BTW, I need to evaluate the results in the finance domain and I see the config py file for financeIQ in configs, but no specific download path. As well as seeing Fin-Eva on the official website but not the relevant config file in opencompass(https://hub.opencompass.org.cn/dataset-detail/Fin-Eva). 
Can you provide the download path of financeIQ for opencompass? And do I need to install Fin-Eva's github repository additionally or can I adapt it to opencompass directly?

> > After you deploy the Qwen/Qwen2.5-14B-Instruct as a service using vLLM or LMDeploy, the `path` above-mentioned is the `model` in the `client.chat.completions.create` of the OpenAI SDK. I suggest that you explore the relevant documentation.
> 
> thanks a lot, I've successfully used llm judge for evaluation. BTW, I need to evaluate the results in the finance domain and I see the config py file for financeIQ in configs, but no specific download path. As well as seeing Fin-Eva on the official website but not the relevant config file in opencompass(https://hub.opencompass.org.cn/dataset-detail/Fin-Eva). Can you provide the download path of financeIQ for opencompass? And do I need to install Fin-Eva's github repository additionally or can I adapt it to opencompass directly?

OpenCompass does not currently support downloading the financeIQ dataset. Please ensure it is prepared in your local directory.