zhiminy/EvalEng huggingface dataset:
Repository: open-compass/opencompass
Issue: [Bug] mmlu_pro结果正则提取出错
Root cause: Algorithmic Error

### 先决条件

- [x] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [x] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

None

### 重现问题 - 代码/配置示例

None

### 重现问题 - 命令或脚本

None

### 重现问题 - 错误信息

mmlu_pro输出结果提取时，以`Answer: Let's think step by step.`开头的回答提取出来的选项都是`L`，导致结果和分数计算错误

### 其他信息

建议修改 `first_option_postprocess`

```python
def first_option_postprocess(text: str, options: str, cushion=True) -> str:
    text = text.replace("Answer: Let's think step by step.", '')
    ....
```

```
def first_option_postprocess(text: str, options: str, cushion=True) -> str:
    text = text.replace("Answer: Let's think step by step.", '')
    ....
```

我按照这个修改后，结果提取看起来是正常了的，但是这么修改可能不太合理，所以没有提交PR；辛苦看一下是否有更合适的修改方式。

Thanks for your report, we will check this bug and reply soon.

Can you provide a running command for this bug?

我的有一些改动，就是使用`Qwen2.5-VL-Instruct`测评`mmlu_pro`，结果中就会出现此问题；我尝试使用`Qwen2.5-0.5B-Instruct`也有此问题。如果尝试复现可以使用此命令：
```
python run.py eval.py
```

`eval.py`内容如下：
```python
from mmengine.config import read_base

with read_base():
    from opencompass.configs.datasets.mmlu_pro.mmlu_pro_gen_cdbebf import mmlu_pro_datasets

datasets = [*mmlu_pro_datasets]

from opencompass.models import HuggingFacewithChatTemplate

model_path = '/path/to/Qwen2.5-0.5B-Instruct'

models = [
    dict(
        type=HuggingFacewithChatTemplate,
        path=model_path,
        tokenizer_path=model_path,
        max_seq_len=7168,
        max_out_len=1024,
        batch_size=16, 
        run_cfg=dict(num_gpus=1), 
    )
]
```


这个问题还存在，核心原因是mmlu_pro eval时的正则表达式有问题，提取了第一个符合格式`answer: x` 的字母X作为回答。

mmlu_pro的[eval_cfg](https://github.com/open-compass/opencompass/blob/main/opencompass/configs/datasets/mmlu_pro/mmlu_pro_0shot_cot_gen_08c1de.py#L52):
```python
    mmlu_pro_eval_cfg = dict(
        evaluator=dict(type=AccEvaluator),
        pred_postprocessor=dict(
            type=match_answer_pattern,
            answer_pattern=r'(?i)ANSWER\s*:\s*([A-P])')
    )
```
对应的[match_answer_pattern](https://github.com/open-compass/opencompass/blob/main/opencompass/utils/text_postprocessors.py#L241)：
```python
def match_answer_pattern(response_text: str, answer_pattern: str):
    match = re.search(answer_pattern, response_text)
    extracted_answer = match.group(1) if match else ''
    return extracted_answer
```
因此复现bug的一个例子如下：
```python
import re

answer_pattern=r'(?i)ANSWER\s*:\s*([A-P])'


def match_answer_pattern(response_text: str, answer_pattern: str):
    match = re.search(answer_pattern, response_text)
    extracted_answer = match.group(1) if match else ""
    return extracted_answer


RAW_STR = "The Final answer: ANSWER: F"
out = match_answer_pattern(RAW_STR, answer_pattern)
print(out)
```
输出结果是A，匹配的是`The Final answer: A` 这部分。

一个可能的修改方式是修改answer_pattern为`r'(?s).*(?i)ANSWER\s*:\s*([A-P])'`,简单测试是OK的，更多情况需要进一步验证
Repository: open-compass/opencompass
Issue: [Bug] huamneval测试Deepseek-R1结果过低
Root cause: Configuration Error

### Prerequisite

- [x] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [x] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

评测模型：deepseek-R1 671B 
评测数据集：humaneval
评测结果

<img width="382" alt="Image" src="https://github.com/user-attachments/assets/b99c7cf5-7936-48b5-bec6-e8905e238503" />

<img width="1375" alt="Image" src="https://github.com/user-attachments/assets/53fd6102-1425-4654-a7f0-0d68cb2d2b46" />

### Reproduces the problem - code/configuration sample

from mmengine.config import read_base

with read_base():
    from opencompass.configs.datasets.humaneval.humaneval_gen import humaneval_datasets

from opencompass.models import DeepSeekR1API
from opencompass.partitioners import NaivePartitioner
from opencompass.runners import LocalRunner
from opencompass.tasks import OpenICLInferTask

# 定义API模型的元模板
api_meta_template = dict(
    round=[
        dict(role='HUMAN', api_role='user'),
        dict(role='BOT', api_role='assistant', generate=True),
    ],
)

# 定义DeepSeek-R1模型配置
models = [
    dict(
        type=DeepSeekR1API,
        abbr='deepseek-r1',
        path='DeepSeek-R1',
        url='',
        meta_template=api_meta_template,
        query_per_second=1,
        max_seq_len=4096,
        max_out_len=1024,
        temperature=0.5,
        presence_penalty=1.03,
        frequency_penalty=1.0,
        top_p=0.95,
        batch_size=1,
        run_cfg=dict(num_gpus=0),  
    ),
]

# 使用HumanEval数据集
datasets = humaneval_datasets

# 定义任务
task = dict(type=OpenICLInferTask)

# 定义任务和运行器
infer = dict(
    partitioner=dict(type=NaivePartitioner),
    runner=dict(
        type=LocalRunner,
        max_num_workers=1,
        task=task,
    ),
    task=task,
)

# 配置工作目录
work_dir = './outputs/deepseek_r1_humaneval' 

### Reproduces the problem - command or script

运行命令：
python ./opencompass/run.py ./deepseek_r1_test.py

### Reproduces the problem - error message

<img width="358" alt="Image" src="https://github.com/user-attachments/assets/e2b83c40-2bde-4236-acb1-5ae00d90e14e" />

### Other information

_No response_
Repository: open-compass/opencompass
Issue: [Feature] aime2025的llm测评
Root cause: Unimplemented Feature Gap

### 描述该功能

aime2025的测评方式是大模型测评，但是aime2024生成的结果只是数字，不包含复杂的latex公式，为什么不直接使用aime2024的测评方式呢

### 是否希望自己实现该功能？

- [x] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

aime2025的测评方式是大模型测评，但是aime2025生成的结果只是数字，不包含复杂的latex公式，为什么不直接使用aime2024的测评方式呢

We found that the reasoning LLM cannot follow the instruction for the specific format requirement, and the rule-based evaluator will introduce some errors in cases such as matrices. We believe that LLM verification will be a new trend and we will take it as the future benchmark.

Feel free to add a prompt version for rule-based aime2025, looking forward to seeing your contribution.
Repository: open-compass/opencompass
Issue: opencompass/tasks/outer_eval/alpacaeval.py中 example.json 内容是什么格式？框架中没有看到样例
Root cause: Documentation Deficiency

https://github.com/open-compass/opencompass/blob/854c6bf025ed53e332ae58a7ee66807eae48618d/opencompass/tasks/outer_eval/alpacaeval.py#L65

同问，这个文件起什么作用？我遇到了下面的报错：
```
100%|██████████| 101/101 [6:42:31<00:00, 306.94s/it]
100%|██████████| 101/101 [6:42:31<00:00, 239.12s/it]
05/09 21:27:43 - OpenCompass - WARNING - Default to dump eval details, it might take extraspace to save all the evaluation details. Set --dump-eval-details False to skip the details dump
05/09 21:27:43 - OpenCompass - INFO - Partitioned into 1 tasks.
05/09 21:27:48 - OpenCompass - INFO - Try to load the data from /home/ecust/.cache/opencompass/./data/subjective/alpaca_eval

Map:   0%|          | 0/805 [00:00<?, ? examples/s]
Map: 100%|██████████| 805/805 [00:00<00:00, 8842.21 examples/s]
Traceback (most recent call last):
  File "/home/ecust/zhangmu/opencompass/run.py", line 4, in <module>
    main()
  File "/home/ecust/zhangmu/opencompass/opencompass/cli/main.py", line 385, in main
    runner(tasks)
  File "/home/ecust/zhangmu/opencompass/opencompass/runners/base.py", line 38, in __call__
    status = self.launch(tasks)
  File "/home/ecust/zhangmu/opencompass/opencompass/runners/local.py", line 120, in launch
    cmd = task.get_command(cfg_path=param_file, template=tmpl)
  File "/home/ecust/zhangmu/opencompass/opencompass/tasks/outer_eval/alpacaeval.py", line 123, in get_command
    }).run()
  File "/home/ecust/zhangmu/opencompass/opencompass/tasks/outer_eval/alpacaeval.py", line 63, in run
    with open(
FileNotFoundError: [Errno 2] No such file or directory: '/home/ecust/zhangmu/opencompass/data/subjective/alpaca_eval/example.json'
```
Repository: open-compass/opencompass
Issue: [Feature] 希望增加多机多卡测试的功能
Root cause: Unimplemented Feature Gap

### 描述该功能

我使用的是 vllm 框架，需要能够使用多级多卡进行模型评估测试。

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

I support you. Since now, I don't think opencompass havs ready2use multi-gpu framework yet. So for example, qwen qwq-32b model is not supported for vllm fast inference. Many internal bugs have to be resolved~

> I support you. Since now, I don't think opencompass havs ready2use multi-gpu framework yet. So for example, qwen qwq-32b model is not supported for vllm fast inference. Many internal bugs have to be resolved~

I've had success with the openai service on 8 gpus. you can try it 

> I support you. Since now, I don't think opencompass havs ready2use multi-gpu framework yet. So for example, qwen qwq-32b model is not supported for vllm fast inference. Many internal bugs have to be resolved~

I'm running qwq-32b model with local model for vllm fast inference on 2 gpus. 

> > I support you. Since now, I don't think opencompass havs ready2use multi-gpu framework yet. So for example, qwen qwq-32b model is not supported for vllm fast inference. Many internal bugs have to be resolved~
> 
> I'm running qwq-32b model with local model for vllm fast inference on 2 gpus.

can you tell me what's your command to run TP?
Repository: open-compass/opencompass
Issue: [Feature] Can base model start the service via openai for ppl evaluation?
Root cause: Unimplemented Feature Gap

### Describe the feature

base model over 7/8b requires multiple gpu parallelism on my gpu, already tested instruct model e.g. qwen2_5-72b-instruct can be used for multi-DP evaluation with multiple gpu via openai method (--max-num-workers 8, the reason for using this method is that --hf-gpu-nums doesn't actually work). To summarize, how can base model use openai service method for ppl evaluation?

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!
Repository: open-compass/opencompass
Issue: [Feature] Does OpenCompass support multiple machines with multiple gpus (e.g. 2 machines with 8 gpus total 16 gpus)?
Root cause: Unimplemented Feature Gap

### Describe the feature

I would like to accelerate long text reviews of large parameter models by using multiple machines with multiple gpus.
What command should I use to start the above service?

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

We currently support multi-GPU but do not support multi-machine setups. You can refer to the configuration related to GPU quantity setup in the `Quick Start` section of the documentation:
https://doc.opencompass.org.cn/get_started/quick_start.html


> We currently support multi-GPU but do not support multi-machine setups. You can refer to the configuration related to GPU quantity setup in the `Quick Start` section of the documentation: https://doc.opencompass.org.cn/get_started/quick_start.html

thanks, i've already used opencompass to eval my task, since you said that opencompass does not support multi-machine then how the deepseek-r1-671B finished the eval task on opencompass

We are deploying the deepseek-r1-671B as an API service for evaluation.
Repository: open-compass/opencompass
Issue: [Feature] 增加 benbench 评测集
Root cause: Unimplemented Feature Gap

### Describe the feature

发现在pr中有benbench相关的改进但是很久没有合并到主分支，是否可以近期合并一下benbench的相关测试

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

We will address PR requests in the near future.
Repository: open-compass/opencompass
Issue: [Bug] 多卡测试llama-3-8b-vllm，精度为0
Root cause: Configuration Error

### Prerequisite

- [x] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [x] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

python run.py configs/vllm/eval_llama3_vllm.py

### Reproduces the problem - code/configuration sample

from mmengine.config import read_base

with read_base():
    from ..datasets.ARC_c.ARC_c_gen_1e0de5 import ARC_c_datasets
    #from ..datasets.ARC_e.ARC_e_gen_1e0de5 import ARC_e_datasets
    from ..summarizers.example import summarizer

datasets = sum([v for k, v in locals().items() if k.endswith("_datasets") or k == 'datasets'], [])
work_dir = './outputs/llama3/'

from opencompass.models import VLLM

models = [
    dict(
        type=VLLM,
        abbr='llama-3-8b-vllm',
        path="./models/Meta-Llama-3-8B",
        model_kwargs=dict(tensor_parallel_size=4),
        max_out_len=100,
        max_seq_len=2048,
        batch_size=16,
        generation_kwargs=dict(temperature=0),
        run_cfg=dict(num_gpus=4, num_procs=1),
    )
]

### Reproduces the problem - command or script

四张卡测试ARC精度，精度为0

### Reproduces the problem - error message

![Image](https://github.com/user-attachments/assets/94ccc4f2-3bb7-42e4-afc6-2b703299d8dd)

### Other information

_No response_

Hi @normlpl, would you check the predictions you get from this model?

try max_out_len=1024 or larger.
Repository: open-compass/opencompass
Issue: 可以提供下关于Qwen-7B-Chat模型评测的代码吗？我这边复现不了表里的结果?
Root cause: Documentation Deficiency

### Describe the feature

ceval                                   -          naive_average     gen     48.97
agieval                                 -          naive_average     gen     26.44
mmlu                                    -          naive_average     gen     45.22
GaokaoBench                             -          weighted_average  gen     49.25

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

+需要

Please take a look at this:

https://github.com/InternLM/opencompass/issues/220#issuecomment-1681932513

Feel free to re-open if needed.
Repository: open-compass/opencompass
Issue: [Feature] How about add bfcl evaluation to the repo
Root cause: Unimplemented Feature Gap

### 描述该功能

So far, I don't think opencompass has bfcl already. I was wondering if it can be added in future

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

As a hint, look up something like https://github.com/ShishirPatil/gorilla/blob/main/berkeley-function-call-leaderboard/README.md

> As a hint, look up something like https://github.com/ShishirPatil/gorilla/blob/main/berkeley-function-call-leaderboard/README.md

We consider adding the evaluation of agent capabilities later, such as function call. You are welcome to contribute at that time.
Repository: open-compass/opencompass
Issue: [Bug] The ifeval evaluation on 541 cases returns unexpected low score.
Root cause: Validation Gap

### Prerequisite

- [x] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [x] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

from opencompass.models import VLLMwithChatTemplate

models = [
    dict(
        type=VLLMwithChatTemplate,
        abbr='qwa-32b',
        path='path to QwQ-32B',
        model_kwargs=dict(tensor_parallel_size=4),
        max_out_len=32768,
        max_seq_len=131072,
        batch_size=4,
        generation_kwargs=dict(temperature=0),
        run_cfg=dict(num_gpus=4),
    )
]

### Reproduces the problem - code/configuration sample

# export CUDA_VISIBLE_DEVICES=0,1,2,3
export CUDA_VISIBLE_DEVICES=1,2,3,6
export VLLM_WORKER_MULTIPROC_METHOD=spawn
nohup python3 run.py --models vllm_qwq_32b_preview --datasets IFEval_gen_3321a3 --debug >lmd2.log 2>&1 &


### Reproduces the problem - command or script

from opencompass.openicl.icl_prompt_template import PromptTemplate
from opencompass.openicl.icl_retriever import ZeroRetriever
from opencompass.openicl.icl_inferencer import GenInferencer
from opencompass.datasets import IFEvalDataset, IFEvaluator

ifeval_reader_cfg = dict(
    input_columns=['prompt'], output_column='reference')

ifeval_infer_cfg = dict(
    prompt_template=dict(
        type=PromptTemplate,
        template=dict(round=[
            dict(
                role='HUMAN',
                prompt='{prompt}'),
        ])),
    retriever=dict(type=ZeroRetriever),
    inferencer=dict(type=GenInferencer, max_out_len=32768))

ifeval_eval_cfg = dict(
    evaluator=dict(type=IFEvaluator),
    pred_role='BOT',
)

ifeval_datasets = [
    dict(
        abbr='IFEval',
        type=IFEvalDataset,
        path='data/ifeval/ifeval.jsonl',
        reader_cfg=ifeval_reader_cfg,
        infer_cfg=ifeval_infer_cfg,
        eval_cfg=ifeval_eval_cfg)
]


### Reproduces the problem - error message

raw format
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-------------------------------
Model: qwa-32b
IFEval: {'Prompt-level-strict-accuracy': 34.011090573012936, 'Inst-level-strict-accuracy': 48.80095923261391, 'Prompt-level-loose-accuracy': 37.52310536044362, 'Inst-level-loose-accuracy': 52.15827338129496}

### Other information

Do you know where is the problem, as qwq-32b official score on the ifeval dataset is something 83+. BTW, I'm using the ifeval dataset from source https://huggingface.co/datasets/google/IFEval
Repository: open-compass/opencompass
Issue: [Bug] 最新版本的默认summarizer 会在平均分上产生冗余结果
Root cause: Documentation Deficiency

### 先决条件

- [x] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [x] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

![Image](https://github.com/user-attachments/assets/dadd08b8-fe36-494e-a05e-5f000a346f07)



### 重现问题 - 代码/配置示例

![Image](https://github.com/user-attachments/assets/bd50426e-f9ff-45da-88ff-a2eac9f794c0)



### 重现问题 - 命令或脚本

python3 run.py --models hf_qwen2_5_7b  --datasets cmmlu_gen_c13365  --dump-eval-details


### 重现问题 - 错误信息

![Image](https://github.com/user-attachments/assets/dadd08b8-fe36-494e-a05e-5f000a346f07)

这里算平均分的地方感觉完全没必要产生2次，希望能检查一下相关的逻辑

### 其他信息

_No response_

Hello @linbeyoung, thank you for your question. "naive_average" is a default metric we set for the `DefaultSummarizer` which takes the aggregate average of the summary group. It can turn out to be equivalent to the individual metric if there is only a single dataset within the summary group. You can filter out the metric in the resulting csv if it's not relevant for your use case.
Repository: open-compass/opencompass
Issue: [Feature] Does opencompass supports doing evaluation for rag like dify?
Root cause: Unimplemented Feature Gap

### 描述该功能

Doing evaluation for rag like dify.

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

Hi, you might consider wrapping the model that uses RAG into an API, so that you can directly invoke the API to use opencompass for evaluation.

Solved by this:
https://github.com/lzskyline/OpenDify/tree/main
Repository: open-compass/opencompass
Issue: [Feature] 推理指引文档链接失效
Root cause: Documentation Deficiency

### 描述该功能


<img width="993" alt="image" src="https://github.com/InternLM/opencompass/assets/75015153/9bf6ae0f-6aaf-4a8c-816d-a5617a24a840">

https://opencompass.org.cn/mmbench-submission

推理指引文档的链接已经失效

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

 Thanks for the report, we will fix this link issue soon.

Fixed
Repository: open-compass/opencompass
Issue: [Bug] OpenAISDK类的定义没有加@MODELS.register_module()的修饰符
Root cause: Validation Gap

### Prerequisite

- [x] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [x] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

@MODELS.register_module()
class OpenAISDK(OpenAI):

    def __init__(
        self,
        path: str = 'gpt-3.5-turbo',
        max_seq_len: int = 16384,
        query_per_second: int = 1,
        rpm_verbose: bool = False,
        retry: int = 2,
        key: str | List[str] = 'ENV',
        org: str | List[str] | None = None,
        meta_template: Dict | None = None,
        openai_api_base: str | List[str] = OPENAISDK_API_BASE,
        openai_proxy_url: Optional[str] = None,
        mode: str = 'none',
        logprobs: bool | None = False,
        top_logprobs: int | None = None,
        temperature: float | None = None,
        tokenizer_path: str | None = None,
        extra_body: Dict | None = None,
        verbose: bool = False,
        status_code_mappings: dict = {},
    ):

### Reproduces the problem - code/configuration sample

@MODELS.register_module()
class OpenAISDK(OpenAI):

    def __init__(
        self,
        path: str = 'gpt-3.5-turbo',
        max_seq_len: int = 16384,
        query_per_second: int = 1,
        rpm_verbose: bool = False,
        retry: int = 2,
        key: str | List[str] = 'ENV',
        org: str | List[str] | None = None,
        meta_template: Dict | None = None,
        openai_api_base: str | List[str] = OPENAISDK_API_BASE,
        openai_proxy_url: Optional[str] = None,
        mode: str = 'none',
        logprobs: bool | None = False,
        top_logprobs: int | None = None,
        temperature: float | None = None,
        tokenizer_path: str | None = None,
        extra_body: Dict | None = None,
        verbose: bool = False,
        status_code_mappings: dict = {},
    ):

### Reproduces the problem - command or script

opencompass/opencompass/models/openai_api.py文件中
OpenAISDK类的定义没有加@MODELS.register_module()的修饰符

### Reproduces the problem - error message

KeyError: 'opencompass.models.OpenAISDK is not in the opencompass::model registry. Please check whether the value of `opencompass.models.OpenAISDK` is correct or it was registered as expected. More details can be found at https://mmengine.readthedocs.io/en/latest/advanced_tutorials/config.html#import-the-custom-module'

### Other information

_No response_

Hi @StyleAIPro, would you mind providing the command you ran to reproduce this error?