zhiminy/EvalEng huggingface dataset:
Repository: open-compass/opencompass
Issue: [Bug] 部分数据集中对question进行增广时使用浅拷贝会导致选项与答案错位
Root cause: Algorithmic Error

### 先决条件

- [x] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [x] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

大部分数据集在对数据进行复制增广时使用类似```data_list = data * n_repeats```的语句对数据复制多份，这样得到的```data_list```是浅拷贝，改动其中一个元素会同步改动其他复制出来的元素。以下是一个例子

```python
data_dict_list = [
    { "value": 1},
]

data_dict_list = data_dict_list * 4

# 改动第一个元素的“value”值
data_dict_list[0]["value"] = 2

print(data_dict_list)
# 打印结果后所有元素的"value"值均被改变
# --> [{'value': 2}, {'value': 2}, {'value': 2}, {'value': 2}]
```

比如gpqa数据集中直接使用 ```data_list = data * n_repeats``` 对数据进行复制增广后，下面的扰乱选项会导致只有1/4的数据正确评测，其余3/4的数据正确答案与选项错位，最终结果是模型在3/4的数据上等价于随机选答案
https://github.com/open-compass/opencompass/blob/35ec307c6bcafb95a35878dbe3b24d9faf3a311d/opencompass/datasets/gpqa.py#L88


一种正确的方式是直接使用copy库的deepcopy
```python
data_repeats = [copy.deepcopy(item) for item in data for _ in range(n_repeats)]
```

### 重现问题 - 代码/配置示例

见上

### 重现问题 - 命令或脚本

见上

### 重现问题 - 错误信息

见上

### 其他信息

_No response_

Thanks for the report, we will check this issue.
Repository: open-compass/opencompass
Issue: [Feature] humaneval_postprocess历史无效的单测清理或case更新
Root cause: Validation Gap

### Describe the feature

**【问题背景】**
这边在本地拉取和构建opencompass的过程中，发现test_humaneval.py文件对应的单测无法正常运行，经过检查后，发现原单测所测试的humaneval_postprocess函数已经在 2024.06.28， commit id为 a32f21a3的一次提交中被迭代掉了，取而代之的是humaneval_postprocess_v2、humaneval_postprocess_v3

**【希望确认的点】**
humaneval_postprocess_v2函数 与 humaneval_postprocess_v3函数的逻辑看起来相当简洁，核心是正则表达式是否符合预期，我不确定项目团队是否有计划为这两个函数添加新的单测，如果没有的话或许可以直接删除test_humaneval，如果有的话可能需要为这两个函数涉及一套新的验证case
如果有我可以贡献力量的地方我会非常乐意创建PR

### Will you implement it?

- [x] I would like to implement this feature and create a PR!
Repository: open-compass/opencompass
Issue: [Feature] 支持多个api endpoint
Root cause: Unimplemented Feature Gap

### 描述该功能

当资源充分时，是否有可能支持多个类openai endpoint（vllm）

如下，`openai_api_base`是否可以允许传入多个？

```
from opencompass.models import OpenAISDK

api_meta_template = dict(
    round=[
        dict(role='HUMAN', api_role='HUMAN'),
        dict(role='BOT', api_role='BOT', generate=True),
    ],
    reserved_roles=[dict(role='SYSTEM', api_role='SYSTEM')],
)

models = [
    dict(
        abbr='Meta-Llama-3-8B-Instruct-LMDeploy-API',
        type=OpenAISDK,
        key='EMPTY', # API key
        openai_api_base='http://0.0.0.0:23333/v1', # 服务地址
        path='Meta-Llama-3-8B-Instruct ', # 请求服务时的 model name
        tokenizer_path='meta-llama/Meta-Llama-3.1-8B-Instruct', # 请求服务时的 tokenizer name 或 path, 为None时使用默认tokenizer gpt-4
        rpm_verbose=True, # 是否打印请求速率
        meta_template=api_meta_template, # 服务请求模板
        query_per_second=1, # 服务请求速率
        max_out_len=1024, # 最大输出长度
        max_seq_len=4096, # 最大输入长度
        temperature=0.01, # 生成温度
        batch_size=8, # 批处理大小
        retry=3, # 重试次数
    )
]
```

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

很好，我看到它了，似乎需要结合数据分片一起使用？
Repository: open-compass/opencompass
Issue: 在没有 ppl 的 dataset 中如何测试 base model？
Root cause: Documentation Deficiency

### 先决条件

- [x] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [x] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

无

### 重现问题 - 代码/配置示例

无

### 重现问题 - 命令或脚本

无

### 重现问题 - 错误信息

无

### 其他信息

_No response_
Repository: open-compass/opencompass
Issue: 是否计划对DeepSeek R1 及诸多蒸馏模型提供支持

### 描述该功能

是否计划对DeepSeek R1 及诸多蒸馏模型提供支持

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

可以直接测的

> 可以直接测的

似乎没有看到对应的config

        dict(
        type=TurboMindModelwithChatTemplate,
        abbr='DeepSeek-R1-Distill-Qwen-32B',
        path='deepseek-ai/DeepSeek-R1-Distill-Qwen-32B',
        engine_config=dict(session_len=32768, max_batch_size=16, tp=4),
        gen_config=dict(top_k=1, temperature=0.1, top_p=1, max_new_tokens=32768, do_sample=True),
        max_seq_len=32768,
        max_out_len=32768,
        batch_size=16,
        run_cfg=dict(num_gpus=4),
    ) 供参考

多谢

---- 回复的原邮件 ----
| 发件人 | ***@***.***> |
| 发送日期 | 2025年02月11日 17:53 |
| 收件人 | open-compass/opencompass ***@***.***> |
| 抄送人 | xujunrt ***@***.***>,
Author ***@***.***> |
| 主题 | Re: [open-compass/opencompass] 是否计划对DeepSeek R1 及诸多蒸馏模型提供支持 (Issue #1864) |

    dict(
    type=TurboMindModelwithChatTemplate,
    abbr='DeepSeek-R1-Distill-Qwen-32B',
    path='deepseek-ai/DeepSeek-R1-Distill-Qwen-32B',
    engine_config=dict(session_len=32768, max_batch_size=16, tp=4),
    gen_config=dict(top_k=1, temperature=0.1, top_p=1, max_new_tokens=32768, do_sample=True),
    max_seq_len=32768,
    max_out_len=32768,
    batch_size=16,
    run_cfg=dict(num_gpus=4),
) 供参考


—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: ***@***.***>

Feel free to re-open if needed.
Repository: open-compass/opencompass
Issue: [Feature] 请问如何加载本地模型进行测评
Root cause: Documentation Deficiency

### Describe the feature

opencompass --models hf_internlm2_5_1_8b_chat --datasets demo_gsm8k_chat_gen
您好，看到您举例的这个方法似乎只能加载线上模型，如果我想加载微调过的本地模型，该如何操作呢？

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!
Repository: open-compass/opencompass
Issue: [Feature] 是否可以有输出结果后，单独去计算一下评分的功能
Root cause: Unimplemented Feature Gap

### 描述该功能

在测试数据集过程中，整个模型输出都已经输出并记录成json，但是在计算分数的过程中报错导致没有结果。现在是否可以直接调用哪个脚本计算一下得分？

### 是否希望自己实现该功能？

- [x] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

确实重新请求一次耗时太长了，如果能利用已有的结果评测就好了，实现应该不难，就是把步骤拆出来。希望官方给个demo文档

同意, infer完后的Prediction结果都保存成功了, 但是eval的时候报错. 修改Bug后如果再评测就要再进行一轮Infer, 又要花一堆时间, 希望官方给个demo文档哇.

同需要这个功能

这个功能支持了吗
Repository: open-compass/opencompass
Issue: [Bug] https://github.com/open-compass/opencompass/blob/main/configs/eval_academic_leaderboard_202412.py找不到了
Root cause: Documentation Deficiency

### 先决条件

- [x] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [x] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

目录https://github.com/open-compass/opencompass/blob/main/configs/eval_academic_leaderboard_202412.py找不到了

### 重现问题 - 代码/配置示例

[Bug] https://github.com/open-compass/opencompass/blob/main/configs/eval_academic_leaderboard_202412.py找不到了

### 重现问题 - 命令或脚本

[Bug] https://github.com/open-compass/opencompass/blob/main/configs/eval_academic_leaderboard_202412.py找不到了

### 重现问题 - 错误信息

[Bug] https://github.com/open-compass/opencompass/blob/main/configs/eval_academic_leaderboard_202412.py找不到了

### 其他信息

_No response_

examples/eval_academic_leaderboard_202412.py
Repository: open-compass/opencompass
Issue: [Feature] expose `max_task_size` in run.py, for quick debug
Root cause: Unimplemented Feature Gap

### Describe the feature

Currently, if I want to quick debug my evaluation process with only a few (e.g., 5) data samples, I have to hard-code the *per-task* cfg files. Allowing users to simply pass this parameter into run.py would be helpful.

Another brute workaround: backup the original dataset and temporally clip the full-size test.jsonl into only a few lines.

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!
Repository: open-compass/opencompass
Issue: [Bug] The relative path in tools/list_configs.py
Root cause: Architectural Constraint

### Prerequisite

- [x] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [x] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

Tag 0.4.0

### Reproduces the problem - code/configuration sample

``` python
python tools/list_configs.py llama
```

### Reproduces the problem - command or script

``` python
python tools/list_configs.py llama
```

### Reproduces the problem - error message

null

### Other information

Since the folder `configs` has moved, the relative path in `tools/list_configs.py` needs to be modified.

只需要调用match_files加层opencompass目录就可以解决了:models = match_files('opencompass/configs/models/', args.pattern, fuzzy=True)

Also datasets = match_files('opencompass/configs/datasets/', args.pattern, fuzzy=True)
Repository: open-compass/opencompass
Issue: [Feature] 目前是否有适配Codeforces、SWE Verified、Aider-Polyglot这些在R1中出现的数据集的计划呢？
Root cause: Unimplemented Feature Gap

### Describe the feature

这段时间在测试R1模型，有不少数据集是现在opencompass还未支持的，不知道目前是否有计划能进行适配呢？
![Image](https://github.com/user-attachments/assets/03b2d654-8daa-4a4c-890a-4f2e2830af9b)

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

蹲蹲回复

+1

+1

+1
Repository: open-compass/opencompass
Issue: [Feature] 我需要对本地部署的Qwen-110B模型进行MMLU基准测试，请问该怎么操作呢？
Root cause: Documentation Deficiency

### Describe the feature

我已经将项目下载到本地，并且将数据集也下载到本地了
git clone https://github.com/open-compass/opencompass opencompass
cd opencompass
pip install -e .

wget https://github.com/open-compass/opencompass/releases/download/0.2.2.rc1/OpenCompassData-core-20240207.zip
unzip OpenCompassData-core-20240207.zip

我在opencompass 文件夹下配置了configs，其中configs里面创建了py文件，文件如下：
from opencompass.models import OpenAI
from opencompass.partitioners import NaivePartitioner
from opencompass.runners import LocalRunner
from opencompass.tasks import OpenICLInferTask
from opencompass.datasets import MMLUDataset

# 配置本地Qwen模型API
model = OpenAI(
    abbr='qwen-110b-chat',
    path='qwen-110b-chat',
    key='EMPTY',  
    query_per_second=1, 
    max_seq_len=4096,
    api_base='http://localhost:8080/v1',  
    meta_template=None,
    retry=10  # 增加重试次数
)

# 配置MMLU数据集
datasets = [
    MMLUDataset(
        path='./data/OpenCompassData-core-20240207/mmlu', 
        name='stem', 
        split='test'
    )
]

# 配置工作流
work_dir = 'outputs/qwen_mmlu' 

infer = {
    'partitioner': {'type': NaivePartitioner},
    'runner': {
        'type': LocalRunner,
        'max_workers': 2, 
    },
    'task': {'type': OpenICLInferTask}
}

请问我配置的正确吗？我该如何进行mmlu基准测试？

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

I have same question, how to run a dataset evaluation on a local model instance.
Repository: open-compass/opencompass
Issue: [Feature] Need to identify status stop status in the dlc task
Root cause: Validation Gap

### Describe the feature

currently, If user stop dlc infer or eval task. The main process cannot be stopped. 

It seems that the current code only handles success and fail situations, Status "Stopped" should be also handled！

![Image](https://github.com/user-attachments/assets/68c52b29-7bb0-485e-9f9f-6f20ce3a19b6)

### Will you implement it?

- [x] I would like to implement this feature and create a PR!

add into pr.
Repository: open-compass/opencompass
Issue: [Feature] Can we support terminate dlc and volc tasks when oc evaluation task is terminated.
Root cause: Unimplemented Feature Gap

### Describe the feature

currently If we kill or terminate a oc evaluation task with dlc or volc runners, the dlc or volc tasks is not stopped automatically.

**Can we recieve kill or terminate signal to stop dlc or volc tasks automatically?**

1. start a oc task with dlc or volc runner. such as opencompass xxx.py
2. kill the process by ctrl+c or by kill command
3. the process recieve the kill or terminate signel, then handle dlc or volc tasks into stop, then exit the evaluation process. 

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!
Repository: open-compass/opencompass
Issue: [Bug] 使用max-num-worker使得ssh断连
Root cause: Resource Mishandling

### Prerequisite

- [x] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [x] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

我需要并行评测的我的模型，因为评估单个example的时间太长，所以我参考了 #1755 中的代码，使用```CUDA_VISIBLE_DEVICES=6,7 opencompass --models vllm_qwen2_5_0_5b_instruct --datasets triviaqa_gen -a vllm --max-num-worker 2```这种方式进行并行。然而每次当我启动此代码时，这将导致我的ssh直接断连，并返回报错`shell request failed on channel 0`，感觉该问题与opencompass的code有关。

### Reproduces the problem - code/configuration sample

CUDA_VISIBLE_DEVICES=6,7 opencompass --models vllm_qwen2_5_0_5b_instruct --datasets triviaqa_gen -a vllm --max-num-worker 2

### Reproduces the problem - command or script

CUDA_VISIBLE_DEVICES=6,7 opencompass --models vllm_qwen2_5_0_5b_instruct --datasets triviaqa_gen -a vllm --max-num-worker 2

### Reproduces the problem - error message

shell request failed on channel 0

### Other information

 

Can you launch the evaluation with --debug?

Yes, when running with --debug option, the ssh connection is fine. However, it will only use one GPU instead of the number I set in CUDA_VISIBLE_DEVICES and max-num-worker.

Actually, I have no idea about this bug. Can you use tmux and then try to remove --debug?
Repository: open-compass/opencompass
Issue: [Bug] Take too much time on MATH-500 dataset evaluation
Root cause: Configuration Error

### Prerequisite

- [x] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [x] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

it can inference correctly

### Reproduces the problem - code/configuration sample

python run.py     --datasets math_500_gen     --hf-type base     --hf-path /home/maoshizhuo/2025/deepseek-Qwen-1.5B     --debug     --max-out-len 32768
02/25 23:53:14 - OpenCompass - INFO - Loading math_500_gen: /home/maoshizhuo/2025/opencompass/opencompass/configs/./datasets/math/math_500_gen.py
02/25 23:53:14 - OpenCompass - INFO - Loading example: /home/maoshizhuo/2025/opencompass/opencompass/configs/./summarizers/example.py
02/25 23:53:14 - OpenCompass - INFO - Current exp folder: outputs/default/20250225_235314
02/25 23:53:14 - OpenCompass - WARNING - SlurmRunner is not used, so the partition argument is ignored.
02/25 23:53:14 - OpenCompass - INFO - Partitioned into 1 tasks.
02/25 23:53:16 - OpenCompass - WARNING - Only use 1 GPUs for total 4 available GPUs in debug mode.
02/25 23:53:16 - OpenCompass - INFO - Task [deepseek-Qwen-1.5B_hf/math-500]
02/25 23:53:33 - OpenCompass - INFO - Try to load the data from /home/maoshizhuo/.cache/opencompass/./data/math/
02/25 23:53:33 - OpenCompass - INFO - Start inferencing [deepseek-Qwen-1.5B_hf/math-500]
11%|███████████████                                                                                                                        | 7/63 [13:49:33<118:18:44, 7605.80s/it]

### Reproduces the problem - command or script

python run.py     --datasets math_500_gen     --hf-type base     --hf-path /home/maoshizhuo/2025/deepseek-Qwen-1.5B     --debug     --max-out-len 32768

### Reproduces the problem - error message

need too much time to get result about 131 hours

### Other information

有什么办法加速推理吗？我注意到vllm可以加速推理，但是由于其集成了量化技术，得到的精度并不准确，我希望得到准确的结果并加速。我的实验环境有4张V100-32G GPU。谢谢！

If your model is a chat model, try to use `--hf-type chat`, this will use chat_template of the model. On the other hand, under the hood, OC uses HF to generate, try to call the original HF generate for one example to see if it also takes that long time.

Please check the prediction to find if there is a repeat pattern in response. And reduce the --max-out-len. Also, you can remove the --debug and use four workers.

The best option is to use vllm or lmdeploy, because the math requires the model to generate the reasoning process.

> Please check the prediction to find if there is a repeat pattern in response. And reduce the --max-out-len. Also, you can remove the --debug and use four workers.

Could you please help me solve this issue? https://github.com/open-compass/opencompass/issues/1929
Repository: open-compass/opencompass
Issue: [Feature] 是否有计划兼容python3.11
Root cause: External Dependency Breakage

### 描述该功能

最近在一些新的benchmark上发现不少仓库都使用3.11作为默认py版本，有一些内容确实和3.10有所不同，目前opencompass还是只能支持3.10，是否可以计划逐步兼容3.11？

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

For the moment, we do not have the plan to upgrade to higher version of Python. We will have a look internally to see the possibility for upgrade.

从我个人的经验来说，实际上opencompass唯一不支持3.10以上python的是[`pyext`](https://github.com/refi64/PyExt/tree/master)这个库，只需要一个很小的修改，但是官方没有更新。
我解决的办法是从fork的repo中安装好，再去装opencompass的环境。我目前使用python 3.12.9，运行良好。

pip install git+https://gitee.com/xana/PyExt.git@a95f488490fc57ec17d0c00a99c6bc0a4726824f
Repository: open-compass/opencompass
Issue: [Feature] Offering help with Unit Tests

Hi everyone! I noticed a lot of Pull Requests, particularly in relation to new features and tests.

I'd like to help improve the test coverage using [CodeBeaver](https://www.codebeaver.ai/), an AI agent that I built (it's free for open source!). To demonstrate what's possible, I've created a PR:
[[Feature] Add HLE (Humanity's Last Exam) dataset - Unit Tests](https://github.com/CodeBeaverAI/opencompass/pull/2)
that adds comprehensive tests to the last Pull Request that was opened here: [[Feature] Add HLE (Humanity's Last Exam) dataset](https://github.com/open-compass/opencompass/pull/1902).

## What the PR demonstrates:

* Increases coverage by +100.00% for `tests/test_datasets_info.py`, `tests/test_OlympiadBench.py`, and `tests/test_OlympiadBench_categories.py`
* Adds comprehensive tests with 3 new test files: 
  - `tests/test_datasets_info.py`
  - `tests/test_OlympiadBench.py`
  - `tests/test_OlympiadBench_categories.py`
* All new tests pass successfully, ensuring reliability

On top of that: If you check the PR you will see that CodeBeaver does not only go for coverage - it also adds all the edge cases it can think of (we make it iterate 3 times on that alone).

## How it works:

CodeBeaver analyzes your code changes in PRs
* Automatically generates and updates tests
* Opens PRs with new/updated tests
* Helps catch bugs before they reach production

## Next steps

If this interests the maintainer team, I'm happy to:

* Walk through how the tests were generated
* Help set up automated test generation for future PRs (takes ~5 minutes)
* Focus on specific areas you'd like to improve coverage for

Let me know what you think! Happy to adapt this approach based on the project's needs.
Repository: open-compass/opencompass
Issue: [Feature] Cannot find a huggingafce source code to reload model找不到加载hf模型的源代码
Root cause: Documentation Deficiency

### Describe the feature

我需要加载自定义的INT8量化模型以开展数学推理能力的评测
我现在只需要把加载hf模型的源代码
model = Qwen2ForCausalLM.from_pretrained('/home/maoshizhuo/2025/deepseek-Qwen-1.5B', torch_dtype=torch.bfloat16, device_map='auto')
改成
model = Int8Qwen2ForCausalLM.from_pretrained('/home/maoshizhuo/smoothquant/int8_models/deepseek-Qwen-1.5B-smoothquant_ds_Qwen2_1.5B_2048', torch_dtype=torch.float16, device_map='auto')
        load_parameters(model,'./models/model_params_ds_qwen2_1.5B.pth')
就可以开始推理了。
但我找遍了opencompass/models下的模型，并没有模型使用huggingface的transformers库加载的，但是在使用手册里面是有这个的，并且--hf-path就是可以加载hf模型，但我找不到哪里有model = Qwen2ForCausalLM.from_pretrained或者AutoModelForCausalLM?

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

@msz12345 There is a `HuggingFaceBaseModel` and `HuggingFacewithChatTemplate` class in `opencompass.models` which uses `AutoModelForCausalLM` under the hood. Does that work for you?
Repository: open-compass/opencompass
Issue: [Bug] 使用deepseek-v2-lite-chat进行humaneval测试时，出现问题，无论是在官网还是本地部署测试分数只有1.22，其他模型都正常
Root cause: Interface Contract Mismatch

### Prerequisite

- [x] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [x] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

![Image](https://github.com/user-attachments/assets/b5045993-c2b7-45a3-bf42-f33c481b5d2e)

![Image](https://github.com/user-attachments/assets/26ca7112-f20b-43cc-a1d5-4f0250a4b1a6)

![Image](https://github.com/user-attachments/assets/b04a0388-0b9e-402c-9818-b824bb51cf29)

### Reproduces the problem - code/configuration sample

![Image](https://github.com/user-attachments/assets/d0330a28-820f-47cf-827b-7bb728a3927e)
使用该配置

### Reproduces the problem - command or script

出现的问题

### Reproduces the problem - error message

。

### Other information

_No response_

其他的模型无论是在线测试还是我在离线测试分数都正常

Thank you for the report. We will investigate this issue.


![Image](https://github.com/user-attachments/assets/7938e47b-8336-4de1-962c-e7d517a3afe7)
对了我使用evalscope+opencompass推理结果也是比较正常的

I also find this problem. Plz fix it.