zhiminy/EvalEng huggingface dataset:
Repository: open-compass/opencompass
Issue: [Bug] LongBench v2 dataset config is incorrect
Root cause: Configuration Error

### Prerequisite

- [x] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [x] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

Not relevant

### Reproduces the problem - code/configuration sample

opencompass：https://github.com/open-compass/opencompass/blob/main/opencompass/configs/datasets/longbenchv2/longbenchv2_gen_75fbba.py#L19
```
prompt='Please read the following text and answer the questions below.\n <text> \n {context} \n </text> \n \n What is the correct answer to this question: {question} \n \n Choices: \n (A) {choice_A} \n (B) {choice_B} \n (C) {choice_C} \n (D) {choice_D} \n Let’s think step by step. Based on the above, what is the single, most likely answer choice? Format your response as follows: "The correct answer is (insert answer here)'
```
longbenchv2：https://github.com/THUDM/LongBench/blob/main/prompts/0shot_cot_ans.txt
```
Please read the following text and answer the questions below.

The text is too long and omitted here.

What is the correct answer to this question: $Q$
Choices:
(A) $C_A$
(B) $C_B$
(C) $C_C$
(D) $C_D$

Let’s think step by step: $COT$

Based on the above, what is the single, most likely answer choice? Format your response as follows: "The correct answer is (insert answer here)".
```

OpenCompasse code **Missing “.**

### Reproduces the problem - command or script

Not relevant

### Reproduces the problem - error message

Not relevant

### Other information

_No response_
Repository: open-compass/opencompass
Issue: [Feature] 关于多卡加载模型的问题
Root cause: Documentation Deficiency

### 描述该功能

举例：
使用两张40G的A100进行chatglm6b的评测任务时，同一个模型会被均分到两个卡上，推理速度反而不如单卡推理快
如：launch OpenICLInfer[chatglm2-6b-hf/tnews-dev_1] on GPU 0,1

我认为更合理的方式应该是卡0和卡1各自加载一个模型，并分别推理不同的文件，但不知道怎样达到这种效果
launch OpenICLInfer[chatglm2-6b-hf/tnews-dev_0] on GPU 0
launch OpenICLInfer[chatglm2-6b-hf/tnews-dev_1] on GPU 1

对于相关参数的调整有一点疑惑，不知道如何调整
我理解关于这里可调的参数有以下4个
config中的
num_gpus，num_procs
以及参数中的，max-num-workers 和 -max-workers-per-gpu

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

For single dataset, DP is not supported currently. `num_gpus` is for loading bigger model, `num_procs` should be always 1 for hf models.
If you want to speed up, you could set `num_gpus=1` and alter `--max-partition-size` according to the dataset length, the runner will split the dataset automatically and run in parallel. 
Please try the `--dry-run` option in `run.py` before your experiment.
Repository: open-compass/opencompass
Issue: [Feature] 想用此项目,评估多模态大模型的纯文本能力,这个项目可以评估吗?比如intervl2.5
Root cause: Documentation Deficiency

### Describe the feature

想用此项目评估多模态大模型的纯文本能力,这个项目可以评估吗?比如intervl2.5

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

Yes, you can serve the model via API or use backend(such as LMDeploy).
You can find one example here: https://github.com/open-compass/opencompass/blob/main/opencompass/configs/models/internvl/lmdeploy_internvl_2_5_38b.py

Feel free to re-open if needed.

这是来自QQ邮箱的假期自动回复邮件。
 
您好，我最近正在休假中，无法亲自回复您的邮件。我将在假期结束后，尽快给您回复。
Repository: open-compass/opencompass
Issue: [Feature] Add mmlu-redux
Root cause: Unimplemented Feature Gap

### Describe the feature

Thanks for your contribution to this all-in-one evaluation kit. I find Qwen3 has used a MMLU variant, [MMLU-redux](https://github.com/aryopg/mmlu-redux), to conduct evaluation. But I can't find any relevant information about this dataset in our community. Will this dataset be considered to support?

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

After reviewing this dataset, I find the corrected answer is not totally an index for "ABCD" like mmlu. It is stored as a string, which can also be a multiple correct answer, or a text string which tells no correct answer in the choices, etc. To introduce this dataset, the [LLM Judger](https://github.com/open-compass/opencompass/blob/main/docs/en/advanced_guides/llm_judge.md) should be used.

Welcome to contribute to this dataset via Pull Request. You can refer to MMLU to implement this dataset. Additionally, you can implement a custom post process function or use the LLM as judge.
Repository: open-compass/opencompass
Issue: [Feature] add AI/ML API as provider
Root cause: Unimplemented Feature Gap

### Describe the feature

Hi!

I'm Sergey from the Integrations team over at [AI/ML API](https://aimlapi.com/), a startup with 150K+ users, providing over 300 AI models in one place

Your project looks dope, so we'd like a native integration. We already got integrations with Langflow, AutoGPT, and LiteLLM - so our integrations team is pretty seasoned :)

Say you're interested, and we'll test the compatibility, update the code/docs to include us, create a PR and add a tutorial on using opencompass with AI/ML API to our docs

If you guys give us a green light, we could get on it 👷

### Will you implement it?

- [x] I would like to implement this feature and create a PR!

No problem, welcome to contribute
Repository: open-compass/opencompass
Issue: [Feature] 请问主观评测脚本支持用本地模型作为judge模型吗？
Root cause: Unimplemented Feature Gap

### Describe the feature

examples/eval_subjective.py
在这个文件中，我把judge_models改为了vllmwithchattemplate的形式，似乎并不能正常评测，alpaca eval的最终输出结果为空。
请问主观评测脚本支持用本地模型作为judge模型吗？

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!
Repository: open-compass/opencompass
Issue: [Feature] 支持cleva的评测吗
Root cause: Unimplemented Feature Gap

### 描述该功能

https://github.com/LaVi-Lab/CLEVA

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

https://github.com/open-compass/opencompass/releases/tag/0.2.2.rc1

Currently, we have no plan for this dataset. Welcome to contribute to this dataset to OC
Repository: open-compass/opencompass
Issue: [Config] PHYBench config 问题
Root cause: Validation Gap

### 描述该功能

https://github.com/open-compass/opencompass/blob/5fd489994757e32bd64d8fbf3136bf71498c2a35/opencompass/configs/datasets/PHYBench/phybench_gen.py#L18

看起来 Remember 前面缺少个 空格

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！
Repository: open-compass/opencompass
Issue: [Bug] OpenICLEvalTask does not combine results split by NumWorkerPartitioner
Root cause: Architectural Constraint

### Prerequisite

- [x] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [x] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

```python
eval = dict(
    partitioner=dict(
       type=NumWorkerPartitioner,
       num_worker=64
   ),
   runner=dict(type=LocalRunner,
               max_num_workers=64,
              task=dict(type=OpenICLEvalTask)),
)
```

This eval config causes the eval phase resulting in individual split files. The files have the same name but with increasing numbering postfixes. Each splits has its very own accuracy calculated by evaluating the samples from within. 

This separation caused difficulty in summing up the accuracy scores, as the samples in the last file is not the same with the rest. 


e.g.
```bash
data_1.json
data_2.json
data_3.json
```

data1.json
```json
{
"accuracy": 90,
"details":[

...
]
}
```

### Reproduces the problem - code/configuration sample

```shell
python run.py examples/evaluation_script.py
```

### Reproduces the problem - command or script

```shell
python run.py examples/evaluation_script.py
```

### Reproduces the problem - error message

the summarizer can not find the desired result file, so the score is empty.

### Other information

_No response_
Repository: open-compass/opencompass
Issue: [Feature] Support Resume Mechanism for Interrupted Inference Tasks
Root cause: Documentation Deficiency

### Describe the feature

Problem Description
Currently, when OpenCompass performs large-scale model inference (infer), if a task is interrupted unexpectedly (e.g., due to resource failures, manual termination, etc.), it requires a full restart and cannot resume from the interruption point. This leads to:

Loss of completed inference results, causing resource waste through redundant computations

Poor fault tolerance for long-running tasks (e.g., full evaluation of billion-parameter models)

Feature Request
Add a resume mechanism for infer to achieve:

✅ Automatic progress recording: Periodically persist completed sample IDs/indices to a checkpoint file during runtime

✅ Intelligent resumption: Automatically detect checkpoint files upon restart and skip completed samples

✅ Progress visualization: Clearly display [Resumed] identifiers and remaining task volume in logs (e.g., Progress: 1200/5000 (resumed))

✅ Flexible control: Explicitly enable resumption via CLI parameters (e.g., --resume) to prevent accidental overwrites

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

Thanks for your suggestions. We currently support resume, you can use `-r latest` to restart the inference
Repository: open-compass/opencompass
Issue: 搞了好久demo一直运行不成功

### Prerequisite

- [X] I have searched [Issues](https://github.com/InternLM/opencompass/issues/) and [Discussions](https://github.com/InternLM/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/InternLM/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

1

### Reproduces the problem - code/configuration sample

1

### Reproduces the problem - command or script

1

### Reproduces the problem - error message

1

### Other information

1

Please provide all the **necessary** information instead of filling the blanks with meaningless information, with which we are not able to identify the reason.
Repository: open-compass/opencompass
Issue: [Bug] 使用不同数量的GPU进行任务评测结果不一致
Root cause: Algorithmic Error

### Prerequisite

- [x] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [x] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

opencompass 0.3.8版本

### Reproduces the problem - code/configuration sample

对于一个相同的任务，使用相同的推理引擎（hf），使用相同的参数配置，并且均采用贪婪策略，如果CUDA_VISIBLE_DEVICE=0，1，2，3来使用4张卡和使用8张卡，并且指定对应的数量的--max-num-worker。但是评估得到的结果最终存在一些差异。但是很奇怪我分别使用0，1，2，3或者使用4567四张卡进行推理时结果又一样，不同的卡数为什么会对评估结果造成影响？ 这是一个Bug还是预期行为

### Reproduces the problem - command or script

指定不同的CUDA_VISIBLE_DEVICE和设置对应数量的--max-num-worker就可以了

### Reproduces the problem - error message

无错误，只是不同数量的GPU相同配置相同参数相同策略时，在同一数据集上评估结果有差异

### Other information

_No response_

Different numbers of GPUs will use different tensor parallel strategies. It may introduce performance differences due to the implementation of some operation kernels.
Repository: open-compass/opencompass
Issue: [Feature] Ascend support accelerating reasoning models (e.g., Qwen3) using LMDeploy or vLLM?
Root cause: Unimplemented Feature Gap

### 描述该功能

Hi,
I would like to ask whether Ascend (e.g., ascend 910b) currently supports using LMDeploy or vLLM to accelerate the inference of reasoning models such as Qwen3.


### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

OpenCompass is agnostic to the inference backend. If vLLM or LMDeploy support Ascend, it's expected to run OpenCompass on it.
Repository: open-compass/opencompass
Issue: FileNotFoundError: Couldn't find a module script at opencompass-main/accuracy/accuracy.py. Module 'accuracy' doesn't exist on the Hugging Face Hub either.
Root cause: Validation Gap

### Prerequisite

- [X] I have searched [Issues](https://github.com/InternLM/opencompass/issues/) and [Discussions](https://github.com/InternLM/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/InternLM/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

1

### Reproduces the problem - code/configuration sample

1

### Reproduces the problem - command or script

1

### Reproduces the problem - error message

1

### Other information

_No response_

Duplicated with #219
Repository: open-compass/opencompass
Issue: [Bug] config file  gsm8k_contamination_ppl_ecdd22 has an importing bug
Root cause: Validation Gap

### Prerequisite

- [x] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [x] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

file  gsm8k_contamination_ppl_ecdd22.py  line 5:
```python
from opencompass.datasets import GSM8KDataset, GSM8KReferenceSkywork
```
There is no defination of class GSM8KReferenceSkywork



### Reproduces the problem - code/configuration sample

no

### Reproduces the problem - command or script

no

### Reproduces the problem - error message

no

### Other information

_No response_
Repository: open-compass/opencompass
Issue: [Bug] agieval_math is not multi-choice question.
Root cause: Configuration Error

### 先决条件

- [x] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [x] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

python

### 重现问题 - 代码/配置示例

python

### 重现问题 - 命令或脚本

agieval_gen_617738.py

### 重现问题 - 错误信息

<img width="795" alt="Image" src="https://github.com/user-attachments/assets/b67261fe-fa4d-4b42-b216-a1aafdfc7efb" />
agieval_math is not a multi-choice question，but prompt is 'math':
        'The following is a Math question. Please select the correct answer.' in agieval_gen_617738.py.

### 其他信息

_No response_
Repository: open-compass/opencompass
Issue: FileNotFoundError: Couldn't find a module script at opencompass-main/accuracy/accuracy.py. Module 'accuracy' doesn't exist on the Hugging Face Hub either.
Root cause: External Dependency Breakage

### Prerequisite

- [X] I have searched [Issues](https://github.com/InternLM/opencompass/issues/) and [Discussions](https://github.com/InternLM/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/InternLM/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

为啥会报这个错呢
FileNotFoundError: Couldn't find a module script at opencompass-mainaccuracy/accuracy.py. Module 'accuracy' doesn't exist on the Hugging Face Hub either.

### Reproduces the problem - code/configuration sample

1

### Reproduces the problem - command or script

1

### Reproduces the problem - error message

1

### Other information

_No response_

在outputs/demo/ .../logs/eval/下面，不知道为啥会报这个错，这个文件需要从哪里下载码

这个是metric，huggingface 应该可以自动下载的，参考https://huggingface.co/spaces/evaluate-metric/accuracy  ，可以在命令行试试能不能跑链接给的案例


Feel free to re-open if needed.
Repository: open-compass/opencompass
Issue: [Bug] 官网的指导手册有更新嘛？和现在master的代码完全对不上，新版本的内容多久能更新，官网对应的版本是多少
Root cause: Documentation Deficiency

### 先决条件

- [x] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [x] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

RT. 忍不住想问问

### 重现问题 - 代码/配置示例

无

### 重现问题 - 命令或脚本

无

### 重现问题 - 错误信息

无

### 其他信息

无

Agree more，the guidelines for API evaluation are almost dead, there is no clear way to use it

Sorry for the inconvenience. Please provide the specific errors and we would like to update the corresponding material.

是的，根据官方的指导手册，使用API调用，使用本地文件数据集都没有很好的指导手册。是否可以给出详细的说明呢
Repository: open-compass/opencompass
Issue: [Feature] The model evaluation is too slow, is this option to set enable_thinking=False?
Root cause: Documentation Deficiency

### 描述该功能

The model evaluation is too slow. Can we turn off the model thinking to improve the evaluation speed? Is this option available for configuration

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

Set  `enable_thinking`  to  `False` in `extra_body`in  a vllm api service like:

```
qwen3_235b_a22b_model = dict(
    abbr="qwen_3_235b_a22b_thinking", # Used to identify the model configuration
    key="YOUR_SERVE_API_KEY",
    openai_api_base="YOUR_SERVE_API_URL",
    type=OpenAISDK, # The model configuration types, commonly used such as OpenAISDK, TurboMindModelwithChatTemplate, HuggingFacewithChatTemplate
    path="Qwen/Qwen3-235B-A22B",
    temperature=0.6,
    meta_template=dict(
    round=[
        dict(role='HUMAN', api_role='HUMAN'),
        dict(role='BOT', api_role='BOT', generate=True),
    ],
),
    query_per_second=1,
    max_out_len=32000,
    max_seq_len=32768,
    batch_size=8,
    retry=10,
    extra_body={
        'chat_template_kwargs': {'enable_thinking': False},
    }, # Additional configurations of the model, such as the option in Qwen3 series to control whether they thinks or not
    pred_postprocessor=dict(type=extract_non_reasoning_content), # adding this pred_postprocessor can extract the non-reasoning content from models that output with a think tag
)
```
Repository: open-compass/opencompass
Issue: [Bug] 目前只有python3.10版本，3.11啥时候能支持
Root cause: Environment Incompatibility

### 先决条件

- [x] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [x] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

无

### 重现问题 - 代码/配置示例

无

### 重现问题 - 命令或脚本

无

### 重现问题 - 错误信息

无

### 其他信息

_No response_

Hi, 

I found that removing 'pyext' from `requirement/runtime.txt `makes it work with Python 3.11 or above.
Repository: open-compass/opencompass
Issue: [Feature] 可视化工具
Root cause: Unimplemented Feature Gap

### Describe the feature

请问，opencompass有结果文件可视化分析平台么？
类似evalScope的[Evalscope Dashboard]，

https://evalscope.readthedocs.io/zh-cn/v0.16.3/get_started/visualization.html#id7
Repository: open-compass/opencompass
Issue: 支持qwen3的测评吗
Root cause: Unimplemented Feature Gap

### Describe the feature

支持qwen3的测评吗 

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

You can refer to this [page](https://opencompass.readthedocs.io/en/latest/notes/academic.html)

好的，希望opencompass以后自动支持更多新模型
Repository: open-compass/opencompass
Issue: [Feature] MindSpore framework support for OpenCompass
Root cause: Unimplemented Feature Gap

### Describe the feature

# Motivation
Currently, PyTorch is the only supported framework for OpenCompass. We would like to propose supporting different frameworks such as MindSpore on Ascend NPU in addition to PyTorch for the following reasons:
- Each framework has its own user groups and ecosystems. Supporting MindSpore framework will make it easier for Ascend NPU users to utilize OpenCompass. For example, the evaluation of models natively trained on Ascend NPU + MindSpore framework will be seamlessly supported.
- As the AI framework with the most native support for Ascend NPU, MindSpore has carried out a lot of performance optimizations for inference scenarios, such as graph/op fusion and integrated compilation, as well as developer friendly Triton op support. Supporting MindSpore framework for OpenCompass will also bring better performance benefits to Ascend NPU users.


# Design
## Background
- OpenCompass currently supports multiple inference backend frameworks. In addition to the default HuggingFace Transformers, it also supports more efficient inference acceleration libraries such as LMDeploy and vLLM. When executing evaluation tasks, OpenCompass supports switching between different backend inference services via command-line parameters or inference acceleration service APIs.
- MindSpore framework currently supports multiple LLM serving frameworks, such as MindIE and vLLM. For vLLM, MindSpore provides adaptive compatibility via the vLLM-MindSpore Plugin, enabling efficient inference for models supported in the MindSpore Transformers library.
## Recommended high-level design
Considering that OpenCompass's existing architecture already has good support for vLLM, we propose supporting MindSpore framework by adding a vLLM-MindSpore backend，while maintaining full compatibility with the current architecture. After supporting the MindSpore framework, the overall invocation stack diagram of OpenCompass is as follows:
<img width="1179" height="345" alt="Image" src="https://github.com/user-attachments/assets/00e367bc-f1dd-41ef-816b-6e974a659aec" />

# RoadMap
We hope this work will be included in the H2 Roadmap of OpenCompass. We have the current tentative schedule:
- 25Q3：Complete the detailed design， finish the prototype verification based on the Qwen model, create a code branch for MindSpore.
- 25Q4：Full support of all functions, merge the code into the master branch, complete the documentation, officially release.

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

Thanks for your contribution, we would like to provide essential support.
Repository: open-compass/opencompass
Issue: [Feature] is there any way to evaluate an agent?
Root cause: Unimplemented Feature Gap

### 描述该功能

is there any way to evaluate an agent?
btw how to use transbench?

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

We are about to open-source an evaluation framework for assessing the AGENT capabilities of LLMs. Stay tuned for updates.
As for TransBench, regrettably, it is not currently on our development roadmap for support.
Repository: open-compass/opencompass
Issue: [Feature] 请问现在支持internlm/Intern-S1-mini的测评了吗
Root cause: Documentation Deficiency

### Describe the feature

internlm/Intern-S1-mini的测评

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

Please refer https://opencompass.readthedocs.io/en/latest/user_guides/interns1.html 

请问有没有不需要部署成API形似的办法进行测评
Repository: open-compass/opencompass
Issue: [Feature] Support pass@1 evaluation for multi predictions in MathEvaluator
Root cause: Unimplemented Feature Gap

### Describe the feature

When using a Hugging Face model with the parameter num_return_sequences set greater than 1, the output column “predictions” becomes a list instead of a string. As a result, the MathEvaluator always returns an accuracy of 0, regardless of whether the prediction is correct. It would be beneficial if the score function could handle list-type inputs and evaluate pass@1 using multiple predictions, similar to the approach mentioned in the DeepSeek-R1 technical report.

### Will you implement it?

- [x] I would like to implement this feature and create a PR!