zhiminy/EvalEng huggingface dataset:
Repository: open-compass/opencompass
Issue: [Feature] 请求添加BigCodeBench评测集
Root cause: Unimplemented Feature Gap

### 描述该功能

如题，添加BigCodeBench能方便评测代码在复杂问题的性能，其中包括full和hard子集，能够有complete和instruct两个版本
https://github.com/bigcode-project/bigcodebench

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

Thanks for the suggestions, we will add bigcodebench in this month.

> Thanks for the suggestions, we will add bigcodebench in this month.

How's it going now :)
Repository: open-compass/opencompass
Issue: [Bug] mmlupro 正则提取错误
Root cause: Algorithmic Error

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

None

### 重现问题 - 代码/配置示例

https://github.com/open-compass/opencompass/blob/main/configs/datasets/mmlu_pro/mmlu_pro_gen_cdbebf.py

### 重现问题 - 命令或脚本

None

### 重现问题 - 错误信息

正则提取错误导致得分不可信

### 其他信息

_No response_
Repository: open-compass/opencompass
Issue: [Feature] MMLU CEVAL MATH test results for some models

### Describe the feature

Is there any “**official results**” come out by OpenCompass?

I'd like to compare and double check.

I need results for Qwen2, Qwen2.5, GLM4, Llama3, Llama3.1

THANK YOU

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

Hi, thanks for your interest. We will provide the performance of the recent open-source models on the latest optimized prompts this week. Stay tuned.
cc @MaiziXiao
Repository: open-compass/opencompass
Issue: [Feature] Support Math23k
Root cause: Unimplemented Feature Gap

### Describe the feature

· https://github.com/SCNU203/Math23k
Math23K is a dataset consisting of 23,164 elementary school Chinese math application problems. These questions are collected from various online education websites. The problems in this dataset focus on solving a linear equation with one variable. In addition, the dataset includes 2,317 test samples for evaluation purposes.

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

Welcome to contribute this dataset into OpenCompass via a pull request. If you encounter any problems while supporting this feature, feel free to re-open if needed.
Repository: open-compass/opencompass
Issue: [Feature] dataset for humaneval-multipl
Root cause: Unimplemented Feature Gap

### Describe the feature

where could I find the data of humaneval-multipl?

No valid url for ./data/multi-data/humaneval_multipl-e/

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

Thanks for the report. We will support the automatic download for this dataset in next version.

any update?
Repository: open-compass/opencompass
Issue: [Feature] 请问在主观评测上是否支持Azure OpenAI API
Root cause: Unimplemented Feature Gap

### 描述该功能

我希望能够在主观评测中支持Azure OpenAI API

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

API类都是可以自己实现的，模仿OpenAI的那个类写一个相应的接口即可

Welcome to support Azure OpenAI API and contribute with a Pull Request. If you encounter any problems while supporting this feature, feel free to re-open if needed.
Repository: open-compass/opencompass
Issue: [Bug] DO NOT Use relative import
Root cause: Architectural Constraint

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

https://github.com/open-compass/opencompass/blob/main/opencompass/models/__init__.py

relative import caused wrong package import, you should use absolute import in such a complex project.

### 重现问题 - 代码/配置示例

```
from .accessory import LLaMA2AccessoryModel
```

### 重现问题 - 命令或脚本

```
from .accessory import LLaMA2AccessoryModel
```

### 重现问题 - 错误信息

relative import a outside module instead of opencompass.models.xxx

you should use absolute import

### 其他信息

change to 
```
from opencompass.models.accessory import LLaMA2AccessoryModel
```

https://google.github.io/styleguide/pyguide.html

> Do not use relative names in imports. Even if the module is in the same package, use the full package name. This helps prevent unintentionally importing a package twice.

Thanks for the suggestions. We will consider this improvement.
Repository: open-compass/opencompass
Issue: [Feature] 请问使用API评测如何支持自定义数据集？
Root cause: Documentation Deficiency

### Describe the feature

> command：
```
(opencompass) root@node1:~/user/OpenCompass/opencompass# python run.py configs/vllm-glm4-9b-chat-custom.py
```

> vllm-glm4-9b-chat-custom.py
```
from opencompass.models import OpenAISDK
from mmengine.config import read_base

with read_base():
    from .datasets.cmmlu.cmmlu_gen import cmmlu_datasets

datasets = []
datasets = cmmlu_datasets

api_meta_template = dict(
    round=[
        dict(role='HUMAN', api_role='HUMAN'),
        dict(role='BOT', api_role='BOT', generate=True),
    ],
    reserved_roles=[dict(role='SYSTEM', api_role='SYSTEM')],
)

models = [
    dict(
        abbr='glm-4-9b-chat-vllm-API',
        type=OpenAISDK,
        key='EMPTY',
        openai_api_base='http://localhost:port/v1', 
        path='glm-4-9b-chat',
        tokenizer_path='/root/user/models/glm-4-9b-chat',
        rpm_verbose=True, 
        meta_template=api_meta_template, 
        query_per_second=10, 
        max_out_len=1024,
        max_seq_len=4096,
        temperature=0.01,
        batch_size=8,
        retry=3,
    )
]
```
想在不更改OpenCompass代码的情况下实现，但是文档示例只有以下用法：
```
python run.py \
    --models hf_llama2_7b \
    --custom-dataset-path xxx/test_qa.jsonl \
    --custom-dataset-data-type qa \
    --custom-dataset-infer-method gen
```

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

You can change the models with the API models.

@Jimmy-L99 You can set your custom dataset configs in your `vllm-glm4-9b-chat-custom.py` file by appending to the `datasets` list like so:
```
datasets = [
    {"path": "xxx/test_mcq.csv", "data_type": "mcq", "infer_method": "ppl"},
    {"path": "xxx/test_qa.jsonl", "data_type": "qa", "infer_method": "gen"},
]
```

then run your task with the following command:
```
python run.py configs/vllm-glm4-9b-chat-custom.py
```
or
```
opencompass configs/vllm-glm4-9b-chat-custom.py
```

![Image](https://github.com/user-attachments/assets/00b04a0b-a1d2-46db-9134-8129a194075a)
使用这种方法也不行，报错。
Repository: open-compass/opencompass
Issue: 您好，请问L-Eval的主观题最终得分是使用rougeLsum这个分数吗？还有就是L-Eval数据集缺少了codeU和sci_fi有相关的评测配置文件么？
Root cause: Documentation Deficiency

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

您好，请问L-Eval的主观题最终得分是使用rougeLsum这个分数吗？还有就是L-Eval数据集缺少了codeU和sci_fi有相关的评测配置文件么？
![image](https://github.com/user-attachments/assets/b9b1fc8c-7800-4c8b-a2c4-86d5d69ea0a0)


### Reproduces the problem - code/configuration sample

您好，请问L-Eval的主观题最终得分是使用rougeLsum这个分数吗？还有就是L-Eval数据集缺少了codeU和sci_fi有相关的评测配置文件么？
![image](https://github.com/user-attachments/assets/62b48579-fbf9-4b19-964b-f46412df9f81)


### Reproduces the problem - command or script

您好，请问L-Eval的主观题最终得分是使用rougeLsum这个分数吗？还有就是L-Eval数据集缺少了codeU和sci_fi有相关的评测配置文件么？
![image](https://github.com/user-attachments/assets/d2231455-8656-4fe2-95af-317f6886e6ba)


### Reproduces the problem - error message

您好，请问L-Eval的主观题最终得分是使用rougeLsum这个分数吗？还有就是L-Eval数据集缺少了codeU和sci_fi有相关的评测配置文件么？
![image](https://github.com/user-attachments/assets/b6eccba2-8fc4-466f-bf49-bc07b8bb1e40)


### Other information

您好，请问L-Eval的主观题最终得分是使用rougeLsum这个分数吗？还有就是L-Eval数据集缺少了codeU和sci_fi有相关的评测配置文件么？
![image](https://github.com/user-attachments/assets/61c9655f-7a30-4ea8-9f54-8a3c18db3f22)
Repository: open-compass/opencompass
Issue: 为什么会有两个config路径分别为./config和./opencompass/config.好像第二个是起作用的为什么还要第一个？
Root cause: Architectural Constraint

### 描述该功能

为什么会有两个config路径分别为./config和./opencompass/config.好像第二个是起作用的为什么还要第一个？

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

opencompass/configs has been introduced in recent versions to facilitate the usage via `pip install opencompass`. We will remove the atomic files in configs in v0.4.0.
Repository: open-compass/opencompass
Issue: HumanEvalPlusEvaluator使用的evalplus版本是多少的
Root cause: Documentation Deficiency

### Describe the feature

请问一下opencompass0.3.0的HumanEvalPlusEvaluator使用的evalplus版本是多少的


### Will you implement it?

- [ ] I would like to implement this feature and create a PR!
Repository: open-compass/opencompass
Issue: [Feature] Evaluate multiple metrics simultaneously
Root cause: Unimplemented Feature Gap

### Describe the feature

Is it possible to evaluate multiple metrics at the same time, such as bleu and rouge?

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

You can modify the evaluator for the specific dataset and support multiple metrics simultaneously.

Thanks for your reply, but I think assembling multiple evaluators should be a basic ability.

> Thanks for your reply, but I think assembling multiple evaluators should be a basic ability.

@ezeli You can evaluate multiple metrics for multiple datasets by appending each dataset + evaluator_cfg combination to the dataset config list. Note that each evaluator must support the corresponding dataset in each pair for this to work.
Repository: open-compass/opencompass
Issue: [Bug] 推理完成了，但是因为插件版本，在做测评的时候报错了，但是预测的哪些文件都在，怎么重新继续做预测？还有就是运行的时候默认好像是多有的数据集都会测试
Root cause: External Dependency Breakage

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

'Python': '3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0]',
 'TorchVision': '0.16.0a0',
 'lmdeploy': "not installed:No module named 'lmdeploy'",
 'numpy_random_seed': 2147483648, 
 'opencompass': '0.3.7+',
 'sys.platform': 'linux',
 'transformers': '4.37.2'

### Reproduces the problem - code/configuration sample

from gradio_client import Client, handle_file

### Reproduces the problem - command or script

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python run.py ./configs/eval_llama3_instruct.py --datasets mmlu ifeval --hf-num-gpus 8 --model-kwargs torch_dtype=torch.bfloat16

### Reproduces the problem - error message

ImportError: cannot import name 'handle_file' from 'gradio_client'

### Other information

推理完成之后因为插件版本报错测评全是‘--’，但是所有推理的文件都在，怎么继续评测？

加一个--reuse latest就可以了
Repository: open-compass/opencompass
Issue: [Bug] mbpp 数据量少了，只有500条；谷歌论文中是974条《Program Synthesis with Large Language Models》
Root cause: Documentation Deficiency

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

1

### 重现问题 - 代码/配置示例

1

### 重现问题 - 命令或脚本

1

### 重现问题 - 错误信息

1

### 其他信息

1

OC仅采用 MBPP的 test 数据集共 500 条
https://huggingface.co/datasets/google-research-datasets/mbpp/viewer/full/test
Repository: open-compass/opencompass
Issue: [Feature] 是否打算支持mamba或者mamba2的测评吗

### Describe the feature

是否打算支持mamba或者mamba2的测评吗

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

You can deploy it as an Openai service and use Openai class to evaluate it

Also we support the Huggingface model, welcome to contribute this model into OpenCompass

@MaiziXiao 有打算支持Mamba2吗，因为网络的原因我只能在本地调用。
Repository: open-compass/opencompass
Issue: [Bug] gsm8k_gen_17d0dc-BOT思考提示语错误
Root cause: Validation Gap

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

python run.py conf/eval.py



### 重现问题 - 代码/配置示例

#conf/eval.py

from mmengine.config import read_base

with read_base():
    from .datasets.gsm8k_base_gen import gsm8k_datasets


#conf/datasets/gsm8k_base_gen
from mmengine import read_base

with read_base():
    from opencompass.configs.datasets.gsm8k.gsm8k_gen_17d0dc import gsm8k_datasets

### 重现问题 - 命令或脚本

gsm8k_gen_17d0dc 的line15的BOT思考有错
His team also scores 6 3 pointers, meaning they scored 8*3= 24 points in 3 pointers\n
应该是
His team also scores 8 3 pointers, meaning they scored 8*3= 24 points in 3 pointers\n

### 重现问题 - 错误信息

4 shots 提示 有错误

### 其他信息

可能是编写错误

Thanks, we will fix this error.

@liushz  Please check this error and update the configuration.
Repository: open-compass/opencompass
Issue: 您好，请问OpenCompass最新公开学术榜单中，各个评测数据集分别是使用哪些配置文件的，还有数据集下载地址在哪里？特别是LiveCodeBench评测是使用哪个时间段数据集评测的？
Root cause: Documentation Deficiency

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

您好，请问OpenCompass最新公开学术榜单中，各个评测数据集分别是使用哪些配置文件的，还有数据集下载地址在哪里？特别是LiveCodeBench评测是使用哪个时间段数据集评测的？
![image](https://github.com/user-attachments/assets/1b7d2663-1e50-4eb0-8690-d06ae79fad4f)



### Reproduces the problem - code/configuration sample

您好，请问OpenCompass最新公开学术榜单中，各个评测数据集分别是使用哪些配置文件的，还有数据集下载地址在哪里？特别是LiveCodeBench评测是使用哪个时间段数据集评测的？
![image](https://github.com/user-attachments/assets/7b38d0a5-1d64-4dfa-9bdd-29b889edf934)


### Reproduces the problem - command or script

您好，请问OpenCompass最新公开学术榜单中，各个评测数据集分别是使用哪些配置文件的，还有数据集下载地址在哪里？特别是LiveCodeBench评测是使用哪个时间段数据集评测的？
![image](https://github.com/user-attachments/assets/efd49c2a-5b39-4529-ac59-9648bd15da70)


### Reproduces the problem - error message

您好，请问OpenCompass最新公开学术榜单中，各个评测数据集分别是使用哪些配置文件的，还有数据集下载地址在哪里？特别是LiveCodeBench评测是使用哪个时间段数据集评测的？
![image](https://github.com/user-attachments/assets/762376fb-0839-4cb4-87f7-8c5561a64084)


### Other information

您好，请问OpenCompass最新公开学术榜单中，各个评测数据集分别是使用哪些配置文件的，还有数据集下载地址在哪里？特别是LiveCodeBench评测是使用哪个时间段数据集评测的？
![image](https://github.com/user-attachments/assets/e7b75112-67f8-4d1c-9100-02ffec4a84a2)


https://github.com/open-compass/opencompass/blob/main/configs/eval_academic_leaderboard_202412.py

LiveCodeBench is V1(Code Generation). We will update it with V4 in the next version.

> LiveCodeBench is V1(Code Generation). We will update it with V4 in the next version.

Thanks a lot.

> https://github.com/open-compass/opencompass/blob/main/configs/eval_academic_leaderboard_202412.py

![image](https://github.com/user-attachments/assets/3f2d7270-6825-46e2-bb3c-bc816afb995d)
these files can't find in the source code.

You need install opencompass in source code mode, via `pip install -e.`.

> You need install opencompass in source code mode, via `pip install -e.`.

aime2024数据集.zip压缩包下载后打开里面的aime.jsonl文件，只有30道题，请问有没有下载错？

> > You need install opencompass in source code mode, via `pip install -e.`.
> 
> aime2024数据集.zip压缩包下载后打开里面的aime.jsonl文件，只有30道题，请问有没有下载错？

请问这个aime2024.zip在哪里下载？能否请教一下

> aime2024

应该没有。 https://huggingface.co/datasets/Maxwell-Jia/AIME_2024 也是30条
Repository: open-compass/opencompass
Issue: [Bug] FileNotFoundError: [Errno 2] No such file or directory: './data/math/test_prm800k_500.json'
Root cause: Environment Incompatibility

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

from ..opencompass.configs.datasets.math.math_prm800k_500_0shot_cot_gen import (
        math_datasets,
    )

### 重现问题 - 代码/配置示例

from ..opencompass.configs.datasets.math.math_prm800k_500_0shot_cot_gen import (
        math_datasets,
    )

### 重现问题 - 命令或脚本

python -u run.py configs/eval_academic_leaderboard_202412_lz.py

### 重现问题 - 错误信息

FileNotFoundError: [Errno 2] No such file or directory: './data/math/test_prm800k_500.json'

### 其他信息

'./data/math/test_prm800k_500.json'这个数据集似乎不在OpenCompassData-complete-20240207.zip里面，请问可以提供最新的data吗？

Currently, opencompass will  download the dataset automatically. Or you can find the download link in: https://github.com/open-compass/opencompass/blob/main/opencompass/utils/datasets_info.py

Feel free to re-open if needed.

遇到同样的问题，测试的时候只遇到了can't find，没有自动下载

添加export DATASET_SOURCE=ModelScope 后会自动下载

> 添加export DATASET_SOURCE=ModelScope 后会自动下载

The downloaded dataset is [math.json](https://modelscope.cn/datasets/opencompass/math/file/view/master/math.jsonl?id=26227&status=1), which includes 5000 whole datasets.
It is NOT MATH-500 dataset.

@tonysy Please re-open this issue.

I believe the issue is that the `math500` dataset is not included in the `data/math` directory of the pre-downloaded dataset recommended in the README:

```bash
wget https://github.com/open-compass/opencompass/releases/download/0.2.2.rc1/OpenCompassData-core-20240207.zip
unzip OpenCompassData-core-20240207.zip
```

I was able to resolve this by deleting the local `./data/math` directory and re-running the evaluation command. This action triggered an automatic download of the latest `math` dataset, which correctly includes `math500`.

@tonysy, it would be helpful if `opencompass` could update the `OpenCompassData-core-20240207.zip` file with the latest dataset to avoid this issue for new users.
Repository: open-compass/opencompass
Issue: [Feature] 如何添加评价指标
Root cause: Documentation Deficiency

### Describe the feature

评价指标是否能增加推理速度、吞吐量等指标，或者能否出一个教程演示如何增加自定义指标？感谢

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

推理速度、吞吐量是推理引擎的指标，和compass聚焦的评估数据集是不一样的任务，我认为没有必要。

目前自定义指标可以继承已有的准确率或者BLEU/ROUGE等实现来自定义，但是这个实现只能对推理结果和ground truth作比较，不涉及推理性能的指标

> 推理速度、吞吐量是推理引擎的指标，和compass聚焦的评估数据集是不一样的任务，我认为没有必要。
> 
> 目前自定义指标可以继承已有的准确率或者BLEU/ROUGE等实现来自定义，但是这个实现只能对推理结果和ground truth作比较，不涉及推理性能的指标

好的，感谢~
Repository: open-compass/opencompass
Issue: [Feature] 支持基于ray的多卡多机评测
Root cause: Unimplemented Feature Gap

### Describe the feature

- 在一些平台上slurm不便于使用
- 基于ray能方便的组集群进行大规模大模型的测评

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

We currently support VOLC for ByteDance cluster, DLC for Aliyun PAI cluster. You can implement a new runner for Ray. Also welcome to contribute your implementation into OpenCompass.

楼主请问现在有新的进展吗

> 楼主请问现在有新的进展吗

可以用ray部署服务，然后调API的方式来实现这个功能
Repository: open-compass/opencompass
Issue: [Bug] 缺乏game24以及其他数据集
Root cause: Unimplemented Feature Gap

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

python run.py \
    --datasets korbench_mixed_gen_d00bdd bbh_gen TheoremQA_5shot_gen_6f0af8 NPHardEval_gen hellaswag_gen \

### 重现问题 - 代码/配置示例

python run.py \
    --datasets korbench_mixed_gen_d00bdd bbh_gen TheoremQA_5shot_gen_6f0af8 NPHardEval_gen hellaswag_gen \

### 重现问题 - 命令或脚本

python run.py \
    --datasets korbench_mixed_gen_d00bdd bbh_gen TheoremQA_5shot_gen_6f0af8 NPHardEval_gen hellaswag_gen \

### 重现问题 - 错误信息

Please make sure  `./data/NPHardEval/GCP/` is correct

lack of some datasets like game24 and nphard

### 其他信息

缺乏一些数据集

Thanks for the report. We are working on the dataset validity check and will update the dataset recently.

> Thanks for the report. We are working on the dataset validity check and will update the dataset recently.

@tonysy @liushz @nuochenpku @changlan @hzhwcmhf OpenFinData和FinanceIQ这两个数据集目前也出现了上述问题，问题应该在于`opencompass/utils/datasets_info.py`，里面缺失一些数据集的下载方式，请尽快上传缺失数据集至`http://opencompass.oss-cn-shanghai.aliyuncs.com/datasets/data/`并更新代码。谢谢你们的贡献！

> Thanks for the report. We are working on the dataset validity check and will update the dataset recently.

hi, any update? i need evaluate financeIQ and don't know how to download it. or I can just download from HF but is it compatible with opencompass?

Lots of datasets are missing. I met the same problems in 1) FinanceIQ, 2) LawBench, 3) MedBench/DrugCA, 4) AGIEval/data/v1/

Could you please fix it?

Lots of datasets are missing. I met the same problems in 1) FinanceIQ, 2) LawBench, 3) MedBench/DrugCA, 4) AGIEval/data/v1/

Could you please fix it?
Repository: open-compass/opencompass
Issue: [Feature] 请问是否能有一个汇总，就是opencompass包含的所有评测数据集、这些数据集的metrics和链接(paper/github)？
Root cause: Documentation Deficiency

### 描述该功能

建议同上，对于我这种刚开始接触的外行来说比较友好。

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

Thanks for your suggestion. We have an internal version for the collection of the dataset and paper. We hope to release it soon in the documentation.

I appreciate a lot！
Repository: open-compass/opencompass
Issue: Miss using GPT3 in GPT4.py?
Root cause: Configuration Error

Maybe miss using gpt3.5 in the gpt4 setting.

https://github.com/open-compass/opencompass/blob/ffdc91752354ce688c327af0abd0bd9c9e6a31c5/opencompass/configs/models/openai/gpt_4.py#L12
Repository: open-compass/opencompass
Issue: [Feature] 数据集ppl及gen使用场景建议
Root cause: Documentation Deficiency

### Describe the feature

请问能否添加一个解释各数据集(较为常用的)在基座和chat模型上的使用场景？
例如，在官网只看到了chat_OC15常用来评测对话模型
请问基座模型一般使用哪些数据集呢？以及基座模型在什么数据集应该使用ppl，什么数据集应该使用gen呢？
如果能够在主页添加此类解释对新手会更为友好，十分感谢！

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

PPL are mainly used for the base model, which is typically hard to follow the instruction. Gen is compatible with base model and instruct model

> PPL are mainly used for the base model, which is typically hard to follow the instruction. Gen is compatible with base model and instruct model

thks for reply but, one thing I'm not sure about is that the manual on the official website says that the base model uses ppl for MCQ tasks and gen mode for everything else, but I opened the configs in the github project to find the settings for the datasets and found that some of them have both gen and ppl, so I'm not sure which config to use for the base model is the most appropriate one.
Repository: open-compass/opencompass
Issue: [Feature] 请问是否支持用megatron-deepspeed 框架预训练大模型？

### Describe the feature

请问是否支持用megatron-deepspeed 框架预训练大模型？

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

Our main focus is HF/vLLM/LMDeploy and OpenAISDK. Feel free to add your model implementation in `models` for new model configuration.
Repository: open-compass/opencompass
Issue: [Bug] Need Help! FileNotFoundError: [Errno 2] No such file or directory: './data/math/test_prm800k_500.json' #1797
Root cause: Environment Incompatibility

> 遇到同样的问题，测试的时候只遇到了can't find，没有自动下载 

 _Originally posted by @linbeyoung in [#1797](https://github.com/open-compass/opencompass/issues/1797#issuecomment-2608761956)_

把这个路径（/root/.cache/opencompass/data/math）下的 cache 删除，重新 run 一下

Feel free to re-open if needed.

另外现有的规则评测DeepSeek-R1和其他o1类模型，会因为他们的思维过程太长而无法匹配出正确答案，这块有计划更新适配o1的评测规则吗？

~/.cache/opencompass/data 下并没有math这个文件夹，创建后也不行，或者有一个下载链接吗我下载到本地移动到./data/math里也行

辛苦看到的话可以再回复一下

same issue
@MaiziXiao 

Please solve this problem instead of simply closing it. Math500 is a very commonly used evaluation dataset.