zhiminy/EvalEng huggingface dataset:
Repository: open-compass/opencompass
Issue: [Bug] which version of the dataset should be selected When evaluating the Llama3 model, 
Root cause: Configuration Error

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

OpenCompass v0.2.5
python 3.10
Transformers 4.41.0
Troh 2.1.2

### Reproduces the problem - code/configuration sample

Original Llama3 configuration, only path was updated locally.

### Reproduces the problem - command or script

Llama3-8b base
python run.py --models hf_llama3_8b --datasets triviaqa_wiki_gen_d18bf4 squad20_gen_1710bc agieval_mixed_2f14ad
Llam3-8b-instruct
python run.py --models hf_llama3_8b_instruct --datasets humaneval_gen_a82cae

### Reproduces the problem - error message

There is a big gap between using tools and official websites

Llama3-8b base:
  OpenCompass:
  dataset,version,metric,mode,llama-3-8b-hf
  triviaqa_wiki_0shot,de8113,score,gen,30.95
  triviaqa_wiki_1shot,eaf81e,score,gen,63.31
  **triviaqa_wiki_5shot,e0f87f,score,gen,54.61**
  triviaqa_wiki_25shot,c60113,score,gen,57.75
  squad2.0,c6a034,score,gen,37.03
  Official website:
  ![image](https://github.com/open-compass/opencompass/assets/46083912/24d18ac1-cf3b-4e56-a0f3-9107dbfb1133)
  

Llam3-8b-instruct:
  OpenCompass
  openai_humaneval,8e312c,humaneval_pass@1,gen,54.88
  Official website:
  ![image](https://github.com/open-compass/opencompass/assets/46083912/64f0c0bb-8afd-4db3-8018-27d8bca6eda6)

And the official website contrast has the very big disparity, 2.0 datasets not released I'm not sure if I should choose these datasets or what configuration errors are causing this gap
  

### Other information

_No response_
Repository: open-compass/opencompass
Issue: opencompass公开榜单更新[Feature] 

### 描述该功能

之前提到说上周会对opencompass的公开评测集榜单进行更新，请问现在有更新吗，能否发下链接。想参考一下评测结果

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

已部分更新，请参考 https://github.com/open-compass/opencompass/pull/1206

有具体的榜单吗，例如https://rank.opencompass.org.cn/leaderboard-llm/?m=24-01这个
Repository: open-compass/opencompass
Issue: [Bug] 不支持python3.10以上安装
Root cause: External Dependency Breakage

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

python 3.11

### 重现问题 - 代码/配置示例

.

### 重现问题 - 命令或脚本

pip install opencompass

### 重现问题 - 错误信息

Collecting pyext (from opencompass)
  Using cached pyext-0.7.tar.gz (7.8 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [9 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/data/ruanjh/tmp/pip-install-oxjvkmov/pyext_7fd78ecb368043fd8e5011c0ea165836/setup.py", line 6, in <module>
          import pyext
        File "/data/ruanjh/tmp/pip-install-oxjvkmov/pyext_7fd78ecb368043fd8e5011c0ea165836/pyext.py", line 117, in <module>
          oargspec = inspect.getargspec
                     ^^^^^^^^^^^^^^^^^^
      AttributeError: module 'inspect' has no attribute 'getargspec'. Did you mean: 'getargs'?
      [end of output]

### 其他信息

如何在python3.10+的版本上使用opencompass？

我也遇到了相同的问题

我也遇到了这个问题， 还没有人回复呀

3.12也是

参考：https://github.com/refi64/PyExt/pull/3

可以参考https://github.com/open-compass/opencompass/pull/1985

如果后续还有人碰到这个问题，可以先通过@ShaohonChen 提交的PR修改安装pyExt，不清楚opencompass为什么会依赖一个如此古早的package，项目的依赖管理可能还有调整空间。
```bash
pip install git+https://github.com/refi64/PyExt.git@a95f488490fc57ec17d0c00a99c6bc0a4726824f
```

> 如果后续还有人碰到这个问题，可以先通过[@ShaohonChen](https://github.com/ShaohonChen) 提交的PR修改安装pyExt，不清楚opencompass为什么会依赖一个如此古早的package，项目的依赖管理可能还有调整空间。
> 
> pip install git+https://github.com/refi64/PyExt.git@a95f488490fc57ec17d0c00a99c6bc0a4726824f

有用
Repository: open-compass/opencompass
Issue: [Feature] Difficulty in Evaluating Custom Models with OpenCompass
Root cause: Unimplemented Feature Gap

### Describe the feature

Dear OpenCompass Team,

I've encountered a challenge with OpenCompass when trying to evaluate a custom model that I developed. Currently, it seems that any action I want to perform with OpenCompass must be executed within the OpenCompass repository itself, which can be quite difficult.

My main concern is that OpenCompass is not designed to facilitate executions in external repositories. Typically, model development is done within personal repositories, and it would be impractical to rewrite the model in the OpenCompass repository.

I would like to suggest considering an approach similar to that of fairseq, which uses a registration mechanism to manage models and datasets. This could make it easier for users to work with their custom models. Besides, instead of running an evaluation command inside OpenCompass like 
```
python run.py --dataset ... --models ...
```
it would be more user-friendly to execute a command within the user's own repository, such as 
```
opencompass --dataset ... --models ....
```

This change could significantly improve the user experience by streamlining the process of model evaluation.

Thank you for considering this suggestion. I am looking forward to your response and any potential updates regarding this matter.

Best regards,
Jiadi Jiang

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

Thanks for the suggesiton, we will provide cli command in next version
Repository: open-compass/opencompass
Issue: [Bug] 大佬们，这个函数好像写的有问题，只能解析出来[BEGIN]到[DONE]中间的代码，然而基座模型最先输出的代码不是以[BEGIN]开头的。
Root cause: Validation Gap

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

如题

### Reproduces the problem - code/configuration sample

如题

### Reproduces the problem - command or script

如题

### Reproduces the problem - error message

如题

### Other information

如题

Which function do you mean?
Repository: open-compass/opencompass
Issue: [Bug] llama3 8b 基座模型在ARC-C PPL数据集上的评估，accuracy只有41，不正常
Root cause: Algorithmic Error

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

python run.py --models hf_llama3_8b --datasets ARC_c_ppl

### 重现问题 - 代码/配置示例

无

### 重现问题 - 命令或脚本

无

### 重现问题 - 错误信息

无

### 其他信息

llama3 8b的基座模型，在ARC-C GEN数据集上表现正常。不知道为啥
Repository: open-compass/opencompass
Issue: [Bug] Find `scikit-learn` version conflict in `requirements/runtime.txt` and `requirements/extra.txt`
Root cause: Configuration Error

In `requirements/runtime.txt`, it is necessary as `scikit_learn==1.2.1`
https://github.com/open-compass/opencompass/blob/main/requirements/runtime.txt#L37

But in `requirements/extra.txt`,it is set as `scikit-learn==1.5`
https://github.com/open-compass/opencompass/blob/main/requirements/extra.txt#L3C1-L3C18

which will cause conflict

Thanks for the report. We will fix this conflict in next version.
Repository: open-compass/opencompass
Issue: [Feature] 没有找到“subjective”中的compassarena、compassbench、creationbench等数据
Root cause: Documentation Deficiency

### 描述该功能

请问上述数据去哪里找？能否提供链接或者发布相应数据。期待您的回答。

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

These datasets are internal datasets, thanks for the attention.
Repository: open-compass/opencompass
Issue: [Feature] 为啥我开始评测一直卡在这里
Root cause: Documentation Deficiency

### Describe the feature

![image](https://github.com/open-compass/opencompass/assets/34667896/6e9003f9-1b4c-4332-9487-ef206f83e637)


### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

You can use --debug to check the error details or check the log in `ouputps` folder
Repository: open-compass/opencompass
Issue: [Bug] 在eval_qwen_7b这个base模型评估中，为何出现leaderboard.qwen中的gen的数据集，基础模型不应该是ppl的评估方式吗
Root cause: Documentation Deficiency

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

无

### Reproduces the problem - code/configuration sample

无

### Reproduces the problem - command or script

无

### Reproduces the problem - error message

无

### Other information

如题。

Please refere https://github.com/open-compass/opencompass/blob/main/configs/models/qwen/README.md. The leaderboard on our website will be updated in July.

好的，谢谢。

@tonysy 这个链接里边：https://github.com/open-compass/opencompass/blob/main/configs/models/qwen/README.md 
的Base Models中，也是存在gen测试集的，比如：
<img width="883" alt="11111111" src="https://github.com/open-compass/opencompass/assets/27861805/7fdd9893-dff7-43c6-90e6-ea3b5803a1bd">
base model的评估不是应该都是ppl的评估方式吗，麻烦帮忙解答一下
Repository: open-compass/opencompass
Issue: 有人配置过mmlu_pro数据集么？求分享代码~
Root cause: Validation Gap

### 描述该功能

求配置mmlu_pro数据集的代码逻辑~

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

最新的代码好像支持了，但是运行起来报错，不知道有没有人试了

> 最新的代码好像支持了，但是运行起来报错，不知道有没有人试了

opencompass 整理过后的 mmlu_pro数据集哪里下载？

> > 最新的代码好像支持了，但是运行起来报错，不知道有没有人试了
> 
> opencompass 整理过后的 mmlu_pro数据集哪里下载？

TIGER-Lab/MMLU-Pro Huggingface下载的，opencompass的也是直接调用load_datasets下载huggingface数据集

I attempted to test the locally deployed Deepseek-r1-AWQ model using the MMLU-pro dataset with the following code:

/opencompass/opencompass/configs/models/deepseek/deepseek_awq.py
from opencompass.models import DeepseekAPI

api_meta_template = dict(round=[
    dict(role='HUMAN', api_role='HUMAN'),
    dict(role='BOT', api_role='BOT', generate=True),
], )

models = [
    dict(
        abbr='Deepseek-R1',
        type=DeepseekAPI,
        path='deepseek-reasoner',
        key=
        '123456789',  # The key will be obtained from $OPENAI_API_KEY, but you can write down your key here as well
        url='http://0.0.0.0:8052/v1/chat/completions',
        meta_template=api_meta_template,
        query_per_second=1,
        max_out_len=8192,
        max_seq_len=8192,
        batch_size=4),
]

/opencompass/opencompass/configs/671b_q4.py
from mmengine.config import read_base

with read_base():
    from opencompass.configs.datasets.mmlu_pro.mmlu_pro_0shot_cot_gen_08c1de import \
        mmlu_pro_datasets
    from opencompass.configs.models.deepseek.deepseek_awq import \
        models as deepseek_R1_671b_q4_instruct_model
    from opencompass.configs.summarizers.mmlu_pro import summarizer

datasets = sum([
    v
    for k, v in locals().items() if k.endswith('_datasets') or k == 'datasets'
], [])
models = sum([v for k, v in locals().items() if k.endswith('_model')], [])

work_dir = 'outputs/debug/671b_q4_mmlu_pro'

Startup command is:
python run.py ./opencompass/configs/671b_q4.py --debug^C

When the program first starts running, all GPUs are operating normally with utilization around 95%, and new request tasks can be observed in the VLLM logs. However, after running for some time, the program appears to pause unexpectedly - all GPU utilization drops to 0% and no new request tasks are recorded in the VLLM logs. This is quite strange. I tried setting retry=10 but it didn't resolve the issue.
Repository: open-compass/opencompass
Issue: [Feature] whats the difference between mbpp and deprecated_mbpp ?

### Describe the feature

![image](https://github.com/open-compass/opencompass/assets/97608046/fff64ad1-66ae-4d5f-887a-eb15516bf4e2)
Could you please clarify the difference between mbpp and deprecated_mbpp? I couldn't find a detailed description in the documentation or community resources.

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

The config with `deprecated` means the config is not being used and will be removed soon.

Feel free to re-open if needed.
Repository: open-compass/opencompass
Issue: phi3模型评测[Feature] 
Root cause: Documentation Deficiency

### 描述该功能

想请问下，opencompass有对phi3模型做评测吗

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

Actually we support eval phi3-models in opencompass 

see here https://github.com/open-compass/opencompass/tree/main/configs/models/phi
feel free to reopen it if needed
Repository: open-compass/opencompass
Issue: Regarding the description of ppl and gen usage scenarios on the Open website[Feature] 

### Describe the feature

对于两种评测模式ppl和gen，官网是这样描述的：“从使用上来说，基座模型的单项选择题和部分具有选择题性质的题目会使用 ppl，基座模型的不定项选择和非选择题都会使用 gen。而对话模型的所有题目都会使用 gen，因为许多商用 API 模型不会暴露 ppl 的接口。但也存在例外情况，例如我们希望基座模型输出解题思路过程时 (例如 Let’s think step by step)，我们同样会使用 gen，但总体的使用如下图所示”，这段文字描述和下面给出的表格以及实际的代码是矛盾的。在表格和代码中，多项选择题是用ppl来评估，单项选择
<img width="682" alt="截屏2024-07-05 11 14 33" src="https://github.com/open-compass/opencompass/assets/34769427/ba6a760e-5d13-45a5-a902-37ace4fe1aa6">



### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

另外，我还想请教下ppl和gen方式下进行对比的具体代码和逻辑在哪里可以看到？我对此很感兴趣，谢谢

It only means we use ppl for base model, gen mode is also valid for base model, but it relys on the few-shot example(w/ CoT) and advanced answer extraction. To avoid the complex anwer extraction(post-process), we use ppl as default mode for base models. 

The implementation is in `opencompass/models` , feel free to re-open if needed.
Repository: open-compass/opencompass
Issue: 您好，请问T-Eval的结果中这里的“反思能力”是指T-Eval数据集的中Review这个数据集吗？

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

1

### Reproduces the problem - code/configuration sample

1

### Reproduces the problem - command or script

1

### Reproduces the problem - error message

1

### Other information

您好，请问T-Eval的结果中这里的“反思能力”是指T-Eval数据集的中Review这个数据集吗？
![image](https://github.com/open-compass/opencompass/assets/20581199/d71bd18d-6bb7-4354-b6a5-b12d97774de8)


是的

> 是的

明白，多谢！
Repository: open-compass/opencompass
Issue: 您好，请问一下OpenCompass大语言模型月度榜单是使用评测数据集中dev数据集测试的还是使用test数据集测试的？
Root cause: Documentation Deficiency

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

1

### Reproduces the problem - code/configuration sample

1

### Reproduces the problem - command or script

1

### Reproduces the problem - error message

1

### Other information

您好，请问一下OpenCompass大语言模型月度榜单是使用评测数据集中dev数据集测试的还是使用test数据集测试的？
![image](https://github.com/open-compass/opencompass/assets/20581199/6f257ba7-2c93-48d7-9f98-9feba83bb8eb)


闭源Test集

Feel free to re-open if needed.

> 闭源Test集

多谢。
Repository: open-compass/opencompass
Issue: [Bug] 不支持windows系统运行吗
Root cause: Environment Incompatibility

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

CUDA_VISIBLE_DEVICES=0 python run.py --models hf_qwen1_5_7b --datasets cmmlu_ppl --hf-num-gpus 1 --debug

### 重现问题 - 代码/配置示例

CUDA_VISIBLE_DEVICES=0 python run.py --models hf_qwen1_5_7b --datasets cmmlu_ppl --hf-num-gpus 1 --debug

### 重现问题 - 命令或脚本

CUDA_VISIBLE_DEVICES=0 python run.py --models hf_qwen1_5_7b --datasets cmmlu_ppl --hf-num-gpus 1 --debug

### 重现问题 - 错误信息

signal.SIGALRM is not available on this platform
signal.SIGALRM is not available on this platform
07/10 17:09:43 - OpenCompass - ERROR - D:\share_vmware\my_work\ai_station_project\opencompass\opencompass\tasks\openicl_eval.py - _score - 243 - Task [qwen1.5-7b-hf/cmmlu-chinese_civil_service_exam]: No predictions found.
07/10 17:09:43 - OpenCompass - INFO - time elapsed: 2.86s
07/10 17:09:44 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/10 17:09:44 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval


### 其他信息

signal.SIGALRM is not available on this platform
signal.SIGALRM is not available on this platform
07/10 17:09:43 - OpenCompass - ERROR - D:\share_vmware\my_work\ai_station_project\opencompass\opencompass\tasks\openicl_eval.py - _score - 243 - Task [qwen1.5-7b-hf/cmmlu-chinese_civil_service_exam]: No predictions found.
07/10 17:09:43 - OpenCompass - INFO - time elapsed: 2.86s
07/10 17:09:44 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/10 17:09:44 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval


目前OpenCompass还没有适配Windows
Repository: open-compass/opencompass
Issue: 问题咨询：关于ceval和nq这两类数据是如何计算分数的？
Root cause: Documentation Deficiency

### 描述该错误

ceval是单选题，每个科目题目数目不一样，模型推测出结果之后，如果做对一题就加分吗？然后再归一化为百分制？
nq之类问答题，是如何计算分数的呢？

在discussion区提问，响应比较慢，提到issue区试试。多谢。

### 环境信息

python3.10

### 其他信息

_No response_

1. C-Eval has one of `A` / `B` / `C` / `D` as the reference answer. If the model prediction after the postprocess (such as 'take the first capital letter') is the same as the reference answer, then this question is considered correct, wrong otherwise. Each subset will get an accuracy with a total of 100. The final score is a naive average over all subsets, each has a weight 1.
2. Dataset such as NaturalQuestions uses Exact-Match. Each question in NaturalQuestions has a list of str as reference answer. If the model prediction is exactly same as any one of the references, then this question is considered correct. The final score is an accuracy with a total of 100.

You are also encouraged to explore the dataset config itself.

https://github.com/InternLM/opencompass/blob/262ab794fb52084e4494c281210e652706ce1280/configs/datasets/nq/nq_gen_2463e2.py#L17
Repository: open-compass/opencompass
Issue: [Feature] MBPP数据集的CN版本 mbpp_cn 能够release一下吗 ORZ，感谢大佬
Root cause: Unimplemented Feature Gap

### Describe the feature

数据集：
mbpp_cn

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

同求
Repository: open-compass/opencompass
Issue: 评测结果的稳定性如何保证？ 
Root cause: Configuration Error

### Describe the feature

大模型答案的生成和温度等参数有关？ 所以，opencompass的榜单模型，是保证每次生成的答案都是固定的吗？ 

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

Primarily, our decoding phase employs greedy search, and fixed generation parameters (like temperature and so on) for every model ensuring theoretical consistency in the responses generated. However, there are nuances to consider that can subtly alter outcomes. For instance, the presence or absence of inference accelerators such as VLLM or LMDeploy can introduce slight variations in the results due to their impact on the processing efficiency and potential optimizations they offer.

> Primarily, our decoding phase employs greedy search, and fixed generation parameters (like temperature and so on) for every model ensuring theoretical consistency in the responses generated. However, there are nuances to consider that can subtly alter outcomes. For instance, the presence or absence of inference accelerators such as VLLM or LMDeploy can introduce slight variations in the results due to their impact on the processing efficiency and potential optimizations they offer.

我通过调用api服务测试，发现每次测试的结果分数都不一样，本地模型路径的方式加载测试是一样的，调用api测试是哪里设置不对吗，temperature=0，top_k=-1,top_p=1,可以查看下面issue，谢谢大佬
https://github.com/open-compass/opencompass/issues/1634
Repository: open-compass/opencompass
Issue: [Feature] 是否支持断点继续的功能以避免重复测评？
Root cause: Documentation Deficiency

### 描述该功能

![image](https://github.com/user-attachments/assets/fd3a349c-dbb5-4fd1-be17-ae104e637df5)
正如lm-eval所支持的那样，希望opencompass能够支持类似功能

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

您好，是支持此功能的，可以参考文档的[这里](https://opencompass.readthedocs.io/zh-cn/latest/user_guides/experimentation.html#:~:text=%2Dr%3A%20%E9%87%8D%E7%94%A8%E5%B7%B2%E6%9C%89%E7%9A%84%E6%8E%A8%E7%90%86%E7%BB%93%E6%9E%9C%E3%80%82%E5%A6%82%E6%9E%9C%E5%90%8E%E9%9D%A2%E8%B7%9F%E6%9C%89%E6%97%B6%E9%97%B4%E6%88%B3%EF%BC%8C%E5%88%99%E4%BC%9A%E5%A4%8D%E7%94%A8%E5%B7%A5%E4%BD%9C%E8%B7%AF%E5%BE%84%E4%B8%8B%E8%AF%A5%E6%97%B6%E9%97%B4%E6%88%B3%E7%9A%84%E7%BB%93%E6%9E%9C%EF%BC%9B%E5%90%A6%E5%88%99%E5%88%99%E5%A4%8D%E7%94%A8%E6%8C%87%E5%AE%9A%E5%B7%A5%E4%BD%9C%E8%B7%AF%E5%BE%84%E4%B8%8B%E7%9A%84%E6%9C%80%E6%96%B0%E7%BB%93%E6%9E%9C%E3%80%82)

Feel free to re-open if needed.
Repository: open-compass/opencompass
Issue: [dataset] Request for compassbench_v1.1.public dataset
Root cause: Unimplemented Feature Gap

### 描述该功能

In version 0.2.6 of OC, the **compassbench_20_v1_1_public** dataset configuration was added. May I ask where I can obtain the corresponding **compassbench_v1.1.public** dataset?

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

Thanks for your attention, we will release the CompassBench v1.2 Public recently(this week or next week), stay tuned.
Repository: open-compass/opencompass
Issue: 请教下，如何设置完成多级多卡间并行的测试相同的模型的不同checkpoint呢？
Root cause: Documentation Deficiency

### Describe the feature

无

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

Just consider these checkpoint as different models.

```python
models = [
    dict(abbr='xxx-step-1000', path='xxx-step-1000.pth', ...),
    dict(abbr='xxx-step-2000', path='xxx-step-2000.pth', ...),
    dict(abbr='xxx-step-3000', path='xxx-step-3000.pth', ...),
    ...
]
```

Then run the config in a multi-machine multi-gpu way, please refer to [quick start](https://opencompass.readthedocs.io/en/latest/get_started.html#quick-start)