zhiminy/EvalEng huggingface dataset:
Repository: open-compass/opencompass
Issue: [Feature] Add Math Dataset as in "Let’s Verify Step by Step"
Root cause: Unimplemented Feature Gap

### Describe the feature

Some community models have adopted the methods from the "Let’s Verify Step by Step" paper for training, thus there is a risk of Math data leakage. 

https://arxiv.org/pdf/2305.20050.pdf

![image](https://github.com/open-compass/opencompass/assets/17680578/724cf35a-66ce-4367-a4e8-294ce76a360c)

We may consider using a testing method with only 500 samples for reference.

On the other hand, our current evaluation approach cannot measure the extent of data leakage at all. Whether there is a need to treat Math separately is also worth discussing.

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

Ping myself

update?
Repository: open-compass/opencompass
Issue: [Feature] slurm 资源使用优化
Root cause: Documentation Deficiency

### Describe the feature

slurm 的使用需要优化

1.调度的时候怎么指定计算节点？怎么约束使用的GPU量？
比如我的分区有32张卡，但是我希望只在16张运行，剩下的留给其他同学使用，要怎么做，现在的策略好像是一股脑全部提交上去。read the doc里貌似没有说到相关的指令。

2.亟需增加中止slurm残留进程功能，不然会浪费很多资源，自己去杀也很麻烦，很多集群都限制了kill任务的频率。

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

第一个问题，已经找到了配置方法：
from opencompass.runners import SlurmRunner
from opencompass.tasks import OpenICLInferTask

infer = dict(
    # ...
    runner=dict(
        type=SlurmRunner,
        task=dict(type=OpenICLEvalTask),  # 待运行任务
        max_num_workers=16,  # 最大同时评测任务数
        retry=2,  # 任务失败的重试次数，可以避免意外发生的错误
    ),
)

https://opencompass.readthedocs.io/en/latest/user_guides/evaluation.html

第二个问题，还是得搞啊 
#415 

We have improved the slurm runner in the internal gitlab version within the Lab, feel free to contact me if needed.
Repository: open-compass/opencompass
Issue: gsm8k评测结果是空，
Root cause: Validation Gap

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

环境正确，在ceval和mmlu是正常的输出

### Reproduces the problem - code/configuration sample

1419 dataset    version    metric    mode    raccoon_gsm8k
1420 ---------  ---------  --------  ------  ---------------
1421 gsm8k      -          -         -       -

### Reproduces the problem - command or script

跑了1天多的评测任务，结果输出个空，心态崩了

### Reproduces the problem - error message

1

### Other information

_No response_

Please provide the log in the prediction folder.

Feel free to re-open if needed.
Repository: open-compass/opencompass
Issue: 在ceval,mmlu, gaokao 上都速度异常（3个任务2卡22h还未结束），单个样本速度正常。
Root cause: Configuration Error

              我遇到了类似问题，在ceval,mmlu, gaokao 上都速度异常（3个任务2卡22h还未结束），单个样本速度正常。

outputs 输出发现评估重复，比如profession_law只有170个，但输出文件中有 15 个 xxxx_mmlu_profession_law_[0-15]，每个95个样本，且样本有重复。请问是评估方式不对吗，还是 sizepartition 这里有 bug？

```
python run.py ./configs/eval_xxx.py --mode eval -w outputs/ --slurm --partition xxx --debug
```

_Originally posted by @Desein-Yang in https://github.com/InternLM/opencompass/issues/30#issuecomment-1630409625_
            

@Desein-Yang 请提供 outputs 底下的完整 config。

> @Desein-Yang 请提供 outputs 底下的完整 config。


[20230710_114742.py.zip](https://github.com/InternLM/opencompass/files/12013682/20230710_114742.py.zip)


如附件，模型用的是 13b llama 路径做了脱敏处理

<img width="735" alt="image" src="https://github.com/InternLM/opencompass/assets/17928873/562672a7-808f-4968-99d9-4faa35227473">

log 信息中生成多个任务

请看一下是不是 `.cache/dataset_size.json` 中关于 mmlu 数据集的数据条目记录有问题。我们在进行任务 partition 的时候，会在第一次运行时进行数据缓存，如果第一次缓存的值有问题的话，后续的执行也都会有影响。如果有问题，直接把 `.cache/dataset_size.json` 删掉，重跑即可。
另外，我们 mmlu 测的是 test split，professional_law 有 1534 条数据，跟 16*95 的数字是可以大致对上的。val split 是 170 条数据。

```
python run.py ./configs/eval_xxx.py --mode eval -w outputs/ --slurm --partition xxx --debug
```
when running with `--debug`, only 1 GPU would be used at a time.


> .cache/dataset_size.json

明白了，非常感谢，请问你们测 MMLU 的大致运行时间能否提供做个参考




> ```
> python run.py ./configs/eval_xxx.py --mode eval -w outputs/ --slurm --partition xxx --debug
> ```
> 
> when running with `--debug`, only 1 GPU would be used at a time.

Thanks. If the distributed model requires 2 GPU and 1 proc to distributed inference, does it still use 1 GPU to debug ?

> Thanks. If the distributed model requires 2 GPU and 1 proc to distributed inference, does it still use 1 GPU to debug ?

Sorry, I expressed it incorrectly, it should be `--debug` followed by a single process run, which would use two GPUs if your model requires two GPUs.

Feel free to re-open this issue if needed.
Repository: open-compass/opencompass
Issue: [Bug] not support python3.11 due to faiss-gpu version
Root cause: External Dependency Breakage

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

python3.11

### Reproduces the problem - code/configuration sample

it will throw an error during installation for python3.11

```
ERROR: No matching distribution found for faiss-gpu==1.7.2
```

### Reproduces the problem - command or script

pip install opencompass for python3.11

### Reproduces the problem - error message

ERROR: No matching distribution found for faiss-gpu==1.7.2

### Other information


Thanks for the bug reporting, we will check this bug and fix it.

We have remove the dependency of the faiss-gpu in the default setting. Feel free to re-open if needed.
Repository: open-compass/opencompass
Issue: [Feature] Support pass mode in command cli
Root cause: Unimplemented Feature Gap

### Describe the feature

allow huggingface.py mode to be passed in command cli. https://github.com/open-compass/opencompass/blob/9db56526382944560dde85b2b026dae82df6a2f7/opencompass/models/huggingface.py#L69

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!
Repository: open-compass/opencompass
Issue: [Feature] Support dataset limit for quick development
Root cause: Unimplemented Feature Gap

### Describe the feature

lm-evaluation-harness supports a feature `limit` to get users test with very few examples for a quick experiment. It might be useful for opencompass to have one. https://github.com/EleutherAI/lm-evaluation-harness/blob/f2de86094b4bb4f93636f2a0b0fa749de51aa58a/main.py#L29

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

Thanks for the sharing, I think this feature is useful and we will add this feature into our backlog.
Repository: open-compass/opencompass
Issue: 请问在哪里修改送入模型的template呢，还有拼接方式，我得做模型的适配
Root cause: Documentation Deficiency

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

1

### Reproduces the problem - code/configuration sample

1

### Reproduces the problem - command or script

1

### Reproduces the problem - error message

1

### Other information

_No response_

You can edit the template in the dataset's config. Please provide more information if you have met the specific problem.

Feel free to re-open if needed.
Repository: open-compass/opencompass
Issue: [Feature] 可以评估模型的推理速度，吞吐量，资源占用，高并发等性能吗？
Root cause: Unimplemented Feature Gap

### 描述该功能

1. 有时候需要对比两个模型在相同条件下的响应速度，或者推理速度，可以有量化的数字来评估吗？
2. 对比两个模型的资源占用情况，比如相同条件下显存使用情况，cpu，内存使用情况等。

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

Thanks for the attention. We focus on the evaluation of the model's capability currently.
Repository: open-compass/opencompass
Issue: 请问我想对cmmlu这类选择题的模板进行更改，但是system变量无法赋值
Root cause: Documentation Deficiency

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

 cmmlu_infer_cfg = dict(
 
         ice_template=dict(
              type=PromptTemplate,
              template=dict(
                  begin=[dict(role='SYSTEM', fallback_role='HUMAN', prompt='以下是关于{_ch_name}的单项选择题，请直接给出正确答案的选项。'),'</E>'],
                 #begin="</E>",
                  round=[
                      dict(
                          role="HUMAN",
                          prompt=
                          f"USER: {{question}}\nA. {{A}}\nB. {{B}}\nC. {{C}}\nD. {{D}}"
                      ),
                      dict(role="BOT", prompt=' ASSISTANT: 答案是: {answer}
'),
                  ]),
              ice_token="</E>",
         ),
)

### Reproduces the problem - code/configuration sample

请问怎么让begin里的_ch_name进行赋值呢，我看只有round里面会赋值。或者我应该怎么写比较好呢。”以下是关于{_ch_name}的单项选择题，请直接给出正确答案的选项“在fewshot中只在最前面出现一次

### Reproduces the problem - command or script

1

### Reproduces the problem - error message

1

### Other information

1

我希望的模板是
以下是关于{法律}的单项选择题，请直接给出正确答案的选项。
{qustion1}
{A}
{B}
{C}
{D}
答案是:{answer}
{qustion2}
{A}
{B}
{C}
{D}
答案是:{answer}
'''
{question}
{A}
{B}
{C}
{D}
答案是:


这是基本的python语法，如果你不会的话，可以简单地学习下python的for循环语法，

以下是源代码
```
cmmlu_all_sets = list(cmmlu_subject_mapping.keys())

cmmlu_datasets = []
for _name in cmmlu_all_sets:
    _ch_name = cmmlu_subject_mapping[_name]
    cmmlu_infer_cfg = dict(
        ice_template=dict(
            type=PromptTemplate,
            template=dict(
                begin='</E>',
                round=[
                    dict(
                        role='HUMAN',
                        prompt=
                        f'以下是关于{_ch_name}的单项选择题，请直接给出正确答案的选项。\n题目：{{question}}\nA. {{A}}\nB. {{B}}\nC. {{C}}\nD. {{D}}'
                    ),
                    dict(role='BOT', prompt='答案是: {answer}'),
                ]),
            ice_token='</E>',
        ),
        retriever=dict(type=FixKRetriever, fix_id_list=[0, 1, 2, 3, 4]),
        inferencer=dict(type=GenInferencer),
    )
```
以下是主要要关注的地方
```
for _name in cmmlu_all_sets:  # 这里遍历 cmmlu_all_sets 集合中的所有值
    _ch_name = cmmlu_subject_mapping[_name]  # 这里取对应类目的分类名称
```
Repository: open-compass/opencompass
Issue: [Bug] buggy implement for rouge on chinese dataset
Root cause: Algorithmic Error

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

1

### Reproduces the problem - code/configuration sample

1

### Reproduces the problem - command or script

1

### Reproduces the problem - error message

1

### Other information

There is actually English words in LCSTS. It seems to me like they ignored the Chinese part and only calculated the ROUGE for the English & numeric parts.

Fixed in https://github.com/open-compass/opencompass/pull/459
Repository: open-compass/opencompass
Issue: [Feature] 建议增加Embedding模型评测
Root cause: Unimplemented Feature Gap

### 描述该功能

Embedding模型在知识的召回起到至关重要的作用，针对Embedding的专业评测非常有价值。

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

Are there any specific examples of the embedding models we are talking here? I'm not very familiar with this field.

请教下，你这边复现的结果和榜单上是一致的吗？


> ### 描述该功能
> Embedding模型在知识的召回起到至关重要的作用，针对Embedding的专业评测非常有价值。
> 
> ### 是否希望自己实现该功能？
> * [ ]  我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

Welcome more detailed suggestions about evaluation of embedding model.

> 请教下，你这边复现的结果和榜单上是一致的吗？

We may provide more help if you can provide the information of the evaluation experiments, like the specific model and dataset
Repository: open-compass/opencompass
Issue: [Feature] Will opencompass supports vllm as backend in the future?
Root cause: Unimplemented Feature Gap

### Describe the feature

Whether opencompass will support vllm during inference pipeline?

Although lmdeploy support a lot models, but I may test some other models such as BLOOM, GPT BigCode, Falcon etc. So I wonder if OpenCompass will support vllm as backend officially in the future?

If I implement it by myself, are there any suggestions or guidelines?

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

Thanks for the suggestion, we will add this feature into our backlog.

If I implement it by myself, are there any suggestions or guidelines?

https://github.com/open-compass/opencompass/pull/484
https://github.com/open-compass/opencompass/pull/490
These two PRs are related to this feature, feel free to raise question if needed.
Repository: open-compass/opencompass
Issue: [Feature] Any plans to rank for the 13B models?
Root cause: Unimplemented Feature Gap

### Describe the feature

We are curious how far can 13B models go compared with 7B models. Do you have plans to list them in the leader-boards? Some recommendations for the models: LLAMA-13B, Baichuan-13B, WizardLM-13B, Vicuna-13B and etc.

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

Thanks for the attention. We will update the rank of 13B models in this week.
Repository: open-compass/opencompass
Issue: [Feature] add acc_norm evaluation
Root cause: Unimplemented Feature Gap

### Describe the feature

lm-evaluation-harness supports ```acc_norm``` evaluation, which is used in [huggingface leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
```
ARC: 25-shot, arc-challenge (acc_norm)
HellaSwag: 10-shot, hellaswag (acc_norm)
```
```acc_norm``` is calculated by the result (answer logits sum) divided by answer length
```python
acc_norm = 1.0 if np.argmax(results / completion_len) == gold else 0.0
```
In ARC and hellaswag datasets, different answers have different lengths, so longer answers are likely to have larger logits sum, so it should be normalized with answer length to give a accurate prediction.

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!
Repository: open-compass/opencompass
Issue: [Bug] Failed to reproduce llama2-70b-base on triviaqa 
Root cause: Configuration Error

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

其他任务正常

### 重现问题 - 代码/配置示例


from opencompass.openicl.icl_prompt_template import PromptTemplate
from opencompass.openicl.icl_retriever import ZeroRetriever
from opencompass.openicl.icl_inferencer import GenInferencer
from opencompass.datasets import TriviaQADataset, TriviaQAEvaluator

triviaqa_reader_cfg = dict(
    input_columns=['question'],
    output_column='answer',
    train_split='dev',
    test_split='dev')

triviaqa_infer_cfg = dict(
    prompt_template=dict(
        type=PromptTemplate,
        template=dict(
            round=[
                dict(role='HUMAN', prompt='Question: {question}\nAnswer: '),
            ], )),
    retriever=dict(type=ZeroRetriever),
    inferencer=dict(type=GenInferencer, max_out_len=50))

triviaqa_eval_cfg = dict(
    evaluator=dict(type=TriviaQAEvaluator), pred_role='BOT')

triviaqa_datasets = [
    dict(
        type=TriviaQADataset,
        abbr='triviaqa',
        path='./data/triviaqa/',
        reader_cfg=triviaqa_reader_cfg,
        infer_cfg=triviaqa_infer_cfg,
        eval_cfg=triviaqa_eval_cfg)
]


### 重现问题 - 命令或脚本

CUDA_VISIBLE_DEVICES=0,1,2,3 python run.py configs/eval_llama2_70b_turbomind.py -w outputs/llama2-70b

### 重现问题 - 错误信息

在triviaqa数据集llama2-70b-base复现结果为3.8（榜单上是70.7）

### 其他信息

是否是配置文件有问题？

能否提供一下模型输出的文件

How about evaluating llama2-70b-base without turbormind?





> How about evaluating llama2-70b-base without turbormind?

The result which use huggingface generate function is the same as llama2-70b-base with turbormind.(3.8)

Similar with llama2-7b.
(BTW, I'm using a local llama2-7b-hf, while comparing to the Llama-2-7B  on leaderboard.)
leaderboard: 52.8 (triviaqa_gen_3e39a5)
my results: 2.15 (triviaqa_gen_3e39a5), 52.44 (triviaqa_gen_0356ec)

my model outputs:
[llama-2-7b-hf.zip](https://github.com/open-compass/opencompass/files/13848591/llama-2-7b-hf.zip)


Fixed in latest version, welcome to try. Feel free to re-open if needed.
Repository: open-compass/opencompass
Issue: [Feature] ensure greedy decoding with huggingface model
Root cause: Unimplemented Feature Gap

### Describe the feature

something like

```python
from transformers.generation import GenerationConfig
self.model.generation_config = GenerationConfig.from_pretrained(path, trust_remote_code=True)
self.model.generation_config.do_sample = False
```

### Will you implement it?

- [X] I would like to implement this feature and create a PR!

Finished.
Repository: open-compass/opencompass
Issue: [Bug] ModuleNotFoundError: No module named 'evaluate'
Root cause: Environment Incompatibility

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

python 3.8.10
torch 1.14.0
mmengine 0.8.2
opencv-python 4.8.0.74

### Reproduces the problem - code/configuration sample

python run.py --datasets ceval_ppl mmlu_ppl --hf-path /data/juicefs_ai_3mp/public_data/PTM/3MP/llama-7b-hf --model-kwargs device_map='auto' --tokenizer-kwargs padding_side='left' truncation='left' use_fast=False --max-out-len 100 --max-seq-len 2048 --batch-size 8 --no-batch-padding --num-gpus 1

### Reproduces the problem - command or script

python run.py --datasets ceval_ppl mmlu_ppl --hf-path /data/juicefs_ai_3mp/public_data/PTM/3MP/llama-7b-hf --model-kwargs device_map='auto' --tokenizer-kwargs padding_side='left' truncation='left' use_fast=False --max-out-len 100 --max-seq-len 2048 --batch-size 8 --no-batch-padding --num-gpus 1

### Reproduces the problem - error message

ModuleNotFoundError: No module named 'evaluate'

### Other information

opencompass/openicl/icl_evaluator/icl_hf_evaluator.py:5 导入evaluate报错

Please try `pip3 install evaluate`.

You are also welcome to follow the install instruction illustrated here: https://github.com/open-compass/opencompass/tree/main#%EF%B8%8F-installation

Feel free to re-open if needed.
Repository: open-compass/opencompass
Issue: [Feature] 代码相关数据集评测保留更详细的评测信息
Root cause: Documentation Deficiency

### 描述该功能

对于HumanEval这样的代码相关的数据集，评测时保存哪些错了哪些对了，以及具体的执行报错信息（traceback或者至少error message，能指向具体的代码行最好），这样方便查找问题，以及排除后处理等的影响、区分一些trivial的错误（如缺少import）等。如果能针对错误类型再做一个类似报表的统计，那就更好了。

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

Thanks for the feature request, we will add this feature into our backlog of Q4. PR are also welcomed! Thanks again.

Have you ever tested the performance of APIs such as GPT on the human eval dataset, and how did you test it?

> Have you ever tested the performance of APIs such as GPT on the human eval dataset, and how did you test it?

Please check our documentation for more details.

You can use `--dump-eval-details` currently.https://github.com/open-compass/opencompass/blob/001e77fea236276aa8018b34cd23076145ab1672/run.py#L127

 Feel free to re-open if needed.
Repository: open-compass/opencompass
Issue: [Bug] agieval's score in your website ,how to reproduce it
Root cause: Configuration Error

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

official environment

### Reproduces the problem - code/configuration sample

CUDA_VISIBLE_DEVICES='0,1,2,3,4,5,6,7' python run.py --hf-path baichuan2-7b-base --datasets agieval_gen_a0c741 --model-kwargs device_map='auto' --tokenizer-kwargs padding_side='left' truncation='left' use_fast=False --max-out-len 100 --max-seq-len 4096 --batch-size 8 --no-batch-padding --num-gpus 1  --max-partition-size 15000

### Reproduces the problem - command or script

CUDA_VISIBLE_DEVICES='0,1,2,3,4,5,6,7' python run.py --hf-path baichuan2-7b-base --datasets agieval_gen_a0c741 --model-kwargs device_map='auto' --tokenizer-kwargs padding_side='left' truncation='left' use_fast=False --max-out-len 100 --max-seq-len 4096 --batch-size 8 --no-batch-padding --num-gpus 1  --max-partition-size 15000

### Reproduces the problem - error message

![image](https://github.com/open-compass/opencompass/assets/79788571/09238271-48c0-4292-b5cc-d8a1e3be6114)
when i calculate the average value of it , it result to 23.7 , which is 34.6 in your website 
![image](https://github.com/open-compass/opencompass/assets/79788571/df2c8faa-c50a-4a6c-90e3-2564b22be19d)


### Other information

_No response_

Please try with `--datasets agieval_mixed_2f14ad`


Feel free to re-open if needed.
Repository: open-compass/opencompass
Issue: [Feature] I try to use instruction tuning to fine-tune llama model and get a new alpaca model. How to use compass to evaluate the local alpaca model on MMLU and other datasets
Root cause: Documentation Deficiency

### Describe the feature

How to use compass to evaluate the local alpaca model on MMLU and other datasets

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

LLama format or huggingface format?

Hi,

huggingface format