zhiminy/EvalEng huggingface dataset:
Repository: open-compass/opencompass
Issue: 运行时，出现 No module named 'opencompass'错误
Root cause: Environment Incompatibility

### Describe the bug

按照第一个例子执行：
python run.py configs/eval_demo.py -w outputs/demo
上报如下错误：
Traceback (most recent call last):
  File "/root/project/opencompass-0.1.0/opencompass/tasks/openicl_eval.py", line 10, in <module>
    from opencompass.registry import (ICL_EVALUATORS, MODELS, TASKS,
ModuleNotFoundError: No module named 'opencompass'

看了下opencompass-0.1.0/opencompass/tasks/openicl_eval.py的代码：
from opencompass.registry import (ICL_EVALUATORS, MODELS, TASKS,TEXT_POSTPROCESSORS)
感觉文件夹的层次不对？
这块需要怎么配置，能让openicl_eval.py直接看到两层之外的opencompass目录呢？

python小白，请大侠帮忙，感谢

### Environment

python3.10

### Other information

_No response_

问题应该已经解决；
安装依赖时，需要使用pip install -e .
我误用了 pip install -r requirement
导致没有将opencompass安装到环境中

但是我用了pip install -e .，还是会报这个错
Repository: open-compass/opencompass
Issue: [Feature] Why is the leaderboard called "Multi-modal Modal Leaderboard"? 
Root cause: Documentation Deficiency

### 描述该功能

"Multi-modal" and "Modal" are very duplicate words...This somehow does not make a lot of sense...
Why not rename it as "Multi-modal Model Leaderboard" directly?

### 是否希望自己实现该功能？

- [X] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

@kennymckormick Please check if there is a typo.

Hi, @zhimin-z , that seems to be a typo (modal -> model), we are going to fix it.
Repository: open-compass/opencompass
Issue: 了解下， 如何处理 TheoremQA, TriviaQA and Xsum 这些数据集用于本框架的预测呢？
Root cause: Documentation Deficiency

### 描述该错误

了解下， 如何处理 TheoremQA, TriviaQA and Xsum 这些数据集用于本框架的预测呢？

### 环境信息

无

### 其他信息

无

I do not fully understand what you want to know. I assume these codes might be helpful:

https://github.com/InternLM/opencompass/blob/3715be6595e413b542f94f1c988948c18e5f3619/opencompass/datasets/TheoremQA.py#L19-L26

https://github.com/InternLM/opencompass/blob/3715be6595e413b542f94f1c988948c18e5f3619/opencompass/datasets/triviaqa.py#L36-L59

https://github.com/InternLM/opencompass/blob/3715be6595e413b542f94f1c988948c18e5f3619/opencompass/datasets/xsum.py#L34-L36


我想了解下这些数据集文件要从哪儿获取？以及如何预处理来适配当前opencompass框架？

We just uploaded a new `.zip` containing TheoremQA / TriviaQA / Xsum, please refer to 

https://github.com/InternLM/opencompass/releases/tag/0.1.1

```bash
wget https://github.com/InternLM/opencompass/releases/download/0.1.1/OpenCompassData.zip
unzip OpenCompassData.zip
```

That's great， thanks
Repository: open-compass/opencompass
Issue: 执行python run.py configs/eval_demo.py -w outputs/demo 速度异常
Root cause: Documentation Deficiency

### 描述该错误

按照文档中的快速上手，数据及使用的是ceval，模型为OPT-125M；
运行python run.py configs/eval_demo.py -w outputs/demo时，特别慢，没有在GPU上运行，不知道是我哪里的问题；
log打印：
![image](https://github.com/InternLM/opencompass/assets/17903509/f500604c-911e-479c-906e-805904184234)

GPU情况：
![image](https://github.com/InternLM/opencompass/assets/17903509/68cedbd7-5d63-49b8-ac61-bcfcbddd8abe)

一直卡着，没有进度。

### 环境信息

python3.10

### 其他信息

_No response_

![image](https://github.com/InternLM/opencompass/assets/18586273/57f824a0-6e80-499a-9e46-5e5a75235ace)

The infer tasks finished at 22:45, the `nvidia-smi` had no GPU info because Eval Tasks do not need GPU。

For Infer Task, 'gen' mode LLM itself is pretty slow especially pretrain models will generate text until it hits the generation length limit example the demo takes around 90 mins on my test machine with one 1660ti GPU.


> The infer tasks finished at 22:45, the `nvidia-smi` had no GPU info because Eval Tasks do not need GPU。
> 
> For Infer Task, 'gen' mode LLM itself is pretty slow especially pretrain models will generate text until it hits the generation length limit example the demo takes around 90 mins on my test machine with one 1660ti GPU.

上面截图的状态是跑eval_demo.py配置，但是将数据集修改为ceval中的两门课（目的是简化数据，打通流程）。所以，infer任务很快就完成了；等看GPU状态时，已经跑完；

你这里解释完之后，就清晰啦。Eval任务就应该在CPU上跑；

另外，后来我又将Ceval的所有科目都打开，开始运行，发现又非常慢；
最终原因：ceval数据已经在本地了，但是好像每次都去huggingface上下载，超时之后，才从本地下载，导致每次执行到：
Found cached dataset csv (/root/.cache/huggingface/datasets/csv/default-e2ed8a58cfad59df/0.0.0/eea64c71ca8b46dd3f537ed218fc9bf495d5707789152eb2764f5c78fa66d59d)
时都卡好几分钟；

这个问题的原因是，应该设置下面环境变量：
export HF_DATASETS_OFFLINE=1
export TRANSFORMERS_OFFLINE=1
export HF_EVALUATE_OFFLINE=1

这里也抛出来，供后面同学参考。
再次感谢。
Repository: open-compass/opencompass
Issue: 关于 医疗方面 MedBench， 在连接模型测试时的问题
Root cause: Documentation Deficiency

### Describe the feature

对于 医疗方面的 MedBench 排行，https://medbench.opencompass.org.cn/leaderboard ，有点问题，请opencompass的相关朋友解答下：
1、opencompass排行中，有一个qwen-chat-72b条目，这是本地部署了开源的qwen-chat-72b 然后测试的？还是连接阿里的api测试的呢？如果是本地部署的开源的qwen-chat，那用的是哪个版本，v1、v1.5版本？如果是连api，那么在线连接的是哪个版本的模型呢？开源还是闭源的呢？
2、什么时候测试的呢？我看发布日期和更新日期都是2024/02/20，这里标的“发布日期”、“更新日期”分别是指啥呢？
3、在做这个MedBench 测试时，具体连接各个模型或api的相关代码，是在哪个文件里呢？

### Will you implement it?

- [x] I would like to implement this feature and create a PR!

Thanks for your attention. 
Please @xmshi-trio check this issue and share your idea.

Please contact medbench@pjlab.org.cn for more information

您好！感谢对MedBench的关注。
1. qwen-chat-72b是用户在本地部署，然后做一遍推理，然后提交结果到MedBench上，具体哪个版本我们需要后续联系该用户确认。
2. “发布日期”是模型的发布日期，“更新日期是”是提交结果的日期
3. 目前仅支持本地推理，然后上传。本地推理需要用户根据自己的模型自行进行。

feel free to reopen it if needed
Repository: open-compass/opencompass
Issue: [Bug]  Evaluations on Mistral-7B-v0.1 couldn't be reproduced
Root cause: External Dependency Breakage

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

None

### Reproduces the problem - code/configuration sample

Mistral-7B[https://huggingface.co/mistralai/Mistral-7B-v0.1]，我用11月的master代码& data，以及现在的master & data，跑下来的结果差不多
MMLU = 53.3
TydiQA= 25.5

跟opencompass官网披露的差的比较多(MMLU=64.1, TydiQA=35.7)

If you can reproduce the result same as opencompass shown, please tell me how to do it, thanks.



### Reproduces the problem - command or script

None

### Reproduces the problem - error message

the details of tydiqa:
dataset,version,metric,mode,Mistral-7B-v0.1
--
tydiqa-goldp_arabic,3d688e,exact_match,gen,18.78
tydiqa-goldp_arabic,3d688e,f1,gen,49.09
tydiqa-goldp_bengali,ed3379,exact_match,gen,1.77
tydiqa-goldp_bengali,ed3379,f1,gen,10.05
tydiqa-goldp_english,38da40,exact_match,gen,24.32
tydiqa-goldp_english,38da40,f1,gen,32.82
tydiqa-goldp_finnish,4adb40,exact_match,gen,29.92
tydiqa-goldp_finnish,4adb40,f1,gen,35.08
tydiqa-goldp_indonesian,7b2a2a,exact_match,gen,15.93
tydiqa-goldp_indonesian,7b2a2a,f1,gen,25.91
tydiqa-goldp_japanese,8e7cb9,exact_match,gen,19.12
tydiqa-goldp_japanese,8e7cb9,f1,gen,19.48
tydiqa-goldp_korean,5be5b3,exact_match,gen,12.32
tydiqa-goldp_korean,5be5b3,f1,gen,30.24
tydiqa-goldp_russian,4070c2,exact_match,gen,10.34
tydiqa-goldp_russian,4070c2,f1,gen,28.68
tydiqa-goldp_swahili,97110f,exact_match,gen,17.43
tydiqa-goldp_swahili,97110f,f1,gen,25.72
tydiqa-goldp_telugu,df948c,exact_match,gen,0.15
tydiqa-goldp_telugu,df948c,f1,gen,1.71
tydiqa-goldp_thai,3e6b28,exact_match,gen,1.01
tydiqa-goldp_thai,3e6b28,f1,gen,21.51
tydiqa-goldp,-,exact_match,gen,13.74
tydiqa-goldp,-,f1,gen,25.48



### Other information

align the data metric on  Mistral-7B-v0.1

using the latest Mistral-7B model can solve this issue.
Repository: open-compass/opencompass
Issue: 关于mode为eval的疑问
Root cause: Documentation Deficiency

### Describe the feature

之前评测了好几个数据集然后进程中断了。
文件夹下有这三个文件：
<img width="173" alt="image" src="https://github.com/InternLM/opencompass/assets/72484291/2a0e0205-33c2-4d2c-b27e-b6a9ce3b3c7c">
prediction里为每个任务的输出，我该怎么评测结果，尝试加上--mode eval -r 20230724_021658参数后生成的csv文件都是‘--’值

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

I'm not sure as well, but you can navigate to `logs` directory and read the logs under `eval/` to figure out the reason

I don't know how to use mode ‘eval’ to evaluate 'predictions',can you give me a command line example？It would be better if there are more detailed examples, thanks a lot！

i found the reason, thanks!
Repository: open-compass/opencompass
Issue: 关于大海捞针测试的问题
Root cause: Documentation Deficiency

### 描述该功能

看config中cdme数据集配置时有一个'tokenizer_model': 'gpt-4'参数，想请问一下这里的这个参数是什么作用呢，如果要测试自己的模型也要用gpt-4嘛 还是需要修改成自己模型tokenizer的路径呢？感谢回答
![2024-02-26 14-40-08屏幕截图](https://github.com/open-compass/opencompass/assets/150245861/0cfabc60-cf44-4475-837b-a28c4a1896ae)


### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！
Repository: open-compass/opencompass
Issue: [Feature] Will OpenCompass provide the CPM Model evaluation result ?
Root cause: Unimplemented Feature Gap

### 描述该功能

Hi OpenCompass guys,

Will the CPM model test result updated in https://rank.opencompass.org.cn/leaderboard-llm ?

Thanks






### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

Thanks for your suggestions, we will update CPM in the next 1-2 weeks.
Repository: open-compass/opencompass
Issue: [Bug] FileNotFoundError: [Errno 2] No such file or directory: 'data/subjective/mtbench.json'
Root cause: Environment Incompatibility

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

data/subjective/mtbench.json 找不到, OpenCompassData-complete-20240207.zip 和 OpenCompassData-complete.zip 中都没有.


### Reproduces the problem - code/configuration sample

1

### Reproduces the problem - command or script

1

### Reproduces the problem - error message

```

Traceback (most recent call last):
  File "/var/work/run.py", line 361, in <module>
    main()
  File "/var/work/run.py", line 303, in main
    tasks = partitioner(cfg)
  File "/var/work/opencompass/partitioners/base.py", line 80, in __call__
    tasks = self.partition(**model_and_dataset_args,
  File "/var/work/opencompass/partitioners/size.py", line 90, in partition
    comb['datasets'] = sorted(comb['datasets'],
  File "/var/work/opencompass/partitioners/size.py", line 91, in <lambda>
    key=lambda x: self.get_cost(x),
  File "/var/work/opencompass/partitioners/size.py", line 214, in get_cost
    dataset = build_dataset_from_cfg(dataset)
  File "/var/work/opencompass/utils/build.py", line 13, in build_dataset_from_cfg
    return LOAD_DATASET.build(dataset_cfg)
  File "/usr/local/lib/python3.10/site-packages/mmengine/registry/registry.py", line 570, in build
    return self.build_func(cfg, *args, **kwargs, registry=self)
  File "/usr/local/lib/python3.10/site-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg
    obj = obj_cls(**args)  # type: ignore
  File "/var/work/opencompass/datasets/base.py", line 12, in __init__
    self.dataset = self.load(**kwargs)
  File "/var/work/opencompass/datasets/subjective/mtbench.py", line 173, in load
    with open(filename, 'r', encoding='utf-8') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'data/subjective/mtbench.json'
```

### Other information

_No response_

Thanks for your reminding, we will add this data file to the OpenCompassData ASAP

Updated, refer to https://github.com/open-compass/opencompass/releases/tag/0.2.2.rc1. Feel free to contact me if you have any other problems

Feel free to re-open if needed.
Repository: open-compass/opencompass
Issue: [Bug] One mistake in the pipeline of MMBench 
Root cause: Documentation Deficiency

### Describe the bug

In [https://opencompass.readthedocs.io/en/latest/MMBench.html](https://opencompass.readthedocs.io/en/latest/MMBench.html#how-to-construct-the-inference-prompt):
```
if data_sample['context'] is None:
    prompt = data_sample['context'] + ' ' + data_sample['question'] + ' ' + data_sample['options']
else:
    prompt = data_sample['question'] + ' ' + data_sample['options']
```
It should be 
```
if data_sample['context'] is not None:
```


### Environment

No anomaly

### Other information

Also, the evaluation pipeline of MMbench is not complete. Can you release the code of **Circular Evaluation Strategy** and  **ChatGPT-involved Choice Extraction**?  How should we submit the results to the leaderboard?  The hyperlink of instruction https://opencompass.org.cn/mmbench  seems not correct.
![image](https://github.com/InternLM/opencompass/assets/21003563/fd8c1c5e-0e87-4aee-9549-6d566e3197f8)


You can follow the intructions in https://opencompass.org.cn/leaderboard-multimodal for submitting your predictions

That looks like a typo, we will fix it in our documentations. In our code, the logic is:
```python
if data_sample['context'] is not None:
    prompt = data_sample['context'] + ' ' + data_sample['question'] + ' ' + data_sample['options']
else:
    prompt = data_sample['question'] + ' ' + data_sample['options']
```

Closed, feel free to re-open this issue if you have more questions.

Hi,  whether the models on the leaderboard add a template (header, role) when evaluating,  like "roles=("Human", "Assistant")," in Vicuna.
Repository: open-compass/opencompass
Issue: [Bug] 官网1.0评测数据集链接失效
Root cause: External Dependency Breakage

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

windows

### 重现问题 - 代码/配置示例

暂无

### 重现问题 - 命令或脚本

暂无

### 重现问题 - 错误信息

点击数据集链接，显示There is nothing here!

### 其他信息

_No response_
Repository: open-compass/opencompass
Issue: [Bug] Wrong hints language for AGIEval `jec-qa-kd` and `jec-qa-ca`
Root cause: Configuration Error

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I have modified the code (config is not considered code), or I'm working on my own tasks/models/datasets.

### Environment

not necessary

### Reproduces the problem - code/configuration sample

not necessary

### Reproduces the problem - command or script

not necessary

### Reproduces the problem - error message

not necessary

### Other information

Related discussion: https://github.com/open-compass/opencompass/discussions/712
Prompts for `jec-qa-kd` and `jec-qa-ca` have a Chinese intro and an English hint, which is not consistent.

Thanks for the bug report. We will check this bug.

@tonysy https://github.com/open-compass/opencompass/blob/caf1cf8a1739756a4074f30d9162c1d2876d9037/configs/datasets/agieval/agieval_gen_64afd3.py#L101 here, these two datasets should be included in `agieval_chinese_sets`
Repository: open-compass/opencompass
Issue: [Feature] Support F1-score for DROP dataset
Root cause: Unimplemented Feature Gap

### 描述该功能

谷歌的Gemini, Claude 3 发布的页面上，DROP使用都是F1-score

<img width="1157" alt="image" src="https://github.com/open-compass/opencompass/assets/18586273/9a6c49fb-8e37-4df3-a220-8097c18a3ca1">

<img width="1048" alt="image" src="https://github.com/open-compass/opencompass/assets/18586273/83d3c25d-6538-4d52-8212-98df9d7c95ae">


官方实现： https://github.com/allenai/allennlp-reading-comprehension/blob/master/allennlp_rc/eval/drop_em_and_f1.py

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！
Repository: open-compass/opencompass
Issue: [Feature] Support QuALITY dataset
Root cause: Unimplemented Feature Gap

### 描述该功能

在Claude3中加入了QuALITY: Question Answering with Long Input Texts, Yes!”这个长文测试集（平均5k token），是一个人工标注且质量比较高的测试集，希望支持

Arxiv : https://arxiv.org/abs/2112.08608
Github: https://github.com/nyu-mll/quality

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

https://github.com/open-compass/opencompass/pull/976
Repository: open-compass/opencompass
Issue: [Bug]  HumanEval eval报错
Root cause: Validation Gap

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

官方

### 重现问题 - 代码/配置示例

官方

### 重现问题 - 命令或脚本

官方

### 重现问题 - 错误信息

Traceback (most recent call last):
  File "/home/ma-user/modelarts/user-job-dir/conda_activate/opencompass/opencompass/tasks/openicl_eval.py", line 363, in <module>
    inferencer.run()
  File "/home/ma-user/modelarts/user-job-dir/conda_activate/opencompass/opencompass/tasks/openicl_eval.py", line 109, in run
    self._score()
  File "/home/ma-user/modelarts/user-job-dir/conda_activate/opencompass/opencompass/tasks/openicl_eval.py", line 202, in _score
    icl_evaluator = ICL_EVALUATORS.build(self.eval_cfg['evaluator'])
  File "/home/ma-user/anaconda3/envs/opencompass-240305/lib/python3.10/site-packages/mmengine/registry/registry.py", line 570, in build
    return self.build_func(cfg, *args, **kwargs, registry=self)
  File "/home/ma-user/anaconda3/envs/opencompass-240305/lib/python3.10/site-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg
    obj = obj_cls(**args)  # type: ignore
  File "/home/ma-user/modelarts/user-job-dir/conda_activate/opencompass/opencompass/datasets/humaneval.py", line 54, in __init__
    from human_eval.evaluation import \
  File "/home/ma-user/modelarts/user-job-dir/conda_activate/opencompass/human-eval/human_eval/evaluation.py", line 10, in <module>
    from human_eval.execution import check_correctness
  File "/home/ma-user/modelarts/user-job-dir/conda_activate/opencompass/human-eval/human_eval/execution.py", line 59
    result.append("passed")
IndentationError: expected an indented block after 'with' statement on line 47

### 其他信息

_No response_

你解决了吗，尝试用pip install human-eval可以解决问题，但是指标差距太大。请问你怎么解决的呢 @nyBball 

在human_eval下的excution.py文件中48行增加行代码exec(check_program, exec_globals)，如下所示：
            try:
                exec_globals = {}
                with swallow_io():
                    with time_limit(timeout):
                        exec(check_program, exec_globals)

> 你解决了吗，尝试用pip install human-eval可以解决问题，但是指标差距太大。请问你怎么解决的呢 [@nyBball](https://github.com/nyBball)

您解决指标问题了吗，我也遇到在humaneval+llama2-7b上pass@1只有1.22的问题。同时mbpp上的指标也明显低
Repository: open-compass/opencompass
Issue: [Bug] 评测C-Eval数据集时，选择test目录下测试数据，但是最后得分非常低，接近于0分
Root cause: Documentation Deficiency

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

1

### Reproduces the problem - code/configuration sample

1

### Reproduces the problem - command or script

1

### Reproduces the problem - error message

1

### Other information

评测C-Eval数据集时，选择test目录下测试数据，但是最后得分非常低，接近于0分
请问是不是C-Eval数据集评测时，是不是需要自己根据模型的回答，将回答拿到C-Eval官网上计算分数？

Please submit the results on C-Eval official website for test acc. We only have answer of the val set.

> Please submit the results on C-Eval official website for test acc. We only have answer of the val set.

Thanks for your reply.
Repository: open-compass/opencompass
Issue: [Feature] Dev dataset for MedBench
Root cause: Unimplemented Feature Gap

### 描述该功能

Current MedBench dataset contains only test examples, is it possible to release a dev dataset?

### 是否希望自己实现该功能？

- [x] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！
Repository: open-compass/opencompass
Issue: [Feature] 数据集的中、英分类以及公开/自建分类的展示；样本数统计

### 描述该功能

Hi, team

1. 目前opencompass支持的数据集在github pape和官网数据集社区都有支持的数据集的展示，但是比较不方便的一点是，无法得知哪些是中文的数据集，其中又哪些是自建的、哪些是公开的。
2. 另外，能否同时展示每个数据集的样本数统计，或者退一步中、英文分类/自建、公开分类的样本数统计？目前只看到github上有写40,0000+questions。

如果已存在相关的功能请告知，抱歉打扰，谢谢。

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

Thanks for your great suggestions, we will add this feature into the backlog.
Repository: open-compass/opencompass
Issue: [Feature] 请问Base模型测评ppl任务什么时候可以支持vllm？
Root cause: Documentation Deficiency

### 描述该功能

如题，请问Base模型和相应任务测评什么时候可以支持vllm，使用hf测评速度太难以接受了

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

Actually we already support VLLM and LMDeploy, see in https://github.com/open-compass/opencompass/blob/3098d788455dc785e6830f8c69eb9d1010c0cce1/opencompass/models/vllm.py#L14

> Actually we already support VLLM and LMDeploy, see in
> 
> https://github.com/open-compass/opencompass/blob/3098d788455dc785e6830f8c69eb9d1010c0cce1/opencompass/models/vllm.py#L14

不好意思，我没有表述清楚。就是base模型测评一般使用这个ppl测试，但是ppl-based 的任务不支持vllm
![image](https://github.com/open-compass/opencompass/assets/97608046/d1f6123f-5937-4b6b-aa0b-cc58be22e322)
这个有什么好的办法吗？

I'm sorry but the Inference backends like VLLM and LMDeploy generally do not support PPL.

> I'm sorry but the Inference backends like VLLM and LMDeploy generally do not support PPL.

In fact, vllm already provided the feature of returning logits for prompts:
https://github.com/vllm-project/vllm/pull/1328

opencompass can use this feature to infer PPL for a given prompt.

> > I'm sorry but the Inference backends like VLLM and LMDeploy generally do not support PPL.
> 
> In fact, vllm already provided the feature of returning logits for prompts: [vllm-project/vllm#1328](https://github.com/vllm-project/vllm/pull/1328)
> 
> opencompass can use this feature to infer PPL for a given prompt.

@bittersweet1999  应该是的。您好，可以再关注一下这个问题吗？可以参考EleutherAI/lm-evaluation-harness的评测框架，他们应该使用了该方法来支持使用vllm计算ppl。https://github.com/EleutherAI/lm-evaluation-harness/blob/28ec7fa950346b5a895e85e1f3edd5648168acc4/lm_eval/models/vllm_causallms.py#L183-L184
![image](https://github.com/open-compass/opencompass/assets/97608046/cf1ae951-7106-4ad8-8602-bd2790e55f51)
这使得他们的ppl评测非常迅速。
opencompass如果能支持该项功能将非常有帮助。非常感谢您耐心的解答。

Hi we have supported vllm and lmdeploy get ppl right now, for more information please see in here https://github.com/open-compass/opencompass/pull/1003
Repository: open-compass/opencompass
Issue: [Feature] Support LiveCodeBench
Root cause: Unimplemented Feature Gap

### LiveCodeBench

[Github](https://github.com/LiveCodeBench/LiveCodeBench)
[HomePage](https://livecodebench.github.io/)


数据集优点：
1. humaneval 与 mbpp 题目过于基础， 该数据集更难
2. 来源于近期的code比赛，数据污染问题上还好很多
3. 除了**写代码**任务，还有 **结果预测**， **代码修复**， **代码执行**。更加全面的衡量一个模型的代码能力


### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

We have supported APPS and TACO. Also we will release the LeetcodeBench(2023) built by ourself recently.

Cool! 

Looking forward to your work！
Repository: open-compass/opencompass
Issue: [Bug] 0.2.2.rc1 版本的数据里面找不到 needlebench 
Root cause: Unimplemented Feature Gap

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

无须

### 重现问题 - 代码/配置示例

无须

### 重现问题 - 命令或脚本

# Download dataset to data/ folder
wget https://github.com/open-compass/opencompass/releases/download/0.2.2.rc1/OpenCompassData-complete-20240207.zip

unzip OpenCompassData-complete-20240207.zip


然后里面没有needlebench文件夹

### 重现问题 - 错误信息

无须

### 其他信息

_No response_

Hi @genggui001  ,

Thanks for pointing out the missing `needlebench` dataset in version 0.2.2.rc1. We apologize for any inconvenience this may have caused. We will include it in the next dataset update as soon as possible.

有没有单独的needlebench的数据可以共享给我一波，感觉用你们的框架测大海捞针比较方便，磕头跪谢

@DseidLi 同样问题，跪求更新，十分感谢！！

[needlebench.zip](https://github.com/open-compass/opencompass/files/14741330/needlebench.zip)


Feel free to re-open if needed.