zhiminy/EvalEng huggingface dataset:
Repository: open-compass/opencompass
Issue: ceval数据集的test模式为啥不能输出到一个json，让我能提交到 官网
Root cause: Documentation Deficiency

### Prerequisite

- [X] I have searched [Issues](https://github.com/InternLM/opencompass/issues/) and [Discussions](https://github.com/InternLM/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/InternLM/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

1

### Reproduces the problem - code/configuration sample

1

### Reproduces the problem - command or script

1

### Reproduces the problem - error message

1

### Other information

1

Here is an example script, a PR is welcomed.

```python
from glob import glob
import json
import os
import os.path as osp
import argparse


def first_capital_postprocess(text: str) -> str:
    for t in text:
        if t in ["A", "B", "C", "D"]:
            return t
    return 'C'

def run(name, input_dir):
    input_paths = glob(input_dir, "ceval-test-*.json")
    overall = {}
    for input_path in input_paths:
        name = input_path.split("ceval-test-")[-1].split(".")[0]
        with open(input_path, "r") as f:
            data = json.load(f)
        results = {}
        for k, v in data.items():
            results[k] = first_capital_postprocess(v["prediction"])
            assert results[k] in ["A", "B", "C", "D"]
        overall[name] = results

    os.makedirs("ceval-submission", exist_ok=True)
    output_path = f"ceval-submission/{name}.json"
    with open(output_path, "w") as f:
        json.dump(overall, f, indent=4)
    print('submission path', output_path)


parser = argparse.ArgumentParser()
parser.add_argument("name", type=str, default="output")
parser.add_argument("input_dir", type=str, default="ceval-test")
args = parser.parse_args()

run(args.name, args.input_dir)
```

3Q very much
Repository: open-compass/opencompass
Issue: does this repo support MAC OS system? [Bug] 
Root cause: Environment Incompatibility

### Prerequisite

- [X] I have searched [Issues](https://github.com/InternLM/opencompass/issues/) and [Discussions](https://github.com/InternLM/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/InternLM/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

I have tried both Python3.8 and Python 3.10
My computer is Mac OS 13.4 with M1 pro

### Reproduces the problem - code/configuration sample

NA

### Reproduces the problem - command or script

```bash
pip install -e .
```


### Reproduces the problem - error message

ERROR: Could not find a version that satisfies the requirement faiss_gpu==1.7.2 (from opencompass) (from versions: 1.5.3, 1.6.0, 1.6.1, 1.6.3, 1.6.4, 1.6.4.post2, 1.6.5, 1.7.0, 1.7.1, 1.7.1.post1, 1.7.1.post2)
ERROR: No matching distribution found for faiss_gpu==1.7.2

### Other information

_No response_

Thanks for the insightful idea, we will add this feature into our backlog.
Also, welcome to contribute this feature.

We have not tried installing OpenCompass on MacOS, and we will look into it.

As for the issue you encountered, `faiss-gpu` is not used when evaluating some of the datasets. You might be able to bypass this error by changing the version dependency of `faiss-gpu` here.

https://github.com/InternLM/opencompass/blob/2a5cef29142547068fa0399b2202103863669e20/requirements/runtime.txt#L9

i see, thanks a lot


Feel free to repoen if needed.
Repository: open-compass/opencompass
Issue: [Feature] Add CMMLU in summarizer 
Root cause: Documentation Deficiency

### Describe the feature

After run cmmlu inference, i  can get all subclass acc in cmmlu, but i can not find cmmlu result in  output summary,  i wonder how to reproduce cmmlu result shown in opencompass leadboard,   for example:  just compute mean acc  of each class ?    

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

Already added over here:

https://github.com/open-compass/opencompass/blob/main/configs/summarizers/groups/cmmlu.py

Sorry for the late FYI.

Feel free to re-open if needed.
Repository: open-compass/opencompass
Issue: [Feature] Support of InstructBlip on MME Dataset
Root cause: Unimplemented Feature Gap

### Describe the feature

For now, only MiniGPT-4 is supported with MME dataset. I wonder if it's possible to further support MLLMs like Instruct Blip and LLaVA.

I also have a problem about the MME Prompt Constructor. Where can I find a reference to set up the MME prompt constructor by myself. I assume prompt design has quite big impact on the benchmarking results.

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

Hi @zycheiheihei .
> For now, only MiniGPT-4 is supported with MME dataset. I wonder if it's possible to further support MLLMs like Instruct Blip and LLaVA.

Since all our MLLMs in opencompass follows a similar evaluation pipeline, just replace the `model` of mini-gpt4 in MME config with model of llava or instructblip and implement a `LLaVAMMEPromptConstructor` following the one of MiniGPT-4 could do the job.

> I also have a problem about the MME Prompt Constructor. Where can I find a reference to set up the MME prompt constructor by myself. I assume prompt design has quite big impact on the benchmarking results.

As shown in the config, the prompt constructor is imported in this line
https://github.com/InternLM/opencompass/blob/c26ecdb1b05baea7bcf34c99ea245ae68a3ada83/configs/multimodal/minigpt_4/minigpt_4_7b_mme.py#L1

And the class is defined here:
https://github.com/InternLM/opencompass/blob/c26ecdb1b05baea7bcf34c99ea245ae68a3ada83/opencompass/multimodal/models/minigpt_4/prompt_constructor.py#L143-L156

We supported custom `system_prompt` and `reply_prompt` for now. And also you could implement your own constructor here. Don't forget to register it in `__init__.py` before import.

Hope it helps. Feel free to ask if anything is unclear.

Thanks for your quick reply!
Repository: open-compass/opencompass
Issue: [Feature] Support llava on seed-bench, mme and more multi-modal benchmarks.
Root cause: Unimplemented Feature Gap

### 描述该功能

Thank you for your awesome work. Is there a plan to support llava on seed-bench, mme and more multi-modal benchmarks?

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

Thanks for the insightful suggestions. We will add this feature into our backlog. Also welcome to contribute these benchs.

We have merged the evaluation of LLaVA, feel free to try it. #331
Repository: open-compass/opencompass
Issue: 在评估internlm7b在c-eval gaokao上评测速度很慢，不知道是为什么？多卡多线程下好像更慢
Root cause: Configuration Error

### 描述该错误

    比较慢

### 环境信息

 比较慢

### 其他信息

_No response_

GPU 利用率正常吗？请提供配置文件

from mmengine.config import read_base

with read_base():
    from ..datasets.GaokaoBench.GaokaoBench_gen import GaokaoBench_datasets
    from ..datasets.agieval.agieval_gen import agieval_datasets
    from ..datasets.ceval.ceval_gen import ceval_datasets


# datasets = [*GaokaoBench_datasets]
datasets = [*ceval_datasets]



from opencompass.models import HuggingFaceCausalLM

models = [
    dict(
        type=HuggingFaceCausalLM,
        abbr='internlm-7b-hf',
        path="./download/models/internlm-7b",
        tokenizer_path='./download/models/internlm-7b',
        tokenizer_kwargs=dict(
            trust_remote_code=True,
            padding_side='left',
            truncation_side='left',
            use_fast=False,
        ),
        max_out_len=100,
        max_seq_len=2048,
        batch_size=96,
        model_kwargs=dict(trust_remote_code=True, device_map='auto'),
        run_cfg=dict(num_gpus=2, num_procs=2),
    )
]


烦请确认下"很慢"到底是多慢。作为参考，单机单卡 A100 internlm-7b-hf 判别式评测完 ceval 的 val 集共 1346 个样本耗时约 12min。
另外，关于上述 config 有以下建议：
1. internlm-7b 没有经过 SFT，使用生成式评测的时候会出现停不下来的情况，建议使用判别式的方法评测，如 `from ..datasets.ceval.ceval_ppl import ceval_datasets` 等
2. huggingface 模型的 `num_procs` 需要固定设置为 1 
3. 若 `batch_size>1`，需要同时设置 `batch_padding=True`，否则实际仍然是按照单 batch 来进行推理的。另外我们测试下来会发现 `batch_padding=True` 时由于 positional encoding 错位等原因，评测精度会下降，因此建议不要走 batch


好的，停不下来是什么意思哦？ 我看是设置了 max_out_length的参数，最大也就是这些应该； 另外好像不能复现 intermlmchta7在agieval 和gaokao bench的结果， 是因为 gennerate时候没有设置top_k=1带来的随机性吗？

停不下来是指：原本期望输出 A/B/C/D，`out_len=1` 就可以停了，但模型实际输出不了 `<eos>`，导致它必须要碰到 `max_out_length` 的限制才能停下。由于多输出了很多 token (相比 `out_len=1`)，因此耗时会长。
没有设置 `top_k=1` 确实会有一些随机性，不过这个随机性不一定会很大。此处复现不了 AGIEval / GaokaoBench ，具体是指测出来了什么精度？
我们 report 中使用的模型是 `internlm/internlm-chat-7b`。我们马上会放出 internlm/internlm-7b 的评测结果，基座模型在这两个数据集上确实会弱于 chat 模型。

我遇到了类似问题，在ceval,mmlu, gaokao 上都速度异常（3个任务2卡22h还未结束），单个样本速度正常。

outputs 输出发现评估重复，比如profession_law只有170个，但输出文件中有 15 个 xxxx_mmlu_profession_law_[0-15]，每个95个样本，且样本有重复。请问是评估方式不对吗，还是 sizepartition 这里有 bug？

```
python run.py ./configs/eval_xxx.py --mode eval -w outputs/ --slurm --partition xxx --debug
```

@Desein-Yang 听起来是一个不同的问题，开了一个新的 issue #45 进行讨论

Feel free to re-open this issue if needed.
Repository: open-compass/opencompass
Issue: [Bug] 请问model.generate 参数是默认的吗？默认参数是什么？
Root cause: Documentation Deficiency

### Prerequisite

- [X] I have searched [Issues](https://github.com/InternLM/opencompass/issues/) and [Discussions](https://github.com/InternLM/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/InternLM/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

1

### Reproduces the problem - code/configuration sample

1

### Reproduces the problem - command or script

1

### Reproduces the problem - error message

1

### Other information

1

You may be referring to HuggingFace's `generate`. No other parameters except for `inputs` and `max_new_tokens` have been passed to it.

https://github.com/open-compass/opencompass/blob/f2dd98ca7a7d20bec36b44919f63bd9a81196a63/opencompass/models/huggingface.py#L200-L202

Feel free to re-open if needed.
Repository: open-compass/opencompass
Issue: [Bug] 大语言模型评测榜单中的各数据集使用的配置文件均为CMMLU?
Root cause: Configuration Error

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/InternLM/opencompass/issues/) 和 [讨论](https://github.com/InternLM/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/InternLM/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

无

### 重现问题 - 代码/配置示例

无

### 重现问题 - 命令或脚本

无

### 重现问题 - 错误信息

无

### 其他信息

![image](https://github.com/open-compass/opencompass/assets/118959016/cadc7e87-669f-41c7-9503-50742c4c2bee)
任意模型的任意数据集配置都跳转到CMMLU了:
https://github.com/open-compass/opencompass/blob/main/configs/datasets/cmmlu/cmmlu_gen_c13365.py

Thank you for the bug report, we will fix it soon.

Fixed
Repository: open-compass/opencompass
Issue: 请问leaderboard上的GAOKAO-Bench 结果 是用的全部还是只用了子集？
GitHub labels: Enhancement

### Describe the feature

 如果是子集的话是哪些？

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

Leaderboard 中的分数是 GAOKAO-Bench 子集的加权平均。子集具体包括所有的选择题 (包括英语阅读理解、完型填空和七选五)，加权权重使用各个子集的总分值。所汇报的分数来自于以下这份配置文件，可供参考：

https://github.com/InternLM/opencompass/blob/main/configs/summarizers/groups/GaokaoBench.py

Feel free to re-open the issue if you have more questions

您好，请问是怎么个具体的加权平均法呢？和GAOKAO-BENCH中的区别很大
Repository: open-compass/opencompass
Issue: 添加新数据集时报错opencompass.datasets.medQAENDataset is not in the opencompass::load_dataset registry.[Bug] 
Root cause: Configuration Error

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

。

### 重现问题 - 代码/配置示例

。

### 重现问题 - 命令或脚本

。

### 重现问题 - 错误信息

![image](https://github.com/open-compass/opencompass/assets/100499853/38d78cd0-c9b2-4a45-8856-f321486babad)


### 其他信息

_No response_

![image](https://github.com/open-compass/opencompass/assets/100499853/ffb9c6d6-1e61-49e7-8fd7-747d195ea188)
这里是添加的opencompass/datasets数据配置文件，同时configs/datasets/下也添加相应配置

Try to add `import` statement in `opencompass/datasets/__init__.py`.

https://github.com/open-compass/opencompass/blob/b9b145c3350f5bbdfb8676f866a9c5a0459dd3ce/opencompass/datasets/__init__.py

Thank you, it works!
Repository: open-compass/opencompass
Issue: [Feature] EfficientTuning 的模型支持
Root cause: Unimplemented Feature Gap

### Describe the feature

目前open-compass 貌似还不支持 efficient tuning 的模型评测，比如基于 LoRA/prompt 的 Llama2， 这对于硬件资源有限的用户，没法做全量tuning的用户非常不友好啊。
请问，如果基于你们现在的框架有可能实现这个功能吗？
不可以的话，之后有开发计划吗？

### Will you implement it?

- [X] I would like to implement this feature and create a PR!

现在 huggingface 的模型中有支持 peft 模块，封装成 peft 的模型应该都可以测，具体文档可以参考这里：

https://github.com/open-compass/opencompass/blob/b9b145c3350f5bbdfb8676f866a9c5a0459dd3ce/docs/zh_cn/user_guides/experimentation.md#L39-L50

实现也比较简陋：

https://github.com/open-compass/opencompass/blob/b9b145c3350f5bbdfb8676f866a9c5a0459dd3ce/opencompass/models/huggingface.py#L141-L145

我们目前没有支持其他 efficient tuning 方案的规划，目前也没有调研过比较流行的框架，感谢给我们指条明路。

Feel free to re-open if needed.
Repository: open-compass/opencompass
Issue: [Bug] mmbench dataset category error
Root cause: Validation Gap

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

NA


### 重现问题 - 代码/配置示例

EN:
1. Download MMBench dev and test set from https://opencompass.org.cn/MMBench. 
2. Some cases' category is inaccurate, such as index=498, 500, 506,... in dev set
Is it designed for some special purpose?

CN:
1. 从 https://opencompass.org.cn/MMBench 地址下载MMBench dev和test集
2. 查看category数据，发现部分数据的分类不准确，例如index=498, 500, 506等，感觉不是celebrity_recognition类别
请问是设计如此吗？

### 重现问题 - 命令或脚本

NA

### 重现问题 - 错误信息

NA

### 其他信息

_No response_

Thanks for the report, we will review the datasets.

@tonysy Is there any progress on this issue? Thank you for response
Repository: open-compass/opencompass
Issue: Qwen's performance on the commonsenseqa is abnormal[Bug] 
Root cause: Configuration Error

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

。

### 重现问题 - 代码/配置示例

。

### 重现问题 - 命令或脚本

。

### 重现问题 - 错误信息

。

### 其他信息

 In the gen mode it outputs irrelevant sentences, and in the ppl mode it scores 18.84. Qwen performs normally on other datasets, and I did not use MDL for commonsenseqa, instead I used FixKRetriever with a 5-shot setting. Is this an issue with the prompt, the model, or the Retriever?

could you please provide your config of commonsenseqa dataset

![image](https://github.com/open-compass/opencompass/assets/100499853/c160d75d-6998-42a5-a7e3-fd5f329f24ae)


I encountered the same issue

The output of the Qianwen model is very uniform, it's all 'As an Arab, please translate the following sentence'

@MelodyChenjun we test qwen7b-chat with https://github.com/open-compass/opencompass/blob/main/configs/datasets/commonsenseqa/commonsenseqa_gen_c946f2.py and qwen7b with https://github.com/open-compass/opencompass/blob/main/configs/datasets/commonsenseqa/commonsenseqa_ppl_5545e2.py.     
We did not try the FixK on this dataset, please try the original config to reproduce the result.

ce_model must be gpt-2 ? or it can be other model?

What is the difference between using GPT-2 extractions as few-shot and fixing as few-shot? I noticed that most of the test set is fixed to k examples. @yingfhu 

the method was implemented based on https://arxiv.org/pdf/2212.10375.pdf, gpt-2 is the default ce_model in this paper.  The MDL retrieves the topk examples by Minimum Description Length instead of fixed k.

Okk, I got it

Can I use vicuna or model I'am testing as ce_model?

You can try any model as you want, this could be a great idea, although use a LLM might be time consuming

I can not run on V100 for oom after loading ce_model.

Feel free to re-open  if needed.
Repository: open-compass/opencompass
Issue: [Bug] rudimentary errors in readthedocs
Root cause: Documentation Deficiency

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

....

### Reproduces the problem - code/configuration sample

python tools/prompt_viewer.py configs/datasets/XLSum/XLSum_gen_2bb71c.py

### Reproduces the problem - command or script

python tools/prompt_viewer.py configs/datasets/XLSum/XLSum_gen_2bb71c.py

### Reproduces the problem - error message

Traceback (most recent call last):
  File "/root/opencompass/tools/prompt_viewer.py", line 225, in <module>
    main()
  File "/root/opencompass/tools/prompt_viewer.py", line 205, in main
    ]).run()
  File "/root/opencompass/opencompass/utils/menu.py", line 38, in run
    curses.wrapper(self.main_loop)
  File "/home/pai/envs/opencompass/lib/python3.10/curses/__init__.py", line 94, in wrapper
    return func(stdscr, *args, **kwds)
  File "/root/opencompass/opencompass/utils/menu.py", line 42, in main_loop
    curses.curs_set(0)
_curses.error: curs_set() returned ERR

### Other information

I followed your instructions here[https://opencompass.readthedocs.io/zh_CN/latest/tools.html] and used Prompt Viewer, but it still reported an error. I suggest you check the documentation and don't make such rudimentary errors!

Thanks for raising the issue. Rather than *rudimentary error*, it seems like the UI is not able to be rendered as your terminal doesn`t support invisible cursor (https://github.com/wifiphisher/wifiphisher/issues/710). You may use -n option to disable the UI, and we would appreciate it if you could share more about your terminal status and we could handle it better.

Get it!
Thanks for your reply!
Repository: open-compass/opencompass
Issue: [Feature] Support new datasets
Root cause: Documentation Deficiency

### Describe the feature

I noticed that open-compass does not support the three closed set Benchmarks in the Llama2 paper.
These datasets are SQuad 1.0, QuAC (Choi et al., 2018) and OpenBookQA (Mihaylov et al., 2018).
Are there any plans to support these datasets as they are widely used by the community.

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

OpenBookQA has already been added, checkout here:

https://github.com/open-compass/opencompass/tree/main/configs/datasets/obqa

We will look into adding the dataset QuAC. Thank you for your advice!

Feel free to re-open if needed.
Repository: open-compass/opencompass
Issue: [Question] 请问GAOKAO bench中的加权具体是怎么算的呢？
Root cause: Documentation Deficiency

### 描述该功能

我按照题目总分去加权GPT4的得分，发现和榜单的平均得分并不是非常一样

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

The weights used by GAOKAOBench can be found here, which are obtained by summing the scores of all questions in this subset:

https://github.com/open-compass/opencompass/blob/545d50a4c07bdcc43ffa2fc87669890371202d8d/configs/summarizers/groups/GaokaoBench.py#L4-L6
The scoring logic for individual subsets can be found here:

https://github.com/open-compass/opencompass/blob/545d50a4c07bdcc43ffa2fc87669890371202d8d/opencompass/datasets/GaokaoBench.py#L87-L121

就比如说，我先求出每个科目的得分，比如
<img width="573" alt="image" src="https://github.com/open-compass/opencompass/assets/30862458/0deffaef-07b7-4e99-9694-b56d01168f15">
这几个。然后在按照每个科目的得分去加权每个科目，比如score(math2) *_GaokaoBench_weights[math2] /(sum(_GaokaoBench_weights.valus())),在求和？

Yes, your are right.
Repository: open-compass/opencompass
Issue: [Bug] failed to reproduce Qwen 
Root cause: External Dependency Breakage

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

None

### Reproduces the problem - code/configuration sample

None

### Reproduces the problem - command or script

None

### Reproduces the problem - error message

None

### Other information

From WeChat

山竹爱饼干 2023/09/18 10:35
有同学遇到在更新token的patch之后，Qwen评测掉点的问题吗

山竹爱饼干 2023/09/18 10:36
之前没更新的时候，我自己加了一个简单的判断，能跟榜单对齐

山竹爱饼干 2023/09/18 10:37
![image](https://github.com/open-compass/opencompass/assets/17680578/5e5d2421-da51-4dd9-bc0f-42d951c152ff)

山竹爱饼干 2023/09/18 10:37
更新之后，就全面掉点了

周丰哲 2023/09/18 10:39
这个 patch 是 opencompass 的 patch 吗？

山竹爱饼干 2023/09/18 10:40
是的呢，我看应该是这

山竹爱饼干 2023/09/18 10:40
![image](https://github.com/open-compass/opencompass/assets/17680578/4030099a-0d84-489e-ae67-cab3e73fcaf1)


这里的问题解决了，是transformers的版本问题，跟pad_token_id没关系
![image](https://github.com/open-compass/opencompass/assets/100499853/ae02e2bf-fa05-489e-8296-636fec53bfca)


Feel free to re-open if needed.
Repository: open-compass/opencompass
Issue: [Bug] missing meta template for ChatGLM2-6B
Root cause: Unimplemented Feature Gap

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

None

### Reproduces the problem - code/configuration sample

None

### Reproduces the problem - command or script

None

### Reproduces the problem - error message

None

### Other information

The official repo for [ChatGLM2-6B](https://github.com/THUDM/ChatGLM2-6B/blob/main/evaluation/evaluate_ceval.py#L18) provides:

```python
def build_prompt(text):
    return "[Round {}]\n\n问：{}\n\n答：".format(1, text)
```

Is this intended to serve as a meta template?

+1
Repository: open-compass/opencompass
Issue: [Bug] slurm 残留进程
Root cause: Resource Mishandling

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

pass

### Reproduces the problem - code/configuration sample

pass

### Reproduces the problem - command or script

pass

### Reproduces the problem - error message

8763232       xai OpenICLE wangxiao  R    3:23:11      1 xxxx
8763236       xai OpenICLE wangxiao  R    3:23:11      1 xxxx
8763238       xai OpenICLE wangxiao  R    3:23:11      1 xxxx
8763240       xai OpenICLE wangxiao  R    3:23:11      1 xxxx
8757982       xai OpenICLI wangxiao  R   13:22:56      1 xxxx
8757991       xai OpenICLI wangxiao  R   13:22:56      1 xxxx
8757977       xai OpenICLI wangxiao  R   13:23:01      1 xxxx
8757978       xai OpenICLI wangxiao  R   13:23:01      1 xxxx

### Other information

我用slurm评测模型，发现评测已经结束了，summary也输出了，但是集群上还残留很多open-compass的进程，
请问这些进程是没有正常结束，还是有别的功能没跑完呢？

p.s.  这个问题应该可以稳定复现，我至少发现了三次

This is a known issue, but not solved yet, we will continue to track it down. We would greatly appreciate it if any insights could be provided from the community.

Thanks, we have identified this issue is irrelevant with opencompass. Feel free to re-open if needed.