zhiminy/EvalEng huggingface dataset:
Repository: open-compass/opencompass
Issue: [Bug] compassrank和数据集readme差距很大，且compassrank的配置文件无法获取
Root cause: Documentation Deficiency

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

none

### 重现问题 - 代码/配置示例

none

### 重现问题 - 命令或脚本

none

### 重现问题 - 错误信息

none

### 其他信息

以mmlu、qwen1.5-chat为例，config  [readme](https://github.com/open-compass/opencompass/tree/main/configs/datasets/mmlu)给的是61.79，而compassrank是41.9，两者差距很大，现在compassrank只显示7月份的结果，无法查看对应的config文件，所以这两者的差距主要来源于哪里呢？compassrank对应的config是否可以获取呢？

We have introduced new prompt setting following the openai, you can use the following configs:

```
    ## Core Set
    # ## Examination
    from opencompass.configs.datasets.mmlu.mmlu_openai_simple_evals_gen_b618ea import mmlu_datasets
    from opencompass.configs.datasets.mmlu_pro.mmlu_pro_0shot_cot_gen_08c1de import mmlu_pro_datasets
    from opencompass.configs.datasets.cmmlu.cmmlu_0shot_cot_gen_305931 import cmmlu_datasets
    # ## Reasoning
    from opencompass.configs.datasets.bbh.bbh_gen_4a31fa import bbh_datasets
    from opencompass.configs.datasets.gpqa.gpqa_openai_simple_evals_gen_5aeece import gpqa_datasets
    # ## Math
    from opencompass.configs.datasets.math.math_0shot_gen_393424 import math_datasets
    # ## Coding
    from opencompass.configs.datasets.humaneval.humaneval_gen_8e312c import humaneval_datasets
    # ## Instruction Following
    from opencompass.configs.datasets.IFEval.IFEval_gen_3321a3 import ifeval_datasets
```

Feel free to re-open if needed.

感谢，请问现在7月的compassrank用的是哪个commit版本的代码？我用现在最新的版本的会有些值不一样。另外想问一下这些setting的来源是那里呢？

Would you like to provide the detail information of the performance gap(which dataset and which model)? 
0-shot Cot setting originates from the openai/simpel-eval

> Would you like to provide the detail information of the performance gap(which dataset and which model)? 0-shot Cot setting originates from the openai/simpel-eval

qwen2-7b-instruct
![捕获](https://github.com/user-attachments/assets/b1f59745-07c9-4a62-995e-3803f29b1fd7)


We use LMDeploy as the inference backend.
Repository: open-compass/opencompass
Issue: 您好，请问OpenCompass官网开源榜单中，以下这些数据集分别是0-shot测试还是5-shot测试或者其他X-shot？
Root cause: Documentation Deficiency

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

1

### Reproduces the problem - code/configuration sample

1

### Reproduces the problem - command or script

1

### Reproduces the problem - error message

1

### Other information

MMLU、MMLU-Pro、CMMLU、BBH、GQPA-Dimand、MATH、HumanEval、IFEval

![image](https://github.com/user-attachments/assets/005d3ede-5911-4cdd-a09c-edc11a727709)


Please refer https://github.com/open-compass/opencompass/issues/1434

> Please refer #1434

多谢，请问IFEval数据集评测返回结果中，哪个是IFEval最终得分？
![image](https://github.com/user-attachments/assets/13fc5d24-6f0c-4e12-bf3a-fc8ffd43ff59)


> > Please refer #1434
> 
> 多谢，请问IFEval数据集评测返回结果中，哪个是IFEval最终得分？ ![image](https://private-user-images.githubusercontent.com/20581199/359815031-13fc5d24-6f0c-4e12-bf3a-fc8ffd43ff59.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ0MDE3NzIsIm5iZiI6MTcyNDQwMTQ3MiwicGF0aCI6Ii8yMDU4MTE5OS8zNTk4MTUwMzEtMTNmYzVkMjQtNmYwYy00ZTEyLWJmM2EtZmM4ZmZkNDNmZjU5LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODIzVDA4MjQzMlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWEwODQwZjA0NGNlMzQzY2FmNGQ4ZDZmYWIwMDk1NjNhYzVmNDllZjA5M2FiYWE3MjY2YzVjMDdmNWQ0OWY5MDcmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.1lw_3JBkDcLnptmfNYPIjW-Pw-14b-vYjH3xf-pDEO0)

一般是取第一个作为得分，你看一些paper或者model card里面会说明

> 取第一个作为得分，你

Thanks a lot.
Repository: open-compass/opencompass
Issue: [Feature] 为什么SanitizedMBPPDataset是取7:264这个范围进行评测？
Root cause: Documentation Deficiency

### 描述该功能

![image](https://github.com/user-attachments/assets/fa444049-3a25-4ab4-9061-9eab7710a051)
1. 我在检查mbpp结果的时候发现SanitizedMBPPDataset取了其中257进行评测，但是这部分我在其他评测中很少看到类似的评法，不知道有什么paper说明这个问题吗？

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

The first few examples are left out for the few shot evluation settings. 

那264到387这部分的题目呢？我看evalplus这部分的题目也是在评测中的

> 那264到387这部分的题目呢？我看evalplus这部分的题目也是在评测中的
SanitizedMBPP应该就264道
Repository: open-compass/opencompass
Issue: [Feature] Doc and CLI Improvment. 
Root cause: Documentation Deficiency

### Describe the feature

- Use chat model and gen mode for primary example in Doc
- Can use provide the CLI command for these two functions(list dataset, list model)?  
One example: https://github.com/InternLM/xtuner/blob/main/setup.py 

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!
Repository: open-compass/opencompass
Issue: [Feature] Evaluation efficiency optimization(SciCode)
Root cause: Unimplemented Feature Gap

### Describe the feature

1. Improve the evaluation efficiency for SciCode
2. Reduce the cost for h5 data loading 

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

@HariSeldon0 Please resolve this issue
Repository: open-compass/opencompass
Issue: [Feature] Support evaluation with vllm/lmdeploy openai-compatible api via CLI
Root cause: Unimplemented Feature Gap

### Describe the feature

Support use the following pattern:

```bash
OPENAI_API_KEY=xxxx BASE_URL=xxx opencompass xxxxxxxxx
```

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!
Repository: open-compass/opencompass
Issue: 我想知道如果我做了self consistent，是如何进行众多答案的确定和选择的
Root cause: Documentation Deficiency

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

0

### Reproduces the problem - code/configuration sample

0

### Reproduces the problem - command or script

0

### Reproduces the problem - error message

0

### Other information

主要是针对于gsm8k或者mmlu这一类单独答案的问题，可以指出相关的脚本文件吗
Repository: open-compass/opencompass
Issue: [Feature] Support BigCodeBench
Root cause: Unimplemented Feature Gap

### Describe the feature

BigCode (Hugging Face and ServiceNow Research) released a new large-scale benchmark, BigCodeBench, for code generation with diverse function calls and complex instructions, covering 1140 expert-annotated tasks. It has been officially used by DeepSeek and CodeGeeX4. BigCodeBench is considered a better alternative for HumanEval and other function-level code generation benchmarks (see [here](https://x.com/_philschmid/status/1816514989982908591)).

- [Preprint](https://arxiv.org/abs/2406.15877)
- [GitHub](https://github.com/bigcode-project/bigcodebench)
- [Leaderboard](https://huggingface.co/spaces/bigcode/bigcodebench-leaderboard)


### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

Thanks for the suggestions, would you like to contribute this bench into OpenCompass?

Thanks for asking! I'm quite busy these two months, but I might be able to take a look in October if no one else has the bandwidth to add the bench.
Repository: open-compass/opencompass
Issue: [Bug] A problem when processing the prediciton of mmlu pro
Root cause: Configuration Error

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

1

### Reproduces the problem - code/configuration sample

1

### Reproduces the problem - command or script

1

### Reproduces the problem - error message

1

### Other information

  In the ```mmlu_pro``` dataset, the answer template example is: ```A: Let's think step by step. A characteristic of a ring is R is $n$ if the statement $ka = 0$ for all $a\in 2Z$ implies that $k$ is a multiple of $n$. Assume that $ka = 0$ for all $a\in 2Z$ for some $k$. In particular $2k = 0$. Hence $k=0$ and $n =0$. The answer is (A).```. 
  The answer is at the end, but in ```configs/datasets/mmlu_pro/mmlu_pro_gen_cdbebf.py```， ```pred_postprocessor=dict(type=first_option_postprocess, options='ABCDEFGHIJKLMNOP'),``` . It means extract the first litter as the answer. It should be changed to ```last_option_postprocess```

`first_option_postprocess` uses REGEX to match the "first option". So it will not return the very first character but the first matched answer.
Repository: open-compass/opencompass
Issue: [Bug] RuntimeError: configs is a namespace package, which is invalid for get_install_path
Root cause: Configuration Error

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我修改了代码（配置不视为代码），或者我正在处理我自己的任务/模型/数据集。

### 环境

opencompass configs/api_examples/eval_api_gpt.py

### 重现问题 - 代码/配置示例

11

### 重现问题 - 命令或脚本

11

### 重现问题 - 错误信息

Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/mmengine/utils/package_utils.py", line 55, in get_installed_path
    pkg = get_distribution(package)
  File "/opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py", line 512, in get_distribution
    dist = get_provider(dist)
  File "/opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py", line 386, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py", line 952, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py", line 813, in resolve
    dist = self._resolve_dist(
  File "/opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py", line 854, in _resolve_dist
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'configs' distribution was not found and is required by the application

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/bin/opencompass", line 33, in <module>
    sys.exit(load_entry_point('opencompass', 'console_scripts', 'opencompass')())
  File "/home/yuhui.zuo/opencompass-nomi/opencompass/cli/main.py", line 220, in main
    cfg = get_config_from_arg(args)
  File "/home/yuhui.zuo/opencompass-nomi/opencompass/utils/run.py", line 82, in get_config_from_arg
    config = Config.fromfile(args.config, format_python_code=False)
  File "/opt/conda/lib/python3.10/site-packages/mmengine/config/config.py", line 494, in fromfile
    raise e
  File "/opt/conda/lib/python3.10/site-packages/mmengine/config/config.py", line 492, in fromfile
    cfg_dict, imported_names = Config._parse_lazy_import(filename)
  File "/opt/conda/lib/python3.10/site-packages/mmengine/config/config.py", line 1071, in _parse_lazy_import
    root_path = get_installed_path(package)
  File "/opt/conda/lib/python3.10/site-packages/mmengine/utils/package_utils.py", line 66, in get_installed_path
    raise RuntimeError(
RuntimeError: configs is a namespace package, which is invalid for `get_install_path`

### 其他信息

_No response_

import configs 路径问题

为什么根目录下还有一个opencompass，并且结构和根目录是差不多的，我也遇到了这个问题，请问你是怎么解决的呢？
Repository: open-compass/opencompass
Issue: [Bug] configs/datasets中path有的写的是相对路径，有的是Datasets_ids,会导致后续数据集加载错误，按目前版本都应该是Datasets_ids
Root cause: Configuration Error

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我修改了代码（配置不视为代码），或者我正在处理我自己的任务/模型/数据集。

### 环境

cuda=12.1、torch=2.4、python3.10

### 重现问题 - 代码/配置示例

        OpenFinData_datasets.append(
            dict(
                type=OpenFinDataDataset,
                path='./data/openfindata_release',
                name=_name,
                abbr='OpenFinData-' + _name,
                reader_cfg=dict(
                    input_columns=['question', 'A', 'B', 'C'],
                    output_column='answer'),
                infer_cfg=OpenFinData_infer_cfg,
                eval_cfg=OpenFinData_eval_cfg,
            ))
ARC_c_datasets = [
    dict(
        abbr='ARC-c',
        type=ARCDataset,
        path='opencompass/ai2_arc-dev',
        name='ARC-Challenge',
        reader_cfg=ARC_c_reader_cfg,
        infer_cfg=ARC_c_infer_cfg,
        eval_cfg=ARC_c_eval_cfg,
    )
]

### 重现问题 - 命令或脚本

python run.py --datasets --models 

### 重现问题 - 错误信息

        OpenFinData_datasets.append(
            dict(
                type=OpenFinDataDataset,
                path='./data/openfindata_release',
                name=_name,
                abbr='OpenFinData-' + _name,
                reader_cfg=dict(
                    input_columns=['question', 'A', 'B', 'C'],
                    output_column='answer'),
                infer_cfg=OpenFinData_infer_cfg,
                eval_cfg=OpenFinData_eval_cfg,
            ))
ARC_c_datasets = [
    dict(
        abbr='ARC-c',
        type=ARCDataset,
        path='opencompass/ai2_arc-dev',
        name='ARC-Challenge',
        reader_cfg=ARC_c_reader_cfg,
        infer_cfg=ARC_c_infer_cfg,
        eval_cfg=ARC_c_eval_cfg,
    )
]

### 其他信息

本地加载的方式，path应该都传Dataset_ids，通过Dataset_ids找到地址,而不是相对路径

Thanks for the report, we will fix this issue. Actually, our design is compatible with `id` or relative path. Some datasets need `get_data_path(path, local_mode=True)`.

@tonysy 请问这个问题还有在跟进吗？例如`opencompass/ai2_arc-dev`这个dataset id，似乎已经在huggingface仓库中删除了？在modelscope中也不存在，请问该如何对齐呢？

@tonysy 请问这个问题还有在跟进吗？例如opencompass/ai2_arc-dev这个dataset id，似乎已经在huggingface仓库中删除了？在modelscope中也不存在，请问该如何对齐呢？
Repository: open-compass/opencompass
Issue: [Feature] 多模态模型测试
Root cause: Documentation Deficiency

### 描述该功能

你好，我这边想测试多模态的模型，该如何添加配置

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

测多模态模型是使用这个库哈：https://github.com/open-compass/VLMEvalKit

Feel free to re-open if needed.
Repository: open-compass/opencompass
Issue: [Bug] opencompass/configs/models/hf_llama/vllm_llama_series.py这个文件的第13行和第14行结尾少了逗号
Root cause: Validation Gap

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

python==3.10
vllm==0.5.3.post1
opencompass==0.3.1

### 重现问题 - 代码/配置示例

https://github.com/open-compass/opencompass/blob/95aad6c28263a347a8873244eafcdb238f669025/opencompass/configs/models/hf_llama/vllm_llama_series.py#L2-L17

### 重现问题 - 命令或脚本

CUDA_VISIBLE_DEVICES=2,3,4,5 python run.py configs/eval_needlebench.py --max-num-workers 16

### 重现问题 - 错误信息

opencompass/configs/models/hf_llama/vllm_llama_series.py", line 13, in <module>     ('llama-3.1-8b-instruct-vllm', 'meta-llama/Meta-Llama-3.1-8B-Instruct', 1) TypeError: 'tuple' object is not callable

### 其他信息

_No response_

Thanks for pointing out. Fix in https://github.com/open-compass/opencompass/pull/1486
Repository: open-compass/opencompass
Issue: [Feature] 添加mlc-llm后端支持
Root cause: Unimplemented Feature Gap

### 描述该功能

我注意到opencompass已经为lmdeploy和vllm提供支持，能否支持使用mlc-llm进行推理？
https://github.com/mlc-ai/mlc-llm

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

We can consider adding this to our backlog, @liushz.

> We can consider adding this to our backlog, @liushz.

thanks a lot
Repository: open-compass/opencompass
Issue: [Feature] mtbench基线结果
Root cause: Unimplemented Feature Gap

### Describe the feature

官方可以像其他数据集一样提供一些基线结果和命令，用于参照吗

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

Thank you for your suggestion. We will release a judgellm soon and will provide the baseline performance
Repository: open-compass/opencompass
Issue: [Feature] 咱们oc啥时候能支持下lighteval的评测呀？
Root cause: Unimplemented Feature Gap

### 描述该功能

希望oc可以支持下lighteval方式的评测~~~
1. lighteval在比较小的训练量下能更好的看出metric的变化。

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

Would you like to share the link of lighteval?

链接如下：https://github.com/huggingface/lighteval
https://huggingface.co/datasets/HuggingFaceFW/fineweb/blob/main/lighteval_tasks.py
Repository: open-compass/opencompass
Issue: [Feature] 为什么调整batch_size之后，显存占用并没有提升呢
Root cause: Unimplemented Feature Gap

### Describe the feature

评估模型的时候发现，调整batch_size,gpu显存占用并没有变化，batch_size不起作用是吧

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!
Repository: open-compass/opencompass
Issue: [Bug] qwen的base模型为什么需要generate配置文件呢
Root cause: Validation Gap

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

![image](https://github.com/user-attachments/assets/3caa49ba-8e3b-4fdf-bba6-5e69ab62cb40)
用的官方实验

### Reproduces the problem - code/configuration sample

from opencompass.models import HuggingFaceBaseModel

models = [
    dict(
        type=HuggingFaceBaseModel,
        abbr='qwen-14b-hf',
        path='./models/qwen-14B-base',
        max_out_len=1024,
        batch_size=8,
        run_cfg=dict(num_gpus=4),
    )
]


### Reproduces the problem - command or script

opencompass ./configs/eval_qwen_14b_base.py

### Reproduces the problem - error message

![image](https://github.com/user-attachments/assets/eaf2bf98-5e73-4b21-86c5-aa0cdd453a07)


### Other information

没有

```generation_config.json``` is essential for all HuggingFace LLMs, whether they are Base models or Chat models. 

Feel free to re-open if needed.
Repository: open-compass/opencompass
Issue: 请教下ceval数据集中计算PPL的方法

### 描述该错误
**中文描述：**
opencompass-3715be6/opencompass/models/huggingface.py中的

def _get_ppl(self, inputs: List[str], mask_length: Optional[List[int]] = None) -> List[float]:     outputs, inputs = self.get_logits(inputs) ....... 
这里给你一个input列表，就可以计算出PPL的值；
首先调用get_logits，返回模型预测的token和input token
这个input 就是按照prompt模板组成的prompt，以ceval为例，就是five-shot的例子加上新的题目以及答案是一个label “A”
我的问题是：为啥将这个input给模型，调用模型的前向传播方法，获取output[0]就是模型的预测呢？
“fiveshot + 新的选择题目+answer ：A“ 这个给模型是想让模型预测啥呢？

大模型小白，希望高手赐教。

**English description：**
In the opencompass-3715be6/opencompass/models/huggingface.py file, there is a function _get_ppl that takes a list of inputs and calculates the value of Perplexity (PPL). It first calls the get_logits function to get the model's predicted tokens and input tokens.

The input list provided to the function is constructed based on a prompt template, such as the "ceval" template. In this case, it consists of a five-shot example, a new question, and the answer labeled as "A".

The question is why this input is given to the model and why the output[0] of the model's forward pass is considered as the model's prediction. What is the intention behind providing the input "fiveshot + new question + answer: A" to the model?

As a novice in large models, I would appreciate guidance from experts.

### 环境信息

python3.10

### 其他信息

_No response_

We recommend using English or English & Chinese for issues so that we could have broader discussion.

> We recommend using English or English & Chinese for issues so that we could have broader discussion.

Okay, from now on, I will ask my questions in English.

> Given a model and an input text sequence, perplexity measures how likely the model is to generate the input text sequence.
> As a metric, it can be used to evaluate how well the model has learned the distribution of the text it was trained on.

We are not predicting anything here but measuring the likelihood of generating the sequence,  which is the literal meaning of `perplexity`.
You could refer to https://huggingface.co/docs/transformers/perplexity for more details.

> > Given a model and an input text sequence, perplexity measures how likely the model is to generate the input text sequence.
> > As a metric, it can be used to evaluate how well the model has learned the distribution of the text it was trained on.
> 
> We are not predicting anything here but measuring the likelihood of generating the sequence, which is the literal meaning of `perplexity`. You could refer to https://huggingface.co/docs/transformers/perplexity for more details.

OK， I got it，thank you very much.
Repository: open-compass/opencompass
Issue: Based on the openai service of the local model deployed by vllm, it is hoped that the sampling parameters top_p, etc. can be passed in when the model is initialized
Root cause: Unimplemented Feature Gap

### Describe the feature

In `opencompass/openai.py`, I only see the input parameters of temperature, which results in me being able to only add sampling parameters to each data set, but not in the model initialization. 

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!
Repository: open-compass/opencompass
Issue: [Feature] How to config to support auto awq int4 llama3 model 
Root cause: Unimplemented Feature Gap

### Describe the feature

I generate a quantization auto awq model base on qllm with llama3,how could i test base on opencompass

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!
Repository: open-compass/opencompass
Issue: [Feature] 数据集版本问题
Root cause: Documentation Deficiency

### 描述该功能

请问我们提供的数据集为什么有这么多的版本？是内容有更新还是其他的原因？怎么确定评测一个模型的时候使用哪个版本呢？谢谢
![image](https://github.com/user-attachments/assets/e1107a05-5add-4a63-8680-7e8e3496720d)


### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

Different version means different prompts and settings. You can use the xxx_gen.py as default.

感谢解答。但是我发现运行
```
# 原始vicuna7b
python run.py --datasets mmlu_gen   \
     --hf-path  lmsys/vicuna-7b-v1.5 \
     --w './work_dirs/vicuna7bdebug'  \
     --max-num-worker 8 --batch-size 2
```
对vicuna1.5进行测试，只有43左右的精度，和mmlu_gen中给出的精度差距较大，请问是什么原因呢？
![image](https://github.com/user-attachments/assets/c422b525-0ae9-41db-8d3e-d6ecb99993b6)
![image](https://github.com/user-attachments/assets/ab700f1a-0824-481b-85e9-0cb825957251)