zhiminy/EvalEng huggingface dataset:
Repository: open-compass/VLMEvalKit
Issue: Enhancing Multi-Choice Question Handling with Case-Sensitive Matching
Root cause: Unimplemented Feature Gap

It might be beneficial to implement a mechanism for exact matching of uppercase option letters in multi-choice questions. This could help avoid confusion caused by the presence of quantifiers like "a" in responses, which might be mistakenly interpreted as indicating multiple choices. Additionally, in cases where multiple letters or multiple instances of "yes"/"no" appear, the system could prioritize the analysis of the first word in the sentence to determine the intended response.

I am also curious about whether the scores currently displayed on the OpenCompass leaderboard have been updated to reflect these latest modifications. Could you provide any information on this?

Hi, @jdy18 ,
For the ideas you proposed,
1. Upper case matching: We are not going to implement it, since the choice labels can also appear as lowercase letters, while quantifiers like "a" can also appear as uppercase letters (when it appears at the beginning of a sentence). Thus implementation may lead to noisy evaluation results. So it would be better to employ the LLM-based matching for these ambiguous cases.
2. Same, the first word may not indicate the actual yes / no label.
3. Yes.
Repository: open-compass/VLMEvalKit
Issue: Are there any problems with using vlmeval for the model evaluation of qwen2/2.5?
Root cause: Configuration Error

I noticed that in the model Settings of config, all QwenModels have set the default value of minpixel. This value is relatively high. Will this have an impact on some benchmarks with lower image resolutions? For example, the performance improvement/decrease caused by the resize of pictures

I guess the reason why `min_pixels=1280*28*28` is this is configured by [qwen2.5 evaluation code](https://github.com/QwenLM/Qwen2.5-VL/blob/d2240f11656bfe404b9ba56db4e51cd09f522ff1/evaluation/mmmu/run_mmmu.py#L53C20-L53C30)

If the config is too high, you can change the `min_pixels` and `max_pixels` based on setting.
Repository: open-compass/VLMEvalKit
Issue: 除了输入image的路径之外，能否将image的tensor作为model.generate()方法的输入？
Root cause: Unimplemented Feature Gap

除了输入image的路径之外，能否将image的tensor作为model.generate()方法的输入？需要做哪些修改？

你好，支持image_tensor作为输出比较复杂，你可以看一下 `vlmeval/inference/py:infer_data`函数，这个函数就介绍了怎么从model, dataset一步步构建prompt，然后调用`model.generate()`方法生成答案的
Repository: open-compass/opencompass
Issue: [Bug] AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
Root cause: External Dependency Breakage

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

ubuntu 22.04

### Reproduces the problem - code/configuration sample

conda create --name opencompass python=3.10 pytorch torchvision pytorch-cuda -c nvidia -c pytorch -y

### Reproduces the problem - command or script

```bash
1. conda create --name opencompass python=3.10 pytorch torchvision pytorch-cuda -c nvidia -c pytorch -y
2. git clone https://github.com/open-compass/opencompass.git
3. cd opencompass
4. pip install  -e .
```

### Reproduces the problem - error message

AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?

### Other information

Please  upgrade python=3.11 when install opencompass, use this command with conda:

conda create --name opencompass python=3.11 pytorch torchvision pytorch-cuda -c nvidia -c pytorch -y


Please provide the detailed config you use
Repository: open-compass/opencompass
Issue: [Feature] Utilizing an Alternate Model for Model Evaluation
Root cause: Documentation Deficiency

### Describe the feature

I currently have a model specifically designed for evaluating answers. During the evaluation phase, is it possible to support the use of an additional model which requires GPU utilization to validate the inference results of one model?

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

Sure, we support use any model as judge model to evaluate your results, for example, in : https://github.com/open-compass/opencompass/blob/02e7eec91131cec2050a0bfee3eb07f37298491f/configs/eval_subjective_judge_pandalm.py#L66
you can change the ```judge_model``` to your model
Repository: open-compass/opencompass
Issue: [Bug] 为什么评估的humaneval分数比使用bigcode-evaluation-harness的分数低？
Root cause: Configuration Error

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

None

### 重现问题 - 代码/配置示例

None

### 重现问题 - 命令或脚本

使用opencompass评测了deepseek-code-1.3B-base模型，在humaneval上的pass@1的分数多次运行仅有约22分左右，但是相同的模型参数下使用bigcode-evaluation-harness却能获得32分左右，接近论文的评测分数、请问是有Bug吗？为什么会出现这样的问题？

### 重现问题 - 错误信息

None

### 其他信息

_No response_

Could you provide the config you used (both deepseek-code-1.3B-base config and HumanEval config used) ?

```
from mmengine.config import read_base
from opencompass.models import HuggingFaceCausalLM


with read_base():
    from .datasets.humaneval.humaneval_gen_8e312c import humaneval_datasets

datasets = []
datasets += humaneval_datasets

models = [
    dict(
        type=HuggingFaceCausalLM,
        abbr='deepseek-coder-1.3b-base-hf',
        path="deepseek-ai/deepseek-coder-1.3b-base",
        tokenizer_path='deepseek-ai/deepseek-coder-1.3b-base',
        model_kwargs=dict(
            device_map='auto',
            trust_remote_code=True,
        ),
        tokenizer_kwargs=dict(
            padding_side='right',
            truncation_side='left',
            trust_remote_code=True,
            use_fast=False,
        ),
        max_out_len=1024,
        max_seq_len=1024,
        batch_size=1,
        run_cfg=dict(num_gpus=1, num_procs=1),
        generation_kwargs=dict(
            num_return_sequences=1,
            do_sample=True,
            top_p=0.95,
            top_k=0,
            temperature=0.2,
        ),
    )
]
````

Why are you evaluating a base model using a chat datasets config? Maybe you want to use the config in 'configs\datasets\humaneval\humaneval_gen_d2537e.py'.
In addition, is using greedy decoding a better choice when using the Pass@1 metric to evaluate the model's performance on HumanEval?

How can I differentiate between base and chat datasets config? A PromptTemplate with the term `round` might be related to a chat scenario？

I did not use greedy decoding. In addition, do you think the sampling strategy is the key to affecting performance?

Is there a recommended configuration (datasets and model) to achieve better performance?

- The round term is truly used for only chat config.
- If you only using Pass@1, greedy decoding could make your results reproducible. But sampling may not affect performance to some extent.
- The default config usually works well, but there is no guarantee.
Repository: open-compass/opencompass
Issue: [Feature] Support PromptCBLUE
Root cause: Unimplemented Feature Gap

### Describe the feature

Is there any plan to support PromptCBLUE, a Chinese medical LLM evaluation benchmark?

https://github.com/michael-wzhu/PromptCBLUE 

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

Welcome to contribute this bench into OpenCompass
Repository: open-compass/opencompass
Issue: [Feature] 请问chatglm目前支持吗，是否有相关文档参考
Root cause: Documentation Deficiency

### Describe the feature

自己试了下，报错蛮多的，想问下是否有官方文档支持清华chatglm系列

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

> 已解决，感谢
Repository: open-compass/opencompass
Issue: [Bug] Long text evaluation parameters are not clear
Root cause: Documentation Deficiency

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

python 3.10.1
OpenCompass 0.2.3
vllm 0.2.3

### Reproduces the problem - code/configuration sample

configs/models/chatglm/vllm_chatglm2_6b_32k.py
from opencompass.models import VLLM

models = [
    dict(
        type=VLLM,
        abbr='chatglm2-6b-32k-vllm',
        path='THUDM/chatglm2-6b-32k',
        max_out_len=512,
        max_seq_len=4096,
        batch_size=32,
        generation_kwargs=dict(temperature=0),
        run_cfg=dict(num_gpus=1, num_procs=1),
    )
]

### Reproduces the problem - command or script

python run.py --model vllm_chatglm2_6b_32k --datasets longbench leval

### Reproduces the problem - error message

The difference between the evaluation result parameters and the document long text evaluation is about 20 points, The score for the document can not be reproduced. 
1. “max_seq_len、max_out_len” Should these two parameters be modified in any way?



### Other information

_No response_

For optimal performance, it is advisable to configure the `max_seq_len` parameter to the highest value feasible, such as 32768 or even higher if possible. As for the `max_out_len`, it typically comes with a preset default value within the dataset configuration. You have the option to adjust this to 256, or you may simply retain the default setting.

Thank you very much. I reproduced most of the scores.

I also need to ask, indicators for rouge1, rouge2,rougeL,rougeLsum subset of the score difference is still very large.
1. What is the reason wow?
2. What are the indicators used in the rank?

![image](https://github.com/open-compass/opencompass/assets/46083912/02cb5ddd-a657-4409-a3be-9eced9507b62)

![image](https://github.com/open-compass/opencompass/assets/46083912/200a6a87-08e8-416d-b2d8-c4e1eac0dce1)


@liushz
Repository: open-compass/opencompass
Issue: [Bug] 使用NumWorkerPartitioner切分推理层和验证层任务后，最总结果指标无法将数据集结果汇总
Root cause: Unimplemented Feature Gap

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我修改了代码（配置不视为代码），或者我正在处理我自己的任务/模型/数据集。

### 环境

opencompass               0.2.1

### 重现问题 - 代码/配置示例

infer = dict(
    partitioner=dict(type=NumWorkerPartitioner, num_worker=8),
    runner=dict(
        type=LocalRunner,
        task=dict(type=OpenICLInferTask),
        max_num_workers=8
    )
)

eval = dict(
    partitioner=dict(type=NumWorkerPartitioner, num_worker=8),
    runner=dict(
        type=LocalRunner,
        task=dict(type=OpenICLEvalTask),
        max_num_workers=100
    )
)

### 重现问题 - 命令或脚本

python run.py \
    configs/$task_name.py \
    -m infer \
    -w $output_dir \
    --dump-eval-details

### 重现问题 - 错误信息

在工作路径中，对于每个模型需要评估的模型和测试集，每个测试集都被切分成了8份，并计算了各自的指标，但在最终的summarize中，结果展示都为“-”，是否使用了NumWorkerPartitioner后，结果无法汇总？

20240410_104105/results/llama2_7b_iter_0173600/AGIEval_0.json
20240410_104105/results/llama2_7b_iter_0173600/AGIEval_1.json
……
20240410_104105/results/llama2_7b_iter_0173600/AGIEval_7.json

### 其他信息

_No response_

@tonysy 

已经通过自己实现NumWorkerSummarizer解决问题

> 已经通过自己实现NumWorkerSummarizer解决问题

请问您是怎么实现的，可以分享一下代码吗？
Repository: open-compass/opencompass
Issue: [Bug] MMBench测试有bug
Root cause: Resource Mishandling

### 描述该错误

一直卡在evaluating，持续好几个小时。
![image](https://github.com/InternLM/opencompass/assets/21003563/0d564723-2999-4f8a-b633-f1618c6ce2f9)


### 环境信息

None

### 其他信息

_No response_

有维护人员的邮箱吗？

 Thanks for reporting the bug, we will hanle this issue as soon as possible. Email address: opencompass@pjlab.org.cn.

Feel free to re-open this issue if more help needed.
Repository: open-compass/opencompass
Issue: [Feature] 自定义数据集命令如何修改评估指标？
Root cause: Unimplemented Feature Gap

### Describe the feature

python run.py \
    --models hf_llama2_7b \
    --custom-dataset-path xxx/test_qa.jsonl \
    --custom-dataset-data-type qa \
    --custom-dataset-infer-method gen
    
 使用这个命令得到的结果得分默认是accuracy。这意味着要完全相同才能算对么？如何替换成别的评估指标呢？
 通过新增配置文件，学习成本有点高。。。

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!
Repository: open-compass/opencompass
Issue: [Feature] 兼容torch_npu
Root cause: Unimplemented Feature Gap

### Describe the feature

目前看到代码中强依赖于torch.cuda，希望可以更改接口并兼容npu卡，即兼容torch_npu。

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

Thanks for the great suggestions. We will add this feature into our backlog.
Repository: open-compass/opencompass
Issue: [Bug] 找不到LongContext的数据集
Root cause: Documentation Deficiency

### 先决条件

- [X] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [X] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

readme里说支持LongContext的几个数据集，像longbench和LEval，但是完整的数据集并没有看到这俩

### 重现问题 - 代码/配置示例

readme里说支持LongContext的几个数据集，像longbench和LEval，但是完整的数据集并没有看到这俩

### 重现问题 - 命令或脚本

readme里说支持LongContext的几个数据集，像longbench和LEval，但是完整的数据集并没有看到这俩

### 重现问题 - 错误信息

readme里说支持LongContext的几个数据集，像longbench和LEval，但是完整的数据集并没有看到这俩

### 其他信息

readme里说支持LongContext的几个数据集，像longbench和LEval，但是完整的数据集并没有看到这俩
Repository: open-compass/opencompass
Issue: [Feature] Qwen1.5 bench result
Root cause: Documentation Deficiency

### Describe the feature

https://hub.opencompass.org.cn/dataset-detail/C-Eval

this page is too old

I want to know is anyone run qwen1.5 on ceval and got a result.

here is my result

|model|result|
|---|---|
|qwen1.5-7b-chat | 70.67 |
|qwen1.5-32b-chat | 81.47 |
|qwen1.5-72b-chat | 82.3 |

i'm not so confident with these results and i want to check some "official result"

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

你是怎么运行qwen1.5-32b-chat，我在model下qwen文件夹没有找到关于qwen1.5-32b-chat的配置文件

@jinmingyi1998 yes, your result can match with ours.

|model | ceval-dev | ceval-test |
|---|---|---|
|qwen1.5-7b-chat | 70.62 | 68.18 |
|qwen1.5-32b-chat | 81.47 | 80.66 |
|qwen1.5-72b-chat | 82.44 | 81.37 |

Here are some of more results for your reference, only the huggingface ones, only the chat ones:

[summary_20240418_140747.csv](https://github.com/open-compass/opencompass/files/15020595/summary_20240418_140747.csv)

* lcbench is a leetcode-weekly-like dataset, not released yet, please ignore
* prompt of hellaswag, TheoremQA and sanitized_mbpp are still problematic, please take it with caution
* math is under 0-shot setting, not 4-shot as in the leaderboard


> 你是怎么运行qwen1.5-32b-chat，我在model下qwen文件夹没有找到关于qwen1.5-32b-chat的配置文件

按需自行修改config即可。
没有就拿7b的复制一下改下模型路径，num_gpu之类的就行了
Repository: open-compass/opencompass
Issue: [Feature]在model文件中没有找到关于Qwen1.5-32b-chat的配置文件 
Root cause: Unimplemented Feature Gap

### Describe the feature

只有7b，14b，72b。这个怎么办呢？

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

You can change the path with 32b-chat, and we will update the new model config soon.

好的谢谢！我自己尝试把微调后32b的qwen模型转成hf格式，然后运行命令如下python run.py --datasets cmmlu_gen \
    --hf-path /root/autodl-tmp/Qwen1.5-32B-Chat \
    --model-kwargs device_map='auto' \
    --tokenizer-kwargs padding_side='left' truncation='left' trust_remote_code=True \
    --max-seq-len 300 \
    --max-out-len 5 \
    --batch-size 8  \
    --num-gpus 1
  。请问我还需要添加其他参数设置吗？比如模板什么的？
Repository: open-compass/opencompass
Issue: [Feature] Support CLongEval dataset
Root cause: Unimplemented Feature Gap

### Describe the feature

CLongEval is a Chinese benchmark for evaluating long-context LLMs, which is characterized by three key features: (1) Sufficient data volume, comprising 7 distinct tasks and 7,267 examples; (2) Broad applicability, accommodating to models with context windows size from 1K to 100K; (3) High quality, with over 2,000 manually annotated question-answer pairs in addition to the automatically constructed labels.

Github Link: https://github.com/zexuanqiu/CLongEval

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

Welcome to contribute this benchmark into OpenCompass

Feel free to reopen it if needed
Repository: open-compass/opencompass
Issue: [documentation] 文档链接失效
Root cause: Documentation Deficiency

### 描述该错误
首页readme文档：
下面展示了快速安装的步骤。有部分第三方功能可能需要额外步骤才能正常运行，详细步骤请参考[安装指南](https://opencompass.readthedocs.io/zh_cn/latest/get_started.html)。
该链接not found了

### 环境信息

None

### 其他信息

_No response_

感谢汇报，我们会尽快修复。
Repository: open-compass/opencompass
Issue: [Feature] llama3‘s prompt template is a little bit wrong
Root cause: Validation Gap

### 描述该功能

’configs/models/hf_llama/hf_llama3_70b_instruct.py‘
where 
```
_meta_template = dict(
    round=[
        dict(role="HUMAN", begin="
user
\n\n", end="
"),
        dict(role="BOT", begin="
assistant
\n\n", end="
", generate=True),
    ],
)
```
should be
```
_meta_template = dict(
    round=[
        dict(role="HUMAN", begin="
user
\n\n", end="
"),
        dict(role="BOT", begin="
assistant
\n\n", end="
", generate=True),
    ],
)
```

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

Thank you for pointing this out. We will fix it immediately
Repository: open-compass/opencompass
Issue: [Feature] 是否可以对arc-e和arc-c的base模型评估增加25-shot with choice
Root cause: Unimplemented Feature Gap

### 描述该功能

with choice和wo choice差几十个点，希望可以对齐llama3中的评估方式

https://github.com/meta-llama/llama3/blob/main/eval_details.md

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！
Repository: open-compass/opencompass
Issue: [Bug]internLM-7b 跑openai_humaneval数据，在eval阶段跑错
Root cause: Validation Gap

### 描述该错误

控制台打印：
![image](https://github.com/InternLM/opencompass/assets/17903509/a4a8cfbf-0aa3-4594-b455-cfac8857cc05)

outputs/internlm_7b/20230726_232201/logs/eval/internlm-7b-hf/openai_humaneval.out文件的输入为：

![image](https://github.com/InternLM/opencompass/assets/17903509/f67a7972-edf0-4ab7-9e54-0aab6d7b0236)

完整log为：
[openai_humaneval.out.txt](https://github.com/InternLM/opencompass/files/12179423/openai_humaneval.out.txt)

帮忙看下是怎么回事？多谢

### 环境信息

python3.10
cuda11.8

### 其他信息

_No response_

Have you done this in installation of openai/humaneval:
```
Please read the comments in human_eval/execution.py lines 48-57 to understand the potential risks of executing the model generation code. 
If you accept these risks, uncomment line 58 to enable code execution evaluation.
```

please read the comments carefully.
![image](https://github.com/InternLM/opencompass/assets/18586273/7ce1f952-bfbc-4e4f-8541-4c94d39a98f8)
Repository: open-compass/opencompass
Issue: [Feature] 请问在使用VLLM测评模型humaneval时，batch_size 不同导致 测评结果有区别是为什么？
Root cause: Algorithmic Error

### Describe the feature

![image](https://github.com/open-compass/opencompass/assets/97608046/1f85dc02-373a-4d2b-a1da-ca2ce0b153d0)
![image](https://github.com/open-compass/opencompass/assets/97608046/da63e2a7-a0b7-4ee3-9923-7c5ecf0c84b6)
在batch_size 分别为128，64，16的情况下，deepseek 1.3B 的P@1 分别是31.71、30.49、29.27 
请问这是为什么？

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!
Repository: open-compass/opencompass
Issue: 问题咨询：关于ceval数据集以及新增类似ceval数据集的问题
Root cause: Documentation Deficiency

### 描述该错误

1、data/ceval/包含formal_ceval和release_ceval两个文件夹，分别都是什么用途？评测internLM-7b时，使用的是formal_ceval下的数据。使用release_ceval的数据在数据集合配置glm文件夹下，是评测GLM的模型的时候使用release_ceval数据吗？
2、data/ceval/formal_ceval下分别有dev，test，val三个文件夹，CEvalDataset类将这些数据load到了下面变量：
`return DatasetDict({
            'val': val_dataset,
            'dev': dev_dataset,
            'test': test_dataset
        })`
这块后续是如何评测的呢？测试集、开发集，验证集这三类数据如何使用？
按照ceval官方的说法，测试机没有答案，需要将测试结果提交他们网站才能拿到得分，opencompass是如何操作的呢？
3、opencompass是否支持问答题的评测，如果支持，哪个数据集是问答题的例子？

建议opencompass开个论坛，我们可以到论坛里去交流相关技术问题。

感谢~


### 环境信息

无。

### 其他信息

_No response_

1. `formatl_ceval` is the released version of C-Eval, the same as the one downloaded from C-Eval [official](https://cevalbenchmark.com/index.html) . `release_ceval`, on the other hand, is legacy data, which should be deleted. Configuration in the glm folder is out-of-date and buggy, please use `formatl_ceval` in all scenarios.
2. `dev` is used as few shot examples. Scores of 5-shot `dev` and `val` is reported on the [opencompass](https://opencompass.org.cn/leaderboard-llm) official site. `test` is used only for inference mode. As the answer to `val` is not released, the trial to evaluate on `test` will get `0`s.
3. All datasets which have `qa` in the name is probably a question-answer datasets. You may take [TriviaQA](https://github.com/InternLM/opencompass/blob/main/configs/datasets/triviaqa/triviaqa_gen_2121ce.py) & [NatualQuestions](https://github.com/InternLM/opencompass/blob/main/configs/datasets/nq/nq_gen_c788f6.py) as examples.
4. Thank you for your suggestion on the discourse thing. We will look into it.

We just activated the Discussion panel for OpenCompass. You are welcome to discuss general topics then.
Repository: open-compass/opencompass
Issue: 有没有一些能够减少显存的操作，我在推理128k的needlebench时，8张A100仍然显存爆炸[Feature] 
Root cause: Unimplemented Feature Gap

### Describe the feature

下面是我的一些参数配置
from opencompass.models import HuggingFaceCausalLM
from opencompass.models import VLLM
_meta_template = dict(
    round=[
        dict(role="HUMAN", begin="
user
\n\n", end="
"),
        dict(role="BOT", begin="
assistant
\n\n", end="
", generate=True),
    ],
)

models = [
    dict(
        type=HuggingFaceCausalLM,
        abbr="llama-3-8b-instruct-hf",
        path="/opt/218/models/Meta-Llama-3-8B-Instruct-NTK",
        model_kwargs=dict(device_map="auto"),
        tokenizer_kwargs=dict(
            padding_side="left",
            truncation_side="left",
            use_fast=False,
        ),
        meta_template=_meta_template,
        max_out_len=128,
        max_seq_len=122880,
        batch_size=1,
        run_cfg=dict(num_gpus=7, num_procs=1),
        generation_kwargs={"eos_token_id": [128001, 128009]},
        # batch_padding=True,
    )
]

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

You can try LMDeploy
Repository: open-compass/opencompass
Issue: /home/wumao/opencompass-main/opencompass/summarizers/needlebench.py 的sizes是不是少了一个128k  [Bug] [Feature] 
Root cause: Validation Gap

### Describe the feature

导致报错：
Traceback (most recent call last):
  File "/home/wumao/opencompass-main/run.py", line 4, in <module>
    main()
  File "/home/wumao/opencompass-main/opencompass/cli/main.py", line 354, in main
    summarizer.summarize(time_str=cfg_time_str)
  File "/home/wumao/opencompass-main/opencompass/summarizers/needlebench.py", line 540, in summarize
    save_results_to_plots(output_path)
  File "/home/wumao/opencompass-main/opencompass/summarizers/needlebench.py", line 215, in save_results_to_plots
    folder_path = os.path.join(plot_path, dataset_mapping_dict[dataset_abbr])
KeyError: '2needle_en_128k'

### Will you implement it?

- [X] I would like to implement this feature and create a PR!

Thank you for your careful attention to this matter! I have updated the code accordingly in #1143.