zhiminy/EvalEng huggingface dataset:
Repository: open-compass/opencompass
Issue: [Bug]The program appears to be paused.
Root cause: Resource Mishandling

### 先决条件

- [x] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [x] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

python=3.10

### 重现问题 - 代码/配置示例

I attempted to test the performance of a locally deployed DeepSeek-R1-AWQ model on the MMLU-pro dataset.

I used the VLLM framework to deploy the DeepSeek-R1-AWQ model with the following command:


vllm serve /local/DeepSeek-R1-AWQ --max-model-len 65536 --max-num-batched-tokens 65536 --trust-remote-code --tensor-parallel-size 8 --served-model-name deepseek-reasoner --api-key ${api_key} --enable-chunked-prefill --enable-prefix-caching --gpu-memory-utilization 0.95

Model configuration:

from opencompass.models import DeepseekAPI

api_meta_template = dict(round=[
    dict(role='HUMAN', api_role='HUMAN'),
    dict(role='BOT', api_role='BOT', generate=True),
], )

models = [
    dict(
        abbr='Deepseek-R1',
        type=DeepseekAPI,
        path='deepseek-reasoner',
        key='123456789',  # Placeholder key (actual key should be secured)
        url='http://0.0.0.0:8052/v1/chat/completions',
        meta_template=api_meta_template,
        query_per_second=1,
        max_out_len=8192,
        max_seq_len=8192,
        batch_size=8),
]

Other configurations:


from mmengine.config import read_base

with read_base():
    from opencompass.configs.datasets.mmlu_pro.mmlu_pro_0shot_cot_gen_08c1de import \
        mmlu_pro_datasets
    from opencompass.configs.models.deepseek.deepseek_awq import \
        models as deepseek_R1_671b_q4_instruct_model
    from opencompass.configs.summarizers.mmlu_pro import summarizer

datasets = sum([
    v
    for k, v in locals().items() if k.endswith('_datasets') or k == 'datasets'
], [])
models = sum([v for k, v in locals().items() if k.endswith('_model')], [])

work_dir = 'outputs/debug/671b_q4_mmlu_pro'



### 重现问题 - 命令或脚本

Execution command:

python run.py ./opencompass/configs/671b_q4.py

### 重现问题 - 错误信息

When the program first starts running, all GPUs are operating normally with utilization around 95%, and new request tasks can be observed in the VLLM logs. However, after running for some time, the program appears to pause unexpectedly - all GPU utilization drops to 0% and no new request tasks are recorded in the VLLM logs. This is quite strange. I tried setting retry=10 but it didn't resolve the issue.

### 其他信息

_No response_

When I added the --debug parameter, the output content shows:

![Image](https://github.com/user-attachments/assets/0b40cbee-1c17-40f8-9d21-a95129ba3e38)

The interface remains stuck at this state without any changes, and no errors are printed. However, when accessing the locally deployed API through alternative methods, it functions normally.
Repository: open-compass/opencompass
Issue: [Feature] Add F1-Score For Drop Dataset
Root cause: Unimplemented Feature Gap

### Describe the feature

I found that most of the models such as llama use f1-score to evaluate the drop dataset, the official llama 3.1-8b measured a drop score of 50+, and I only scored 18.4 in the opencompass.

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!
Repository: open-compass/opencompass
Issue: [Bug] The AIME2024 data provided by OpenCompass is inconsistent with Hugging Face
Root cause: External Dependency Breakage

![Image](https://github.com/user-attachments/assets/9eae5b0c-5ab5-4bd4-a952-0967201cc294)



@liushz 

Please provide the dataset  url for the right part.

It seems like the right version provided a structural description for the triangle, the original problem and answer are same.

From https://github.com/open-compass/opencompass/blob/main/opencompass/utils/datasets_info

/aime2024": {
        "url":
        "http://opencompass.oss-cn-shanghai.aliyuncs.com/datasets/data/aime.zip",
        "md5": "fbe2d0577fc210962a549f8cea1a00c8",
    },




> It seems like the right version provided a structural description for the triangle, the original problem and answer are same.

On other problems, there are also inconsistent situations that can lead to subtle differences in evaluation results.

> > It seems like the right version provided a structural description for the triangle, the original problem and answer are same.
> 
> On other problems, there are also inconsistent situations that can lead to subtle differences in evaluation results.

![Image](https://github.com/user-attachments/assets/ebfd8032-a7c6-430a-b8a1-f819ed36fba4)

The AIME2024 questions are collected by our own members, and for this question, we believe the question can be answered without the image, because according to the question, there is only one arrangement of these two rectangles​ can fit the requirements, when segment DC is located on the right side of segment EF (i.e., a different configuration from the original diagram), the condition that points A, D, H, G lie on the same circle cannot be satisfied. Consequently, through logical deduction, we can arrive at the original arrangement of the two rectangles.

If you have further questions, feel free to reply.
Repository: open-compass/opencompass
Issue: [Bug] setup.py中python版本>=3.8，但代码使用3.10+才有的的 union types as X | Y，导致3.8，3.9版本运行报错
Root cause: Validation Gap

### Prerequisite

- [x] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [x] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

python3.9
Repository: open-compass/opencompass
Issue: [Bug] c-eval数据集下载报错
Root cause: Configuration Error

### 先决条件

- [x] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [x] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

'opencompass': '0.4.1+'

### 重现问题 - 代码/配置示例

from mmengine.config import read_base
with read_base():
    from opencompass.configs.datasets.ceval.ceval_gen_2daf24 import ceval_datasets  # ceval数据集

datasets = ceval_datasets

### 重现问题 - 命令或脚本

none

### 重现问题 - 错误信息

  File "XXXXX\opencompass\utils\datasets.py", line 90, in download_dataset
    assert dataset_name, f'No valid url for {data_path}!\n' + \
AssertionError: No valid url for ./data/ceval/formal_ceval!
Please make sure  `./data/ceval/formal_ceval` is correct

### 其他信息

_No response_

借楼发一个相同问题，辛苦一并帮忙看看，感谢感谢：

with read_base():
    # 直接从预设的数据集配置中读取所需的数据集配置
    from .datasets.aime2024.aime2024_gen_6e39a4 import aime2024_datasets

报错：
AssertionError: No valid url for ./data/aime.jsonl!
Please make sure  `./data/aime.jsonl` is correct

怎么解决的？

首先下载数据集在本地可以解决. 如下: 
```
# Download dataset to data/ folder
wget https://github.com/open-compass/opencompass/releases/download/0.2.2.rc1/OpenCompassData-core-20240207.zip
unzip OpenCompassData-core-20240207.zip
```
Repository: open-compass/opencompass
Issue: [Feature]
Root cause: Unimplemented Feature Gap

### 描述该功能

现在全模态模型越来越多，考虑单独出一个全模态模型的版单吗

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

Thanks for your insightful suggestion, we will take this idea into consideration.
Repository: open-compass/opencompass
Issue: [Bug] I get aime2024 score 10 points on qwen 2.5 32b instruct
Root cause: Configuration Error

### Prerequisite

- [x] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [x] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

I use the latest vllm 0.8.4 and followed the installation instructions on the opencompass home page.

### Reproduces the problem - code/configuration sample

models = [
    dict(
        type=VLLMwithChatTemplate,
        abbr='qwq-32b',
        path='Qwen2___5-32B-Instruct',
        model_kwargs=dict(tensor_parallel_size=2),
        max_out_len=32768,
        max_seq_len=131072,
        batch_size=8,
        generation_kwargs=dict(temperature=0,max_tokens=32768),
        run_cfg=dict(num_gpus=2),
    )
]


### Reproduces the problem - command or script

[aime2024.json](https://github.com/user-attachments/files/19893230/aime2024.json)

### Reproduces the problem - error message

the final summary gives me score 10 on aime2024 no matter how I run the code. I think it's quite different from the official report which is something 26.7. I don't know why. The max_out_len is set 32768 and max_seq_len=131072. I also set temperature =0 for greedy search. BTW, the output sequence from qwen 2.5-32b-instruct is something short which is weired.


[aime2024.json](https://github.com/user-attachments/files/19893230/aime2024.json)

### Other information

_No response_

which prompt version used for this evaluation?

My result with the same model is 13.33.How to get the prompt version? my opencompass version is 0.4.2
Repository: open-compass/opencompass
Issue: [Feature] Missing config file for LLM as Judge
Root cause: Documentation Deficiency

### Describe the feature

官网提供的五档大模型评测示例文件缺失：
https://opencompass.readthedocs.io/zh-cn/latest/advanced_guides/compassbench_intro.html
请问目前大模型评测方式是有更新吗，我还处于官网所述的只支持AorB回复进行打分的方式
请麻烦提供一下五档打分的评测方式案例，感谢！


### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

Please refer to this for the subjective evaluation https://opencompass.readthedocs.io/zh-cn/latest/advanced_guides/subjective_evaluation.html
Repository: open-compass/opencompass
Issue: [Feature] 是否支持昇腾910B npu 进行多卡评测
Root cause: Unimplemented Feature Gap

### Describe the feature

是否支持昇腾910B的多卡测试

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

If the transformer or pytorch supports multi-npu, it should work also.

> If the transformer or pytorch supports multi-npu, it should work also.如果 transformer 或 pytorch 支持多 npu，它也应该可以工作。

thank you，请问那NPU如何设置多卡并行评测呢，应该设置什么参数，我尝试了设置num_work和num_gpu,但是都只用了NPU0，其它NPU处于 空闲状态

> > If the transformer or pytorch supports multi-npu, it should work also.如果 transformer 或 pytorch 支持多 npu，它也应该可以工作。
> 
> thank you，请问那NPU如何设置多卡并行评测呢，应该设置什么参数，我尝试了设置num_work和num_gpu,但是都只用了NPU0，其它NPU处于 空闲状态

用vllm，或者hf的话v4_33.py那个文件把is_npu_available注释掉改为import torch_npu; from torch_npu.contrib import transfer_to_npu

> > > If the transformer or pytorch supports multi-npu, it should work also.如果 transformer 或 pytorch 支持多 npu，它也应该可以工作。
> > 
> > 
> > thank you，请问那NPU如何设置多卡并行评测呢，应该设置什么参数，我尝试了设置num_work和num_gpu,但是都只用了NPU0，其它NPU处于 空闲状态
> 
> 用vllm，或者hf的话v4_33.py那个文件把is_npu_available注释掉改为import torch_npu; from torch_npu.contrib import transfer_to_npu

请问对于hf的话，是只能在 debug 模式正常使用吗？我不使用 debug模式好像还是只能单卡运行

> > > If the transformer or pytorch supports multi-npu, it should work also.如果 transformer 或 pytorch 支持多 npu，它也应该可以工作。如果 transformer 或 pytorch 支持多 npu，它也应该可以工作。如果 transformer 或 pytorch 支持多 npu，它也应该可以工作。
> > 
> > 
> > thank you，请问那NPU如何设置多卡并行评测呢，应该设置什么参数，我尝试了设置num_work和num_gpu,但是都只用了NPU0，其它NPU处于 空闲状态
> 
> 用vllm，或者hf的话v4_33.py那个文件把is_npu_available注释掉改为import torch_npu; from torch_npu.contrib import transfer_to_npu用 vllm，或者 hf 的话 v4_33.py 那个文件把 is_npu_available 注释掉改为 import torch_npu;从 torch_npu.contrib 导入 transfer_to_npu

请问您在这么设置的时候有没有遇到显存分配不均匀，比如单机8卡，最后一张7卡只停留在加载模型阶段
Repository: open-compass/opencompass
Issue: [Feature] 能否支持qwen3的评测
Root cause: Unimplemented Feature Gap

### Describe the feature

请问能否支持下qwen3的评测

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

OpenCompass can support the evaluation of Qwen3. You can develop the relevant config yourself, or wait for us to merge the Qwen3 config later.

借楼问一下，是否支持DistilQwen2.5-7B，我下载了 PAI/DistilQwen2.5-7B-Instruct，但是执行错误。仅仅替换了模型路径，vllm 后端和hg 后端都出错了

We have supported Qwen3 https://github.com/open-compass/opencompass/pull/2126/files

> 借楼问一下，是否支持DistilQwen2.5-7B，我下载了 PAI/DistilQwen2.5-7B-Instruct，但是执行错误。仅仅替换了模型路径，vllm 后端和hg 后端都出错了

Please try to load this model in transformers first to make sure it can be loaded successfully.

Feel free to re-open if needed.
Repository: open-compass/opencompass
Issue: Please Add average scores for C-Eval 
Root cause: Documentation Deficiency

### 描述该功能

when i test LLMs on C-Eval, i focus more on average score on validation sets, please add this function. And there are also C-Eval Hard datasets, please add this datasets in C-Eval. You can refer to https://github.com/SJTU-LIT/ceval

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

It has been implemented in the summarizer module. Just include this line in your config and rerun your experiment:

https://github.com/InternLM/opencompass/blob/4fc1701209adad832f7fca91640fac34331b8e29/configs/eval_internlm_7b.py#L9

@Leymore We need to have a doc for summarizer.

> @Leymore We need to have a doc for summarizer.

ok, thanks a lot!
Repository: open-compass/opencompass
Issue: [Feature] 测评集测试源文件找不到。
Root cause: Documentation Deficiency

### Describe the feature

我在执行下面命令下载数据集源文件的时候，发现部分测评集无法自动下载源文件，请问如果发生这样的情形是需要自己搜索对应的测评集源文件吗（我有找过，但是其中还是有一部分找不到）？如果是这样怎么确定自己的源文件就是该测评集的测试文件，就是说是正确对应上的。
python run.py --models  model_name--datasets dataset_name --debug --dry-run

### Will you implement it?

- [x] I would like to implement this feature and create a PR!

Please provide the dataset_name you can't download. Thank you! 

下载数据集参考此issue

[leval数据集问题](https://github.com/open-compass/opencompass/issues/1825)

我正在测试全量的测评集，总结完毕后回复。

BTW，已经跑通40+测评集
Repository: open-compass/opencompass
Issue: [Feature] 能否支持ProverBench这个评估呢？
Root cause: Unimplemented Feature Gap

### Describe the feature

https://huggingface.co/datasets/deepseek-ai/DeepSeek-ProverBench

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

We will schedule a merge if necessary. You’re also welcome to submit a PR. Thank you.

Feel free to re-open if needed.
Repository: open-compass/opencompass
Issue: [Bug] MBPP的训练集(train split)为什么是0-10条，在hf上是600-974条
Root cause: Configuration Error

### Prerequisite

- [x] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [x] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

No need

### Reproduces the problem - code/configuration sample

No need

### Reproduces the problem - command or script

No need

### Reproduces the problem - error message

No need

### Other information

![Image](https://github.com/user-attachments/assets/ee40b401-063a-4f34-b486-32e40749f55b)
如图，请问这是 bug 还是正常的？
Repository: open-compass/opencompass
Issue: [Feature] 使用OpenAISDK请求时并行请求
Root cause: Unimplemented Feature Gap

### Describe the feature

假设我有4个显卡，分别使用vllm部署了4个7b的不同的模型，我期望它可以并行请求这些模型。而不是请求完第一个的全部评测，才去请求第二个。

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

OpenAI SDK is agnostic to the inference backend. You can use vllm or sglang to serve different models, then start an OpenCompass task with `models` in config consisting of the details of each model.

> OpenAI SDK 与推理后端无关。您可以使用 vllm 或 sglang 来服务不同的模型，然后使用`models`包含每个模型详细信息的配置启动 OpenCompass 任务。

但是他实质上是串行的。我测试我部署了多个vllm service。分别使用不同的显卡，监控这些vllm服务的日志会发现，他创建了4个task，但是是先执行第一个task全部结束才执行第二个。因此4个vllm服务同时只有一个在更新日志。

> OpenAI SDK 与推理后端无关。您可以使用 vllm 或 sglang 来服务不同的模型，然后使用`models`包含每个模型详细信息的配置启动 OpenCompass 任务。

我尝试了使用opencompass启动离线版的vllm，和我使用vllm serve部署http。均得到了相同的结果。最终我是用docker启动了4个opencompass任务每个对应一个vllm才真正并行执行task

The `--debug` flag will result in the disabling of parallel execution. It is advisable to attempt removing this flag.


> The `--debug` flag will result in the disabling of parallel execution. It is advisable to attempt removing this flag.

SO important info......
Repository: open-compass/opencompass
Issue: [Bug] Seems index is useless
Root cause: Validation Gap

### Prerequisite

- [x] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [x] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

https://github.com/open-compass/opencompass/blob/main/opencompass/runners/local.py#L163

### Reproduces the problem - code/configuration sample

.

### Reproduces the problem - command or script

.

### Reproduces the problem - error message

.

### Other information

_No response_

What are the details of this issue?

```
            def submit(task, index):
                task = TASKS.build(dict(cfg=task, type=self.task_cfg['type']))
                num_gpus = task.num_gpus
                assert len(gpus) >= num_gpus

```
->
```
            def submit(task):  # remove index, it is useless
                task = TASKS.build(dict(cfg=task, type=self.task_cfg['type']))
                num_gpus = task.num_gpus
                assert len(gpus) >= num_gpus
```
Repository: open-compass/opencompass
Issue: [Bug] parse_entries or entry?
Root cause: Documentation Deficiency

### Prerequisite

- [x] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [x] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

.

### Reproduces the problem - code/configuration sample

.

### Reproduces the problem - command or script

.

### Reproduces the problem - error message

<img width="1653" alt="Image" src="https://github.com/user-attachments/assets/c5e23d42-c9ad-450a-99d9-3206e270d8a9" />

https://github.com/open-compass/opencompass/blob/main/opencompass/openicl/icl_inferencer/icl_gen_inferencer.py#L152


Perhaps we should move line 152 to line 156? In its current position, it might confuse readers into thinking we intend to pass parsed_entries into self.model.generate_from_template().

### Other information

_No response_
Repository: open-compass/opencompass
Issue: [Feature] MixtralKit缺少相应的eval_mixtral.py文件
Root cause: Unimplemented Feature Gap

### Describe the feature

在github项目https://github.com/open-compass/MixtralKit中，一直缺少eval_mixtral.py文件，导致项目无法正常实现在openCompass中测评mixtral的功能。项目中也有多个关于此的issue，但一直没有得到回应。
恳求官方能够提供eval_mixtral.py文件，或在openCompass中直接添加该文件。

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

Please consider use the official transformers to evaluate mixtral model.
Repository: open-compass/opencompass
Issue: [Feature] 纯吐槽，Bug太多了，更新不及时！

### 描述该功能

纯发一个issue吐槽一下，太难用了，bug还多。基本没有一个能正常跑下来全流程。要经过无数次debug。
服了，到底还有没有人维护啊！

### 是否希望自己实现该功能？

- [ ] 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！

从0开始，我已经正确跑通40多个了，一把辛酸泪。

Please provide the details of the bug.

。。。。。。 纯纯造bug，用api推理，报错说推理文件不存在。。。。。还没有文档。。。翻代码不如换工具算了

https://opencompass.readthedocs.io/zh-cn/latest/dataset_statistics.html

这个数据集页面怎么是空的？

> 。。。。。。 纯纯造bug，用api推理，报错说推理文件不存在。。。。。还没有文档。。。翻代码不如换工具算了

换什么工具，有推荐的吗

> > 。。。。。。 纯纯造bug，用api推理，报错说推理文件不存在。。。。。还没有文档。。。翻代码不如换工具算了
> 
> 换什么工具，有推荐的吗

用EvalScope或者根据评测集手动实现一个模型调用到评测间的转换，一般评测集起码会提供拿到结果后的评测脚本

> 。。。。。。 纯纯造bug，用api推理，报错说推理文件不存在。。。。。还没有文档。。。翻代码不如换工具算了

有啥好用的推荐吗，为用lm-eval也碰到问题了

> > 。。。。。。 纯纯造bug，用api推理，报错说推理文件不存在。。。。。还没有文档。。。翻代码不如换工具算了
> 
> 有啥好用的推荐吗，为用lm-eval也碰到问题了

比较合理的方案是看目标评测集的输出格式是啥样的，对齐输出格式，然后让大模型给你写一个评测脚本，也就是借助大模型纯手写一套稳定版的，他们这个只是说有这个功能，能不能用不好说

可以试试这个工具，熟悉opencompass的人会非常容易上手：
https://gitee.com/aisbench/benchmark
Repository: open-compass/opencompass
Issue: [Feature] What is the evaluation configuration of the qwen3 model in the leaderboard?
Root cause: Documentation Deficiency

### Describe the feature

I would like to know the qwen3-series evaluation config in this leaderboard [https://rank.opencompass.org.cn/leaderboard-llm-academic/?m=REALTIME](url)
such as is it greedy sampling or t=0.6 the same as the qwen3 recommended
### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

Hi, @KADCA21 
We use the evaluation config as follows:
Do_Sampling=True, Temperature=0.6, TopK=20, TopP=0.95, Max_out_len=32K

> Hi, [@KADCA21](https://github.com/KADCA21) We use the evaluation config as follows: Do_Sampling=True, Temperature=0.6, TopK=20, TopP=0.95, Max_out_len=32K

tx!
Repository: open-compass/opencompass
Issue: [Feature] 能否提供qwen3评测的配置文件
Root cause: Unimplemented Feature Gap

### Describe the feature

目前基于opencompass评测qwen3，发现结果和榜单中的结果无法对齐，能否提供qwen3的配置文件，便于复现结果

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

https://github.com/open-compass/opencompass/issues/2093 

https://github.com/open-compass/opencompass/pull/2126/files
Repository: open-compass/opencompass
Issue: [Bug] NeedleBench - Unable to reproduce the results for Multi-Reasoning
Root cause: Unimplemented Feature Gap

### Prerequisite

- [x] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [x] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I have modified the code (config is not considered code), or I'm working on my own tasks/models/datasets.

### Environment

Not relevant

### Reproduces the problem - code/configuration sample

Not relevant

### Reproduces the problem - command or script

Not relevant

### Reproduces the problem - error message

Not relevant

### Other information

Hi! 
I'm trying to reproduce the NeedleBench results for the multi-reasoning category from this [paper](https://arxiv.org/pdf/2407.11963) and I'm having a hard time. It seems like the seed is not correctly used in the randomness, because the dataset that is created always returns the same 10 needles in the whole dataset. Also, even when fixing that, I tried to run the results with Qwen2.5 7b, and get very good results (>80%) instead of the really bad results (<25%) from the paper. I've tried the 32k version, with various depths, and 10 repeats per file.

Is this a good place to fill this bug? Thank you! 

Hi, I am really sorry for seeing your message so late. I sincerely apologize for the delay.

The core reason for the issue you encountered is that my PR has not been merged yet, so the latest code is not available in the main branch at this moment. You can check the related PR here: https://github.com/open-compass/opencompass/pull/2090

We will do our best to get this new code merged into the main branch as soon as possible.
Again, I am truly sorry for the inconvenience and the late reply. Thank you so much for your patience!


Update (June 18): The PR (#2090) has now been merged into main. You can pull the latest main branch and test it. Thanks for your patience!


Thank you very much for addressing the issue! I'll let you know if I encounter an issue when I try it again.