Repository: open-compass/opencompass
Issue: [Bug] HLE推理索引错误，hle wrong index prediction
Root cause: Configuration Error

### 先决条件

- [x] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [x] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

.

### 重现问题 - 代码/配置示例

.

### 重现问题 - 命令或脚本

python run.py --models qwen3_1_7b --datasets hle_llmverify_academic

### 重现问题 - 错误信息

.

### 其他信息

<img width="675" height="320" alt="Image" src="https://github.com/user-attachments/assets/1105407c-254d-4e34-b56a-80c2d772c0e5" />

The opencompass leaderboard 'HLE was extremely low, I suspect it was wrong too. see the pic. The prompt was not put into the model correctly, it was an off-by-one错误或者数据对齐问题

Is the screenshot you provided from the `predictions` folder or the `results` folder? In addition, it seems that `qwen3_1_7b` is not in the main branch code of OpenCompass. Please provide your detailed model configuration.

it was from predictions, and my understanding is that the results folder
are built based on predictions folder.

here are the two files（one is capture） about my model configuration. Is
there any wrong? actually it might be these two's problems. For
opencompass's provided datasets, i can correctly get like 80% but 20% would
have those like index 错位 problems. Thanks for your help.

[image: image.png]



Myhs_phz ***@***.***> 于2025年10月24日周五 17:03写道：

> *Myhs-phz* left a comment (open-compass/opencompass#2295)
> <https://github.com/open-compass/opencompass/issues/2295#issuecomment-3441969810>
>
> Is the screenshot you provided from the predictions folder or the results
> folder? In addition, it seems that qwen3_1_7b is not in the main branch
> code of OpenCompass. Please provide your detailed model configuration.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/open-compass/opencompass/issues/2295#issuecomment-3441969810>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/BCJJFXZGHVCE7MFDK6YNSRL3ZHTO3AVCNFSM6AAAAACKCWR25SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTINBRHE3DSOBRGA>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>


It seems unable to obtain the image from your reply. Please resend it through a form other than e-mail.

can i have your wechat? this is a vital problem to me i want to try to fix it.

Now, I am encountering this off-by-one indexing problem for every single
dataset on opencompass. First, I edited the code in icl_gen_inferencer by
ai because it says 'ValueError: not enough values to unpack (expected 2,
got 1)', and then no matter how I trying to fix it, it simply would go for
the 1,3,5... index of prompt being an answer like 'A', 'B' .



Myhs_phz ***@***.***> 于2025年10月27日周一 16:18写道：

> *Myhs-phz* left a comment (open-compass/opencompass#2295)
> <https://github.com/open-compass/opencompass/issues/2295#issuecomment-3450009574>
>
> It seems unable to obtain the image from your reply. Please resend it
> through a form other than e-mail.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/open-compass/opencompass/issues/2295#issuecomment-3450009574>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/BCJJFX57CDSI4DJSRPSUNLT3ZXIODAVCNFSM6AAAAACKCWR25SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTINJQGAYDSNJXGQ>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>


did all these issues occur after you modified the code? was there an error before making the modification? if so, will there be errors for all different model configs? In addition, it seems that read-write conflicts may also contribute to this phenomenon. For the former, please provide your specific model config in the code snippet for us to reproduce. For the latter, please carefully check your work directory and duplicate tasks.

I have fixed this issue, it was my model config lacking the meta-template
or the batch-size, I have not found out which was the reason or maybe both
were. However now I followed the official way of doing it and it got fixed.
I was thinking batch size was useless so i deleted it , but i made a
mistake.
Thanks a lot anyway

Myhs_phz ***@***.***> 于2025年10月28日周二 14:45写道：

> *Myhs-phz* left a comment (open-compass/opencompass#2295)
> <https://github.com/open-compass/opencompass/issues/2295#issuecomment-3454869901>
>
> did all these issues occur after you modified the code? was there an error
> before making the modification? if so, will there be errors for all
> different model configs? In addition, it seems that read-write conflicts
> may also contribute to this phenomenon. For the former, please provide your
> specific model config in the code snippet for us to reproduce. For the
> latter, please carefully check your work directory and duplicate tasks.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/open-compass/opencompass/issues/2295#issuecomment-3454869901>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/BCJJFX62XCVUXVYCRHH3OET3Z4GI7AVCNFSM6AAAAACKCWR25SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTINJUHA3DSOJQGE>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>


Feel free to re-open if needed.