Repository: open-compass/opencompass
Issue: [Bug] Wrong scores on siqa dataset
Root cause: Validation Gap

### Prerequisite

- [x] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [x] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

{'CUDA available': True,
 'CUDA_HOME': '/usr/local/cuda',
 'GCC': 'x86_64-linux-gnu-gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0',
 'GPU 0,1,2,3,4,5,6,7': 'NVIDIA H20',
 'MMEngine': '0.10.7',
 'MUSA available': False,
 'NVCC': 'Cuda compilation tools, release 12.4, V12.4.99',
 'OpenCV': '4.11.0',
 'PyTorch': '2.7.1+cu126',
 'PyTorch compiling details': 'PyTorch built with:\n'
                              '  - GCC 11.2\n'
                              '  - C++ Version: 201703\n'
                              '  - Intel(R) oneAPI Math Kernel Library Version '
                              '2024.2-Product Build 20240605 for Intel(R) 64 '
                              'architecture applications\n'
                              '  - Intel(R) MKL-DNN v3.7.1 (Git Hash '
                              '8d263e693366ef8db40acc569cc7d8edf644556d)\n'
                              '  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
                              '  - LAPACK is enabled (usually provided by '
                              'MKL)\n'
                              '  - NNPACK is enabled\n'
                              '  - CPU capability usage: AVX512\n'
                              '  - CUDA Runtime 12.6\n'
                              '  - NVCC architecture flags: '
                              '-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90\n'
                              '  - CuDNN 90.5.1\n'
                              '  - Magma 2.6.1\n'
                              '  - Build settings: BLAS_INFO=mkl, '
                              'BUILD_TYPE=Release, '
                              'COMMIT_SHA=e2d141dbde55c2a4370fac5165b0561b6af4798b, '
                              'CUDA_VERSION=12.6, CUDNN_VERSION=9.5.1, '
                              'CXX_COMPILER=/opt/rh/gcc-toolset-11/root/usr/bin/c++, '
                              'CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 '
                              '-fvisibility-inlines-hidden -DUSE_PTHREADPOOL '
                              '-DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER '
                              '-DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM '
                              '-DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK '
                              '-DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC '
                              '-Wall -Wextra -Werror=return-type '
                              '-Werror=non-virtual-dtor '
                              '-Werror=range-loop-construct '
                              '-Werror=bool-operation -Wnarrowing '
                              '-Wno-missing-field-initializers '
                              '-Wno-unknown-pragmas -Wno-unused-parameter '
                              '-Wno-strict-overflow -Wno-strict-aliasing '
                              '-Wno-stringop-overflow -Wsuggest-override '
                              '-Wno-psabi -Wno-error=old-style-cast '
                              '-fdiagnostics-color=always -faligned-new '
                              '-Wno-maybe-uninitialized -fno-math-errno '
                              '-fno-trapping-math -Werror=format '
                              '-Wno-stringop-overflow, LAPACK_INFO=mkl, '
                              'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, '
                              'TORCH_VERSION=2.7.1, USE_CUDA=ON, USE_CUDNN=ON, '
                              'USE_CUSPARSELT=1, USE_GFLAGS=OFF, USE_GLOG=OFF, '
                              'USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, '
                              'USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, '
                              'USE_OPENMP=ON, USE_ROCM=OFF, '
                              'USE_ROCM_KERNEL_ASSERT=OFF, \n',
 'Python': '3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]',
 'TorchVision': '0.22.1+cu126',
 'lmdeploy': "not installed:No module named 'lmdeploy'",
 'numpy_random_seed': 2147483648,
 'opencompass': '0.5.0+',
 'sys.platform': 'linux',
 'transformers': '4.56.1'}

### Reproduces the problem - code/configuration sample

from opencompass.models import OpenAISDK
from mmengine.config import read_base

with read_base():
    from opencompass.configs.datasets.ARC_c.ARC_c_gen import ARC_c_datasets
    from opencompass.configs.datasets.ARC_e.ARC_e_gen import ARC_e_datasets
    from opencompass.configs.datasets.gsm8k.gsm8k_gen import gsm8k_datasets
    from opencompass.configs.datasets.obqa.obqa_gen import obqa_datasets
    from opencompass.configs.datasets.piqa.piqa_gen import piqa_datasets
    from opencompass.configs.datasets.siqa.siqa_gen import siqa_datasets
    from opencompass.configs.datasets.winogrande.winogrande_gen import winogrande_datasets
    from opencompass.configs.datasets.race.race_gen import race_datasets
    from opencompass.configs.datasets.math.math_gen import math_datasets

datasets = ARC_c_datasets + ARC_e_datasets + gsm8k_datasets + obqa_datasets + piqa_datasets + \
    siqa_datasets + winogrande_datasets + race_datasets + math_datasets

models = [
    dict(
        abbr='DS-R1-Distill-Llama-70B-ht-sym-fp8',
        type=OpenAISDK,
        key='EMPTY',
        openai_api_base='http://127.0.0.1:12071/v1',
        path='/mnt/models/DS-R1-Distill-Llama-70B_ht_sym_fp8',
        tokenizer_path='/mnt/models/DS-R1-Distill-Llama-70B_ht_sym_fp8',
        rpm_verbose=True,
        query_per_second=32,
        max_out_len=2048,
        max_seq_len=16384,
        temperature=0,
        batch_size=1024,
        retry=2,
    )
]

work_dir = 'outputs/70b_sym_fp8/arc_ce_2048'

### Reproduces the problem - command or script

CUDA_VISIBLE_DEVICES=4,5,6,7 python -m vllm.entrypoints.openai.api_server \
    --model /mnt/models/DS-R1-Distill-Llama-70B_ht_sym_fp8 \
    --port 12071 \
    --tensor-parallel-size 4 \
    --gpu-memory-utilization 0.95 \
    --max-model-len 16384 \
    --dtype auto

### Reproduces the problem - error message

No error message

### Other information

I evaluated several quantized Deepseek-R1-distill-Llama-70B models with opencompass 0.5.0 and vllm 0.10.0 and found that opencompass gave wrong scores on siqa dataset. In the evaluation of some low-precision quantized models, the scores of siqa dataset were always around 35 points while the scores of other dataset were zero. I check one of the prediction file and found meaningless output from the model but siqa still got the score 32.91. The following is the result:
dataset,version,metric,mode,DS-R1-Distill-Llama-70B-ht-sym-fp8
ARC-c,1e0de5,accuracy,gen,0.00
ARC-e,1e0de5,accuracy,gen,0.00
gsm8k,4fb824,accuracy,gen,0.00
openbookqa,2e1c32,accuracy,gen,0.00
openbookqa_fact,60bd99,accuracy,gen,0.00
piqa,1194eb,accuracy,gen,0.00
**siqa,e78df3,accuracy,gen,32.91**
winogrande,458220,accuracy,gen,0.00
race-middle,9a54b6,accuracy,gen,0.00
race-high,9a54b6,accuracy,gen,0.00

In this case, the model output 2048 backslashes for each question in each dataset. The siqa prediction result is shown below:

<img width="1864" height="908" alt="Image" src="https://github.com/user-attachments/assets/d0416365-cc3f-408e-a324-8801550ad7e2" />

Please check whether there's something wrong in processing siqa evaluation.