Repository: open-compass/opencompass
Issue: [Bug] Wrong eval of some tasks(word_sorting and web_of_lies) of bbh(big bench hard) on a model deployed locally.
Root cause: Validation Gap

### Prerequisite

- [x] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [x] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I have modified the code (config is not considered code), or I'm working on my own tasks/models/datasets.

### Environment

```python
{'CUDA available': False,
 'GCC': 'gcc (GCC) 10.3.1',
 'MMEngine': '0.10.7',
 'MUSA available': False,
 'OpenCV': '4.11.0',
 'PyTorch': '2.8.0+cpu',
 'PyTorch compiling details': 'PyTorch built with:\n'
                              '  - GCC 13.3\n'
                              '  - C++ Version: 201703\n'
                              '  - Intel(R) MKL-DNN v3.7.1 (Git Hash '
                              '8d263e693366ef8db40acc569cc7d8edf644556d)\n'
                              '  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
                              '  - LAPACK is enabled (usually provided by '
                              'MKL)\n'
                              '  - NNPACK is enabled\n'
                              '  - CPU capability usage: DEFAULT\n'
                              '  - Build settings: BLAS_INFO=open, '
                              'BUILD_TYPE=Release, '
                              'COMMIT_SHA=a1cb3cc05d46d198467bebbb6e8fba50a325d4e7, '
                              'CXX_COMPILER=/opt/rh/gcc-toolset-13/root/usr/bin/c++, '
                              'CXX_FLAGS=-ffunction-sections -fdata-sections '
                              '-fvisibility-inlines-hidden -DUSE_PTHREADPOOL '
                              '-DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI '
                              '-DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON '
                              '-DUSE_PYTORCH_QNNPACK '
                              '-DAT_BUILD_ARM_VEC256_WITH_SLEEF -DUSE_XNNPACK '
                              '-DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC '
                              '-DC10_NODEPRECATED -Wall -Wextra '
                              '-Werror=return-type -Werror=non-virtual-dtor '
                              '-Werror=range-loop-construct '
                              '-Werror=bool-operation -Wnarrowing '
                              '-Wno-missing-field-initializers '
                              '-Wno-unknown-pragmas -Wno-unused-parameter '
                              '-Wno-strict-overflow -Wno-strict-aliasing '
                              '-Wno-stringop-overflow -Wsuggest-override '
                              '-Wno-psabi -Wno-error=old-style-cast '
                              '-faligned-new -Wno-maybe-uninitialized '
                              '-fno-math-errno -fno-trapping-math '
                              '-Werror=format -Wno-dangling-reference '
                              '-Wno-error=dangling-reference '
                              '-Wno-stringop-overflow, LAPACK_INFO=open, '
                              'TORCH_VERSION=2.8.0, USE_CUDA=OFF, '
                              'USE_CUDNN=OFF, USE_CUSPARSELT=OFF, '
                              'USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, '
                              'USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, '
                              'USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, '
                              'USE_ROCM_KERNEL_ASSERT=OFF, USE_XCCL=OFF, '
                              'USE_XPU=OFF, \n',
 'Python': '3.11.13 (main, Jun  5 2025, 13:03:00) [GCC 11.2.0]',
 'lmdeploy': "not installed:No module named 'lmdeploy'",
 'numpy_random_seed': 2147483648,
 'opencompass': '0.4.2+f30aea8',
 'sys.platform': 'linux',
 'transformers': '4.55.0'}
```

### Reproduces the problem - code/configuration sample

model config(a glm-4-9b model deployed locally, url and tokenizer_path leaved blank for privacy, I dont know whether the problem can be reproduced with other models):
```python
from opencompass.models import OpenAI
from opencompass.models import OpenAISDK


api_meta_template = dict(
    round=[
            dict(role='HUMAN', api_role='HUMAN'),
            dict(role='BOT', api_role='BOT', generate=True),
    ],
)

models = [
    dict(
        abbr='GLM4',
        type=OpenAISDK,
        key='EMPTY', # API key
        openai_api_base='xxx',  # Service address
        path='glm4-9b-0710',  # Model name for service request
        tokenizer_path='xxx', # The tokenizer name or path, if set to `None`, uses the default `gpt-4` tokenizer
        rpm_verbose=True,  # Whether to print request rate
        meta_template=api_meta_template,  # Service request template
        query_per_second=1,  # Service request rate
        max_out_len=1024,  # Maximum output length
        max_seq_len=4096,  # Maximum input length
        temperature=0.01,  # Generation temperature
        batch_size=64,  # Batch size
        retry=3,  # Number of retries
    )
]
```

python scripts:
```python
from mmengine.config import read_base

with read_base():
    from opencompass.configs.datasets.bbh.bbh_0shot_nocot_gen_9c32f6 import bbh_datasets

datasets = bbh_datasets
models = glm4
```

Because only ```web_of_lies``` and ```word_sorting``` tasks are evaluated incorrectly, you can change ```opencompass.configs.datasets.bbh.bbh_0shot_nocot_gen_9c32f6``` as below for reproduce simplicity
```python
import os
from opencompass.openicl.icl_prompt_template import PromptTemplate
from opencompass.openicl.icl_retriever import ZeroRetriever
from opencompass.openicl.icl_inferencer import GenInferencer
from opencompass.openicl.icl_evaluator import AccEvaluator
from opencompass.datasets import BBHDataset, BBHEvaluator, bbh_mcq_postprocess, bbh_freeform_postprocess, BBHEvaluator_mcq

bbh_reader_cfg = dict(input_columns=['input'], output_column='target')



singular_datasets = [
    'word_sorting',
    'web_of_lies'
]

bbh_datasets = []

for _name in singular_datasets:
    
    bbh_infer_cfg = dict(
        prompt_template=dict(
            type=PromptTemplate,
            template=dict(round=[
                dict(
                    role='HUMAN',
                    prompt=
                    f"Follow the given examples and answer the question.\n\nQuestion: {{input}}\n You must give your final answer by starting with 'So the answer is' "
                )
            ])),
        retriever=dict(type=ZeroRetriever),
        inferencer=dict(type=GenInferencer, max_out_len=512))
    bbh_eval_cfg = dict(
        evaluator=dict(type=BBHEvaluator), 
        pred_role='BOT'
    )

    bbh_datasets.append(
        dict(
            type=BBHDataset,
            path='opencompass/bbh',
            name=_name,
            abbr='bbh-' + _name,
            reader_cfg=bbh_reader_cfg,
            infer_cfg=bbh_infer_cfg.copy(),
            eval_cfg=bbh_eval_cfg.copy()))
```

### Reproduces the problem - command or script

```shell
opencompass examples/eval_api_glm.py
```


### Reproduces the problem - error message

1. Problem1: In web_of_lies task, when model predicts 'no' or 'no, xxxx' and the target is 'No', it is judged as wrong answer, which is irrational. The result score is 0；
2. Problem2: In word_sorting task, similar to the problem1, when model predicts 'A, B, C' and the target is 'A B C', it is judged as wrong answer, which is irrational. The reuslt score is only 9.6;

<img width="633" height="211" alt="Image" src="https://github.com/user-attachments/assets/f14ade5d-2e43-4595-90c6-eb9e84960e80" />

word_sorting:
<img width="449" height="285" alt="Image" src="https://github.com/user-attachments/assets/87aff97d-ad45-4dfd-ab85-831ccb2ef5c7" />

web_of_lies:
<img width="670" height="285" alt="Image" src="https://github.com/user-attachments/assets/3bc1dc9b-cce8-4d17-8769-08d9db403853" />

### Other information

I have solved this problem and get corrected eval result of web_of_lies and word_sorting. Below are my solutions.

First, add postprocessor for datasets and predictions in ```bbh_eval_cfg```, like below:
```python
bbh_eval_cfg = dict(
        evaluator=dict(type=BBHEvaluator), 
        pred_postprocessor=dict(type=bbh_freeform_postprocess),
        dataset_postprocessor=dict(type=bbh_freeform_postprocess),
        pred_role='BOT'
    )
``` 

Then, we need to do some change to ```opencompass/opencompass/datasets/bbh.py```
Replace the line 62 with ``` return ans.replace(',', '').split(',')[0].upper()```.
Explantation: I find my model prefer using ',' as split character, so I replace ',' with '' to align with answers. This solve the problem of word_sorting. '.upper()' make sure the answer and prediction are all uppercase. And I found my model sometimes predict like 'Yes, xxx', so I use split(',')[0] to scissor the latter part. 

After adaption above, I get corrected result below:
 
<img width="603" height="231" alt="Image" src="https://github.com/user-attachments/assets/434c0684-423c-4e29-b68b-5807ef24d447" />