Repository: open-compass/opencompass
Issue: [Bug] Qwen3评测分数低
Root cause: Configuration Error

### 先决条件

- [x] 我已经搜索过 [问题](https://github.com/open-compass/opencompass/issues/) 和 [讨论](https://github.com/open-compass/opencompass/discussions) 但未得到预期的帮助。
- [x] 错误在 [最新版本](https://github.com/open-compass/opencompass) 中尚未被修复。

### 问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

### 环境

```
{'CUDA available': False,
 'GCC': 'gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0',
 'MMEngine': '0.10.6',
 'MUSA available': False,
 'OpenCV': '4.11.0',
 'PyTorch': '2.5.1',
 'PyTorch compiling details': 'PyTorch built with:\n'
                              '  - GCC 10.2\n'
                              '  - C++ Version: 201703\n'
                              '  - Intel(R) MKL-DNN v3.5.3 (Git Hash '
                              '66f0cb9eb66affd2da3bf5f8d897376f04aae6af)\n'
                              '  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
                              '  - LAPACK is enabled (usually provided by '
                              'MKL)\n'
                              '  - NNPACK is enabled\n'
                              '  - CPU capability usage: NO AVX\n'
                              '  - Build settings: BLAS_INFO=open, '
                              'BUILD_TYPE=Release, '
                              'CXX_COMPILER=/opt/rh/devtoolset-10/root/usr/bin/c++, '
                              'CXX_FLAGS=-ffunction-sections -fdata-sections '
                              '-D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 '
                              '-fvisibility-inlines-hidden -DUSE_PTHREADPOOL '
                              '-DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI '
                              '-DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON '
                              '-DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK '
                              '-DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC '
                              '-Wall -Wextra -Werror=return-type '
                              '-Werror=non-virtual-dtor -Werror=bool-operation '
                              '-Wnarrowing -Wno-missing-field-initializers '
                              '-Wno-type-limits -Wno-array-bounds '
                              '-Wno-unknown-pragmas -Wno-unused-parameter '
                              '-Wno-strict-overflow -Wno-strict-aliasing '
                              '-Wno-stringop-overflow -Wsuggest-override '
                              '-Wno-psabi -Wno-error=old-style-cast '
                              '-Wno-missing-braces -fdiagnostics-color=always '
                              '-faligned-new -Wno-unused-but-set-variable '
                              '-Wno-maybe-uninitialized -fno-math-errno '
                              '-fno-trapping-math -Werror=format '
                              '-Wno-stringop-overflow, LAPACK_INFO=open, '
                              'TORCH_VERSION=2.5.1, USE_CUDA=OFF, '
                              'USE_CUDNN=OFF, USE_CUSPARSELT=OFF, '
                              'USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, '
                              'USE_GLOG=OFF, USE_GLOO=ON, USE_MKLDNN=ON, '
                              'USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, '
                              'USE_OPENMP=ON, USE_ROCM=OFF, '
                              'USE_ROCM_KERNEL_ASSERT=OFF, \n',
 'Python': '3.10.2 (main, Jan  8 2025, 06:13:36) [GCC 9.4.0]',
 'TorchVision': '0.20.1',
 'lmdeploy': "not installed:No module named 'lmdeploy'",
 'numpy_random_seed': 2147483648,
 'opencompass': '0.3.9+',
 'sys.platform': 'linux',
 'transformers': '4.51.3'}
```

### 重现问题 - 代码/配置示例

评测Qwen3-8B时分数较低，这是我添加的配置
```
from opencompass.models import VLLMwithChatTemplate

models = [
    dict(
        type=VLLMwithChatTemplate,
        abbr='qwen3-8b-instruct-vllm',
        path='/Qwen-Series/Qwen3-8B-Instruct',
        model_kwargs=dict(tensor_parallel_size=1),
        max_out_len=4096,
        batch_size=8,
        generation_kwargs=dict(temperature=0),
        run_cfg=dict(num_gpus=1),
    )
]
```
在评测humaneval(humaneval_gen_8e312c)和mbpp(sanitized_mbpp_gen_830460)时发现开启think的qwen3分数比qwen2.5-7b以及不开启think的qwen3-8b低
然后我尝试在这两个数据的Dataset中对pred进行</think>截断只留取</think>后的部分，出来的分数还是如上最低
例如humanevaluator的score方法中添加 
```
def score...

    def truncate_string(s):
        end_tag = "</think>"
        index = s.find(end_tag)
        return s[index + len(end_tag):] if index != -1 else s

        for preds, refer in zip(predictions, references):
            preds = truncate_string(preds).strip()
```


### 重现问题 - 命令或脚本

```
python run.py --models vllm_qwen3_8b_instruct --datasets humaneval_gen_8e312c --max-num-workers 8
```

### 重现问题 - 错误信息

测试出来的分数如下: 
qwen3-8b-woCoT: openai_humaneval: 83.54, sanitized_mbpp: 77.43
qwen3-8b-withCoT: openai_humaneval: 57.93, sanitized_mbpp: 73.93


### 其他信息

_No response_

For thinking mode, use Temperature=0.6, TopP=0.95, TopK=20, and MinP=0 (the default setting in generation_config.json). DO NOT use greedy decoding, as it can lead to performance degradation and endless repetitions. For more detailed guidance, please refer to the [Best Practices](https://huggingface.co/Qwen/Qwen3-8B#best-practices) section.

For non-thinking mode, we suggest using Temperature=0.7, TopP=0.8, TopK=20, and MinP=0. For more detailed guidance, please refer to the [Best Practices](https://huggingface.co/Qwen/Qwen3-8B#best-practices) section.

from https://huggingface.co/Qwen/Qwen3-8B

> For thinking mode, use Temperature=0.6, TopP=0.95, TopK=20, and MinP=0 (the default setting in generation_config.json). DO NOT use greedy decoding, as it can lead to performance degradation and endless repetitions. For more detailed guidance, please refer to the [Best Practices](https://huggingface.co/Qwen/Qwen3-8B#best-practices) section.
> 
> For non-thinking mode, we suggest using Temperature=0.7, TopP=0.8, TopK=20, and MinP=0. For more detailed guidance, please refer to the [Best Practices](https://huggingface.co/Qwen/Qwen3-8B#best-practices) section.
> 
> from https://huggingface.co/Qwen/Qwen3-8B

Thanks for the reply, but I'm not using greedy search, using temp=0 to make sure the results are reproducible. The core of my problem is thinking mode using opencompass the way it's currently reviewed doesn't seem to fit well on ifeval and mbpp

For Qwen3 series model, using greedy search may cause serious repeated patterns or endless prediction until the max length. Please try the sampling strategy.

> For thinking mode, use Temperature=0.6, TopP=0.95, TopK=20, and MinP=0 (the default setting in generation_config.json). DO NOT use greedy decoding, as it can lead to performance degradation and endless repetitions. For more detailed guidance, please refer to the [Best Practices](https://huggingface.co/Qwen/Qwen3-8B#best-practices) section.
> 
> For non-thinking mode, we suggest using Temperature=0.7, TopP=0.8, TopK=20, and MinP=0. For more detailed guidance, please refer to the [Best Practices](https://huggingface.co/Qwen/Qwen3-8B#best-practices) section.
> 
> from https://huggingface.co/Qwen/Qwen3-8B

Hi, I have used Temperature=0.6, TopP=0.95, TopK=20, and MinP=0 for thinking mode and  and Temperature=0.7, TopP=0.8, TopK=20, and MinP=0 for non-thinking mode and re-evaled openai_humaneval and mbpp datasets. But the scores for the thinking mode review for both datasets were still low (humaneval 61.59 and sanitized_mbpp 73.15), and the result for the non-thinking mode was humaneval 85.37 and sanitized_mbpp 74.71

> For Qwen3 series model, using greedy search may cause serious repeated patterns or endless prediction until the max length. Please try the sampling strategy.

Hi, I have used Temperature=0.6, TopP=0.95, TopK=20, and MinP=0 for thinking mode and  and Temperature=0.7, TopP=0.8, TopK=20, and MinP=0 for non-thinking mode and re-evaled openai_humaneval and mbpp datasets. But the scores for the thinking mode review for both datasets were still low (humaneval 61.59 and sanitized_mbpp 73.15), and the result for the non-thinking mode was humaneval 85.37 and sanitized_mbpp 74.71

Hi, @GenerallyCovetous，
Maybe you can try a longer `max_out_len` and use the `pred_postprocessor` provided in OpenCompass.
Here is an example:

```
from opencompass.utils.text_postprocessors import extract_non_reasoning_content

model = dict(
        type=TurboMindModelwithChatTemplate,
        abbr='qwen_3_0.6b_thinking',
        path='Qwen/Qwen3-0.6B',
        engine_config=dict(session_len=32768, max_batch_size=16, tp=1),
        gen_config=dict(
            top_k=20, temperature=0.6, top_p=0.95, do_sample=True,enable_thinking=True
        ),
        max_seq_len=32768,
        max_out_len=32000,
        batch_size=16,
        run_cfg=dict(num_gpus=1),
        pred_postprocessor=dict(type=extract_non_reasoning_content)
    ),
```




建议检查一下输出的result文件夹的数据，对结果进行分析，多数情况下为无法正确识别代码块与生成信息不完整导致。

建议用llm judge，纯规则提取判定的分数会比较低。